🪄
Installation
A complete guide to install Episilia
- 1.AWS, GCP, Digital Ocean are supported out of the box. For installation in a different cloud provider or in your own datacenter, any S3 compatible Block Storage can be supplied, such as minio server.
- 2.Helm version 3.0+ (Helm CLI)
- 3.Kubectl CLI
- 4.A running Kubernetes cluster. Minimum of 2 cores and 4 GB memory required
- 5.Kafka for sourcing logs. Other topics to be created during installation
- 6.S3 credentials and a bucket to store index and data files
- 7.AVX2 support needed on compute machines
On linux (or unix machines) information about your cpu is in /proc/cpuinfo. You can extract information from there by hand, or with a grep command (grep flags /proc/cpuinfo). Also most compilers will automatically define AVX2 so you can check for that too
Repo URL: https://episilia.gitlab.io/episilia-helm/release
helm repo add <NAME> <URL>
$ helm repo add episilia https://episilia.gitlab.io/episilia-helm/release
$ helm repo list or helm repo ls
NAME URL
episilia https://episilia.gitlab.io/episilia-helm/release
$ helm search repo episilia
Update global values in the episilia/episilia-cpanel master chart values.yaml file. All configurable values are explained in the section below titled "Configuration".
Inspect the values before installing application use below:
$ helm inspect values episilia/episilia-cpanel > episilia_values.yaml
$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia
helm install episilia episilia/episilia-cpanel -f episilia_values.yaml --dry-run
$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --dry-run --set global.client.license.key=episilia
helm install <RELEASE NAME> <CHART>
$ helm install episilia episilia/episilia-cpanel -f episilia_values.yaml
or
$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia
Listing installed helm charts
$ helm ls
Listing pods
$ kubectl get pods
List services:
$ kubectl get services
Access Grafana Logs Browser using grafana cluster IP. Redirect Grafana IP as needed.
Default USERNAME and PASSWORD is admin
Episilia can be configured with the following values
Docker image for episilia.
imageTag: &release "1.1.4"
Enable required servers.
episilia-log-indexer, episilia-search, episilia-gateway needs to be enabled by default.
Disable grafana if using your own Grafana instance
episilia-log-indexer:
enabled: true
episilia-search:
enabled: true
episilia-search-fixed:
enabled: false
episilia-gateway:
enabled: true
episilia-log-indexer-opt:
enabled: false
episilia-alert:
enabled: false
grafana:
enabled: true
env denotes the client identified environment. Examples include dev / qa / prod
global:
client:
id: episilia-helm
env: test-helm
license:
key: episilia
Common ops config goes below.
ops:
log:
debug: on # Enable to get debug logs in all the servers
license:
url: episilia-cpanel:8080
cpanel:
data:
publish:
interval:
seconds: 300 # Time interval in which cpanel will be pushing metrics to console
All kafka specific configuration goes below.
kafka:
metadata:
broker:
list: localhost:9092 # The kafka broker
group:
search: episilia-search-group # Kafka consumer-group for search
cpanel: episilia-cpanel-group # Kafka consumer-group for cpanel
logwatcher: episilia-logwatcher-group # Kafka consumer-group for log-watcher
logwatcher.tail: episilia-logwatcher-tail-group # Kafka consumer-group for log-watcher-tail
logwatcher.alert: episilia-logwatcher-alert-group # Kafka consumer-group for log-watcher-alert
topic:
index:
live: stagefiles-topic # Topic for internal publish indexed files - stage.topic
optimized: optfiles-topic # Topic for internal optimize.topic: publish file names post optimization
optimize:
request: stagefolder-topic #optimize.request.topic send folders to optimize
cpanel:
in: cpaneld-topic # Internal topic cpanel.data.topic
out: cpaneld-topic # Internal topic cpanel.data.topic
alert:
response:
out: alert_out
indexer:
logs:
topics: logs # Topic from where logs are loaded.
group: episilia-indexer-group # Kafka consumer-group for indexer
S3 bucket and folder details goes below.
datastore:
s3:
accesskey: "" # filestorage access key (eg: AWS S3 access key)
secretkey: "" # filestorage secret key (eg: AWS S3 secret key)
region: "" # filestorage region (eg: AWS S3 region)
endpoint:
url: "" # filestorage endpoint URL (eg: AWS S3 endpoint URL)
sign:
payload: true
bucket: episilia-bucket # filestorage bucket (eg: AWS S3 bucket)
folder: episilia-folder # filestorage folder URL (eg: AWS S3 folder)
work:
folder: work-folder # Folder name to store s3 limit/write results
url:
prefix: s3:// # filestorage prefix URL (default to AWS S3)
Config for indexer and optimizer.
indexer:
image:
repository: episilia/episilia-log-indexer # docker image of episilia-log-indexer
tag: *release
replicaCount: "1" # kubernetes pod replicas of episilia-log-indexer
resources:
limits:
cpu: 800m # cpu limit on episilia-log-indexer
memory: 1024Mi # memory limit on episilia-log-indexer
requests:
cpu: 400m # cpu request on episilia-log-indexer
memory: 300Mi # memory request on episilia-log-indexer
logs:
source: kafka # source: kafka # s3 or kafka
schema:
appid:
fixed: "defaultApp" # If appid is a fixed string
keys: "project.app_id" # label(s) for app identifier
tenantid:
fixed: "defaultTenant" # If tenantid is a fixed string
keys: "" # label(s) for tenant identifier
message:
key: "log" # actual log message key
timestamp:
key: "time" # timestamp key
formats: "%Y-%m-%dT%H:%M:%S" #to specify timestamp format (ex: %Y-%m-%dT%H:%M:%S )
exclude: "time" # labels to be excluded from the list
optimizer:
replicaCount: "1" # kubernetes pod replicas of episilia-optimizer
resources:
limits:
cpu: 800m # cpu limit on episilia-optimizer
memory: 1024Mi # memory limit on episilia-optimizer
requests:
cpu: 500m # cpu request on episilia-optimizer
memory: 300Mi # memory request on episilia-optimizer
Config for alert server .
alert:
resources:
limits:
cpu: 500m # cpu limit on episilia-optimizer
memory: 600Mi # memory limit on episilia-optimizer
requests:
cpu: 300m # cpu request on episilia-optimizer
memory: 300Mi # memory request on episilia-optimizer
rules.file.url: #s3 path of alert rules file
prometheus.gateway: localhost:5070 # push-gateway url
Config for search server goes below.
search:
image:
repository: episilia/episilia-search # docker image of episilia-search
tag: *release
replicaCount: "1" # kubernetes pod replicas of episilia-search
resources:
limits:
cpu: "1" # cpu limit on episilia-search
memory: 2048Mi # memory limit on episilia-search
requests:
cpu: 500m # cpu request on episilia-search
memory: 600Mi # memory request on episilia-search
api:
timeout:
seconds: 40 # timeout for search while querying
live:
from:
hours: 48 # hours from when the required index blocks should be loaded
to:
hours: 0 # hours till when the required index blocks should be loaded, Note: value to be "0" to get instant logs
labels:
exclude: "@timestamp,log" # Lables excluded from grafana dropdown GUI.
Config for historic search server.
fixedSearch:
bucket: "" # s3 bucket for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.bucket value as default
folder: "" # s3 folder for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.folder value as default
replicaCount: "1" # kubernetes pod replicas of historic episilia-search
resources:
limits:
cpu: "1" # cpu limit on historic episilia-search
memory: 1024Mi # memory limit on historic episilia-search
requests:
cpu: 500m # cpu request on historic episilia-search
memory: 600Mi # memory request on historic episilia-search
fixed:
from:
yyyymmddhh: "2021092100" # the date from when the required index blocks should be loaded
to:
yyyymmddhh: "2021092202" # the date till when the required index blocks should be loaded
api:
timeout:
seconds: 40 # timeout for search while querying
labels:
exclude: "@timestamp,log" # Lables excluded from grafana dropdown GUI.
Gateway specific configuration goes below.
gateway:
image:
repository: episilia/episilia-gateway # docker image of episilia-gateway
tag: *release
service:
type: ClusterIP # service type of episilia-gateway
replicaCount: "1" # kubernetes pod replicas of episilia-gateway
resources:
limits:
cpu: 500m # cpu limit on episilia-gateway
memory: 600Mi # memory limit on episilia-gateway
requests:
cpu: 300m # cpu request on episilia-gateway
memory: 200Mi # memory request on episilia-gateway
search:
timeout:
seconds: 40 # timeout of the query from gateway
Control Panel specific configuration goes below.
cpanel:
ops:
healthchecks:
interval:
mins: "5" # time interval in which metrics are pushed
exclude:
list: "" # to exclude the specific metrics
api:
access: # Unique keys to access the episilia console
key: token
token: random
post:
server: "https://console.episilia.com/publish_cpanel_data" # Url of the console
get:
server: ""
grpc:
port: "8080"
metrics:
port: "8090"
If PV is enabled, configure the same below.
persistence:
enabled: false # to enable PVC
mountPath: "/data" # mount path of PVC
storageClassName: do-block-storage # storage class name (differs on the cloud services that are used)
accessModes:
- ReadWriteOnce # access modes
size: "40Gi" # size of PVC which will be mounted to episilia-search for live search
historicSize: "20Gi" # size of PVC which will be mounted to episilia-search for historic search
# annotations: {}
finalizers:
- kubernetes.io/pvc-protection
# selectorLabels: {}
# subPath: ""
# existingClaim:
External Resources
Upload a chart to Kubernetes
helm install <RELEASE NAME> <CHART>
helm install <CHART> —generate-name
helm install <NAME> <CHART> —dry-run --debug
Remove a chart repository:
helm repo remove | rm <NAME>
helm repo remove episilia
To view template of the chart, before installation
helm template <CHART>
helm template episilia/episilia-cpanel
Passing keywords at runtime during installation
helm install episilia episilia/episilia-log-indexer --set image.repository=<imagename> --set image.tag=<tagname>
Upgrade the chart with a specific release
helm upgrade <RELEASE> <CHART>
helm upgrade episilia episilia/episilia-cpanel
flag -i or –install can be specified to run an install before if a release by this name doesn’t already exist. Otherwise, perform a rollback. If revision is not specified, the chart is rolled back to the previous version.
helm rollback <RELEASE> <REVISION>
helm rollback episilia --revision 1
View all historical revisions for a given release
helm history <RELEASE>
Uninstall a release
helm uninstall <RELEASE>
helm uninstall episilia
Last modified 1yr ago