🪄Installation

A complete guide to install Episilia

Prerequisites

  1. AWS, GCP, Digital Ocean are supported out of the box. For installation in a different cloud provider or in your own datacenter, any S3 compatible Block Storage can be supplied, such as minio server.

  2. Helm version 3.0+ (Helm CLI)

  3. Kubectl CLI

  4. A running Kubernetes cluster. Minimum of 2 cores and 4 GB memory required

  5. Kafka for sourcing logs. Other topics to be created during installation

  6. S3 credentials and a bucket to store index and data files

  7. AVX2 support needed on compute machines

On linux (or unix machines) information about your cpu is in /proc/cpuinfo. You can extract information from there by hand, or with a grep command (grep flags /proc/cpuinfo). Also most compilers will automatically define AVX2 so you can check for that too

Using Helm

Step 1 : Add Helm Repo

Repo URL: https://episilia.gitlab.io/episilia-helm/release

$ helm repo add <NAME> <URL>
$ helm repo add episilia https://episilia.github.io/episilia-deploy/

Listing chart repositories:

$ helm repo list or helm repo ls
  NAME        URL 
  episilia   https://episilia.github.io/episilia-deploy/

Searching for charts in the repository:

$ helm search repo episilia

Step 2 : Configure values

Update global values in the episilia/episilia-spike master chart values.yaml file. All configurable values are explained in the section below titled "Configuration".

Inspect the values before installing application use below:

$ helm inspect values episilia/episilia-spike > episilia_values.yaml

$ helm install episilia episilia/episilia-spike --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia

Step 3 : Perform dry run

helm install  episilia episilia/episilia-spike -f episilia_values.yaml --dry-run
$ helm install episilia episilia/episilia-spike --set global.client.id=episilia-client --set global.client.env=dev --dry-run --set global.client.license.key=episilia

Step 4 : Install Helm Repo

helm install <RELEASE NAME> <CHART>
$ helm install episilia episilia/episilia-spike  -f episilia_values.yaml
                              or      
$ helm install episilia episilia/episilia-spike --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia

Listing installed helm charts

 $ helm repo ls

Listing pods

 $ kubectl get pods

List services:

 $ kubectl get services

Step 5 : Access Spike-UI Logs Browser

Access Spike-UI Logs Browser using spike LoadBalancer IP. Redirect Spike-UI IP as needed.

The default USERNAME is admin and the Password is the license key.

Configuration

Episilia can be configured with the following values

Image tag

Docker image for episilia.

imageTag: &release "3.0.0"

Enabling server nodes

Enable required servers.

episilia-spike, episilia-spike-ui, episilia-log-indexer, episilia-optimizer, episilia-search and episilia-gateway need to be enabled by default.

redpanda:
  enabled: false
episilia-log-indexer:
  enabled: true
episilia-log-indexer-s3:
  enabled: false
episilia-log-indexer-opt:
  enabled: true
episilia-search:
  enabled: true
episilia-search-fixed:
  enabled: false
episilia-search-ondemand:
  enabled: false
episilia-gateway:
  enabled: true
episilia-spike-ui:
  enabled: true

License and Client id

License key and client ID can be obtained from Episilia

Please fill out the form and the team will get back.

env denotes the client-identified environment. Examples include dev / qa / prod

global:
  client:
    id: episilia-helm
    env: test-helm
    license:
      key: episilia
    release:
      version: *release
    arn: ""    #to annotate service account "optional"

All common ops

Common ops config goes below.


ops:
    log:
      debug: "off" # Enable to get debug logs in all the servers
      metrics:
        publish:
          interval:
            seconds: "300" # Time interval in which cpanel will be pushing metrics to console
      state:
        publish:
          interval:
            seconds: "10" # Time interval in which all servers will be pushing metrics to console
    monitor:
      memory:
        max:
          mb: "1024"     #   modify max memory for searching queries based on the memory usage of each search server

Kafka config

All kafka specific configuration goes below.

 kafka:
    group:
      search: episilia-search-group                  #Kafka consumer-group for search
      spike: episilia-spike-group                    #Kafka consumer-group for spike
      cpanel: episilia-cpanel-group                  #Kafka consumer-group for cpanel
      gateway:
        tail: episilia-gw-tail-group                #Kafka consumer-group for gateway 
      logwatcher:
        alert: episilia-lw-alert-group               #Kafka consumer-group for alert 
        tail: episilia-lw-tail-group                 #Kafka consumer-group for gateway 
      s3log:
        files: episilia-s3files-group                #Kafka consumer-group for s3logs
      
    topic:
      index:
        live: episilia-stagefiles              #Topic to publish indexed files - stage.topic
        optimized: episilia-optfiles           # Topic for optimize.topic:publish file names post optimization
        labels: episilia-indexlabels                      #Topic to publish labels from indexer
      optimize:
        request: episilia-stagefolder            #optimize.request.topic send folders to optimize
      s3log:
        files: episilia-s3logs                          #Topic from where s3 logs files are loaded.
      cpanel:
        out: episilia-cpanel-out                           #Internal topic cpanel.data.topic
      
      tail:
        request:
          in: episilia-tail-in                          #Incoming requests for tail requests
        response:
          out: episilia-tail-out                        #publish results for tail
      alert:
        response:
          out:episilia-alert-out                       #publish alerts for spike
    indexer:
      broker:
        list: redpanda:9092              #The kafka broker for logs. If this is not set, it will use the default broker
        security:
          mode: none                           #values are [none|login|oauth|kerberos]
          protocol: SASL_SSL                   #plaintext, ssl, sasl_plaintext, sasl_ssl
        sasl:
          mechanism: SCRAM-SHA-512   #PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,OAUTHBEARER,GSSAPI
          username: episilia
          password: episilia123
        rack:
          aware: "false"                       #enable "true" for rack awareness

      logs:
        topics: episilia-logs                 #Topic from where logs are loaded.
      group: episilia-indexer-group           #Kafka consumer-group for indexer
      
    internal:
      broker:
        list: redpanda:9092                   #Kafka broker for internal communication

Datastore

S3 bucket and folder details goes below.

datastore:      
    s3:                        # (stage,final,sourcebucket for s3 log source)
      accesskey: ""            #Filestorage access key (eg: AWS S3 access key)
      secretkey: ""            #Filestorage secret key (eg: AWS S3 secret key)   
      region: ""                       #Filestorage region (eg: AWS S3 region)
      endpoint:
        url:               #Filestorage endpoint URL (eg: AWS S3 endpoint URL)
      sign:
        payload: true      # when using minio bucket disable payload for internal use.
      bucket: episilia-bucket          #Filestorage bucket (eg: AWS S3 bucket)
      folder: episilia-folder       #Filestorage folder URL (eg: AWS S3 folder)
      work:
        folder: work-folder
      url:
        prefix: s3://
      useArn: false           #enable it to access using ARN role
      assumeRole: ""                                    # ARN role
      https: "true"        # when using minio bucket disable https for internal use.

Indexer

Config for indexer and optimizer.

indexer:
    image:
      repository: episilia/log-indexer    #Docker image of episilia-log-indexer
      tag: *release
    replicaCount: "1"          #Kubernetes pod replicas of episilia-log-indexer
                                    annotations:
      deploy:
      service:
    resources:
      limits:
        cpu: "1"                       #CPU limit on episilia-log-indexer 
        memory: 2Gi                    #Memory limit on episilia-log-indexer
      requests:
        cpu: 400m                      #CPU request on episilia-log-indexer
        memory: 300Mi                  #Memory request on episilia-log-indexer
    schema:
      appid:
        fixed: "defaultApp"                       #If appid is a fixed string
        keys: "app_id"                            #Label(s) for app identifier
      tenantid:        
        fixed: "defaultTenant"                   #If tenantid is a fixed string  
        keys: "tenant_id"                       #Label(s) for tenant identifier        
      message:
        key: "log"                                     #Actual log message key
      timestamp:
        key: "time"                                    #Timestamp key
        formats: "%Y-%m-%dT%H:%M:%S"               #Specify timestamp format (ex: %Y-%m-%dT%H:%M:%S )
      exclude: "time"                     #Labels to be excluded from the list
    logs:
      source: kafka                               #Source: S3 or Kafka
    tail:
      enable: "true"                                 #To enable Gateway server
      maxwait:
        ms: "5000"                                     #Time to get tail logs
    
    ops:
      pause: 
        consume:                         #Pauses injest at the below thresholds
          file:
            max:
              count: "100"                         #Applicable for file messages
          record:
            max:
              count: "500000"                      #Applicable for log messages
            size:
              max: 
                mb: "100"                          #Applicable for log messages
      datablock:
        writer:
          count: "1"   #Datablocks zipped and written to disk,in sequence files
      json:
        processor:
          count: "2"                                    #Number of json parsers
    optimize:
      block:
        maxbytes:
          mb: "50"
          
indexers3:
    image:
      repository: episilia/log-indexer    #Docker image of episilia-log-indexer
      tag: *release
    replicaCount: "1"       #Kubernetes pod replicas of episilia-log-indexer-s3
    resources:
      limits:
        cpu: "1"                          #CPU limit on episilia-log-indexer-s3
        memory: 2Gi                    #Memory limit on episilia-log-indexer-s3
      requests:
        cpu: 400m                       #CPU request on episilia-log-indexer-s3
        memory: 300Mi                #Memory request on episilia-log-indexer-s3

optimizer:
    replicaCount: "1"        #Kubernetes pod replicas of episilia-optimizer  
    resources:
      limits:
        cpu: "1"                           #CPU limit on episilia-optimizer
        memory: 2Gi                        #Memory limit on episilia-optimizer
      requests:
        cpu: 500m                          #CPU request on episilia-optimizer        
        memory: 300Mi

Config for alert server .

    alert:
        enable: "false"                                  #To enable alert server
        rules:
          file:
            url: "s3://bucket/folder/file"                #Alert config URL
        prometheus:
          gateway: localhost:5070                         #Push-Gateway URL

Search

Config for live search server goes below.

livesearch:
    image:
      repository: episilia/search              #Docker image of episilia-search
      tag: *release
    replicaCount: "1"               #Kubernetes pod replicas of episilia-search 
    resources:
      limits:
        cpu: "1"                                  #CPU limit on episilia-search
        memory: 2Gi                            #Memory limit on episilia-search
      requests:
        cpu: 500m                               #CPU request on episilia-search
        memory: 600Mi                        #Memory request on episilia-search
    
    api:
      timeout:
        seconds: 60                          #Timeout for search while querying
      request:
        max:
          days: "30"                          # "from" and "to" 30days can be  querying search which can be modified
    
    live:
      from:
        hours: 48         #Hours from when the required index blocks should be loaded
      to:
        hours: 0   #Hours till when the required index blocks should be loaded, Note: value to be "0" to get instant logs

    ops:
      index:
        cache:
          resetonstart: "true"
    
    labels:
      display:
        max:
          count: "1000"             #Labels count displayed in Spike-UI/Grafana

Config for ondemand search server goes below.

ondemandsearch:
    replicaCount: "1"                                  #Kubernetes pod replicas of episilia-search-ondemand
    resources:
      limits:
        cpu: "1"                  #CPU limit on episilia-search-ondemand
        memory: 2Gi            #Memory limit on episilia-search-ondemand
      requests:
        cpu: 500m               #CPU request on episilia-search-ondemand
        memory: 600Mi        #Memory request on episilia-search-ondemand                                
    prewarm:
      enabled: "false"        #Set either yyyymmddhh pair or hours pair. If both are set the hours pair will be considered
      from:
        hours: "2" #Hours from when the required labels should be loaded
        yyyymmddhh: " "    #Date from when the required labels should be loaded (YYYYMMDDHH) or mske it "0" to load from hours
      to:
        hours: "0" #Hours till when the required labels should be loaded
        yyyymmddhh: " "    #Date till when the required labels should be loaded (YYYYMMDDHH) or mske it "0" to load from hours
    ops:
      index:
        cache:
          s3list:
            seconds: "600"    # stage files will be kept as cache in server for specific seconds

Config for historic search server.

 fixedSearch:
    bucket: ""                #S3 bucket for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.bucket value as default
    folder: ""                #S3 folder for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.folder value as default
    replicaCount: "1"      #Kubernetes pod replicas of historic episilia-search 
    resources:
      limits:
        cpu: "1"                         #CPU limit on historic episilia-search
        memory: 2Gi                   #Memory limit on historic episilia-search
      requests:
        cpu: 500m                      #CPU request on historic episilia-search
        memory: 600Mi               #Memory request on historic episilia-search
    fixed:
      from:
        yyyymmddhh: "2021092100"      #Date from when the required index blocks should be loaded (YYYYMMDDHH)
      to:
        yyyymmddhh: "2021092202"      #Date till when the required index blocks should be loaded (YYYYMMDDHH)

    api:
      timeout:
        seconds: 60                          #Timeout for search while querying

Gateway

Gateway specific configuration goes below.

  gateway:
    replicaCount: "1"              #Kubernetes pod replicas of episilia-gateway
    image:
      repository: episilia/gateway            #Docker image of episilia-gateway
      tag: *release
    resources:
      limits:
        cpu: 500m                                #CPU limit on episilia-gateway
        memory: 600Mi                         #Memory limit on episilia-gateway
      requests:
        cpu: 300m                              #CPU request on episilia-gateway        
        memory: 300Mi                       #Memory request on episilia-gateway    
    service: 
      type: "ClusterIP"  
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"

Spike

Spike Server specific configuration goes below.

spike:
    JVM_MAX_SIZE: "1536m"       
    JVM_NEWGEN_MIN_SIZE: "800m"
    JVM_NEWGEN_MAX_SIZE: "900m"
    metadata:
      backfill:
        forceupdate: "false"
        days: "0"  
    s3logs:
      publish:
        seconds: "2"    # time interval gap to fetch newly published files in s3 through fluentd ot other sources.
        partitionwise: "false"
    login:
      mode: local                           #Login through local, google, okta
      local:
        password:
          encryptionkey: "i am groot"
      google:
        clientid: "google-client"
        token: "google-token"
      okta:
        clientid: "okta-client"
        token: "okta-token"
    pulse:
      access:
        key: "token"
        token: "random"
      host: "pulse-url"
      url: "http://pulse-url:50051/"

Spike-UI

Spike UI Browser-specific configuration goes below.

spikeui:
    replicaCount: "1"             #Kubernetes pod replicas of episilia-spike-ui
    image:
      repository: episilia/spike-ui          #Docker image of episilia-spike-ui
      tag: *release
    resources:
      limits:
        cpu: 500m                               #CPU limit on episilia-spike-ui
        memory: 800Mi                        #Memory limit on episilia-spike-ui
      requests:
        cpu: 300m                             #CPU request on episilia-spike-ui        
        memory: 500Mi                      #Memory request on episilia-spike-ui
    service: 
      type: "LoadBalancer"  
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"

Persistence Volume

If PV is enabled, configure the same below.


persistence:
    enabled: "true"
    storageClassName: gp2 # storage class name (differs on the cloud services that are used)
    accessModes:
      - ReadWriteOnce # access modes
    size: "100Gi" # size of PVC which will be mounted to episilia-search for live search
    historicSize: "100Gi" # size of PVC which will be mounted to episilia-search for historic search
    ondemandSize: "100Gi" # size of PVC which will be mounted to episilia-search for ondemand search
    spikeSize: "10Gi" # size of PVC which will be mounted to episilia-spike for spike
    # annotations: {}
    finalizers:
      - kubernetes.io/pvc-protection
    # selectorLabels: {}
    # subPath: ""
    # existingClaim:

```

References

External Resources

Kubectl CLI kubectl

Helm3 CLI helm3

Upload a chart to Kubernetes

helm install <RELEASE NAME> <CHART>

helm install <CHART> —generate-name 

helm install <NAME> <CHART> —dry-run --debug

Remove a chart repository:

helm repo remove | rm <NAME>
helm repo remove episilia

To view template of the chart, before installation

helm  template <CHART>
helm template episilia/episilia-cpanel

Passing keywords at runtime during installation

helm install episilia episilia/episilia-log-indexer --set image.repository=<imagename> --set image.tag=<tagname>

Upgrade the chart with a specific release

helm upgrade <RELEASE> <CHART> 
helm upgrade episilia episilia/episilia-spike

flag -i or –install can be specified to run an install before if a release by this name doesn’t already exist. Otherwise, perform a rollback. If revision is not specified, the chart is rolled back to the previous version.

helm rollback <RELEASE> <REVISION>
helm rollback episilia --revision 1

View all historical revisions for a given release

helm history <RELEASE>

Uninstall a release

helm uninstall <RELEASE>
helm uninstall episilia

PreviousOverviewNextSearching Logs

Last updated