Released on

Features and Enhancements:

  • Cache library - This feature is introduced because parsing (and validating and checking privileges) is expensive in terms of CPU cycles.
  • New Index optimizer node added to optimize/compact indexing in parallel. Now Indexer runs in two modes default mode or “optimizer” mode.
  • Indexing structure better organized
    • bucket/folder/stage (For staging files).
    • bucket/folder/final/YYYYMMDD (Compacted files from optimizer)
  • Search dynamic caching added for data files and index files.
  • Two search modes were introduced: “dynamic” where search reads live data from both indexers and S3. Whereas “static” mode will enable loading historical S3-only logs within a given date range.

Bug fixes:

  • All the Simple Storage Service (s3) calls based issues has been fixed.
  • To simplify indexer configuration: S3 schema.json loading for indexer is completely removed.
  • Indexer should be started on a fresh new bucket as the old structure/files are not compatible with the new indexer release.

Config changes:

  1. 1.
    Configuration Changes Indexer:
    • ops.server.mode - set to "optimizer" only for the optimizer pod run
    • data.optimize.schedule.minutes - Frequency of the optimizer run
    • out.kafka.optimize.topic - The kafka topic where optimize will connect with search
    • data.process.config.path is removed - Schema.json is removed
    • schema.tenantid.keys
    • schema.appid.keys
    • schema.message.key (defaults: "message")
    • schema.timestamp.key
    • schema.timestamp.formats (defaults: "%Y-%m-%dT%H:%M:%S" )
  2. 2.
    Configuration Changes Search:
    • ops.index.cache.path - search local disk file path where index caching is done
    • - search local disk file path where data caching is done
    • search.mode - “dynamic” or “static”
    • search.mode.dynamic.kafka.stage.topic
    • search.mode.daystoinclude
    • search.mode.static.fromdate (example: 20201107)
    • search.mode.static.todate (example: 20201107)
  3. 3.
    Configuration Changes Gateway:
    • clusters - Search cluster node