πŸš€0.5

Released on

Features and Enhancements:

  • Cache library - This feature is introduced because parsing (and validating and checking privileges) is expensive in terms of CPU cycles.

  • New Index optimizer node added to optimize/compact indexing in parallel. Now Indexer runs in two modes default mode or β€œoptimizer” mode.

  • Indexing structure better organized

    • bucket/folder/stage (For staging files).

    • bucket/folder/final/YYYYMMDD (Compacted files from optimizer)

  • Search dynamic caching added for data files and index files.

  • Two search modes were introduced: β€œdynamic” where search reads live data from both indexers and S3. Whereas β€œstatic” mode will enable loading historical S3-only logs within a given date range.

Bug fixes:

  • All the Simple Storage Service (s3) calls based issues has been fixed.

  • To simplify indexer configuration: S3 schema.json loading for indexer is completely removed.

  • Indexer should be started on a fresh new bucket as the old structure/files are not compatible with the new indexer release.

Config changes:

  1. Configuration Changes Indexer:

    • ops.server.mode - set to "optimizer" only for the optimizer pod run

    • data.optimize.schedule.minutes - Frequency of the optimizer run

    • out.kafka.optimize.topic - The kafka topic where optimize will connect with search

    • data.process.config.path is removed - Schema.json is removed

    • schema.tenantid.keys

    • schema.appid.keys

    • schema.message.key (defaults: "message")

    • schema.timestamp.key

    • schema.timestamp.formats (defaults: "%Y-%m-%dT%H:%M:%S" )

  2. Configuration Changes Search:

    • ops.index.cache.path - search local disk file path where index caching is done

    • ops.data.cache.path - search local disk file path where data caching is done

    • search.mode - β€œdynamic” or β€œstatic”

    • search.mode.dynamic.kafka.stage.topic

    • search.mode.dynamic.kafka.final.topic

    • search.mode.daystoinclude

    • search.mode.static.fromdate (example: 20201107)

    • search.mode.static.todate (example: 20201107)

  3. Configuration Changes Gateway:

    • clusters - Search cluster node

Last updated