🚀
0.5
Released on
- Cache library - This feature is introduced because parsing (and validating and checking privileges) is expensive in terms of CPU cycles.
- New Index optimizer node added to optimize/compact indexing in parallel. Now Indexer runs in two modes default mode or “optimizer” mode.
- Indexing structure better organized
- bucket/folder/stage (For staging files).
- bucket/folder/final/YYYYMMDD (Compacted files from optimizer)
- Search dynamic caching added for data files and index files.
- Two search modes were introduced: “dynamic” where search reads live data from both indexers and S3. Whereas “static” mode will enable loading historical S3-only logs within a given date range.
- All the Simple Storage Service (s3) calls based issues has been fixed.
- To simplify indexer configuration: S3 schema.json loading for indexer is completely removed.
- Indexer should be started on a fresh new bucket as the old structure/files are not compatible with the new indexer release.
- 1.Configuration Changes Indexer:
- ops.server.mode - set to "optimizer" only for the optimizer pod run
- data.optimize.schedule.minutes - Frequency of the optimizer run
- out.kafka.optimize.topic - The kafka topic where optimize will connect with search
- data.process.config.path is removed - Schema.json is removed
- schema.tenantid.keys
- schema.appid.keys
- schema.message.key (defaults: "message")
- schema.timestamp.key
- schema.timestamp.formats (defaults: "%Y-%m-%dT%H:%M:%S" )
- 2.Configuration Changes Search:
- ops.index.cache.path - search local disk file path where index caching is done
- ops.data.cache.path - search local disk file path where data caching is done
- search.mode - “dynamic” or “static”
- search.mode.dynamic.kafka.stage.topic
- search.mode.dynamic.kafka.final.topic
- search.mode.daystoinclude
- search.mode.static.fromdate (example: 20201107)
- search.mode.static.todate (example: 20201107)
- 3.Configuration Changes Gateway:
- clusters - Search cluster node
Last modified 1yr ago