Data Store

Data Storage

For efficient operations, managing data latency is critical. It is important that data be captured and process as it is generated. The storage layer is powered by Elasticsearch storage engine, based on the powerful Lucene engine. The engine provides the ability to index the data as it arrives and make it available for query within seconds. The architecture is horizontally scalable and can scale out to handle large number of events and queries distributed across the cluster. This makes it ideal for use as a near real time search platform, where streams of semi-structured data, including log files, OS and network metrics, can be indexed with very small latency.

The Elasticsearch index store can be used in combination with the other data workflow orchestration and analysis components, which are available as part of the Invariant Data Platform. This provides you the best of both worlds - massive data storage and processing power of the platform combined with real-time search and analytics capabilities of NoSQL engine.