Data Ingestion

Key to achieving real time insights is the ability to continuously ingest and process business event data at scale. The data pipeline is built using Apache Kafka, which provides the backbone of the event data pipeline infrastructure. Kafka is a high throughput messaging system which can deliver message stream for low latency analysis.

For near real-time analysis and dashboards, it is important that data be delivered without adding load to the source system. Latency requirements eliminate the batch pull cycles as an option. Invariant makes use of database replication technologies and write ahead logging to overcome these shortcomings and ensure that data is available as soon as it is generated. Invariant provides database adapters, which can be used to collect data and deliver them to target stores in near real time for analysis and dashboards.