Release Notes

1.0.9.15 - 2020-03-15

Leap year fix for old dates - Correct date time transformation for timestamps older than 1582 due to calendar switch

MS SQL Server AD Auth - Use AD authentication for connecting to SQL Server from inventory manager
Add flag to create partition based on local timezone

Current view timestamp - Merge HQL not updating inv_updtd_dtm on updates for EP and CDC

Schema versioning - Removed the need to manually update version to handle schema changes. Admins can auto-increment schema versions

Artifact Generator -
- Users can pass the artifact generation script a flag to distinguish between event publication and CDC streams
- DDL generators can map elements in AVRO payload to user defined columns in destination tables
- Generate all option added to generate all artifacts in a single invocation.

Pipeline now supports Centralized Schema registry for use with ingestion
AVRO payload support
- Discovery data pipelines can now work with AVRO, which is a row-oriented remote procedure call and data serialization framework. It uses JSON for defining data types and protocols, and serializes data in a compact binary format
CDC table specific mapping and data pipeline
Source tables from one database can be mapped to one or more target schema
Kafka 1.1.1 supported by data pipeline

Autorecon and merge process to roll up streamed data into final target tables
- Internal Scheduler for merge
- Hooks for external triggering merge
DDL generator for stream, recon and current view tables
HQL generator for auto-recon and merge
Sqoop generator for sqooping data into recon tables
Data inventory reporting from source RDBMS and Hive tables

Last updated 4 years ago