Documents
  • Invariant Documents
  • Platform
    • Data Platform
      • Install Overview
      • System Requirement
      • Software Requirement
      • Prepare the Environment
      • Installing Ambari Server
      • Setup Ambari Server
      • Start Ambari Server
      • Single Node Install
      • Multi-Node Cluster Install
      • Cluster Install from Ambari
      • Run and monitor HDFS
    • Apache Hadoop
      • Compatible Hadoop Versions
      • HDFS
        • HDFS Architecture
        • Name Node
        • Data Node
        • File Organization
        • Storage Format
          • ORC
          • Parquet
        • Schema Design
      • Hive
        • Data Organization
        • Data Types
        • Data Definition
        • Data Manipulation
          • CRUD Statement
            • Views, Indexes, Temporary Tables
        • Cost-based SQL Optimization
        • Subqueries
        • Common Table Expression
        • Transactions
        • SerDe
          • XML
          • JSON
        • UDF
      • Oozie
      • Sqoop
        • Commands
        • Import
      • YARN
        • Overview
        • Accessing YARN Logs
    • Apache Kafka
      • Compatible Kafka Versions
      • Installation
    • Elasticsearch
      • Compatible Elasticsearch Versions
      • Installation
  • Discovery
    • Introduction
      • Release Notes
    • Methodology
    • Discovery Pipeline
      • Installation
      • DB Event Listener
      • Pipeline Configuration
      • Error Handling
      • Security
    • Inventory Manager
      • Installation
      • Metadata Management
      • Column Mapping
      • Service Configuration
      • Metadata Configuration
      • Metadata Changes and Versioning
        • Generating Artifacts
      • Reconciliation, Merging Current View
        • Running daily reconciliation and merge
      • Data Inventory Reports
    • Schema Registry
  • Process Insight
    • Process Insight
      • Overview
    • Process Pipeline
      • Data Ingestion
      • Data Storage
    • Process Dashboards
      • Panels
      • Templating
      • Alerts
        • Rules
        • Notifications
  • Content Insight
    • Content Insight
      • Release Notes
      • Configuration
      • Content Indexing Pipeline
    • Management API
    • Query DSL
    • Configuration
  • Document Flow
    • Overview
  • Polyglot Data Manager
    • Polyglot Data Manager
      • Release Notes
    • Data Store
      • Concepts
      • Sharding
    • Shippers
      • Filerelay Container
    • Processors
    • Search
    • User Interface
  • Operational Insight
    • Operational Insight
      • Release Notes
    • Data Store
      • Concepts
      • Sharding
    • Shippers
      • Filerelay Container
    • Processors
    • Search
    • User Interface
  • Data Science
    • Data Science Notebook
      • Setup JupyterLab
      • Configuration
        • Configuration Settings
        • Libraries
    • Spark DataHub
      • Concepts
      • Cluster Setup
      • Spark with YARN
      • PySpark Setup
        • DataFrame API
      • Reference
  • Product Roadmap
    • Roadmap
  • TIPS
    • Service Troubleshooting
    • Service Startup Errors
    • Debugging YARN Applications
      • YARN CLI
    • Hadoop Credentials
    • Sqoop Troubleshooting
    • Log4j Vulnerability Fix
Powered by GitBook
On this page
  • 1.0.9.15 - 2020-03-15
  • Fixed
  • Added
  • 1.0.9.13 - 2020-02-24
  • Fixed
  • Added
  • 1.0.9.12 - 2020-02-20
  • Fixed
  • 1.0.9 - 2019-12-20
  • Fixed
  • Changed
  • Added
  • 1.0.8 - 2019-06-15
  • Added
  1. Discovery
  2. Introduction

Release Notes

1.0.9.15 - 2020-03-15

Fixed

  • Leap year fix for old dates - Correct date time transformation for timestamps older than 1582 due to calendar switch

Added

  • MS SQL Server AD Auth - Use AD authentication for connecting to SQL Server from inventory manager

  • Add flag to create partition based on local timezone

1.0.9.13 - 2020-02-24

Fixed

  • Current view timestamp - Merge HQL not updating inv_updtd_dtm on updates for EP and CDC

Added

  • Kafka Offset - add ppl_offset to sort order in CDC current view merge

1.0.9.12 - 2020-02-20

Fixed

  • EP HQL Generator - Add missing delimiter for insert statement in merge

1.0.9 - 2019-12-20

Fixed

  • Schema versioning - Removed the need to manually update version to handle schema changes. Admins can auto-increment schema versions

Changed

  • Artifact Generator -

    • Users can pass the artifact generation script a flag to distinguish between event publication and CDC streams

    • DDL generators can map elements in AVRO payload to user defined columns in destination tables

    • Generate all option added to generate all artifacts in a single invocation.

Added

  • Pipeline now supports Centralized Schema registry for use with ingestion

  • AVRO payload support

  • CDC table specific mapping and data pipeline

  • Source tables from one database can be mapped to one or more target schema

  • Kafka 1.1.1 supported by data pipeline

1.0.8 - 2019-06-15

Added

  • Autorecon and merge process to roll up streamed data into final target tables

    • Internal Scheduler for merge

    • Hooks for external triggering merge

  • DDL generator for stream, recon and current view tables

  • HQL generator for auto-recon and merge

  • Sqoop generator for sqooping data into recon tables

  • Data inventory reporting from source RDBMS and Hive tables

PreviousIntroductionNextMethodology

Last updated 5 years ago

Discovery data pipelines can now work with , which is a row-oriented remote procedure call and data serialization framework. It uses JSON for defining data types and protocols, and serializes data in a compact binary format

AVRO