Data Inventory Reports

The records counts are tracked across the source and target views nightly. Every night, counts of all records created, updated for the day based on the audit columns configured in the dataSourceTables are recorded. In addition, the count of number of inserts, updates and deletes from the incremental table are recorded. The record count of the reconciliation is recorded after the sqooping the data. The counts of the records from the current view are recorded after the completion of the merge activities for the day.

The counts across the data sources, targets are used to validate the data moved by the pipeline compared to changes in the source systems and what gets merged to the current views. A set of command line utilities are provided to check the status of the tables individually or as a set for the last day or for a specific time period.

./run-invmgr-report.sh                                                                                                      "
    --csvReport | --consoleReport | --mailReport | --dailyCount ] <- Action Param   
    [-d YYYY-mm-DD]                              <- Optional Processing Date      
    [-s iinv|isrc]                               <- Source - only for daily count 
    [-t csv/html]                                <- Format - only for mailReport  
   

The report can be displayed on the screen using consoleReport or can be sent to a file using csvReport option. For mailing the report, define the list of email ids in the "recon.properties" configuration file.

The processing date can be specified but defaults to previous day if empty.

In addition to command line query capability administrative configurations are available to email an operational group of any errors in data inventory so operational and environments team can troubleshoot further.