Error Handling

Error handling and fixing errors

Errors are written to the folder defined in the errorlogDir within the pipeline definition. The errors are logged to files named "year.month.day.*.log". Error log contains the offset, error details (an array of errors) and the actual message. Errors can be divided into technical / system errors caused by a multitude of technical reason vs functional mapping, type conversion related issues.

Functional / Mapping errors observed in the system can be fixed using the following means.

One event at a time

• fix the offset that failed after the error condition has been addressed through mapping / type changes

• fix for a range of offsets (filtered by table(s) after the error condition has been addressed through mapping / type changes.

Data for the error fix can be sourced from the broker or the error log record. Please be mindful as the data in broker is only retained for a predefined number of days.