The data pipeline also consists of a series of definitions which are for additional tuning as well as contains the environment specific details.
avrotypemappingdef.yml
orctypemappingdef.yml
brkadapter.properties
inbound.topics.brkadapter.properties
invariant-hdfs-adapter.yml
Mapping configuration
The data pipeline is configured to translate data types between source and target systems based on the nature of HDFS file format. the supported sources are DB2, Oracle and target HDFS formats are AVRO and ORC.
The avrotypemappingdef.yml is used to define the source datatype to Avro data type.
The orctypemappingdef.yml is used to map source datatype to Orc data types
In this definition CHAR, VARCHAR, TIMESTAMP, DATE, XML datatypes from a DB2 source will be mapped to Avro STRING.
brkadapter.properties is used to define the broker properties used to read the database events streamed from the DBMS. This includes the list of brokers, topic and group information used to read from the brokers.
metadata.outbound.broker.topic=testtopic ==> Topic to publish outbound messages , does not apply to database event processing pipeline
metadata.broker.list=brk1.invariant.io:9092,brk2.invariant.io:9092,brk3.invariant.io:9092 ==> list of broker:port information to be used for reading the message set
metadata.message.processer.group=HdfsGrp28Sep2018 ==> Unique group named used for this pipeline.
metadata.auto.offset.reset=earliest ==> Offset to start the read - earliest reads from the begining of the queue when there is not an offset marked from the process group
metadata.max.partition.fetch.bytes=10240000 ==> Maximum size in bytes for fetching message sets from the broker
Broker Topics
inbound.topics.brkadapter.properties file is used to list the topics from which the data will be consumed. The list of topics should be comma separated
metadata.inbound.broker.topic=table1,table2 ==> Topic containing the database events used by the pipeline.
HDFS Adapter Configuration
invariant-hdfs-adapter.yml is used to configure the Hadoop environment variables, target schema as well as credentials used to interact with HDFS.
hadoop.home: /opt/inv/current/hadoop-client/ ==> Hadoop home to source the libraries required by the processer
hive.home: /opt/inv/current/hive-server2/ ==> Hive home to source the libraries required by the processer
connect.hdfs.principal: invapp ==> Hadoop user for the processer to write to hdfs and perform supporting operations
hadoop.conf.dir: /opt/inv/current/hadoop-client/conf/ ==> Hadoop configuration directory to source Hadoop configurations required by the processer
hdfs.url: hdfs://invnmnd.invariant.io:8020 ==> Hadoop Name Node URI
hive.metastore.uris: thrift://invmeta.invariant.io:9083 ==> Hive MetaStore URI
hive.integration: true ==> Update hive metastore with partition , data file information
hive.database: stream ==> target database for writes
tables.dir: stream ==> target directory to persist data files
hadoop.hive.warehouse.basedir: /etl/ ==> Top level directory for stream system
flush.size: 50 ==> Write size to flush data in memory
hdfs.authentication.kerberos: false ==> kerberos based authentication flag
hdfs.namenode.principal: invapp ==> kerberos principal used for hdfs interactions
connect.hdfs.keytab: ./invapp.server.keytab ==> kerberos keytab used for authentication
HDFS_AUTHENTICATION_KERBEROS_CONFIG: false ==> kerberos based authentication flag, param to be deprecated in future updates
partitioner.class:
io.invariant.invhdfsadapter.partitioner.DailyPartitioner ==> partitioning strategy used for filenaming, file splits
locale: Locale.US ==> locale used for pipeline processing
timezone: America/Los_Angeles ==> time zone used for pipeline processing
