# Configuration

## Pipeline System Configurations 

The data pipeline also consists of a series of definitions which are for additional tuning as well as contains the environment specific details.

* avrotypemappingdef.yml
* orctypemappingdef.yml
* brkadapter.properties
* inbound.topics.brkadapter.properties
* invariant-hdfs-adapter.yml

### Mapping configuration

The data pipeline is configured to translate data types between source and target systems based on the nature of HDFS file format. the supported sources are DB2, Oracle and target HDFS formats are AVRO and ORC.

The avrotypemappingdef.yml is used to define the source datatype to Avro data type.

The orctypemappingdef.yml is used to map source datatype to Orc data types

In this definition CHAR, VARCHAR, TIMESTAMP, DATE, XML datatypes from a DB2 source will be mapped to Avro STRING.

#### avrotypemapping.yml

```
dataserdetype: AVRO
```

```yaml
   mapping:
   - STRING:  => Target (Avro ) Data type
     - !dbtypeMapping
       dbtype: DB2          => Source DBMS
       type: [CHAR,VARCHAR,TIMESTAMP,DATE,XML] => Source DBMS data types
   - DECIMAL:
     - !dbtypeMapping
       dbtype: DB2
       type: [DECIMAL]
   - INT:
     - !dbtypeMapping
       dbtype: DB2
       type: [SMALLINT,INTEGER]
   - BIGINT:
     - !dbtypeMapping
       dbtype: DB2
       type: [BIGINT,LONG]

```

In this definition CHAR, VARCHAR, XML datatypes from a DB2 source will be mapped to ORC STRING.

#### orctypemappingdef.yml

```yaml
dataserdetype: ORC
   mapping:
   - STRING:
     - !dbtypeMapping
       dbtype: DB2
       type: [CHAR,VARCHAR,XML]
   - DATE:
     - !dbtypeMapping
       dbtype: DB2
       type: [DATE]
   - TIMESTAMP:
     - !dbtypeMapping
       dbtype: DB2
       type: [TIMESTAMP]
   - DECIMAL:
     - !dbtypeMapping
       dbtype: DB2
       type: [DECIMAL]
   - INT:
     - !dbtypeMapping
       dbtype: DB2
       type: [SMALLINT,INTEGER]
   - BIGINT:
     - !dbtypeMapping
       dbtype: DB2
       type: [BIGINT,LONG]             
```

### Broker Configuration

brkadapter.properties is used to define the broker properties used to read the database events streamed from the DBMS. This includes the list of brokers, topic and group information used to read from the brokers.

```
metadata.outbound.broker.topic=testtopic            ==> Topic to publish outbound messages , does not apply to database event processing pipeline
metadata.broker.list=brk1.invariant.io:9092,brk2.invariant.io:9092,brk3.invariant.io:9092 ==> list of broker:port information to be used for reading the message set
metadata.message.processer.group=HdfsGrp28Sep2018   ==> Unique group named used for this pipeline.
metadata.auto.offset.reset=earliest                 ==> Offset to start the read - earliest reads from the  begining of the queue when there is not an offset marked from the process group
metadata.max.partition.fetch.bytes=10240000         ==> Maximum size in bytes for fetching message sets from the broker
```

### Broker Topics

inbound.topics.brkadapter.properties file is used to list the topics from which the data will be consumed. The list of topics should be comma separated

```
metadata.inbound.broker.topic=table1,table2      ==> Topic containing the database events used by the pipeline.

```

### HDFS Adapter Configuration

invariant-hdfs-adapter.yml is used to configure the Hadoop environment variables, target schema as well as credentials used to interact with HDFS.   

```
hadoop.home: /opt/inv/current/hadoop-client/            ==> Hadoop home to source the libraries required by the processer
hive.home: /opt/inv/current/hive-server2/               ==> Hive home to source the libraries required by the processer
connect.hdfs.principal: invapp                          ==> Hadoop user for the processer to write to hdfs and perform supporting operations
hadoop.conf.dir: /opt/inv/current/hadoop-client/conf/   ==> Hadoop configuration directory to source Hadoop configurations required by the processer
hdfs.url: hdfs://invnmnd.invariant.io:8020             ==> Hadoop Name Node URI
hive.metastore.uris: thrift://invmeta.invariant.io:9083  ==> Hive MetaStore URI
hive.integration: true                                   ==> Update hive  metastore with partition , data file information
hive.database: stream                                    ==> target database for writes 
tables.dir: stream                                       ==> target directory to persist data files
hadoop.hive.warehouse.basedir: /etl/                     ==> Top level directory for stream system
flush.size: 50                                           ==> Write size to flush data in memory
hdfs.authentication.kerberos: false                      ==> kerberos based authentication flag
hdfs.namenode.principal: invapp                          ==> kerberos principal used for  hdfs interactions
connect.hdfs.keytab: ./invapp.server.keytab              ==> kerberos keytab used for authentication
HDFS_AUTHENTICATION_KERBEROS_CONFIG: false               ==>  kerberos based authentication flag, param to be deprecated in future updates
partitioner.class:
io.invariant.invhdfsadapter.partitioner.DailyPartitioner ==> partitioning strategy used for filenaming, file splits
locale:  Locale.US                                       ==> locale used for pipeline processing
timezone: America/Los_Angeles                            ==> time zone used for pipeline processing
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.invariant.io/content-insight/overview/configuration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.