DB Event Listener

Configuring the DB event listener for discovery pipeline

Configuring DB Event Pipeline with Apache Kafka

The pipeline configuration can be managed by updating the application properties file. The relevant sections are described below

# ------------------------------------------------------
# Invariant DB Event Pub JMS Processor
# ------------------------------------------------------
spring.application.name=invariant-dbevent-mdp
spring.jta.log-dir=./target/tx-object-store

Queue Type

Specify the source Broker type - valid values are WebspgereMQ or ActiveMQ. Next define the broker connection details such as URL, user, password in order to connect to the event publication source.

## Queue Type - WebsphereMQ, ActiveMQ
process-insight.qm.type= ActiveMQ

# ActiveMQ Connection
process-insight.activemq.broker.url=tcp://localhost:61616
process-insight.activemq.broker.username=admin
process-insight.activemq.broker.password=aaaaaaaaaa==:bbbbbbbb/cccccc==

JMS Source

Next specify the queue manager and channel for both the source data queues - ODS and BPM.

# Event Consumer Details
process-insight.mq.ods.queueManager= DEV_ODS_QM
process-insight.mq.ods.host= localhost
process-insight.mq.ods.port= 1416
process-insight.mq.ods.channel= INV_ODSQ_SVRCONN
process-insight.mq.ods.inputQ= SENDQA
process-insight.mq.ods.enabled=true

process-insight.mq.bpm.queueManager= DEV_BPM_QM
process-insight.mq.bpm.host= localhost
process-insight.mq.bpm.port= 1417
process-insight.mq.bpm.channel= INV_BPMQ_SVRCONN
process-insight.mq.bpm.inputQ= SENDQB
process-insight.mq.bpm.enabled=false

Kafka Sink

Finally specify the Kafka sink details including the topic where the data will be written out to

# -- invariant.kafka
process-insight.kafka.brokerlist= localhost:9092
process-insight.kafka.zklist= localhost:2181
process-insight.kafka.sync= synch
process-insight.kafka.producer.qrep.topiclist= QrepSrcJun2016, QrepSrcJun2017
process-insight.kafka.max.request.size=10485760

Logging

Log configuration should be modified for managing log output

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" strict="true">
    <Properties>
        <Property name="filename">/var/log/invariant/dbevent-mdp.log</Property>
        <Property name="PID">????</Property>
        <Property name="LOG_PATTERN">%clr{%d{yyyy-MM-dd HH:mm:ss.SSS}}{faint} %clr{%5p} %clr{${sys:PID}}{magenta}
            %clr{---}{faint} %clr{[%15.15t]}{faint} %clr{%-40.40c{1.}}{cyan} %clr{:}{faint} %m%n%xwEx
        </Property>
    </Properties>
    <Appenders>
        <Console name="STDOUT">
            <PatternLayout pattern="%d{ISO8601} %-5p (%t) [%c{1.}(%M:%L)] %m%n"/>
        </Console>
        <File name="File" fileName="${filename}">
            <PatternLayout
                pattern = "%d{ISO8601} %-5p (%t) [%c{1.}(%M:%L)] %m%n"
            />
        </File>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="STDOUT"/>
            <AppenderRef ref="File"/>
        </Root>
    </Loggers>
</Configuration>