Security

Invariant Discovery pipeline uses Kafka for messaging. There are several approaches to configure Apache Kafka, including user/password combinations, server authentication or mutual SSL authentication using CA certificate. Security has three primary components:

  • Encryption of data in-flight using SSL/TLS - Data is encrypted between producers and Kafka and between consumers and Kafka.

  • Authentication using SSL or SASL - Allows producers and consumers to authenticate to your Kafka cluster and verify their identity.

  • Authorization using ACLs - Once clients are authenticated, Kafka brokers can use access control lists (ACL) to determine whether or not a particular client is authorized to read/write from a topic.

SSL Authentication

‌SSL Auth leverages 2-way authentication. The idea is to also issue certificates to your clients, signed by a certificate authority, which will allow your Kafka brokers to verify the identity of the clients.

SASL Authentication

‌SASL (Simple Authorization Service Layer) is another authentication mechanism, and the following are supported by Kafka.

‌SASL PLAINTEXT

This is a classic username/password combination. These usernames and passwords have to be stored on the Kafka brokers in advance and each change needs to trigger a rolling restart. When using SASL/PLAINTEXT, make sure to enable SSL encryption so that credentials aren’t sent as PLAINTEXT on the network

SASL SCRAM

This is a username/password combination alongside a challenge (salt), which makes it more secure. When using SASL/SCRAM, make sure to enable SSL encryption so that credentials aren’t sent as PLAINTEXT on the network

As described earlier, Kafka authentication can be done using certificates and trust store. It supports both server-only authentication (certificate presented by server) and mutual authentication (both client and server certificates validated).

Generating Certificates

For using TLS with Kafka, you have to provide the following certificates.

  • Certificate Authority (CA) certificate

  • Kafka Server certificate

  • Kafka Client certificate

OpenSSL and Keytool commands can be used to generate certificates and trust store. You may need root access to carry out some of these tasks.

Configuration for Test Setup

Follow these Steps for testing

Create folder on the server /opt/invariant/security. Create key and trust store

Create a topic for testing

[invapp@myserver bin]$ kafka-topics.sh --create --zookeeper localhost:2183 \
      --replication-factor 1 --partitions 1 --topic sectest

Describe the topic

[invapp@myserver bin]$ kafka-topics.sh --zookeeper localhost:2183 \
    --describe --topic sectest

Test setup prior to enabling SSL

[invapp@myserver bin]$ kafka-console-producer.sh --broker-list  \
   myserver.invariant.io:9094  --topic sectest

Read the messages

[invapp@myserver bin]$ kafka-console-consumer.sh --bootstrap-server localhost:9094 \
   --whitelist 'sectest' --from-beginning

From /opt/invariant/security, create the trust store , import the root cert , intermediary cert

[invapp@myserver bin]$ keytool -keystore kafka.truststore.jks -alias CARoot \
       -import -file INV-0.cer

[invapp@myserver bin]$ keytool -keystore kafka.truststore.jks -alias intermediary \
       -import -file INV-2.cer

Setup Server keystore

Create a PKCS#12 file using OpenSSL

Change the following as necessary CN=myserver.invariant.io, OU=TECH, O=INVARIANT, L=SACRAMENTO, ST=CALIFORNIA, C=US

[invapp@ myserver bin]$ openssl pkcs12 -export -name myserver.invariant.io \
       -in inv101.cer -inkey myserver_cert_key.pem -out keystore.p12

Convert it to JKS using

[invapp@myserver bin]$ keytool -importkeystore -destkeystore myserver.keystore.jks \
    -srckeystore keystore.p12 -srcstoretype pkcs12 -alias myserver.invariant.io

Import the root and intermediary certs

[invapp@myserver bin]$ keytool -keystore myserver.keystore.jks  \
         -alias CARoot -import -file INV-0.cer
         
[invapp@myserver bin]$ keytool -keystore myserver.keystore.jks  \
          -alias intermediary -import -file INV-2.cer

Check the contents of the Keystore

[invapp@myserver bin]$ keytool -list -v -keystore myserver.keystore.jks

To set up the server edit the server.properties and add the following

advertised.listeners=PLAINTEXT://myserver.invariant.io:9094, SSL://myserver.invariant.io:9095
ssl.keystore.location=/opt/invariant/security/myserver.keystore.jks
ssl.keystore.password=password01
ssl.key.password=password01
ssl.truststore.location=/opt/invariant/security/kafka.truststore.jks
ssl.truststore.password=password123
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.client.auth=none
ssl.secure.random.implementation=SHA1PRNG

Startup the server and test the setup

openssl s_client -debug -connect myserver.invariant.io:9095 -tls1

Setup the client store

Create the client keystore , CN=myserver.invariant.io, OU=TECH, O=INVARIANT, L=SACRAMENTO, ST=CALIFORNIA, C=US

[invapp@myserver security]$ keytool -genkey -alias localhost \
       -keyalg RSA -keypass password01 -storepass password01 \
       -keystore client.keystore.jks

Import root cert into keystore

[invapp@myserver security]$  keytool -keystore client.keystore.jks  \
      -alias CARoot -import -file INV-0.cer

[invapp@myserver security]$ keytool -keystore client.keystore.jks  \
      -alias intermediary -import -file INV-2.cer

Import the server cert into keystore 

[invapp@myserver security]$  keytool -keystore client.keystore.jks  \
        -alias localhost -import -file myserver.cer

Add SSL for the client in a new file called ssl.properties 

[invapp@myserver security]$  vi ./config/ssl.properties

Add the following entries

 security.protocol=SSL
 ssl.truststore.location=/opt/invariant/security/kafka.truststore.jks
 ssl.truststore.password=[replace password]
 ssl.keystore.location=/opt/invariant/security/client.keystore.jks
 ssl.keystore.password=[replace password]
 ssl.key.password=[replace password]
 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
 ssl.client.auth=none

Test messages both producer and consumer

/opt/invariant/kafka/bin/kafka-console-producer.sh \
      --broker-list myserver.invariant.io:9095 --topic sectest  \
      --producer.config /opt/invariant/kafka/config/ssl.properties

/opt/invariant/kafka/bin/kafka-console-consumer.sh \
      --bootstrap-server myserver.invariant.io:9095 \
      --whitelist 'sectest' --from-beginning \
      --consumer.config /opt/invariant/kafka/config/ssl.properties

/opt/invariant/kafka/bin/kafka-console-consumer.sh \
   --bootstrap-server myserver.invariant.io:9094 \
   --whitelist 'sectest' --from-beginning

Pipeline Configuration

The data pipelines, which processes the message from Kafka and write out to HDFS stream can use secured Kafka. The administrator can provide the Kafka security settings in the config file security.brkadapter.properties

security.protocol=SASL_SSL
ssl.truststore.location=/opt/server-sec/kafka.truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/opt/client-sec/client.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
‌

Additional Resources

For more details, refer to

PreviousError HandlingNextInventory Manager

Last updated