Oracle GoldenGate Big Data Adapter: Establishing Secure Connections to Apache Kafka

Introduction

When publishing data to Apache Kafka via the Oracle GoldenGate Big Data Kafka Handler, it is a good practice to establish secure connections in order to protect sensitive data from un-authorized snooping. The Oracle Big Data Kafka Handler leverages encryption and authentication features built-in to Apache Kafka. In this article we shall detail the Oracle GoldenGate Big Data Apache Handler configuration settings to create secure connections.

This document covers functionality present in Oracle GoldenGate version 12.3; which may not be available in earlier product releases.

The concepts, scripts, and information presented in this article are for educational purposes only. They are not supported by Oracle Development or Support, and come with no guarantee or warrant for functionality in any environment other than the test system used to prepare this article. Before applying any changes presented in this article to your environment, you should thoroughly test to assess functionality and performance implications.

Main Article

This is a continuation of the article, Oracle GoldenGate Big Data Adapter: Apache Kafka Producer, which covers the basics of configuring the Oracle GoldenGate Big Data Adapter as an Apache Kafka Producer. Therefore, we shall not be covering that information here.

Before we start configuring the Oracle GoldenGate Big Data Kafka Handler for secure communications it is good to know that Secure Sockets Layer (SSL) was deprecated in June 2015 and should not be used in production implementations. TLS, or Transport Layer Security, is the current industry standard for establishing secure communications protocols. However, for historical reasons, Kafka and Oracle use the term SSL instead of TLS in configuration and code, which can be a bit confusing.

In this article, any reference to SSL is using the TLS protocol.

TLS Authentication

TLS keys are generated and installed on the Apache Kafka Cluster by the end user. To enable TLS security for Producer clients such as the Oracle GoldenGate Big Data Kafka Handler; (1) the Kafka Broker must be configured to accept SSL connections and (2) a keystore and/or truststore must be created for each Kafka Client.

The client truststore is a file that contains certificates of trusted TLS/SSL servers, or of Certificate Authorities trusted to identify servers. The client truststore are used to determine if a connecting server should be trusted. The client keystore contains authentication credentials used to establish a secure connection between two processes.

In order to properly configure the Kafka Handler, we need to know some details about the Apache Kafka Cluster:

1. The truststore file location.
2. The truststore password.
3. If client authentication is required:
a) The keystore file location.
b) The keystore password.
c) The key password.

In my sandbox, the truststore and keystore file locations are:

/home/oracle/kafka.client.keystore.jks
/home/oracle/kafka.client.truststore.jks

The password I used when creating my truststore and keystore is a not very safe or secure: Oracle1!

Now that I have the required information, I can add security to my custom_kafka_producer.properties file:

bootstrap.servers=kafka-0:9092,kafka-0:9093,kafka-0:9094
acks=1
compression.type=gzip
reconnect.backoff.ms=1000
#
value.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
#
# 100KB per partition
batch.size = 102400
linger.ms = 10000
max.request.size = 5024000
send.buffer.bytes = 5024000
#
#TLS Security
security.protocol=SSL
ssl.truststore.location=/home/oracle/kafka.client.truststore.jks
ssl.truststore.password=Oracle1!
ssl.keystore.location=/home/oracle/kafka.client.keystore.jks
ssl.keystore.password=Oracle1!
ssl.key.password=Oracle1!

Start my Replicat that is acting as the Apache Kafka Producer:

GGSCI (kafka-0.localdomain) 12> start rkafka                                                              Sending START request to MANAGER …
REPLICAT RKAFKA starting

GGSCI (kafka-0.localdomain) 13> info rkafka                                                                REPLICAT   RKAFKA    Last Started 2016-07-06 12:04   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:10 ago)
Process ID           8649
Log Read Checkpoint  File ./dirdat/kf000000005
First Record  RBA 0

When transactions from my source database are published to the Kafka Broker, I can see the data coming in by starting a Kafka Consumer process:

[oracle@kafka-0]$ $KAFKA_HOME/bin/kafka-console-consumer.sh –bootstrap-server localhost:9093 –topic oggtopic –new-consumer –consumer.config ./config/client-ssl.properties
TPC.ORDERSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.114000(00000000060000008171ORDERS_ID @(@$Darryl Lookinglass 7318Peachtree StSilver Lake
80910CAUnited States6399421964BDarrylLookinglass@Lookinglass.com@$Darryl Lookinglass 7318Peachtree StSilver Lake
80910CAUnited States@$Darryl Lookinglass 7318Peachtree StSilver Lake
80910CAUnited States@EPay$Darryl Lookinglass:2016-07-06:12:06:16.869126000�?USD�?&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243000(00000000060000009846$ORDERS_PRODUCTS_ID@Y@ @,@DVD-REDCRed Corner@@��@�M@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243001(00000000060000011996$ORDERS_PRODUCTS_ID�Y@ @*@DVD-LTWPLethal Weapon��Q�~A@�z�G��@�E@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243002(00000000060000012926$ORDERS_PRODUCTS_ID�Y@ @*@DVD-LTWPLethal Weapon��Q�~A@��(\�@B@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243003(00000000060000013855$ORDERS_PRODUCTS_IDZ@ @5@PC-SWAT3:SWAT 3: Close Quarters Battle�(\�S@��(\_�@�G@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243004(00000000060000014792$ORDERS_PRODUCTS_ID@Z@ @$@DVD-UNSG2<Under Siege 2 – Dark Territory=
ףp�=@n�@9@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243005(00000000060000015739$ORDERS_PRODUCTS_ID�Z@ @@DVD-RPMK.The Replacement KillersE@��@�@@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243006(00000000060000016689$ORDERS_PRODUCTS_ID�Z@ @0@DVD-CUFI$Courage Under Fire��Q�~C@R���r�@;@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.243007(00000000060000017633$ORDERS_PRODUCTS_ID[@ @4@DVD-BELOVEDBeloved��Q�~K@����U�@�F@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.244000(00000000060000018553$ORDERS_PRODUCTS_ID@[@ @4@DVD-BELOVEDBeloved��Q�~K@�(\��@?@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.244001(00000000060000019472$ORDERS_PRODUCTS_ID�[@ @8@PC-DISC.Disciples: Sacred Lands�V@��@Q@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.244002(00000000060000020422$ORDERS_PRODUCTS_ID�[@ @@DVD-MATRThe Matrix��Q��C@=
ף�@�J@&TPC.ORDERS_PRODUCTSI42016-07-06 16:06:15.99725242016-07-06T12:06:23.244003(00000000060000021342$ORDERS_PRODUCTS_ID\@ @(@DVD-DHWV2Die Hard With A Vengeance��Q��C@R���ޠ@K@2TPC.ORDERS_STATUS_HISTORYI42016-07-06 16:06:15.99725242016-07-06T12:06:23.245000(000000000600000228920ORDERS_STATUS_HISTORY_IDORDERS_IDDATE_ADDED@ @�?:2016-07-06:12:06:16.898211000�?BOrder received, customer notified

The Apache Kafka documentation lists additional options that may need to be set, depending upon the Kafka Broker configuration. These additional configuration options are:

1. ssl.provider – The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
2. ssl.cipher.suites – A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.
3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 – If required, specify at least one of the protocols configured on the broker side.
4. ssl.truststore.type=JKS
5. ssl.keystore.type=JKS

 

SASL Authentication

SASL authentication requires a Kerberos server properly configured to provide Kafka Producer authentication and the Kafka server properly configured as a Kerberos client. The setup and configuration of the Kerberos server and client is beyond the scope of this document. However, if your organization is using Kerberos and your Kerberos and Kafka Administrators have properly configured the server and client; the following changes to my custom_kafka_producer.properties file will authentication the Oracle GoldenGate Big Data Apache Handler connection via the SASL protocol:

bootstrap.servers=kafka-0:9092,kafka-0:9093,kafka-0:9094
acks=1
compression.type=gzip
reconnect.backoff.ms=1000
#
value.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
#
# 100KB per partition
batch.size = 102400
linger.ms = 10000
max.request.size = 5024000
send.buffer.bytes = 5024000
#                                                                                                            #SASL Security
security.protocol=SASL_PLAINTEXT
#
# If Kerberos SSL authentication is enabled
#security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka

 Start my Replicat acting as the Apache Kafka Producer:

GGSCI (kafka-0.localdomain) 16> start rkafka                                                              Sending START request to MANAGER …
REPLICAT RKAFKA starting

GGSCI (kafka-0.localdomain) 17> info rkafka                                                                REPLICAT   RKAFKA    Last Started 2016-07-06 15:22   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:05 ago)
Process ID           15577
Log Read Checkpoint  File ./dirdat/kf000000006
2016-07-06 15:23:08.998727  RBA 51095

When transactions from my source database are published to the Kafka Broker, I can see the data coming in by starting a Kafka Consumer process:

TPC.ORDERSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.732000(00000000060000038862ORDERS_ID$@�?Loren Penton$8562Hermindger AvePhoenix
46370NVUnited States504.555.1212*loren@lorenpenton.com@Loren Penton$8562Hermindger AvePhoenix
46370NVUnited States@Loren Penton$8562Hermindger AvePhoenix
46370NVUnited States@Credit CardMasterCardLoren Penton 96289229417941122.14:2016-07-06:15:23:09.445591000�?USD�?&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.749000(00000000060000039689$ORDERS_PRODUCTS_ID�_@$@@DVD-YGEMYou’ve Got Mail��Q�~A@�(\�n@@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.749001(00000000060000040652$ORDERS_PRODUCTS_ID`@$@(@DVD-DHWV2Die Hard With A Vengeance��Q��C@�G�z��@H@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.749002(00000000060000041587$ORDERS_PRODUCTS_ID `@$@1@DVD-SPEED
Speed��Q��C@R���ސ@;@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.749003(00000000060000042496$ORDERS_PRODUCTS_ID@`@$@@DVD-RPMK.The Replacement KillersE@L�@�Q@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750000(00000000060000043447$ORDERS_PRODUCTS_ID“@$@4@DVD-BELOVEDBeloved��Q�~K@����U�@�V@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750001(00000000060000044366$ORDERS_PRODUCTS_ID�`@$@:@MSIMEXP>Microsoft IntelliMouse Explorer�����<P@�}�@@P@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750002(00000000060000045316$ORDERS_PRODUCTS_ID�`@$@�?MG200MMSMatrox G200 MMS�p=
׿r@R����@�D@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750003(00000000060000046243$ORDERS_PRODUCTS_ID�`@$@”@DVD-UNSGUnder Siege=
ףp�=@�(\6�@�D@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750004(00000000060000047166$ORDERS_PRODUCTS_ID�`@$@<@GT-P1000$Samsung Galaxy TabR���o�@R���o�@@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750005(00000000060000048102$ORDERS_PRODUCTS_IDa@$@<@GT-P1000$Samsung Galaxy TabR���o�@�p=
g��@�S@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750006(00000000060000049038$ORDERS_PRODUCTS_ID a@$@.@DVD-FRANFrantic�A@:�@�O@&TPC.ORDERS_PRODUCTSI42016-07-06 19:23:08.99872742016-07-06T15:23:14.750007(00000000060000049949$ORDERS_PRODUCTS_ID@a@$@(@DVD-DHWV2Die Hard With A Vengeance��Q��C@{�G�~�@L@2TPC.ORDERS_STATUS_HISTORYI42016-07-06 19:23:08.99872742016-07-06T15:23:14.754000(000000000600000508830ORDERS_STATUS_HISTORY_IDORDERS_IDDATE_ADDED@$@�?:2016-07-06:15:23:09.454486000�?BOrder received, customer notified

 

Summary

In this article we presented settings for the Oracle GoldenGate Big Data Adapter that allow for secure connections via TLS and SASL to Apache Kafka Brokers.

For more information on what other articles are available for Oracle GoldenGate please view our index page.

Comments

  1. Bibin John says:

    Team,
    How can in implement a custom authorization frame work before replicating/publishing into kafka like i can provide a user name and password which needs to be validated as a first step after starting replicats.

Add Your Comment