X

Best Practices from Oracle Development's A‑Team

Oracle GoldenGate Big Data Adapter: Establishing Secure Connections to Apache Kafka

Introduction

When publishing data to Apache Kafka via the Oracle GoldenGate Big Data Kafka Handler, it is a good practice to establish secure connections in order to protect sensitive data from un-authorized snooping. The Oracle Big Data Kafka Handler leverages encryption and authentication features built-in to Apache Kafka. In this article we shall detail the Oracle GoldenGate Big Data Apache Handler configuration settings to create secure connections.

This document covers functionality present in Oracle GoldenGate version 12.3; which may not be available in earlier product releases.

The concepts, scripts, and information presented in this article are for educational purposes only. They are not supported by Oracle Development or Support, and come with no guarantee or warrant for functionality in any environment other than the test system used to prepare this article. Before applying any changes presented in this article to your environment, you should thoroughly test to assess functionality and performance implications.

Main Article

This is a continuation of the article, Oracle GoldenGate Big Data Adapter: Apache Kafka Producer, which covers the basics of configuring the Oracle GoldenGate Big Data Adapter as an Apache Kafka Producer. Therefore, we shall not be covering that information here.

Before we start configuring the Oracle GoldenGate Big Data Kafka Handler for secure communications it is good to know that Secure Sockets Layer (SSL) was deprecated in June 2015 and should not be used in production implementations. TLS, or Transport Layer Security, is the current industry standard for establishing secure communications protocols. However, for historical reasons, Kafka and Oracle use the term SSL instead of TLS in configuration and code, which can be a bit confusing.

In this article, any reference to SSL is using the TLS protocol.

TLS Authentication

TLS keys are generated and installed on the Apache Kafka Cluster by the end user. To enable TLS security for Producer clients such as the Oracle GoldenGate Big Data Kafka Handler; (1) the Kafka Broker must be configured to accept SSL connections and (2) a keystore and/or truststore must be created for each Kafka Client.

The client truststore is a file that contains certificates of trusted TLS/SSL servers, or of Certificate Authorities trusted to identify servers. The client truststore are used to determine if a connecting server should be trusted. The client keystore contains authentication credentials used to establish a secure connection between two processes.

In order to properly configure the Kafka Handler, we need to know some details about the Apache Kafka Cluster:

1. The truststore file location.
2. The truststore password.
3. If client authentication is required:
a) The keystore file location.
b) The keystore password.
c) The key password.

In my sandbox, the truststore and keystore file locations are:

/home/oracle/kafka.client.keystore.jks
/home/oracle/kafka.client.truststore.jks

The password I used when creating my truststore and keystore is a not very safe or secure: Oracle1!

Now that I have the required information, I can add security to my custom_kafka_producer.properties file:

bootstrap.servers=kafka-0:9092,kafka-0:9093,kafka-0:9094
acks=1
compression.type=gzip
reconnect.backoff.ms=1000
#
value.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
#
# 100KB per partition
batch.size = 102400
linger.ms = 10000
max.request.size = 5024000
send.buffer.bytes = 5024000
#
#TLS Security
security.protocol=SSL
ssl.truststore.location=/home/oracle/kafka.client.truststore.jks
ssl.truststore.password=Oracle1!
ssl.keystore.location=/home/oracle/kafka.client.keystore.jks
ssl.keystore.password=Oracle1!
ssl.key.password=Oracle1!

Start my Replicat that is acting as the Apache Kafka Producer:

GGSCI (kafka-0.localdomain) 12> start rkafka                                                              Sending START request to MANAGER ...
REPLICAT RKAFKA starting

GGSCI (kafka-0.localdomain) 13> info rkafka                                                                REPLICAT   RKAFKA    Last Started 2016-07-06 12:04   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:10 ago)
Process ID           8649
Log Read Checkpoint  File ./dirdat/kf000000005
First Record  RBA 0

When transactions from my source database are published to the Kafka Broker, I can see the data coming in by starting a Kafka Consumer process:

[oracle@kafka-0]$ $KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic oggtopic --new-consumer --consumer.config ./config/client-ssl.properties
TPC.ORDERS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.114000(00000000060000008171 ORDERS_ID @ (@ $Darryl Lookinglass 7318Peachtree St Silver Lake
80910 CA United States 6399421964 BDarrylLookinglass@Lookinglass.com @ $Darryl Lookinglass 7318Peachtree St Silver Lake
80910 CA United States @ $Darryl Lookinglass 7318Peachtree St Silver Lake
80910 CA United States @EPay $Darryl Lookinglass :2016-07-06:12:06:16.869126000 �? USD �?&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243000(00000000060000009846 $ORDERS_PRODUCTS_ID @Y@ @ ,@ DVD-REDC Red Corner @@ ��@ �M@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243001(00000000060000011996 $ORDERS_PRODUCTS_ID �Y@ @ *@ DVD-LTWP Lethal Weapon ��Q�~A@ �z �G��@ �E@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243002(00000000060000012926 $ORDERS_PRODUCTS_ID �Y@ @ *@ DVD-LTWP Lethal Weapon ��Q�~A@ ��(\�@ B@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243003(00000000060000013855 $ORDERS_PRODUCTS_ID Z@ @ 5@ PC-SWAT3 :SWAT 3: Close Quarters Battle �(\�S@ ��(\_�@ �G@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243004(00000000060000014792 $ORDERS_PRODUCTS_ID @Z@ @ $@ DVD-UNSG2 <Under Siege 2 - Dark Territory =
ףp�=@ n�@ 9@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243005(00000000060000015739 $ORDERS_PRODUCTS_ID �Z@ @ @ DVD-RPMK .The Replacement Killers E@ ��@ �@@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243006(00000000060000016689 $ORDERS_PRODUCTS_ID �Z@ @ 0@ DVD-CUFI $Courage Under Fire ��Q�~C@ R� ��r�@ ;@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.243007(00000000060000017633 $ORDERS_PRODUCTS_ID [@ @ 4@ DVD-BELOVED Beloved ��Q�~K@ ���� U�@ �F@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.244000(00000000060000018553 $ORDERS_PRODUCTS_ID @[@ @ 4@ DVD-BELOVED Beloved ��Q�~K@ �(\��@ ?@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.244001(00000000060000019472 $ORDERS_PRODUCTS_ID �[@ @ 8@ PC-DISC .Disciples: Sacred Lands �V@ ��@ Q@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.244002(00000000060000020422 $ORDERS_PRODUCTS_ID �[@ @ @ DVD-MATR The Matrix ��Q��C@ =
ף�@ �J@&TPC.ORDERS_PRODUCTS I42016-07-06 16:06:15.99725242016-07-06T12:06:23.244003(00000000060000021342 $ORDERS_PRODUCTS_ID \@ @ (@ DVD-DHWV 2Die Hard With A Vengeance ��Q��C@ R� ��ޠ@ K@2TPC.ORDERS_STATUS_HISTORY I42016-07-06 16:06:15.99725242016-07-06T12:06:23.245000(00000000060000022892 0ORDERS_STATUS_HISTORY_ID ORDERS_ID DATE_ADDED @ @ �? :2016-07-06:12:06:16.898211000 �? BOrder received, customer notified

The Apache Kafka documentation lists additional options that may need to be set, depending upon the Kafka Broker configuration. These additional configuration options are:

1. ssl.provider - The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
2. ssl.cipher.suites - A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.
3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 - If required, specify at least one of the protocols configured on the broker side.
4. ssl.truststore.type=JKS
5. ssl.keystore.type=JKS

 

SASL Authentication

SASL authentication requires a Kerberos server properly configured to provide Kafka Producer authentication and the Kafka server properly configured as a Kerberos client. The setup and configuration of the Kerberos server and client is beyond the scope of this document. However, if your organization is using Kerberos and your Kerberos and Kafka Administrators have properly configured the server and client; the following changes to my custom_kafka_producer.properties file will authentication the Oracle GoldenGate Big Data Apache Handler connection via the SASL protocol:

bootstrap.servers=kafka-0:9092,kafka-0:9093,kafka-0:9094
acks=1
compression.type=gzip
reconnect.backoff.ms=1000
#
value.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
#
# 100KB per partition
batch.size = 102400
linger.ms = 10000
max.request.size = 5024000
send.buffer.bytes = 5024000
#                                                                                                            #SASL Security
security.protocol=SASL_PLAINTEXT
#
# If Kerberos SSL authentication is enabled
#security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka

 Start my Replicat acting as the Apache Kafka Producer:

GGSCI (kafka-0.localdomain) 16> start rkafka                                                              Sending START request to MANAGER ...
REPLICAT RKAFKA starting

GGSCI (kafka-0.localdomain) 17> info rkafka                                                                REPLICAT   RKAFKA    Last Started 2016-07-06 15:22   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:05 ago)
Process ID           15577
Log Read Checkpoint  File ./dirdat/kf000000006
2016-07-06 15:23:08.998727  RBA 51095

When transactions from my source database are published to the Kafka Broker, I can see the data coming in by starting a Kafka Consumer process:

TPC.ORDERS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.732000(00000000060000038862 ORDERS_ID $@ �? Loren Penton $8562Hermindger Ave Phoenix
46370 NV United States 504.555.1212 *loren@lorenpenton.com @ Loren Penton $8562Hermindger Ave Phoenix
46370 NV United States @ Loren Penton $8562Hermindger Ave Phoenix
46370 NV United States @ Credit Card MasterCard Loren Penton 96289229417941122.14 :2016-07-06:15:23:09.445591000 �? USD �?&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.749000(00000000060000039689 $ORDERS_PRODUCTS_ID �_@ $@ @ DVD-YGEM You've Got Mail ��Q�~A@ �(\�n@ @&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.749001(00000000060000040652 $ORDERS_PRODUCTS_ID `@ $@ (@ DVD-DHWV 2Die Hard With A Vengeance ��Q��C@ �G�z ��@ H@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.749002(00000000060000041587 $ORDERS_PRODUCTS_ID `@ $@ 1@ DVD-SPEED
Speed ��Q��C@ R� ��ސ@ ;@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.749003(00000000060000042496 $ORDERS_PRODUCTS_ID @`@ $@ @ DVD-RPMK .The Replacement Killers E@ L�@ �Q@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750000(00000000060000043447 $ORDERS_PRODUCTS_ID ``@ $@ 4@ DVD-BELOVED Beloved ��Q�~K@ ���� U�@ �V@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750001(00000000060000044366 $ORDERS_PRODUCTS_ID �`@ $@ :@ MSIMEXP >Microsoft IntelliMouse Explorer �����<P@ �}�@ @P@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750002(00000000060000045316 $ORDERS_PRODUCTS_ID �`@ $@ �? MG200MMS Matrox G200 MMS �p=
׿r@ R� ���@ �D@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750003(00000000060000046243 $ORDERS_PRODUCTS_ID �`@ $@ "@ DVD-UNSG Under Siege =
ףp�=@ �(\6�@ �D@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750004(00000000060000047166 $ORDERS_PRODUCTS_ID �`@ $@ <@ GT-P1000 $Samsung Galaxy Tab R� ��o�@ R� ��o�@ @&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750005(00000000060000048102 $ORDERS_PRODUCTS_ID a@ $@ <@ GT-P1000 $Samsung Galaxy Tab R� ��o�@ �p=
g��@ �S@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750006(00000000060000049038 $ORDERS_PRODUCTS_ID a@ $@ .@ DVD-FRAN Frantic �A@ :�@ �O@&TPC.ORDERS_PRODUCTS I42016-07-06 19:23:08.99872742016-07-06T15:23:14.750007(00000000060000049949 $ORDERS_PRODUCTS_ID @a@ $@ (@ DVD-DHWV 2Die Hard With A Vengeance ��Q��C@ { �G�~�@ L@2TPC.ORDERS_STATUS_HISTORY I42016-07-06 19:23:08.99872742016-07-06T15:23:14.754000(00000000060000050883 0ORDERS_STATUS_HISTORY_ID ORDERS_ID DATE_ADDED @ $@ �? :2016-07-06:15:23:09.454486000 �? BOrder received, customer notified

 

Summary

In this article we presented settings for the Oracle GoldenGate Big Data Adapter that allow for secure connections via TLS and SASL to Apache Kafka Brokers.

For more information on what other articles are available for Oracle GoldenGate please view our index page.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha

Recent Content