Intelligent Devices with Oracle Edge Analytics & MQTT

Introduction

It is becoming increasingly more common to see scenarios where technologies like IoT (Internet of Things) and Event Streaming are combined to create solutions that leverage real-time intelligence for business decision making. Typical scenarios involve monitoring large numbers devices continuously emitting events as a result of some sort of situation detection (commonly detected by specialized sensors) that are processed, in near real-time, by streaming analytics capable technologies such as OSX (Oracle Stream eXplorer).

In this context, OSX is running in some server-side environment such as the Oracle Cloud or On-Premise data centers, and the incoming events must be sent over a network infrastructure in order to be processed. While this seems to be simple, not all use cases allow these device-related events to be easily sent over the network. In fact, most scenarios that leverage sensor-based technologies cannot count on the luxury of having a reliable/stable network infrastructure that connects the edges (sometimes called “The Fog”) with the server-side environment. Ships crossing oceans, cities with poor network bandwidth infrastructure, and cities with regulatory security rules are common examples.

figure_x_bad_network_infrastruecture

There are various alternatives to overcoming these types of network bandwidth limitations. If the device is capable of caching the events for a period of time, then those events could be sent in a batch when the network is available. If they are not capable of caching, they can communicate with a central hub (sitting somewhere near the devices) that will act as concentrator of those events and will dispatch them appropriately. Unfortunately “band-aid” solutions like this normally add a degree of complexity to the development of those devices, with error-prone programming constructs. Not to mention that the events may be sent in a delayed manner where their value will be inherently lost. In order to be considered useful events need to be processed as they happen, and this implies that sometimes the streaming analytics technology needs to be as close to the fog as possible.

Oracle addresses this problem by making the streaming analytics technology from OSX available to be used in the fog, by devices with very restricted hardware. This is possible through the OEA (Oracle Edge Analytics) product, an optimized low memory and disk footprint version of OSX. This product is not necessarily new (it was known as Oracle Event Processing for Java Embedded) but has a new set of features that might provide the missing ingredient to the development of intelligent, event-driven devices, capable of handling pattern matching, event correlation, filtering, and aggregation scenarios.

This article will provide an overview of how to get started with OEA, covering details about installation, domain creation, application deployment, and how to leverage the MQTT protocol in an OEA application. If you need to find more information about OEA please consider reading the Getting Started with Oracle Edge Analytics guide, available under the product documentation.

Installing Oracle Edge Analytics

Installing OEA in a device is very straightforward. It basically involves extracting the contents of the OEA distribution file into a folder. However; every device has its own way to transfer files into its file system, and that part can be tricky sometimes. For the sake of clarity, this article will assume that you are using a Raspberry Pi Model 2 (Pi for short) running the last Noobs operating system. The Pi device used in this article is shown in figure 1.

raspberry_pi_2_model_b

Figure 1: Raspberry Pi Model 2 (ARM 900 MHz CPU with 1GB of RAM) used in this article.

The first step should be creating a folder in your Pi. You must make sure that this folder has read, write and execution privileges; otherwise the product will not work correctly. In order to do these steps, you must have access to your Pi. Most people do this by connecting the device into a Monitor/TV using a HDMI cable. Alternatively, you can setup SSH access in your Pi to be able to connect to it remotely. The second step should be extracting the contents of the OEA distribution file into the folder created. This is shown in figure 2. If you have an OTN (Oracle Technology Network) account, you can get a copy of OEA here.

installingEdgeAnalyticsIntoRaspberryPi

Figure 2: Extracting the contents of the OEA distribution file into a Raspberry Pi folder.

The third step is making sure you have a supported JDK installation in your Pi. Oracle recommends using either JDK 1.8.0_06 (Client JVM Only) or Java Embedded Runtime 1.8.0_33. The easiest way to do this if you are running a Raspbian distribution is using the following command:

apt-get install oracle-java8-jdk

Finally, create an environment variable named JAVA_HOME whose value points to the location where Java is installed. This is very important because most OEA internal scripts expect that this variable is set in order to work. Thus, find where the Java is installed. If you installed the JDK using the apt-get command shown before, then you can query the exact location where it is installed using the dpkg-query command:

dpkg-query -L oracle-java8-jdk | grep /bin/javac

The folder that holds your Java installation will be shown in the output. Copy the entire path, except for the “/bin/javac” in the end. Then, create the JAVA_HOME environment variable using the following command:

export JAVA_HOME=<ORACLE_JAVA8_JDK_INSTALL>

Using the Configuration Wizard

OEA, just like OSX and WebLogic, uses the concept of domains. A domain consists of one or more OEA servers (and their associated resources) that you manage with a single administration server. Since OEA does not support clustering, a typical OEA domain has one single server.

In order to create a domain, you must use the Configuration Wizard. In OEA, the Configuration Wizard only works in silent mode, which is a non-interactive way to create and configure a domain. As such, you need to create a XML file that defines the domain configuration settings. The listing 1 below shows an example this file.

figure_xyz_silent.xml_file

Listing 1: Sample silent.xml file, used to create an OEA domain.

Save the silent.xml file in some folder on your Pi. With this file in place you can invoke the Configuration Wizard from the shell. Execute the following command:

$OEA_HOME/oep/common/bin/config.sh -mode=silent -silent_xml=<SILENT.XML_FILE_LOCATION> -log=/home/pi/logs/createDomain.log

If the Configuration Wizard does not complete successfully, check the log file for more information. The shell command exit code can help you learn more about the outcome of silent execution. When completed successfully, the domain will have been created in the location specified in the DOMAIN_LOCATION parameter of the silent.xml file.

Sample Application Walkthrough

In this section, we are going to show the details behind the sample application used in this article. The use case implemented shows how to automate a hospital room to automatically detect when the temperature drops, avoiding that the patient be uncomfortable with the cold in the room. Once detected that the temperature drops, a signal can be sent to adjust the thermostat, and also calling the nurse to the room.

figure_2.1_patient_in_hospital

From the devices perspective, it would be necessary a sensor capable of measuring temperatures, and a Raspberry Pi that will execute the OEA product with the application. There are several low-cost sensors available in the market, such as the ones commercialized by Vernier. You can check them here.

The sample application uses the built-in REST adapter from OEA to receive events that, in our example, hold data about temperatures. The idea is that the sensor sends readings to the application by posting a JSON payload containing a sensor identifier and temperature value, to an HTTP endpoint exposed by the application through the REST adapter. Listing 2 shows an example of the JSON payload.

figure_xyz_json_payload_sample

Listing 2: Sample JSON payload that sensors must send to the REST endpoint.

Once received by the application, the JSON payload is automatically transformed into an event type and queried by a processor that must perform a filter. The filter essentially discards any temperature reading that contains values below 70 Fahrenheit. The resulting events that match the criteria are then printed out in the output console. Figure 3 shows the EPN (Event Processing Network) of this application.

figure3_epn_only_rest_adapter

Figure 3: EPN of the OEA application that handles temperatures events.

You can download the OEA application project that contains all the artifacts here. This project has been built using Fusion Middleware JDeveloper contained in the SOA Quick Start Installer version 12.2.1. This JDeveloper version can be downloaded here.

Once the project is open in JDeveloper, you can start checking the implementations details. How the JSON payload is automatically transformed into an event type is a good place to start. If you double-click the restAdapter component, you will see the bean declaration that defines it. The jsonMapper property points to another bean that uses the mapper capabilities available in OEA. Specifically; the mapper used is JAXB-based and it is capable of handling JSON payloads and automatically transforms them into event types. The event type used in this case is specified in the property eventTypeName. Listing 3 shows how this is configured.

figure_4_json_2_event_type_mapping_through_rest

Listing 3: Beans configuration for JSON handling through the REST adapter.

Moving forward, you might be curious about how the OEA application handles the event filtering. This is accomplished through a processor named filterTemperatures. This processor uses a simple CQL query to filter out the undesired events. CQL is the acronym of Continuous Query Language, a query language based on ANSI SQL with added constructs that support streaming data and pattern matching. By using CQL, developers can perform event streaming analysis with a high-level programming language. Listing 4 shows the CQL statement that filters the events received by the application.

figure_5_cql_statement_to_filter_events

Listing 4: CQL statement that filters undesired temperature readings.

Finally, you will notice the outputAdapter component. This is a custom adapter, created using the OEA adapter API, that takes the incoming event and print its contents out in the output console. There are no specific details to cover about this component since it was created only for instructional purposes. Normally, the output events are sent downstream using specialized adapters such as REST or MQTT, to implement the thermostat adjustment, for example.

Deployment and Testing

We are going to deploy and test the sample application discussed in the previous section. The first step is making sure that your OEA server is up and running. If the server is not running, you can easily start it using the built-in script created during the domain creation. Navigate to the folder that contains your OEA server. For example; considering the domain created using the listing 1 silent.xml file, you should navigate to the following folder:

/oea/domains/helloWorldDomain/defaultserver

Inside this folder you will see a script named startwlevs.sh. Execute this script in order to start your OEA server. Wait until the server is completely started. This may take several minutes if it is the first time you’ve start the server.

Next, you need to deploy the sample application onto the OEA server. You can either generate a deploy bundle directly from your JDeveloper install or download a pre-generated copy here. Copy the bundle JAR file into some local folder of your Pi. Once you have copied the bundle JAR, navigate to the following folder:

$OEA_HOME/oep/bin

This folder contains several files that are used to perform administration and deployment tasks. For this activity we are going to leverage the wlevsdeploy.jar file which contains built-in functions to perform deployment tasks. To deploy the sample application into the local OEA server, execute this command:

java -jar wlevsdeploy.jar –url http://localhost:9002/wlevsdeployer -user wlevs -password welcome1 –install <APPLICATION_BUNDLE_JAR_FILE_LOCATION>

The command above uses several parameters that are defined during the domain creation, such as the NETIO_PORT, USERNAME and PASSWORD. After the bundle JAR is correctly installed, you should see in the output console of the OEA server messages like this:

<BEA-2045000> <The application bundle “GettingStartedWithOEA” was deployed successfully>

<BEA-2047000> <The application context for “GettingStartedWithOEA” was started successfully>

With the sample application properly installed, it is time to run some tests. Since the application exposes an HTTP endpoint to receive the events containing the temperature readings, you can perform HTTP POST’s onto your Raspberry Pi using its hostname and the NETIO_PORT as the URI. Also remember that the application registers the HTTP endpoint in the /receiveTemperatures context path, so the target URL will be:

http://hostname:9002/receiveTemperatures

For instance, if you are locally connected into your Pi, you could use the cURL command to perform an HTTP POST. Try posting the JSON payload shown in listing 2 using the following command:

figure_xyz_posting_the_json_payload

Once the payload is posted, you should see the following message in the output console of the OEA server:

[TemperatureReading] sensorId = sensor-01, temperature = 72.8

This means that the JSON payload was properly received by the application and that the EPN flow was executed completely, including the last stage which is the outputAdapter component. Go ahead and try sending events that contain temperature values that falls below the 70 Fahrenheit watermark. You will notice that those events will not be printed out in the output console, since they do not satisfy the filter criteria specified in the CQL statement shown in figure 5.

Adding Support to MQTT

Although REST is largely leveraged today by devices that need to communicate to the external world, it is not considered the best technology for that purpose. REST is designed around the Request/Response model, running on top of a stateless protocol such as HTTP. This alone brings a tremendous overhead in terms of payload due the inherently large amount of header data found in HTTP requests and responses. Another important aspect to consider is the lack of reliable messaging over unreliable networks. Most deployed devices rely on extremely low bandwidth such as radio frequency based communication.

There is another problem with using REST. Power is a scarce resource in many edge devices. Every transaction performed by the device must be done efficiently consuming minimal computational cycles in order to save energy, allowing the device to prolong its activities. REST-based architectures tend to consume large amounts of computational cycles by typically using long-running polling techniques, and by hosting a web server (even if it is something lightweight such as Jetty) in the device.

The MQTT protocol was invented to address many of these challenges. MQTT is a M2M (Machine-to-Machine) protocol that allows reliable connectivity using a Publish/Subscribe messaging style; still being very efficient for small footprint devices. MQTT allows devices to communicate efficiently, saving battery and bandwidth while maintaining extremely low latency when communicating with the external world. There are many tests that prove MQTT’s efficiency and you can read more about that in this article written by Stephen Nicholas.

The Oracle A-Team developed a native adapter for OEA which was designed to allow both inbound and outbound connectivity with MQTT-based brokers. This section will describe how this adapter works, and how it can be incorporated into the sample application described so far.

The MQTT adapter is provided for free to use “AS-IS” but without any official support from Oracle. Bug reports, feedback and enhancement requests are welcome, but need to be performed using the comments section of this blog and the A-Team reserves the right of help in the best-effort capacity.

The first thing you need to do is to obtain a copy of the MQTT adapter. You can download it here. Unzip the package contents into a staging folder. Inside it you will have two other packages, the bundle JAR of the adapter and some pre-built samples. Unzip the bundle JAR and copy it to the modules folder of your OEA server. For example, considering the domain created using the listing 1 silent.xml file, the modules folder location should be this:

/oea/domains/helloWorldDomain/defaultserver/modules

Bundle JARs copied to the modules folder are automatically loaded into the OEA runtime system. Thus, being available to be used by any application deployed. You will need to restart your OEA server in order to the changes take effect. Tip: there is a stopwlevs.sh script under the OEA server folder. Alternatively, you can locally stop the OEA server using the following command:

java -jar wlevsadmin.jar -username wlevs -password welcome1 SHUTDOWN

Download the modified version of the OEA application that contains the MQTT support from here. This new version uses the MQTT adapter to establish communication with an internet-hosted MQTT broker. If the Pi device that you are using does not have internet connection, then a local install of a MQTT broker will be required for the tests. There are plenty of MQTT brokers available to use, as you can see here. Figure 4 shows the EPN of the modified version of the application.

figure_6_epn_with_rest_and_mqtt

Figure 4: EPN of the OEA application, now handling both REST and MQTT events.

If you double-click the mqttAdapter component, you will see the bean declaration who defines it. Just like the restAdapter component, it uses the JAXB mapper to handle JSON payloads and automatically transform them into event types. The property that links the MQTT adapter with the mapper is called mapper. But differently from the REST adapter, there are additional properties to be set in the MQTT adapter in order to establish the communication with the broker. The serverURIs property holds the comma-separated list of brokers; the topicName property specifies which topic the adapter should listen to and the qualityOfService property holds the comma-separated list (one for each broker) of QoS values. Listing 5 shows how this is configured.

figure_7_json_and_event_type_mapping_through_rest_and_mqtt

Listing 5: Beans configuration for JSON handling through the REST and MQTT adapters.

Generate a new bundle JAR with the modified version of the application. Alternatively, you can grab a pre-generated copy of the modified version of the application here. Copy the bundle JAR file into some local folder of your Pi. Once you have copied the bundle JAR, it will be necessary to undeploy the current version of the application. This is necessary because both the old and the new version of the application have the same bundle name, which must be unique in the OEA server. To undeploy the application, use the following command:

java -jar wlevsdeploy.jar -url http://localhost:9002/wlevsdeployer -user wlevs -password welcome1 -uninstall GettingStartedWithOEA

The uninstall command takes the bundle name of the application as parameter. This is how the OEA server knows which application must be undeployed. With the older version of the application properly undeployed, you can deploy the new version of the application using the same deploy command you have used before:

java -jar wlevsdeploy.jar –url http://localhost:9002/wlevsdeployer -user wlevs -password welcome1 –install <APPLICATION_BUNDLE_JAR_FILE_LOCATION>

After the bundle JAR is correctly installed, you should see in the output console the same confirmation messages you have saw before, stating that the application was deployed and installed successfully.

In order to test the new version of the application, an MQTT-based client will be necessary. This client must connect to the same broker that the application is connected to and publish messages to the topic named temperatures. This can be accomplished in two ways:

    • Writing your own Client: There is an open-source project called Eclipse Paho that provides APIs to several programming languages such as C/C++, Java, JavaScript, Python, Go and C# .NET.

    • Using a GUI Client Tool: The Eclipse Paho project also provides a graphical utility tool that is intended to be used for testing purposes. There are versions for several operating systems, and you can freely download it here.

For instance, the figure 5 below shows the GUI utility tool from the Eclipse Paho project being used to publish a message to the MQTT broker.

figure_8_testing_the_mqtt_adapter

Figure 5: Using the Eclipse Paho GUI Utility Tool to publish messages to a MQTT broker.

Since the application subscribe to the temperatures topic using the MQTT adapter, it will receive a copy of the message sent and process it using the implemented EPN. You should see the following message in the output console of the OEA server:

[TemperatureReading] sensorId = sensor-02, temperature = 89.88

The MQTT protocol when combined with OEA product can help you to create very powerful IoT projects, that might provides not only development speed, but also great performance. During the development of the MQTT adapter, the A-Team had the chance to perform some stress testing against OEA while running on top of the Pi, and some interesting numbers came up.

This stress test should not be considered as an Oracle official benchmark. It was made available in this article only for information purposes. Moreover, the numbers shown does not reflect the actual limit of the product, nor the Raspberry Pi device. They might be higher or lower depending of several other factors such as network, hardware, coding, algorithm complexity, etc.

The stress test mainly focused in calculating what is the actual processing limit of the Pi device while running an OEA application that process concurrent sensor events. To increase the accuracy of the test, the network latency was reduced as much as possible, so the only hardware resources that would affect the results and therefore, limit scalability, would be CPU and memory. The figure 6 shows the results of this stress test.

figure_9_stress_test_results

Figure 6: Stress test results obtained using OEA, MQTT and Raspberry Pi.

For sending the messages, it was used a MQTT client producer written in Java that uses two threads to concurrently send messages to the topic. Each thread sends 2,000 events per second, totaling 4,000 per second from the consumer point of view. The tests shown that increasing the incoming volume (increasing the number of producer threads or increasing the number of messages sent) did not positively change the consumption throughput, indicating some sort of contention. Hence, it was how the scalability limit was calculated. Curiously, CPU usage did not increased with the incoming volume increase, neither the JVM GC (Garbage Collection) pressure, which may indicate that the network might be limiting factor for the scalability.

Conclusion

This article introduced the OEA product, Oracle’s technology to bring event stream analytics to devices such as Raspberry Pi. Through a series of examples, it was shown how to get the OEA product installed, configured and ready to execute a sample application. The article also showed how to deploy and test this sample application, exploring the administrative capabilities found in the OEA product. Finally, the article showed how to leverage the MQTT protocol in OEA-based applications, through the use of a native MQTT adapter built specifically for OEA.

Event streaming analytics is a very hot topic that has gained even more traction with the explosion of IoT and machine-generated data. However, there are specific use cases that require event data to be analyzed in the fog, with zero connectivity to server-side environments capable of handling the data stream. This example discussed here shows a possible solution that could bring event streaming analytics to devices, even ones with minimal hardware capabilities and network bandwidth constraints.

Comments

  1. Hi Nick,

    Glad you enjoyed the article. Regarding your questions:

    1. Yes, outbound connectivity is supported in the MQTT adapter. Here is a small example that shows how to configure that on OEA:

    https://gist.github.com/anonymous/dfe49bdf57131f03e7f17f28fed9c367

    2. The way the REST adapter works for outbound connections is very similar to the example I’ve give you above. In a matter of fact, when I wrote the MQTT adapter code, I took the REST adapter as inspiration 😉

    Best regards,

    Ricardo Ferreira

  2. Nick Bowman says:

    This was a very useful article and assisted me in getting a simple use-case working where high-frequency sensor data is being sent from multiple mocrocontrollers (Arduino) via bluetooth to a Raspberry Pi. I’ve then used Python scripts to publish data to MQTT and trigger an OEA application using the MQTT inbound adapter.

    I have two questions:

    1. Is there an MQTT outbound adapter available for OEA?

    2. Are there any good examples or guidelines for using the Outbound REST adapter for OEA? I found the getting started guide to be light on detail.

Add Your Comment