Going Mobile with ADF – Implementing Data Caching and Syncing for Working Offline Part I

Introduction

With over 90% of internet traffic now coming from mobile devices, there is a huge pressure on companies to make their customer-facing applications suitable for rendering on smart phones and tablets. A-team is involved with a number of customers who are looking for ways to adapt their existing Oracle ADF applications and make them “mobile-enabled” . This article is the third in a series of articles that describe what we learned from the experiences with these customers both from a technical and a non-technical perspective. Previous articles in this series are Understanding the Options, and Running ADF Faces on Mobile Phones and Tablets. This article introduces very powerful and generic JDeveloper mobile persistence extension developed by the A-Team that you can reuse to quickly and easily implement data caching on the mobile device in a secure way using the SQLite database. The extension also helps you implementing data synchronization with a remote server in case your mobile application should support offline transactions.This article is written in two parts, this part focusses on the runtime persistence architecture, the second part discusses the powerful design-time wizards included in the extension that allow you to generate a fully functional mobile application in just a few minutes.

Main Article

As discussed in the first article in this series, Understanding the Options, ADF Faces web applications cannot be used for working offline on a mobile device. So the technology choice is simple in this case, we need to use ADF Mobile to create an on-device application that supports working in offline mode. Most likely, the mobile application needs to communicate with an existing back-end (ADF) application to get data (“read actions”) and to send transactions based on changes made to the data by the mobile user (“write actions”). We have various degrees in which we can support read and write actions in offline mode, as illustrated by the following picture: DataCachingStrategies   The simplest strategy 1 is to not support working in offline mode, if that strategy is acceptable for you, you can stop reading this article. Strategy 2 supports reading/viewing the data in offline mode, but the user needs to be connected when he/she wants to modify data. This strategy implies you need to cache the data on the device. This strategy is straightforward to implement when using the sample persistence code provided by the A-team, as we will see later. Strategies 3 and 4 are a different ballgame, here you need to keep track of pending transactions that are not yet sent to the server. The sample code provided by us will help you with registering and (re-)sending pending transactions, however the complex issue of data synchronization conflicts and transactions based on stale data needs to be resolved by you, based on the specifics of both your mobile application and the back-end application that serves the data. This article will first discuss how you can disclose the business logic and data from your back-end and consume this in your mobile application. Then we will cover strategy 2: implementing data caching using the SQLIte database. Finally, we will discuss the issues around data synchronization that come with strategies 3 and 4 in more depth.

Disclosing Back-End (ADF) Applications Using Web Services

ADF Mobile can access back-end systems solely through web services. We will not go into the details of creating and consuming these web services as this topic has already been discussed in various articles and videos: Creating web services:

Consuming web services:

When disclosing your back-end system, you need to choose between SOAP and RESTful web services, and within RESTful web services between XML and JSON payload. A-Team recommends to use REST-JSON web services. RESTful services are simply easier to create and use, and the JSON data exchange format is much more efficient than the more verbose XML, reducing the size of the data packages that are sent over the wire, and improving overall performance. In addition, with its origins in JavaScript, JSON nicely integrates with many client-side web frameworks, which might be useful when you want to build some non-ADF web applications on top of your back-end data. You might notice that most of the above links show SOAP-based or REST-XML web services. This might be related to the fact that declarative support for SOAP-based web services in JDeveloper and ADF Mobile is currently better. To consume a RESTful web service you always need to do Java coding, unless you use the A-Team mobile persistence sample introduced below. However, the declarative SOAP-based data control or does not implement data caching, so you still need to do the same amount of Java coding when you go for strategies 2,3 or 4. Furthermore, you will see a strong focus on enhancing support for REST-JSON in upcoming JDeveloper versions. A new REST data control is planned, as well as full declarative support to create REST-JSON web services on top of ADF Business Components. For a sneak preview of how that functionality will look like, you can take a look at this presentation.

Implementing Strategy 2: On-Device Caching Using SQLite Database

To cache data in a secure way on the mobile device, we recommend to use the embedded SQLIte database. This self-contained database can be created on-the-fly by your mobile application, and does not require any installation steps beforehand. Your data can be stored securely by using one of the supported data encryption algorithms. The data can be stored and retrieved using plain JDBC statements. For more information on using SQLite check out this video by ADF product manager Frederic Desbiens. You typically create so-called service object classes that contain logic for calling the web services and for reading data from and writing data to the local database. The service object also contains the CRUD operations that are exposed through an ADF data control to allow you to build your ADF Mobile AMX pages using drag and drop actions. To read data the service object includes getter methods that return an array of so-called data objects. A data object (also known as entity object or domain object) typically maps 1:1 to an underlying database table, and each instance of a data object represents a row of data. The data object contains attribute getter/setter methods that map to columns in the underlying database table. A data object can also contain methods that return a collection other data objects to implement master-detail like object structures.This data caching architecture is illustrated by the picture below.  ServiceAndDataObject

As explained in the links included in the previous section, ADF Mobile provides utility methods to call a REST web service and SOAP web service programmatically. The RestServiceAdapter class can be used to call REST web services. The technique for calling a SOAP-based web service might be a bit confusing: you first create a data control for the web service, and then you programmatically invoke a method on the data control, for example to retrieve the data. So, you do not use the web service data control to create the user interface, it is only created so you can easily invoke the SOAP web service. The data control you create for the service object is the one you will use to build the user interface. With this architecture in place, the main steps to implement on-device data caching include:

  • Create a set of database tables that match the structure of your web service payload.
  • Create a set of data objects that map to your database tables and web service payload
  • Create service objects (with helper classes to better organize your logic) that include functionality to call the web service, and store the payload returned in an array of Java data objects and persist the same data using JDBC in your on-device database so the data will be preserved when the user closes the application.

Writing the code to perform the above steps is not difficult. However, it is tedious work and coding low-level JDBC statements might feel like we are going back 10 years in Java programming. It would be nice to have some lightweight object relational mapping tool that saves us most of this repetitive Java programming, right? Well, this is exactly what the A-team developed as part of a proof of concept for some of their customers.

The A-Team ADF Mobile Persistence Sample

The ADF Mobile Persistence Sample is a lightweight persistence and data synchronization framework developed by the A-team. It is powerful and sophisticated sample code that you can (partially) reuse, extend or copy as desired. You can use it at your own risk, there is no support channel available, but we are always interested in feedback that you can provide by commenting on this article. The sample code contains generic Java code that invokes the web services and performs CRUD operations against the SQLite database. The Java data object classes and data service classes are generated by powerful design-time wizards as explained in the second part of this article. The date service class extends the EntityCRUDService class from the generic runtime library included with the extension. This superclass contains methods to insert, find, update, delete and merge data objects. The implementation of these methods depends on so-called persistence manager classes, also included in the runtime library, that you can plug into your service object. By default, a DBPersistenceManager is configured which implements the service object CRUD operations by constructing SQL SELECT, INSERT,UPDATE and DELETE statements. You can also configure a remote persistence provider which implements the service object CRUD operations by making the appropriate web service calls.  These generic persistence provider classes are able to generate the correct SQL statements and do the conversion from the XML or JSON payload to the data objects and vice versa, by reading a persistence mapping XML file that describes how tables map to data objects, and how columns map to data object attributes and payload attributes. The overall structure of your ADF Model layer looks like this when using the persistence extension to provide CRUD operations for departments:

PersistenceArchitecture

 

In addition to the Java classes, the JDeveloper wizards included in the extension generate the persistence mapping file, as well as the DDL script to create the SQLIte database at runtime. When running the Mobile Business Objects from Web Service Data Control wizard that comes with the extension, the generated Department class looks like this:

package mobile.model;

import java.util.Arrays;
import java.util.List;
import java.util.ArrayList;
import oracle.ateam.sample.mobile.persistence.util.EntityUtils;
import oracle.ateam.sample.mobile.persistence.model.Entity;

public class Department extends Entity {

    private Integer departmentId;
    private String departmentName;
    private Integer managerId;
    private Integer locationId;
    private transient List employeesViewList = createIndirectList("employeesViewList");
    // dummy attr to prevent child query during JSON serialization
    private transient Employee[] employeesView;

    public Integer getDepartmentId() {
        return this.departmentId;
    }

    public void setDepartmentId(Integer departmentId) {
        this.departmentId = departmentId;
    }

    public String getDepartmentName() {
        return this.departmentName;
    }

    public void setDepartmentName(String departmentName) {
        this.departmentName = departmentName;
    }

    public Integer getManagerId() {
        return this.managerId;
    }

    public void setManagerId(Integer managerId) {
        this.managerId = managerId;
    }

    public Integer getLocationId() {
        return this.locationId;
    }

    public void setLocationId(Integer locationId) {
        this.locationId = locationId;
    }

    public void setEmployeesViewList(List employeesViewList) {
        this.employeesViewList = employeesViewList;
    }

    /**
     * This method is called when entity instance is recreated from persisted JSON string in DataSynchAction
     */
    public void setEmployeesViewList(Employee[] employeesViewList) {
        this.employeesViewList = Arrays.asList(employeesViewList);
    }

    public List getEmployeesViewList() {
        return this.employeesViewList;
    }

    public Employee[] getEmployee() {
        List dataObjectList = getEmployeesViewList();
        return (Employee[])dataObjectList.toArray(new Employee[dataObjectList.size()]);
    }

    public void addEmployee(int index, Employee employee) {
        employee.setIsNewEntity(true);
        EntityUtils.generatePrimaryKeyValue(employee, 1);
        employee.setDepartmentId(getDepartmentId());
        getEmployeesViewList().add(index, employee);
    }

    public void removeEmployee(Employee employee) {
        getEmployeesViewList().remove(employee);
    }
}

Apart from getter/setter methods for the department attributes, you can see methods to access employees within the department, and to add or remove an employee.The employeesViewList property uses indirection: the list is only populated once the accessor is called from the user interface.

The generated DepartmentService class looks like this:

package mobile.model.service;

import java.util.ArrayList;
import java.util.List;
import oracle.ateam.sample.mobile.persistence.util.EntityUtils;
import oracle.ateam.sample.mobile.persistence.manager.DataControlPersistenceManager;
import oracle.ateam.sample.mobile.persistence.service.EntityCRUDService;
import mobile.model.Department;

public class DepartmentService extends EntityCRUDService {

    public DepartmentService() {
        DataControlPersistenceManager remotePersistenceManager = new DataControlPersistenceManager();
        setRemotePersistenceManager(remotePersistenceManager);
        super.findAll();
    }

    protected Class getEntityClass() {
        return Department.class;
    }

    protected String getEntityListName() {
        return "department";
    }

    public Department[] getDepartment() {
        List dataObjectList = getEntityList();
        Department[] dataObjects = (Department[])dataObjectList.toArray(new Department[dataObjectList.size()]);
        return dataObjects;
    }

    public void addDepartment(int index, Department department) {
        addEntity(index, department);
    }

    public void removeDepartment(Department department) {
        removeEntity(department);
    }

}

In the constructor we configure the remote persistence provider which is a DataControlPersistenceManager in this example. Other persistence provider provided are RestJSONPersistenceManager, RestXMLPersistenceManager and RestADFBCXMLPersistenceManager. The getEntityClass and getEntityListName are abstract methods in the superclass that must be implemented.The getDepartments method converts the list of departments into an array of departments, because JDK 1.4 is used on the mobile device, which does not support typed collections.The addDepartment and removeDepartment methods are automatically called by ADF Mobile framework when using the standard Create and Delete operations from the data control palette, so we can keep the list of departments up-to-date.

The generic code in the remote persistence provider reads the persistence mapping file to figure out how payload attributes map to data object attributes and database columns, and which web service method must be called for each CRUD operation. Here is a snippet of this persistence mapping file for the Department class:

<?xml version="1.0" encoding="UTF-8"?>
<mobile-object-persistence>
  <class-mapping-descriptors>
    <class-mapping-descriptor>
      <class>mobile.model.Department</class>
      <crud-service-class>mobile.model.service.DepartmentService</crud-service-class>
      <date-format>yyyy-MM-dd</date-format>
      <date-time-format></date-time-format>
      <order-by>DEPARTMENT_NAME</order-by>
      <table>DEPARTMENT</table>
      <primary-key>
        <column-name>DEPARTMENT_ID</column-name>
      </primary-key>
      <attribute-mappings>
        <attribute-mapping type="direct-mapping">
          <attribute-name>departmentId</attribute-name>
          <column name="DEPARTMENT_ID" dataType="NUMERIC"/>
          <payload-attribute-name>DepartmentId</payload-attribute-name>
          <required>true</required>
        </attribute-mapping>
        <attribute-mapping type="direct-mapping">
          <attribute-name>departmentName</attribute-name>
          <column name="DEPARTMENT_NAME" dataType="VARCHAR"/>
          <payload-attribute-name>DepartmentName</payload-attribute-name>
          <required>false</required>
        </attribute-mapping>
        <attribute-mapping type="direct-mapping">
          <attribute-name>managerId</attribute-name>
          <column name="MANAGER_ID" dataType="NUMERIC"/>
          <payload-attribute-name>ManagerId</payload-attribute-name>
          <required>false</required>
        </attribute-mapping>
        <attribute-mapping type="direct-mapping">
          <attribute-name>locationId</attribute-name>
          <column name="LOCATION_ID" dataType="NUMERIC"/>
          <payload-attribute-name>LocationId</payload-attribute-name>
          <required>false</required>
        </attribute-mapping>
        <attribute-mapping type="one-to-many-mapping">
          <attribute-name>employeesViewList</attribute-name>
          <payload-attribute-name>EmployeesView</payload-attribute-name>
          <reference-class>mobile.model.Employee</reference-class>
          <target-foreign-key>
            <column-reference>
              <source-column table="EMPLOYEE" name="DEPARTMENT_ID"/>
              <target-column table="DEPARTMENT" name="DEPARTMENT_ID"/>
            </column-reference>
          </target-foreign-key>
        </attribute-mapping>
      </attribute-mappings>
      <find-all-method name="findDepartments" dataControlName="HRServiceSOAP" deleteLocalRows="true"
                       payloadElementName="result">
        <parameter name="findCriteria" javaType="java.lang.Object"/>
        <parameter name="findControl" javaType="java.lang.Object"/>
      </find-all-method>
      <create-method name="createDepartments" dataControlName="HRServiceSOAP" payloadElementName="result">
        <parameter name="departments" valueProvider="SerializedDataObject" javaType="java.lang.Object"/>
      </create-method>
      <update-method name="updateDepartments" dataControlName="HRServiceSOAP" payloadElementName="result">
        <parameter name="departments" valueProvider="SerializedDataObject" javaType="java.lang.Object"/>
      </update-method>
      <merge-method name="mergeDepartments" dataControlName="HRServiceSOAP" payloadElementName="result">
        <parameter name="departments" valueProvider="SerializedDataObject" javaType="java.lang.Object"/>
      </merge-method>
      <remove-method name="deleteDepartments" dataControlName="HRServiceSOAP">
        <parameter name="departments" valueProvider="SerializedDataObject" javaType="java.lang.Object"/>
      </remove-method>
    </class-mapping-descriptor>

When running the Mobile Business Objects from REST Web Service wizard, the only difference in the generated Java classes is the remote persistence provider that is configured in the DepartmentService class:

public DepartmentService() {
    RestJSONPersistenceManager remotePersistenceManager = new RestJSONPersistenceManager();
    setRemotePersistenceManager(remotePersistenceManager);
    super.findAll();
}

The generated persistence mapping file differs in the methods section. Rather than invoking data control methods, the persistence manager directly invokes RESTful web service resources, which is refelected in the attributes of the department CRUD methods, as you can see below:

<find-all-method uri="/hrdemorest/rest/json/ReadDepsAndEmps" connectionName="HRDemoRest" requestType="GET"
                 deleteLocalRows="true" payloadElementName="DepartmentsView"></find-all-method>
<create-method uri="/hrdemorest/rest/json/WriteDeps" connectionName="HRDemoRest" requestType="PUT"
               payloadElementName="DepartmentsView">
  <parameter name="json" valueProvider="SerializedDataObject"/>
</create-method>
<update-method uri="/hrdemorest/rest/json/WriteDeps" connectionName="HRDemoRest" requestType="PUT"
               payloadElementName="DepartmentsView">
  <parameter name="json" valueProvider="SerializedDataObject"/>
</update-method>
<merge-method uri="/hrdemorest/rest/json/WriteDeps" connectionName="HRDemoRest" requestType="PUT"
              payloadElementName="DepartmentsView">
  <parameter name="json" valueProvider="SerializedDataObject"/>
</merge-method>
<remove-method uri="/hrdemorest/rest/json/RemoveDeps" connectionName="HRDemoRest" requestType="DELETE"
               payloadElementName="DepartmentsView">
  <parameter name="json" valueProvider="SerializedDataObject"/>
</remove-method>

The last step to take before you can create the mobile user interface is to create a data control from the DepartmentService class. You can then generate a default user interface using the Mobile User Interface Generator wizard also included in the A-team persistence extension, or you can manually create the user interface using drag and drop actions from the data control palette as illustrated by the pictures below: 

BuildUserInterfaceNew1


BuildUserInterfaceNew2

 

BuildUserInterfaceNew3

Implementing Strategies 3 and 4: Data Synchronization

As illustrated above, implementing data caching strategy 2 with offline reads and online writes is straightforward to implement with the help of the A-Team persistence extension. Implementing strategies 3 and 4 is much more complex. When you allow offline writes, you have to register pending transactions on the device, and send them later to the remote server. This introduces many issues you have to think about and deal with, for example:

  • The pending transaction might be based on stale data: another user might have changed the same data in between the moment the mobile user created the transaction offline and the moment he sends it to the remote server when he is online again.
  • The pending transaction might be based on data that has been deleted from the server in the meantime.
  • The pending transaction might be refused by the server because some server-side business rules are violated.
  • The order of pending transactions might be important So, if synchronization of one transaction fails, should you continue to try to synchronize the next transaction, possibly changing the transaction order when this next transaction succeeds?

In all these situations, the mobile user needs to be notified somehow about the failed transactions. You can choose to automatically remove failed transactions, or have the mobile user explicitly remove them from the list of pending data actions. This all depends on the type of application and how you want it to be used. To detect a transaction that is based on stale data, you might need to introduce a version number attribute or last modified timestamp which is incremented/updated with each transaction. This version number or last modified timestamp should be part of the payload of the cached transaction in order for the server to check whether no other transactions have been taken place in the meantime on the same data. The ADF Mobile persistence extension helps you to a certain extent with the implementation of strategies 3 and 4. If the device is offline, or the server refuses the transaction for some reason, then the transaction will be registered as a pending transaction by the EntityCRUDService superclass and its helper classes. When the user later on performs another transaction when the device is online again, a new attempt will be made to synchronize the pending transactions, in the order of creation. When the user closes the mobile application, the pending transactions are persisted to the SQLite database (in a separate database table) and restored when the application is started again. So, the housekeeping of pending transactions is taken care of when using the mobile persistence extension. The extension also includes a reusable ADF Mobile feature archive that you can plug into your application to make the pending transactions visible to the mobile user. You can add this feature from the archive to your application, and then you can add a link that invokes the feature. You can enable or render this link conditionally so it is only active or visible when there are pending changes:

<amx:commandLink id="menPsa" disabled="#{!bindings.hasDataSynchActions.inputValue}" text="Pending Sync Actions"
                 actionListener="#{GoToFeature.goToDataSynchFeature}">
  <amx:setPropertyListener id="menspaspl" from="mobile.model.Department"
                           to="#{applicationScope.dataSynchEntity}"/>
  <amx:closePopupBehavior id="mencPsa" popupId="p1" type="action"/>
</amx:commandLink>

DataSynchNew

This is just one way you could present the pending transactions. Feel free to use this reusable feature, or just build your own mechanism to view and handle pending transactions.You can then still use the extension runtime library for internal housekeeping of pending transactions.

Getting Started with the A-Team Mobile Persistence Extension

See the last section of the second part of this article for instructions on getting started with this extension.

Comments

  1. Steven Davelaar?
    the second part of this article is deleted?did you give us the adf-mobile-persistence-sample-install.zip install file?thank you very much.

Add Your Comment