Automatically Disable Proxy Service to avoid overloading OSB

Introduction

This post provides a programmatic way to disable an OSB endpoint and trigger that code automatically using an OSB alert rule.

Main Article

From the design practice point of view, it is always recommended to make your endpoint highly available. With the help of either load balancer or OSB's failover capability across multiple endpoints, you should be able to make your OSB safer and less vulnerable from endpoint errors. In addition, you'd better implement OSB best practice by configuring a Work Manager attached to Proxy Service and enabling throttling control on Business Service. The details of these two features will not be covered in this blog. You can find the details of using work manager with OSB here and How to enable and use throttling control here in the documentation.

However, this is not the whole picture of the story. Once an unexpected problem occurs on endpoint, you certainly want to be noticed for the incident and take some appropriate action to cope with it especially in the case where you have only one endpoint or the endpoint systems reside externally and out of your control.

Now, let's see how can you get notified when the following two common problems occur at endpoint:

• the endpoint becomes unreachable
• the endpoint still alive but the response becomes slow and results in SLA violation

OSB provides a nice feature to offline the unresponsive endpoint URI due to communication error. A Service Level Agreement (SLA) rule can be defined to trigger the alert once endpoint URI is marked as offline. You can also define a SLA regarding response time. If the response time is longer than the defined value, the alert can be fired automatically by OSB.

Once alerted, it is up to you to decide what are the appropriate actions to take. If the problem on the endpoint persists and cannot be resolved
quickly, a possible action you might want to take is to stop the Proxy Service for not receiving requests anymore until the problem is sorted out. You can manually disable the Proxy Service via OSB console or WLST to achieve this.  This blog shows an alternative approach which is to disable Proxy Service automatically via OSB's Java API when SLA is violated due to endpoint abnormalities.

The following steps highlight how to implement the solution:

1. create a SLA rule which triggers alert if the response time is longer than the expected one or the endpoint is unreachable
2. configure alert to be sent to a designated JMS queue
3. create an "administrative" Proxy Service to listen on the JMS queue
4. make a Java callout within the "administrative" Proxy Service to disable the relevant Proxy Service once the alert is received from JMS queue.

The details are as follows:

Create an alert destination to send alert to a designated JMS queue.

Create the Business Service and enable the offline endpoint URI. If the endpoint URI is unreachable, OSB changes its status to offline automatically.

Define the SLA. An alert will be triggered if the max response time > 5000 ms OR the endpoint is marked as offline.

Create the "administrative" Proxy Service listening on the designated JMS queue

The Proxy Service, upon the receipt of alert message, get the JMS headers set by OSB when sending alert to JMS queue. In this blog, we simply use alert message's ServiceName header to find out which Business Service fired this alert.  The XAPTH expression to achieve this is:

$inbound/ctx:transport/ctx:request/tp:headers/tp:user-header[@name="ServiceName"]/@value.

Based on the Business Service name, you can configure some parameters to find out which Proxy Service is supposed to be disabled. And finally, make a Java Callout to disable that Proxy Service. In this blog, the way to determine the Proxy Service is simply hard coded.

The following is a sample Java class that shows how to programatically enable or disable a Proxy Service via Java API. The same code can be applied to Business Service as well.

... public class ServiceManager {     private static JMXConnector initConnection(String hostname, int port, String username, String password) throws IOException,MalformedURLException     {         JMXServiceURL serviceURL = new JMXServiceURL("t3", hostname, port, "/jndi/" + DomainRuntimeServiceMBean.MBEANSERVER_JNDI_NAME);         Hashtable<String, String> h = new Hashtable<String, String>();         h.put(Context.SECURITY_PRINCIPAL, username);         h.put(Context.SECURITY_CREDENTIALS, password);         h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");         return JMXConnectorFactory.connect(serviceURL, h);     }     private static Ref convertServiceURI2Ref(String resType,String serviceuri) {         Ref ref = null;         if((serviceuri.equals("")) || (serviceuri==null))             return ref;         String[] uriData = serviceuri.split("/");         ref = new Ref(resType,uriData);         return ref;     }     public static void changeProxyServiceStatus(String serviceref,boolean status) throws Exception {         JMXConnector conn = null;         SessionManagementMBean sm = null;         String sessionName = "Session.ByApp." + System.currentTimeMillis();         try {             conn = initConnection("localhost", 7001, "weblogic", "welcome1");             MBeanServerConnection mbconn = conn.getMBeanServerConnection();             DomainRuntimeServiceMBean domainService = (DomainRuntimeServiceMBean) MBeanServerInvocationHandler.             newProxyInstance(mbconn, new ObjectName(DomainRuntimeServiceMBean.OBJECT_NAME));             sm = (SessionManagementMBean) domainService.findService(SessionManagementMBean.NAME, SessionManagementMBean.TYPE, null);             sm.createSession(sessionName);             ProxyServiceConfigurationMBean proxyConfigMBean = (ProxyServiceConfigurationMBean) domainService.findService(                          ProxyServiceConfigurationMBean.NAME + "." + sessionName,  ProxyServiceConfigurationMBean.TYPE, null);             Ref ref = convertServiceURI2Ref("ProxyService",serviceref);             String msg = "";             if(!status) {                 proxyConfigMBean.disableService(ref);                 msg="Disabled the Proxy Service : " + serviceref;            }            else {                 proxyConfigMBean.enableService(ref);                 msg="enabled the Proxy Service : " + serviceref;            }            sm.activateSession(sessionName, msg);            System.out.println(msg);            conn.close();      } catch(Exception ex) {          if(null != sm) {              try {                   sm.discardSession(sessionName);             } catch(Exception e) {                 System.out.println("discard session error");             }         }         throw ex;     } finally {         if(null != conn)             try {                 conn.close();             } catch(Exception e) {                  e.printStackTrace();            }       }     }  }

By putting a sleep of 10 seconds at the endpoint which is greater than the defined 5 seconds in SLA, the Proxy Service is automatically disabled as shown below.

Make another round of test by killing the endpoint, the same result (automatically disabled the Proxy Service) outcomes as expected. At the same time, we can also see the endpoint is automatically marked offline by OSB.

This blog demonstrates a way to automatically turn off a Proxy Service when the endpoint is unreachable or the SLA is violated. However, you need to be cautious when considering the automation. The reason is there are so many factors and criteria to consider and evaluate to make the decision of whether or when the Proxy Service should be disabled. Alternatively, you can also break the above sample into two steps for being alerted and disabling Proxy Service via Java API, then add a manual decision making step in between.

Disclaimer: Sample code fragments provided by Oracle in this article are for illustrative purposes only and are supplied "AS IS" without any warranties or support.
Oracle assumes no responsibility or liability for the use of the software, conveys no license or title under any patent, copyright, or mask work right to the product.
Oracle also reserves the right to make changes in the software without notification. Oracle also make no representation or warranty that such sample code will be suitable for the specified use without further testing or modification.