WebCenter Sites Cluster deployment with a single war

The default installation method for WebCenter Sites in deployments with a large number of cluster nodes  is far from ideal. The installation program requires that each cluster node has its own WAR and its own ContentServer installation directory. This leads to issues around manageability and clusterability of the web application.

In this blog I present a workaround that uses one WAR file and one installation folder per cluster. For this is I use parameterization techniques for the configuration files. The basic approach is to do a single node installation and then modify the configuration files in such a way that they use Java system properties that are set per cluster node. Both horizontal and vertical clusters are possible with the described approach.

I have chosen the method of externalization through system properties over different classpaths per cluster node as per my experience: different classpaths leads often to confusion and long issue remedy cycles.

This article assumes that you use WebLogic 10.3 and Sites 11.1.1.6.x or 11.1.1.8, the Linux OS and Oracle Database. It also assumes a good understanding of the WebCenter Sites architecture and installation process and procedures.

As per the WCS installation documentation, when WebLogic is used, the CS and CAS web applications are required to be deployed in “no-stage” mode. WCS writes to the web application directory at runtime.

In summary: the following configuration settings need to be considered and potentially externalized:

  • Single WAR in no-stage mode.
  • Single ContentServer installation folder for all the ini files
  • SSOConfig for casInternal and casExternal URLs.
  • host.properties for CAS’ host identifier.
  • log4j logging files as defined in WEB-INF/classes/log4j.properties
  • Java IO temp folder for temp files per JVM and inCache disk caching.

The values of the externalized configuration settings are provided through JAVA_OPTIONS in the WebLogic startup script. These are the properties that need to be set:

 

Property

CS

CAS

Note

cluster.node.id

YES

YES

Used to identify the unique JVM in the cluster. Used for log4j and for ticket generation in CAS
java.io.tmpdir

YES

YES

Unique TEMP directory per JVM
casUrl

YES

NO

The external location of CAS
casInternalURL

YES

NO

The internal location of CAS

These properties are used in the following files:

  • wc-sites/bin/cas.properties
  • wc-sites/bin/host.properties
  • CS/WEB-INF/classes/log4j.properties
  • CS/WEB-INF/classes/SSOConfig.xml
  • CAS/WEB-INF/classes/log4j.properties
  • CAS/WEB-INF/deployerConfigContext.xml

My physical architecture setup is as follows:

  1. one WebLogic admin server
    1. a 2-node WebLogic cluster for Sites: named sites1 and sites2
    2. a 2-node WebLogic cluster for CAS: named cas1 and cas2.
  2. one Apache server with 4 virtual hosts,
    1. two public internet facing hosts (www.wcsites and cas.wcsites). These are the hosts that visitors connect to and these hostnames should be public. In a larger setup these point to a load-balancer.
    2. two internal hosts for http traffic between CAS and Sites. The internal hosts are used by CAS to validate credentials at Sites and for Sites to validate the CAS ticket.

As you can see I have isolated Sites and CAS on two different clusters/virtualhosts. I have done this because Oracle’s Security best practices require to host CAS on its own stack. It also provides tuning and scaling capabilities of each application independently if required. You can opt to have Sites and CAS applications hosted on the same cluster.

I have created two managed server start scripts (start-cas.sh and start-sites.sh). In this way I can manage the configuration of the managed servers without making any changes to the installer produced WL files. Per managed server additional system properties are set, as well as the PRE_CLASS_PATH. In the PRE_CLASSPATH are cas.properties and host.properties. host.properties takes a system property to produce a unique value per JVM. Please be noted that the JVM settings are not tuned for production. The heap memory size is for instance too low.

Further have the files log4j.properties, SSOConfig.xml, deployerConfigContext.xml property values that are externalized.

Below are all the files listed that need to be modified or created.

CAS web application

/opt/Oracle/Middleware/wc-sites/bin/cas.properties

ACTION: Check that host.name is commented.

# property moved to host.properties file
#host.name=cas

/opt/Oracle/Middleware/wc-sites/bin/host.properties

ACTION: Change the host.name to the cluster node id. Also check that cs.loginUrl is set correctly. The cs.loginUrl is used cluster-wide by CAS.

host.name=${cluster.node.id}
cs.loginUrl=http://sites-internal.localdomain/cs/ContentServer?pagename=fatwire/wem/sso/ssoLogin

/opt/Oracle/Middleware/applications/CAS/WEB-INF/deployerConfigContext.xml

ACTION: Check that the loginUrl property is referencing cs.loginUrl as set in host.properties.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

	<property name="loginUrl" value="${cs.loginUrl}" />

</beans>

/opt/Oracle/Middleware/applications/CAS/WEB-INF/classes/log4j.properties

ACTION: Change the log4j.appender.logfile value to include the cluster.node.id.

log4j.appender.logfile.File=/var/log/wc-sites/${cluster.node.id}-cas.log

CS web application

/opt/Oracle/Middleware/applications/CS/WEB-INF/classes/SSOConfig.xml

ACTION: Check that propertyPlaceholderConfigurer is added and that casUrl and casInternalUrl are externalized.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">

   <bean id="propertyPlaceholderConfigurer"
                class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

   <property name="casUrl" value="${casUrl}" />
   <property name="casInternalURL" value="${casInternalURL}" />

</beans>

/opt/Oracle/Middleware/applications/CS/WEB-INF/classes/log4j.properties

ACTION:  Same as for CAS. Check that log4j.appender.logfile value includes the cluster.node.id.

log4j.appender.FWDefaultAppender.File=/var/log/wc-sites/${cluster.node.id}-sites.log

WebLogic startup scripts 

The following files are sample WebLogic startup scripts and are provided for reference. They are included to show the various system properties that need to be set.

/opt/Oracle/Middleware/user_projects/domains/wcs/start-sites.sh

#!/bin/bash
if [ $# -eq 0 ]
then
  echo "No arguments supplied"
  exit -1
fi

SERVER_NAME=$1
DOMAIN_NAME="wcs"
ADMIN_URL="http://wcs11gr1.localdomain:7001"
DOMAIN_HOME="/opt/Oracle/Middleware/user_projects/domains/wcs"

WLS_USER="weblogic"
export WLS_USER

WLS_PW="Welcome1"
export WLS_PW

EXT_PRE_CLASSPATH="/opt/Oracle/Middleware/applications/CS/WEB-INF/lib/commons-lang-2.4.jar:/opt/Oracle/Middleware/wc-sites/bin"
export EXT_PRE_CLASSPATH

LD_LIBRARY_PATH="/opt/Oracle/Middleware/wc-sites/bin"
export LD_LIBRARY_PATH

export USERS_MEM_ARGS=-Xmx512M

#  set JAVA_OPTIONS=-Dweblogic.attribute=value -Djava.attribute=value

JAVA_OPTIONS="-Dweblogic.security.SSL.trustedCAKeyStore=/opt/Oracle/Middleware/wlserver_10.3/server/lib/cacerts -DserverType=wlx -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=$DOMAIN_HOME/servers/$SERVER_NAME/tmp -DcasUrl="http://cas.wcsites/cas" -DcasInternalURL="http://cas-internal.localdomain/cas" -Dcluster.node.id=${SERVER_NAME} -Dfile.encoding=UTF-8 -Dnet.sf.ehcache.enableShutdownHook=true -Djava.net.preferIPv4Stack=true -DenableErrorPropagation=true"
export JAVA_OPTIONS

JAVA_VM=""

# Export the admin_url whether the user specified it OR it was sent on the command-line

ADMIN_URL="${ADMIN_URL}"
export ADMIN_URL

SERVER_NAME="${SERVER_NAME}"
export SERVER_NAME

#  Call Weblogic Server with our default params since the user did not specify any other ones
${DOMAIN_HOME}/bin/startWebLogic.sh nodebug noderby noiterativedev notestconsole noLogErrorsToConsole &> ${SERVER_NAME}-console.log &

 

/opt/Oracle/Middleware/user_projects/domains/wcs/start-cas.sh

#!/bin/bash
if [ $# -eq 0 ]
then
  echo "No arguments supplied. You need to supply a managed server name."
  exit -1
fi

SERVER_NAME=$1
DOMAIN_NAME="wcs"
ADMIN_URL="http://wcs11gr1.localdomain:7001"
DOMAIN_HOME="/opt/Oracle/Middleware/user_projects/domains/wcs"

WLS_USER="weblogic"
export WLS_USER

WLS_PW="Welcome1"
export WLS_PW

EXT_PRE_CLASSPATH="/opt/Oracle/Middleware/applications/CS/WEB-INF/lib/commons-lang-2.4.jar:/opt/Oracle/Middleware/wc-sites/bin"
export EXT_PRE_CLASSPATH

export USERS_MEM_ARGS=-Xmx256M

#  set JAVA_OPTIONS=-Dweblogic.attribute=value -Djava.attribute=value

JAVA_OPTIONS="-Dweblogic.security.SSL.trustedCAKeyStore=/opt/Oracle/Middleware/wlserver_10.3/server/lib/cacerts -DserverType=wlx -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=$DOMAIN_HOME/servers/$SERVER_NAME/tmp -Dcluster.node.id=${SERVER_NAME} -Dfile.encoding=UTF-8 -Dnet.sf.ehcache.enableShutdownHook=true -Djava.net.preferIPv4Stack=true"
export JAVA_OPTIONS

JAVA_VM=""

# Export the admin_url whether the user specified it OR it was sent on the command-line

ADMIN_URL="${ADMIN_URL}"
export ADMIN_URL

SERVER_NAME="${SERVER_NAME}"
export SERVER_NAME

#  Call Weblogic Server with our default params since the user did not specify any other ones
${DOMAIN_HOME}/bin/startWebLogic.sh nodebug noderby noiterativedev notestconsole noLogErrorsToConsole &> ${SERVER_NAME}-console.log &

The Apache virtual hosts files.

These files are also included for reference, mostly to understand the request routing of the various hosts to the WebLogic cluster nodes.

/etc/httpd/conf.d/a-wl.conf

LoadModule weblogic_module  modules/mod_wl_22.so
LoadModule filter_module modules/mod_filter.so

/etc/httpd/conf.d/sites-internal.conf

<VirtualHost *:80>
  ServerName sites-internal.localdomain
  DocumentRoot /var/www/sites-internal
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /var/www/sites-internal>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  <Location /cs/ContentServer>
    SetHandler weblogic-handler
    WebLogicCluster localhost:8001,localhost:8002
  </Location>

  ErrorLog logs/error-sites-internal.log
  LogLevel warn
  CustomLog logs/access-sites-internal.log combined

  ProxyRequests Off

</VirtualHost>

/etc/httpd/conf.d/cas-internal.conf

<VirtualHost *:80>
  ServerName cas-internal.localdomain
  DocumentRoot /var/www/cas-internal
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /var/www/cas-internal>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  <Location /cas>
    SetHandler weblogic-handler
    WebLogicCluster localhost:8010,localhost:8011
  </Location>

  ErrorLog logs/error-cas-internal.log
  LogLevel warn
  CustomLog logs/access-cas-internal.log combined

  ProxyRequests Off

</VirtualHost>

/etc/httpd/conf.d/sites.conf

<VirtualHost *:80>
  ServerName www.wcsites
  DocumentRoot /var/www/sites
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /var/www/sites>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>
  <Location /cs>
    SetHandler weblogic-handler
    WebLogicCluster localhost:8001,localhost:8002
    FilterChain gzip
  </Location>

  ErrorLog logs/error-sites.log
  LogLevel warn
  CustomLog logs/access-sites.log combined

  ProxyRequests Off

  RewriteLog logs/sites-rewrite
  RewriteLogLevel 0
  RewriteEngine on
  RewriteRule ^/+$ /cs/ [R,L]
</VirtualHost>

/etc/httpd/conf.d/cas.conf

<VirtualHost *:80>
  ServerName cas.wcsites
  DocumentRoot /var/www/cas
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /var/www/cas>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>
  <Location /cas>
  SetHandler weblogic-handler
  WebLogicCluster localhost:8010,localhost:8011
  </Location>

  ErrorLog logs/error-cas.log
  LogLevel warn
  CustomLog logs/access-cas.log combined
  ProxyRequests Off
</VirtualHost>

 Remote Satellite Server

In this description I have not added Remote SatelliteServer. The installation and configuration of Remote SatelliteServer follows a similar pattern as for Sites. For Remote SatelliteServer you only need to change the location of the log files and the Java temp file location as this is the default location where the inCache files are stored.

 Closing remarks

To have log4j properly working, please make sure that both /opt/Oracle/Middleware/applications/CS/WEB-INF/lib and /opt/Oracle/Middleware/applications/CAS/WEB-INF/lib have commons-logging-<version>.jar and log4j-<version>.jar. Otherwise commons-logging will get confused about which classes and configuration to load.

 

Now, where is the warning in all of this. When doing Sites patches and upgrades, the patch and upgrade scripts might override your changes. You should carefully check after each patch and upgrade deployment if the files are changed and if you need to apply the modifications again.

 

Comments

  1. Hi Dolf, very useful this article, can you please share your other properties values in the cas.properties ??

    I am trying to figure out how can I setup cas/CS properties for an scenario where we have LBR, clusters and we need to provide access in the enterprise LAN (internal) but also remote access thorugh internet for example to technology administrators and content providers.

    However it seems like CS only use the internal URL (casInternalURL) for ticket validation, otherwise use the external URL to redirect client browser to cas login.So I dont have a meaning to keep all communication external or internal depending on the initial request (internal.wcsites for example, or external.wcsites).

    Appreciate your feedback !

    Best !!

    Seryein

  2. Mathew Bibby says:

    Hi Dolf,

    Thanks for the reply, I’ve just a quick look through the INI Files to see if they could be common. There are only 2-3 values that I’m unsure about

    cs.eventhost – You could set this to one host out of the cluster, although you’d reduce resilience?
    cs.xmlfolder – Can this be shared between hosts?
    cs.pgcachefolder – Can this be shared between hosts?

    Cheers
    Matt

    • Dolf Dijkstra says:

      Hi Matt,

      All 3 of those values can be the same for all hosts. Eventhost should point to the loal balancer to receive redundancy.
      In futuretense_xcel.ini is also a setting for batchhost. This should point to a single node, or to a load balancer in a sorry-server setup. A sorry-server is a always forwarding to a single node, unless that node is not reachable.

      Dolf

  3. Mathew Bibby says:

    Hi Dolf,

    Just a quick question, how do you get around the inifiles property in the web.xml?

    In my experience we’ve had to create a custom start up script to rewrite the “inifiles” location before starting each JVM from the common WAR to ensure the correct INI’s are loaded for the JVM in question.

    Is their a neater way of doing this? Does substitution work in the web.xml?

    Cheers
    Matt

    • Mathew Bibby says:

      Apologies, the

      inipath
      VALUE

    • Dolf Dijkstra says:

      Matt,

      The inifiles point to a single location, the war is shared amongst all the cluster members. This means that all the ini files are also shared amonst all the members. As far as I know there is nothing host or cluster node specific in the inifiles.
      An alternative is to have a different set inifiles per JVM in a per JVM classpath. You would still need a set of inifiles in the location specified in web.xml, but the ones in the per JVM classpath will be used (if found).

      Dolf

Add Your Comment