Mounting your WebCenter Content NFS file system correctly

Introduction

It is a very well known fact that NFS protocol used to mount file systems in clustered application environments can be prone to issues with things like lock contention, dirty reads, and stale file handle if not configured correctly.  This article is targeted at outlining how a WebCenter Content clustered implementation requires NFS file systems to be mounted in order to operate without issues.

Main Article

NFS V3 and V4 are very common protocols to be used in Unix/Linux environments for mounting shared file systems which are mutually accessible to physical servers which are hosting clustered applications.  WebCenter Content is an application which supports and is commonly deployed in a clustered capacity to provide high availability and fault tolerance.  I have always thought that it was common knowledge on the requirements that WebCenter Content had for mounting shared file systems over NFS protocol but recently I have run into a couple of very high profile implementations where NFS mounts were not correctly configured.  This leads me to believe that the knowledge I thought was common might not be as common as I thought and has inspired me to write this article to ensure that this information is made highly visible via the A-Team chronicles.

In a clustered implementation of WebCenter Content a shared file system is required. This is true even if WebCenter Content has been configured to store managed content in the database using a filestore provider storage rule configured for jdbc storage.   This is the case due to the fact that in addition to content there is application process data that resides on a portion of the shared file system used by WebCenter Content which also needs to be mutually accessible to all nodes in a cluster.  The high level requirements of a shared file system for WebCenter Content no matter what protocol is used are:

• Must be seen as local disk to all nodes in the cluster (i.e. be transparent to the application).
• Must provide the exact same view of the data (file reads/writes/renames/deletes/time stamps, etc...) to every attached cluster node at the same time.
• All member servers of a cluster are time synchronized within 5 seconds. This includes the shared file system server/appliance and all servers where the WebCenter Content application process is running in the same instance/cluster. Setting all members to use the same time (NTP) server is the easiest way to accomplish this.
• File system needs to support atomic operations (write/rename/etc) to ensure that operations complete successfully before they are visible/available to processes running on node which did not initiate the operation

Specific to when NFS protocol is being used for file system mounting and sharing over a standard TCP/IP backbone the following requirements also need to be adhered to in order to avoid performance issues

• The device used for NFS sharing should be a dedicated filer appliance designed for high speed file operations.  This is contrast of using hardware with a full O/S kernel on it which supports being used as a NFS server.  O/S kernel hardware is not designed to handle file operations at the speed required.
• The bandwidth of your TCP/IP backbone connecting your servers hosting the clustered WebCenter Content application cluster to the filer appliance should be at a minimum a 10G-E connection to mitigate network latency and collisions from causing bottlenecks to storage.
• The NFS mount should be configured with locking and file handle and attribute caching disabled.  Options used for this can vary by operating systems but as an example on a Linux kernel these would be nolock, noac, and actimeo=0.  Consult with your server administration team on these to make sure that the correct options are being used for your environment.

It is common that the last requirement of having locking and caching disabled will give most server administrators cause for concern over file contention and file system performance degradation.   These NFS mounting options are required for WebCenter Content due to how it uses a portion of the shared file system as a semaphore for process coordination between nodes in a cluster running the application.

To minimize concerns related to locking, WebCenter Content has been designed to manage file contention and locking on shared file system resources at the application level.  Therefore it needs full control over files without potential of NFS locking conflicting with its application processes.  Disabling attribute and file handle caching is required to ensure that the application processes running on each node is not subject to dirty reads for operations like checking time stamps on files used as part of process coordination/synchronization.  To manage the breadth of impact disabling caching has on operations like

• Content check-in (CHECKIN_NEW service)
• Content retrieval (GET_FILE service)
• Text extraction during indexing

WebCenter Content provides configuration entries that allows the sub-section of the overall shared file system used for process coordination and indexing to be placed on separate NFS mounted volume that has locking and caching disabled.  This enables the NFS volume that stores things like content (e.g. Vault and Weblayout ) and components which do not require caching and locking to be disabled to be on a volume that has caching and locking enabled to mitigate performance issues.  The configuration entries are the following:

• DataDir – Used to partition application data onto its own storage volume to allow file handle caching and locking requirements to be met without affecting other volumes with these settings.
• UserProfileDir – Used to partition user profile data which typically resides under the larger application data onto its own NFS mounted volume that has file handle caching and locking enabled to make disk access to this data faster. (Note: This configuration entry was not  available until the 11.1.1.6 release)
• SearchDir – Used to partition application search processing data  onto its own storage volume to allow file handle caching and locking requirements to be met without affecting other volumes with these settings
• VaultDir – Used to partition application native rendition content data  onto its own storage volume to allow file handle caching and locking requirements to be met without affecting other volumes with these settings
• WebLayoutDir – Used to partition application web rendition content data  onto its own storage volume to allow file handle caching and locking requirements to be met without affecting other volumes with these

The following diagram is used to illustrate the recommended distribution of WebCenter Content directory structure across two NFS volumes to allow WebCenter Content to operate in the most scalable capacity in relation to shared file system access.

Hopefully this article has been able to provide you with good insight on the NFS shared file system requirements that WebCenter Content has and a good explanation behind these requirements.