Remounting DBFS Shared Storage in SOACS and MFTCS Clusters

March 25, 2017 | 6 minute read
Text Size 100%:

Executive Overview

Oracle Database File System (DBFS) is used in providing shared storage mount points within SOACS and MFTCS clusters. DBFS provides a standard file system interface for directories and files on top of database tables[1] . So, by using the database running in the DBCS instance associated with any SOACS or MFTCS instance, DBFS provides a shared network file system similar to NFS that appears like a local file system. This shared network file system provided by DBFS is used for shared storage across nodes in a SOACS or MFTCS cluster, as shown in the diagram below.

As seen from the diagram above, there is a direct dependency on the availability of the database in the DBCS instance for the shared storage in SOACS or MFTCS clusters to be properly functional. In the event the underlying database instance goes down and comes back up later, the shared mount points for the clusters are left in an inconsistent state. To get the shared storage file systems working again, the mount points have to be unmounted and remounted again.

This is quite a standard practice as documented in the public domain [3]. However, it becomes a little more challenging when it comes to running these operations in a PaaS compute node within Oracle Public Cloud (OPC). This blog leverages the same concept and extends it to describe the details of running the operations in an OPC PaaS compute node.

This process can be used until the mount points become immune to database unavailability in a future release of DBFS.

DBFS provides its own set of APIs, based on different technologies, to manage the shared storage mount points. As a result, there are many ways to unmount and remount the DBFS mount points from the client side. The rest of this article describes one way to achieve this goal.

Background

Problem

A standard SOACS or MFTCS instance in Oracle Public Cloud has the following DBFS based shared file system mount points, configured by default:

  • /u01/soacs/dbfs
  • /u01/soacs/dbfs_directio

If the database server goes down and comes up later on, the DBFS mount points in the PaaS compute nodes, mentioned above, can go into an inconsistent state. In such situations, although the mount points appear to be active, attempts to access them results in OS-level error messages (e.g. Numerical result out of range) rendering the file system unusable.

A typical terminal session transcript is shown below as an example.

[opc@mfttest-jcs-wls-1 ~]$ sudo su - oracle
-bash-4.1$ df -h
Filesystem                                          Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root            16G   11G  4.0G  74%   /
tmpfs                                                  7.4G   80K  7.4G   1%    /dev/shm
/dev/xvdb1                                         477M   62M  386M  14% /boot
/dev/mapper/vg_binaries-lv_tools     9.8G  2.3G  7.0G  25%   /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup  20G  222M   19G   2%   /u01/data/backup
/dev/mapper/vg_domains-lv_domains  9.8G  1.3G  8.0G  14%  /u01/data/domains
/dev/mapper/vg_binaries-lv_mw        9.8G  3.2G  6.1G  35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk          2.0G  303M  1.6G  17% /u01/jdk
dbfs-@ORCL:/                                  957M  4.0M  953M   1% /u01/soacs/dbfs
dbfs-@ORCL:/                                  957M  4.0M  953M   1% /u01/soacs/dbfs_directio

-bash-4.1$ ls -l /u01/soacs
ls: cannot access /u01/soacs/dbfs: Numerical result out of range
ls: cannot access /u01/soacs/dbfs_directio: Numerical result out of range
total 16
drwxr-x--- 4 oracle oracle 4096 Dec 15 21:09 BP6
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:05 DBCLIENT
d????????? ? ?      ?         ?            ? dbfs
d????????? ? ?      ?         ?            ? dbfs_directio
drwxr-x--- 2 oracle oracle 4096 Dec 15 21:07 MFT
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:19 tmp
-bash-4.1$

Solution Approach

Overview

The overall solution consists of the following steps to be performed via terminal sessions on each of the MFTCS or SOACS nodes that are DBFS clients for the DBFS shared file systems.

  • Setup the OS environment to run DBFS client
  • Unmount the DBFS mount points
  • Remount the DBFS mount points

DBFS client uses FUSE (Filesystem in Userspace) interface[2] to administer the DBFS mount points. As part of the default provisioning process for PaaS compute nodes, the DBFS client is automatically installed to provide the FUSE interface.

Task and Activity Details

The following sections will walk through the details of the individual steps. All the commands are executed as Linux user, oracle.

The environment consists of a single node MFTCS cluster and the corresponding DBCS instance, as listed below:

  • OPC PaaS Compute running MFTCS Release 12.1.3.0.160719
  • OPC PaaS Compute running DBCS Release 12.1.0.2 Enterprise Edition

I. Setup the environment

Log in to the MFTCS node as oracle and setup the following environment variables:

  • ORACLE_HOME (Oracle database client home contained within MFTCS/SOACS install base)
  • LD_LIBRARY_PATH (Libraries required to run the DBFS client and fuse packages)
  • TNS_ADMIN (Configuration files needed to establish the SQL*Net client connectivity to the database server)

A typical session transcript is shown below

[opc@mfttest-jcs-wls-1 ~]$ sudo su - oracle
-bash-4.1$
-bash-4.1$ export ORACLE_HOME=/u01/app/oracle/middleware/dbclient
-bash-4.1$ export LD_LIBRARY_PATH=/u01/app/oracle/middleware/dbclient/lib
-bash-4.1$ export TNS_ADMIN=/u01/data/domains/MFTTest-_domain/dbfs

II. Unmount the DBFS shared file systems using FUSE command-line utility (fusermount)

We use the command-line utility, fusermount to unmount the shared file systems.

  • fusermount -u <DBFS mount point>

The DBFS client runs as a background OS process but upon unmounting the shared file system, the process is automatically terminated. A dump of file system mount points and active DBFS client processes before and after the unmount operation is shown below to highlight the behavior.

Baseline

-bash-4.1$ df -h
Filesystem                                          Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root            16G   11G  4.0G  74%   /
tmpfs                                                  7.4G   80K  7.4G   1%   /dev/shm
/dev/xvdb1                                        477M   62M  386M  14% /boot
/dev/mapper/vg_binaries-lv_tools    9.8G  2.3G  7.0G  25%   /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup  20G  222M   19G   2%   /u01/data/backup
/dev/mapper/vg_domains-lv_domains 9.8G  1.3G  8.0G  14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw          9.8G  3.2G  6.1G  35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk           2.0G  303M  1.6G  17% /u01/jdk
dbfs-@ORCL:/                                    957M  4.0M  953M   1% /u01/soacs/dbfs
dbfs-@ORCL:/                                    957M  4.0M  953M   1% /u01/soacs/dbfs_directio
-bash-4.1$ ps -fade | grep dbfs
oracle    5403     1  0 Jan09 ?        00:01:27 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
oracle    9613     1  0 16:26 ?        00:00:00 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
oracle   10483 10447  0 17:23 pts/1    00:00:00 grep dbfs

Unmount

-bash-4.1$ fusermount -u /u01/soacs/dbfs
-bash-4.1$ fusermount -u /u01/soacs/dbfs_directio
-bash-4.1$

Verify

-bash-4.1$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root            16G   11G  4.0G  74% /
tmpfs                                                 7.4G   80K  7.4G   1% /dev/shm
/dev/xvdb1                                        477M   62M  386M  14% /boot
/dev/mapper/vg_binaries-lv_tools     9.8G  2.3G  7.0G  25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup   20G  222M   19G   2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains  9.8G  1.3G  8.0G  14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw           9.8G  3.2G  6.1G  35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk           2.0G  303M  1.6G  17% /u01/jdk

-bash-4.1$ ps -fade | grep dbfs
oracle   10552 10447  0 17:28 pts/1    00:00:00 grep dbfs
-bash-4.1$

As noted above in the "Verify" section, the background DBFS client processes and the shared mount points are removed as a result of the unmount operation.

III. Remount the DBFS shared file systems using DBFS Client command-line utility (dbfs_client)

We use the command-line utility, dbfs_client to remount the shared file systems.This utility is bundled within the Oracle DBFS client library, that is automatically installed in a PaaS compute node as part of the standard provisioning process within OPC. Although it takes several arguments in its usage, the syntax used for our purpose is shown below:

  • $ORACLE_HOME/bin/dbfs_client -o wallet /@<DBCSConnector> <DBFS shared mount  point>

First, we navigate to the dbfs subdirectory under $DOMAIN_HOME for PaaS domain and then run the dbfs_client command to remount the DBFS shared file systems. A sample session is shown below.

Remount

-bash-4.1$ cd /u01/data/domains/MFTTest-_domain/dbfs
-bash-4.1$ $ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
-bash-4.1$ $ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
-bash-4.1$

Verify

-bash-4.1$ df -h
Filesystem                                           Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root             16G   11G  4.0G  74% /
tmpfs                                                   7.4G   80K  7.4G   1% /dev/shm
/dev/xvdb1                                          477M   62M  386M  14% /boot
/dev/mapper/vg_binaries-lv_tools        9.8G  2.3G  7.0G  25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup    20G  222M   19G   2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains   9.8G  1.3G  8.0G  14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw          9.8G  3.2G  6.1G  35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk           2.0G  303M  1.6G  17% /u01/jdk
dbfs-@ORCL:/                                   957M  4.0M  953M   1% /u01/soacs/dbfs
dbfs-@ORCL:/                                  957M  4.0M  953M   1% /u01/soacs/dbfs_directio
-bash-4.1$ ls -ltr /u01/soacs
total 16
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:05 DBCLIENT
drwxr-x--- 2 oracle oracle 4096 Dec 15 21:07 MFT
drwxr-x--- 4 oracle oracle 4096 Dec 15 21:09 BP6
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:19 tmp
drwxr-xr-x 3 root   root      0 Mar 22 16:14 dbfs_directio
drwxr-xr-x 3 root   root      0 Mar 22 16:14 dbfs
-bash-4.1$ ps -fade | grep dbfs
oracle   12792     1  0 Mar22 ?        00:00:03 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
oracle   12821     1  0 Mar22 ?        00:00:02 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
oracle   28468 28435  0 16:32 pts/0    00:00:00 grep dbfs

Thus we can verify that all the DBFS shared storage mount points and their corresponding DBFS client background processes are back up and running as expected.

Summary

The method described here is a quick way to restore the DBFS shared storage mount points in case they are rendered inaccessible due to underlying database availability issues in the attached DBCS instance. There can be other options available in future releases and they will be discussed as an addendum here or in a separate post.

Acknowledgements

Dave Berry and Scott Haaland from Oracle PaaS Product Management team have been actively involved in the knowledge transfer of PaaS shared storage with DBFS for many months. Thanks to my team-mate, Sherwood Zern for providing initial pointers on getting the solution validated during a troubleshooting session with an Oracle Cloud customer.

References

  1. 1. Introducing the Oracle Database File System - Oracle Documentation Set
  2. 2. FUSE (Filesystem in Userspace) - GitHub Project  Documentation
  3. 3. Numerical result out of range on DBFS mount - External blog by ronniethedba

Appendix

Linux packages installed to provide the FUSE functionality in a standard PaaS compute node are listed below.

  • gvfs-fuse-1.4.3-26.el6.x86_64
  • fuse-devel-2.8.3-4.0.2.el6.x86_64
  • fuse-libs-2.8.3-4.0.2.el6.x86_64
  • fuse-2.8.3-4.0.2.el6.x86_64
  • kernel-devel-2.6.32-642.3.1.el6.x86_64

Shub Lahiri


Previous Post

Working with Oracle HCM Cloud Talent Data

Siming Mu | 8 min read

Next Post


ICS Connectivity Agent stdout Log Rotation

Greg Mally | 5 min read