Oracle Database File System (DBFS) is used in providing shared storage mount points within SOACS and MFTCS clusters. DBFS provides a standard file system interface for directories and files on top of database tables[1] . So, by using the database running in the DBCS instance associated with any SOACS or MFTCS instance, DBFS provides a shared network file system similar to NFS that appears like a local file system. This shared network file system provided by DBFS is used for shared storage across nodes in a SOACS or MFTCS cluster, as shown in the diagram below.
As seen from the diagram above, there is a direct dependency on the availability of the database in the DBCS instance for the shared storage in SOACS or MFTCS clusters to be properly functional. In the event the underlying database instance goes down and comes back up later, the shared mount points for the clusters are left in an inconsistent state. To get the shared storage file systems working again, the mount points have to be unmounted and remounted again.
This is quite a standard practice as documented in the public domain [3]. However, it becomes a little more challenging when it comes to running these operations in a PaaS compute node within Oracle Public Cloud (OPC). This blog leverages the same concept and extends it to describe the details of running the operations in an OPC PaaS compute node.
This process can be used until the mount points become immune to database unavailability in a future release of DBFS.
DBFS provides its own set of APIs, based on different technologies, to manage the shared storage mount points. As a result, there are many ways to unmount and remount the DBFS mount points from the client side. The rest of this article describes one way to achieve this goal.
A standard SOACS or MFTCS instance in Oracle Public Cloud has the following DBFS based shared file system mount points, configured by default:
If the database server goes down and comes up later on, the DBFS mount points in the PaaS compute nodes, mentioned above, can go into an inconsistent state. In such situations, although the mount points appear to be active, attempts to access them results in OS-level error messages (e.g. Numerical result out of range) rendering the file system unusable.
A typical terminal session transcript is shown below as an example.
[opc@mfttest-jcs-wls-1 ~]$ sudo su - oracle
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root 16G 11G 4.0G 74% /
tmpfs 7.4G 80K 7.4G 1% /dev/shm
/dev/xvdb1 477M 62M 386M 14% /boot
/dev/mapper/vg_binaries-lv_tools 9.8G 2.3G 7.0G 25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup 20G 222M 19G 2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains 9.8G 1.3G 8.0G 14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw 9.8G 3.2G 6.1G 35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk 2.0G 303M 1.6G 17% /u01/jdk
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs_directio
-bash-4.1$ ls -l /u01/soacs
ls: cannot access /u01/soacs/dbfs: Numerical result out of range
ls: cannot access /u01/soacs/dbfs_directio: Numerical result out of range
total 16
drwxr-x--- 4 oracle oracle 4096 Dec 15 21:09 BP6
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:05 DBCLIENT
d????????? ? ? ? ? ? dbfs
d????????? ? ? ? ? ? dbfs_directio
drwxr-x--- 2 oracle oracle 4096 Dec 15 21:07 MFT
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:19 tmp
-bash-4.1$
The overall solution consists of the following steps to be performed via terminal sessions on each of the MFTCS or SOACS nodes that are DBFS clients for the DBFS shared file systems.
DBFS client uses FUSE (Filesystem in Userspace) interface[2] to administer the DBFS mount points. As part of the default provisioning process for PaaS compute nodes, the DBFS client is automatically installed to provide the FUSE interface.
The following sections will walk through the details of the individual steps. All the commands are executed as Linux user, oracle.
The environment consists of a single node MFTCS cluster and the corresponding DBCS instance, as listed below:
Log in to the MFTCS node as oracle and setup the following environment variables:
A typical session transcript is shown below
[opc@mfttest-jcs-wls-1 ~]$ sudo su - oracle
-bash-4.1$
-bash-4.1$ export ORACLE_HOME=/u01/app/oracle/middleware/dbclient
-bash-4.1$ export LD_LIBRARY_PATH=/u01/app/oracle/middleware/dbclient/lib
-bash-4.1$ export TNS_ADMIN=/u01/data/domains/MFTTest-_domain/dbfs
We use the command-line utility, fusermount to unmount the shared file systems.
The DBFS client runs as a background OS process but upon unmounting the shared file system, the process is automatically terminated. A dump of file system mount points and active DBFS client processes before and after the unmount operation is shown below to highlight the behavior.
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root 16G 11G 4.0G 74% /
tmpfs 7.4G 80K 7.4G 1% /dev/shm
/dev/xvdb1 477M 62M 386M 14% /boot
/dev/mapper/vg_binaries-lv_tools 9.8G 2.3G 7.0G 25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup 20G 222M 19G 2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains 9.8G 1.3G 8.0G 14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw 9.8G 3.2G 6.1G 35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk 2.0G 303M 1.6G 17% /u01/jdk
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs_directio
-bash-4.1$ ps -fade | grep dbfs
oracle 5403 1 0 Jan09 ? 00:01:27 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
oracle 9613 1 0 16:26 ? 00:00:00 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
oracle 10483 10447 0 17:23 pts/1 00:00:00 grep dbfs
-bash-4.1$ fusermount -u /u01/soacs/dbfs
-bash-4.1$ fusermount -u /u01/soacs/dbfs_directio
-bash-4.1$
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root 16G 11G 4.0G 74% /
tmpfs 7.4G 80K 7.4G 1% /dev/shm
/dev/xvdb1 477M 62M 386M 14% /boot
/dev/mapper/vg_binaries-lv_tools 9.8G 2.3G 7.0G 25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup 20G 222M 19G 2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains 9.8G 1.3G 8.0G 14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw 9.8G 3.2G 6.1G 35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk 2.0G 303M 1.6G 17% /u01/jdk
-bash-4.1$ ps -fade | grep dbfs
oracle 10552 10447 0 17:28 pts/1 00:00:00 grep dbfs
-bash-4.1$
As noted above in the "Verify" section, the background DBFS client processes and the shared mount points are removed as a result of the unmount operation.
We use the command-line utility, dbfs_client to remount the shared file systems.This utility is bundled within the Oracle DBFS client library, that is automatically installed in a PaaS compute node as part of the standard provisioning process within OPC. Although it takes several arguments in its usage, the syntax used for our purpose is shown below:
First, we navigate to the dbfs subdirectory under $DOMAIN_HOME for PaaS domain and then run the dbfs_client command to remount the DBFS shared file systems. A sample session is shown below.
-bash-4.1$ cd /u01/data/domains/MFTTest-_domain/dbfs
-bash-4.1$ $ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
-bash-4.1$ $ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
-bash-4.1$
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_root 16G 11G 4.0G 74% /
tmpfs 7.4G 80K 7.4G 1% /dev/shm
/dev/xvdb1 477M 62M 386M 14% /boot
/dev/mapper/vg_binaries-lv_tools 9.8G 2.3G 7.0G 25% /u01/app/oracle/tools
/dev/mapper/vg_backup-lv_backup 20G 222M 19G 2% /u01/data/backup
/dev/mapper/vg_domains-lv_domains 9.8G 1.3G 8.0G 14% /u01/data/domains
/dev/mapper/vg_binaries-lv_mw 9.8G 3.2G 6.1G 35% /u01/app/oracle/middleware
/dev/mapper/vg_binaries-lv_jdk 2.0G 303M 1.6G 17% /u01/jdk
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs
dbfs-@ORCL:/ 957M 4.0M 953M 1% /u01/soacs/dbfs_directio
-bash-4.1$ ls -ltr /u01/soacs
total 16
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:05 DBCLIENT
drwxr-x--- 2 oracle oracle 4096 Dec 15 21:07 MFT
drwxr-x--- 4 oracle oracle 4096 Dec 15 21:09 BP6
drwxr-x--- 3 oracle oracle 4096 Dec 15 21:19 tmp
drwxr-xr-x 3 root root 0 Mar 22 16:14 dbfs_directio
drwxr-xr-x 3 root root 0 Mar 22 16:14 dbfs
-bash-4.1$ ps -fade | grep dbfs
oracle 12792 1 0 Mar22 ? 00:00:03 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL -o direct_io /u01/soacs/dbfs_directio
oracle 12821 1 0 Mar22 ? 00:00:02 /u01/app/oracle/middleware/dbclient/bin/dbfs_client -o wallet /@ORCL /u01/soacs/dbfs
oracle 28468 28435 0 16:32 pts/0 00:00:00 grep dbfs
Thus we can verify that all the DBFS shared storage mount points and their corresponding DBFS client background processes are back up and running as expected.
The method described here is a quick way to restore the DBFS shared storage mount points in case they are rendered inaccessible due to underlying database availability issues in the attached DBCS instance. There can be other options available in future releases and they will be discussed as an addendum here or in a separate post.
Dave Berry and Scott Haaland from Oracle PaaS Product Management team have been actively involved in the knowledge transfer of PaaS shared storage with DBFS for many months. Thanks to my team-mate, Sherwood Zern for providing initial pointers on getting the solution validated during a troubleshooting session with an Oracle Cloud customer.
Linux packages installed to provide the FUSE functionality in a standard PaaS compute node are listed below.