X

Best Practices from Oracle Development's A‑Team

To OCI and back with Virtualbox

Mikael Fransson
Consulting Cloud Solutions Architect

Introduction

Virtualbox recent release 6.0 introduced a very exciting and useful feature. The ability to export a VM directly to OCI. There is a blog and a dedicated white paper on the topic (link in the blog to the wp), https://blogs.oracle.com/virtualization/journey-to-oracle-cloud-infrastructure-with-oracle-vm-virtualbox

My intent here is to share some experiences and recommendations on how to use this feature and show an alternative path to the Cloud to overcome a current performance drawback.

Audience

Any user of Virtualbox that wants to move VM's to the cloud. My guess is that it will be mostly developers.

Use cases

This could be for a variety of reasons. Such as:

  • Sharing work with other developers.
  • Get access to much more powerful hardware.
  • Sharing work with QA teams.
  • Even deploying to production.

How does it work?

It's a very similar workflow to how you have always been able to export your VM as an appliance. But instead of exporting to local disk as an OVA file it's now uploading it directly to a defined Object Storage Bucket and if specified it also creates an instance and starts it up at the end of the export process.

Prerequisites and preparations

This is a short overview of what's needed before you can export a VM to OCI. All the details can be found in the white paper referenced above. There is also a link at the bottom under references.

  • Virtualbox 6.*, always use the latest version. This is a general recommendation. On the rare occasion that there is a regression you can always download an older version and install. I have never seen this having an adverse effect on the VM's themselves.
    • See below for upcoming changes in 6.1, currently in beta.
       
  • Collect the following from the OCI console
    • User OCID
    • Tenancy OCID
    • Region name (note that it's not the OCID)
    • Compartment OCID
    • key_file and fingerprint
       
  • Configure the OCI profile on Oracle VM VirtualBox (Virtualbox Manager menu: File > Cloud Profile Manager)
    • Profiles are stored in ~/Library/VirtualBox/oci_config (macOS). If you already have a profile in ~/.oci/config you can import it when you launch the Cloud Profile Manager or the export the first time.
       
  • OCI access
    • An Object Storage Bucket
    • A VCN
      • Use existing or create one with the recommended regional subnets.
         
  • Prepare the VM to be exported. The White Paper linked to in the blog (referenced above) contains a list of steps to be verified and/or performed on the VM before exporting it. After verifying the checklist in the White Paper this is what I noted for my test VM's:
    • For Oracle Linux 7 & 8 and the latest Ubuntu releases that I installed from scratch I noted that:
      • The VNIC's needed to be renamed as described in the White Paper (eth0, eth1, etc.).
      • I always use NAT and Paravirtualized network driver so no changes needed for my VM's.
      • The official vagrant VM's needed no work at all before exporting to OCI.

Does it work?

  • Yes. If all requirements are fulfilled this does work fine.
  • It can take a while to export, upload and provision a VM. It mostly depends on your own uplink speed.
  • Doing the upload from the Vbox Manager GUI locks up the app for the duration of the export.
  • Network timeouts can happen (I have 100Mbps up and down). I saw that occasionally.
  • Exporting from the command line seemed much more stable.

Recommendations

Prepare the VM with an opc user with a known password and add your public ssh key. For vagrant vm's you can use the existing vagrant user and the ssh key that vagrant installs in the vm but the expected user in an OCI compute instance is opc.

Everything you see in the Virtualbox UI corresponds to a CLI function. This means that there is a command-line option to export to OCI. Exporting from the command-line seems faster and I have not experienced any hiccups or crashes. You are also free to continue working in the Virtualbox Manager with other VM's while the export is running in a terminal session.

List the VM's on your machine:

macos$ VBoxManage list vms
"oracleAPEX" {59755c7d-f3e3-4816-b9d8-5f6f27c2e521}
"MyOCI" {c7dc19f7-8282-4190-874e-7b3f958ada11}
"OL8" {9459b74b-021f-42f4-b835-948299a95acb}
"OL7" {90f746ea-6f10-4766-8761-03a8657deeeb}
"Oracle_DB_Developer_VM" {f8f79b26-eebe-499f-8857-9fd3f8fc050a}
"UbuntuServer" {74dd40f4-4191-4c3c-b8fe-117f3f5ecc1a}
macos$

 

Sample command-line call:

VBoxManage export OL7 --output OCI:// \
--cloud 0 \
--vmname VboxOL7 \
--cloudprofile intvravipati-eu-frankfurt-1 \
--cloudbucket mfranssoBucket1 \
--cloudshape VM.Standard2.1 \
--clouddomain mmXc:EU-FRANKFURT-1-AD-1 \
--clouddisksize 60  \
--cloudocivcn ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaaw.... \
--cloudocisubnet ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaa7... \
--cloudkeepobject true \
--cloudlaunchinstance true \
--cloudpublicip true

 

Shrink the VM's disk file

This is optional but on a VM that's been in use for a while you can save many GB's of disk storage used and this save upload time. More on this a topic here: https://www.ateam-oracle.com/methods-of-compacting-a-disk-image-file.

Test cases

I built a few VM's from scratch and I also tested a couple of our official vagrant VM's.

Steps to rename the NIC are basically the same on Oracle Linux and Ubuntu. You need to disable Consistent Network Device Naming and the biosdevname naming scheme. This is done by appending net.ifnames=0 biosdevname=0 to GRUB_CMDLINE_LINUX in the /etc/default/grub file. It's described in detail in the White Paper. The condensed version is:

  • Edit /etc/default/grub
    • Change (or append) GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
    • Rebuild grub with grub-mkconfig -o /boot/grub/grub.cfg
    • Reboot

Emulated vs. Paravirtualized launch mode

When you export from Virtualbox (GUI or command line) the outcome will be a Compute instance that is launched in emulated mode. This has a downside when it comes to performance. I did some simple comparisons with an OL7 Vbox image (7.7) and compared it with the native Oracle Linux 7.7 image on OCI. Both used the same shape, VM.Standard2.1.

I ran the FIO IOPS test suite described here: https://support.binarylane.com.au/support/solutions/articles/1000055889-how-to-benchmark-disk-i-o

Below are the different test runs.

Disclaimer! This is by no means an official benchmark. Anyone can install these and other utilities and test for themselves. Different shapes will also impact the outcome.

Native - VM.Standard2.1

This is a default OCI compute instance

read : io=3073.3MB, bw=9042.6KB/s, iops=2260 , runt=348023msec
write: io=1022.8MB, bw=3009.3KB/s, iops=752 , runt=348023msec

[root@vm1-fra fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [8988K/2992K /s] [2247 /748  iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=31296: Mon Oct 21 11:56:29 2019
  read : io=3073.3MB, bw=9042.6KB/s, iops=2260 , runt=348023msec
  write: io=1022.8MB, bw=3009.3KB/s, iops=752 , runt=348023msec
  cpu          : usr=2.05%, sys=9.00%, ctx=942181, majf=0, minf=5
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=786756/w=261820/d=0, short=r=0/w=0/d=0
 
Run status group 0 (all jobs):
   READ: io=3073.3MB, aggrb=9042KB/s, minb=9042KB/s, maxb=9042KB/s, mint=348023msec, maxt=348023msec
  WRITE: io=1022.8MB, aggrb=3009KB/s, minb=3009KB/s, maxb=3009KB/s, mint=348023msec, maxt=348023msec
 
Disk stats (read/write):
  sda: ios=786245/261788, merge=0/15, ticks=19716155/2511774, in_queue=21836737, util=99.77%
[root@vm1-fra fio-2.0.9]#

 

Emulated - VM.Standard2.1

This compute instance is exported from Virtualbox to OCI and it runs in emulated mode

read : io=3070.2MB, bw=4222.1KB/s, iops=1055 , runt=744475msec
write: io=1025.9MB, bw=1411.2KB/s, iops=352 , runt=744475msec

As you can see it's a %50 drop in IOPS when you run in emulated mode.

[root@vboxol7mac9 fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [4307K/1466K /s] [1076 /366  iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=14291: Mon Oct 21 13:50:09 2019
  read : io=3070.2MB, bw=4222.1KB/s, iops=1055 , runt=744475msec
  write: io=1025.9MB, bw=1411.2KB/s, iops=352 , runt=744475msec
  cpu          : usr=1.06%, sys=4.03%, ctx=1045079, majf=0, minf=5
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=785962/w=262614/d=0, short=r=0/w=0/d=0
 
Run status group 0 (all jobs):
   READ: io=3070.2MB, aggrb=4222KB/s, minb=4222KB/s, maxb=4222KB/s, mint=744475msec, maxt=744475msec
  WRITE: io=1025.9MB, aggrb=1411KB/s, minb=1411KB/s, maxb=1411KB/s, mint=744475msec, maxt=744475msec
 
Disk stats (read/write):
    dm-0: ios=785700/263051, merge=0/0, ticks=40256234/7254145, in_queue=47512199, util=100.00%, aggrios=785512/263096, aggrmerge=450/93, aggrticks=40249304/7255532, aggrin_queue=47504404, aggrutil=100.00%
  sda: ios=785512/263096, merge=450/93, ticks=40249304/7255532, in_queue=47504404, util=100.00%
You have new mail in /var/spool/mail/root
[root@vboxol7mac9 fio-2.0.9]#

 

Restoring performance by importing a Virtualbox VM as a Custom image

Instead of exporting the VM from Virtualbox to OCI you can convert the Virtualbox VM disk from the native vdi format to vmdk. Upload it to your Object Storage bucket and import it as a custom image and select Paravirtualized Mode.

Overview of the process

PV - VM.Standard2.1

This is the exact same Virtualbox VM but the disk is converted to vmdk format, uploaded and imported as described above.

read : io=3072.5MB, bw=9065.6KB/s, iops=2266 , runt=347046msec
write: io=1023.6MB, bw=3020.2KB/s, iops=755 , runt=347046msec

Permance is restored. The whole process can be scripted.

[root@vm-standard2-1-vbox fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [8795K/3032K /s] [2198 /758  iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=13927: Thu Oct 24 13:09:51 2019
  read : io=3072.5MB, bw=9065.6KB/s, iops=2266 , runt=347046msec
  write: io=1023.6MB, bw=3020.2KB/s, iops=755 , runt=347046msec
  cpu          : usr=2.14%, sys=11.20%, ctx=901820, majf=0, minf=5
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=786541/w=262035/d=0, short=r=0/w=0/d=0
 
Run status group 0 (all jobs):
   READ: io=3072.5MB, aggrb=9065KB/s, minb=9065KB/s, maxb=9065KB/s, mint=347046msec, maxt=347046msec
  WRITE: io=1023.6MB, aggrb=3020KB/s, minb=3020KB/s, maxb=3020KB/s, mint=347046msec, maxt=347046msec
 
Disk stats (read/write):
    dm-0: ios=786336/262117, merge=0/0, ticks=19230667/2856610, in_queue=22089741, util=100.00%, aggrios=786541/262183, aggrmerge=0/12, aggrticks=19238447/2856909, aggrin_queue=21755296, aggrutil=99.85%
  sda: ios=786541/262183, merge=0/12, ticks=19238447/2856909, in_queue=21755296, util=99.85%
You have new mail in /var/spool/mail/root
[root@vm-standard2-1-vbox fio-2.0.9]#

 

New features coming in 6.1 (Beta 2 is out)

This section will be updated for the next beta and upcoming production releases.

Some of the new features are:

  • Support for importing a virtual machine from Oracle Cloud Infrastructure

  • Extended support for exporting a virtual machine to Oracle Cloud Infrastructure, allowing the creation of multiple virtual machines without re-uploading.

  • Also added option to export a VM to the cloud using the more efficient variant "paravirtualized"

  • Virtualization core: support for nested hardware-virtualization on Intel CPUs (starting with 5th generation Core i, codename Broadwell)

  • All details here: https://forums.virtualbox.org/viewtopic.php?f=15&t=94724

Export directly in PV mode for better performance

I tested this with the exact same Virtualbox VM as used in the tests above but now using the new launch mode option on export in Beta 2.

The full export syntax

VBoxManage export           <machines> --output|-o <name>.<ovf/ova/tar.gz>
                            [--legacy09|--ovf09|--ovf10|--ovf20|--opc10]
                            [--manifest]
                            [--iso]
                            [--options manifest|iso|nomacs|nomacsbutnat]
                            [--vsys <number of virtual system>]
                                    [--vmname <name>]
                                    [--product <product name>]
                                    [--producturl <product url>]
                                    [--vendor <vendor name>]
                                    [--vendorurl <vendor url>]
                                    [--version <version info>]
                                    [--description <description info>]
                                    [--eula <license text>]
                                    [--eulafile <filename>]
                            [--cloud <number of virtual system>]
                                    [--vmname <name>]
                                    [--cloudprofile <cloud profile name>]
                                    [--cloudbucket <bucket name>]
                                    [--cloudkeepobject <true/false>]
                                    [--cloudlaunchmode EMULATED|PARAVIRTUALIZED]
                                    [--cloudlaunchinstance <true/false>]
                                    [--clouddomain <domain>]
                                    [--cloudshape <shape>]
                                    [--clouddisksize <disk size in GB>]
                                    [--cloudocivcn <OCI vcn id>]
                                    [--cloudocisubnet <OCI subnet id>]
                                    [--cloudpublicip <true/false>]
                                    [--cloudprivateip <ip>]

 

The command used

time VBoxManage export OL7 --output OCI:// \
--cloud 0 \
--vmname VboxOL7mac9 \
--cloudprofile intvravipati-eu-frankfurt-1 \
--cloudbucket mfranssoBucket1 \
--cloudkeepobject true \
--cloudlaunchmode PARAVIRTUALIZED \
--cloudshape VM.Standard2.1 \
--clouddomain mmXc:EU-FRANKFURT-1-AD-1 \
--clouddisksize 60  \
--cloudocivcn ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaawy3... \
--cloudocisubnet ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaalpb... \
--cloudkeepobject true \
--cloudlaunchinstance true \
--cloudpublicip true

 

The execution and its timing

macos$ time VBoxManage export OL7 --output OCI:// \
> --cloud 0 \
> --vmname VboxOL7mac9 \
> --cloudprofile intvravipati-eu-frankfurt-1 \
> --cloudbucket mfranssoBucket1 \
> --cloudkeepobject true \
> --cloudlaunchmode PARAVIRTUALIZED \
> --cloudshape VM.Standard2.1 \
> --clouddomain mmXc:EU-FRANKFURT-1-AD-1 \
> --clouddisksize 60  \
> --cloudocivcn ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaawy3kze... \
> --cloudocisubnet ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaalpb... \
> --cloudkeepobject true \
> --cloudlaunchinstance true \
> --cloudpublicip true
0%...^@^@^@^@10%...^@20%...30%...^@^@^@40%...^@^@^@50%...^@60%...^@^@70%...
^@^@^@^@^@80%...^@^@^@^@^@^@^@^@^@^@^@^@90%...100%
Successfully exported 1 machine(s).
Creating a cloud instance...
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
A cloud instance with id 'ocid1.instance.oc1.eu-frankfurt-
1.antheljro5bbyticxj2p3gbweto2pdtmkax5r6d2l37xlq6k7vk3r75sotoa' (provider 'OCI') was created
 
real    31m31.788s
user    0m2.761s
sys    0m4.167s
macos$

 

Running the same FIO test as above

read : io=3070.9MB, bw=10851KB/s, iops=2712 , runt=289793msec
write: io=1025.2MB, bw=3622.4KB/s, iops=905 , runt=289793msec

IOPS actually a little better than both NATIVE and earlier tests in PV mode

[root@vboxol7mac9 fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [10741K/3684K /s] [2685 /921  iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=30069: Sat Oct 26 12:24:18 2019
  read : io=3070.9MB, bw=10851KB/s, iops=2712 , runt=289793msec
  write: io=1025.2MB, bw=3622.4KB/s, iops=905 , runt=289793msec
  cpu          : usr=2.56%, sys=12.02%, ctx=924372, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=786142/w=262434/d=0, short=r=0/w=0/d=0
 
Run status group 0 (all jobs):
   READ: io=3070.9MB, aggrb=10851KB/s, minb=10851KB/s, maxb=10851KB/s, mint=289793msec, maxt=289793msec
  WRITE: io=1025.2MB, aggrb=3622KB/s, minb=3622KB/s, maxb=3622KB/s, mint=289793msec, maxt=289793msec
 
Disk stats (read/write):
    dm-0: ios=785991/262508, merge=0/0, ticks=16163236/2316169, in_queue=18481330, util=100.00%, aggrios=786142/262550, aggrmerge=0/9, aggrticks=16168156/2316216, aggrin_queue=18201200, aggrutil=99.85%
  sda: ios=786142/262550, merge=0/9, ticks=16168156/2316216, in_queue=18201200, util=99.85%
You have new mail in /var/spool/mail/root
[root@vboxol7mac9 fio-2.0.9]#

 

References

Journey to Oracle Cloud Infrastructure with Oracle VM VirtualBox
https://blogs.oracle.com/virtualization/journey-to-oracle-cloud-infrastructure-with-oracle-vm-virtualbox

White Paper referenced in the blog post above
http://www.oracle.com/us/technologies/virtualization/oracle-vm-vb-oci-export-20190502-5480003.pdf

Image Import/Export
https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/imageimportexport.htm

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha