To OCI and back with Virtualbox
November 7, 2019 | 12 minute read
Mikael Fransson
Introduction
The recent release of Virtualbox version 6.* introduced some very exciting and useful features. The ability to export and import VM's to and from OCI. There is a blog and a dedicated white paper on the topic (link in the blog to the wp), https://blogs.oracle.com/virtualization/journey-to-oracle-cloud-infrastructure-with-oracle-vm-virtualbox
My intent here is to share some experiences and recommendations on how to use this feature and show an alternative path to the Cloud to overcome a current performance drawback.
Audience
Any user of Virtualbox that wants to move VM's to the cloud. My guess is that it will be mostly developers.
Use cases
This could be for a variety of reasons. Such as:
- Sharing work with other developers.
- Get access to much more powerful hardware.
- Sharing work with QA teams.
- Even deploying to production.
How does it work?
It's a very similar workflow to how you have always been able to export your VM as an appliance. But instead of exporting to local disk as an OVA file it's now possible to upload it directly to a defined Object Storage Bucket and if specified the process also creates an instance and starts it up at the end of the export process.
Prerequisites and preparations
This is a short overview of what's needed before you can export a VM to OCI. All the details can be found in the white paper referenced above. There is also a link at the bottom under references.
Virtualbox 6.1.*, always use the latest version. This is a general recommendation. On the rare occasion that there is a regression you can always download an older version and install. I have never seen this having an adverse effect on the VM's themselves. See below for changes in 6.1.*, current release is 6.1.6.
Collect the following from the OCI console
- User OCID
- Tenancy OCID
- Region name (note that it's not the OCID)
- Compartment OCID
- key_file and fingerprint
Configure the OCI profile on Oracle VM VirtualBox (Virtualbox Manager menu: File > Cloud Profile Manager)
- Profiles are stored in ~/Library/VirtualBox/oci_config (macOS).
- If you already have a profile in ~/.oci/config you can import it when you launch the Cloud Profile Manager or the export the first time.
OCI access
- An Object Storage Bucket
- A VCN (Use existing or create one with the recommended regional subnets)
Prepare the VM to be exported. The White Paper linked to in the blog (referenced above) contains a list of steps to be verified and/or performed on the VM before exporting it. After verifying the checklist in the White Paper this is what I noted for my test VM's:
For Oracle Linux 7 & 8 and the latest Ubuntu releases that I installed from scratch I noted that:
- The VNIC's needed to be renamed as described in the White Paper (eth0, eth1, etc.).
- I always use NAT and Paravirtualized network driver so no changes needed for my VM's.
- The official vagrant VM's needed no work at all before exporting to OCI.
Does it work?
- Yes. If all requirements are fulfilled this does work fine.
- It can take a while to export, upload and provision a VM. It mostly depends on your own uplink speed.
- Doing the upload from the Vbox Manager GUI locks up the app for the duration of the export.
- Network timeouts can happen (I have 100Mbps up and down). I saw that occasionally.
- Exporting from the command line seemed much more stable.
Recommendations
Prepare the VM with an opc user with a known password and add your public ssh key. For vagrant vm's you can use the existing vagrant user and the ssh key that vagrant installs in the vm but the expected user in an OCI compute instance is opc.
Everything you see in the Virtualbox UI corresponds to a CLI function. This means that there is a command-line option to export to OCI. Exporting from the command-line seems faster and I have not experienced any hiccups or crashes. You are also free to continue working in the Virtualbox Manager with other VM's while the export is running in a terminal session.
List the VM's on your machine:
|
$ VBoxManage list vms |
For the sample command below we need a value for the --clouddomain parameter. We can find out valid values via oci cli.
|
$ oci iam availability-domain list |
Sample command-line call (the time command is just there so that I can see the time it took to run the export):
|
time VBoxManage export OL7 --output OCI:// \ |
time is a command on linux and macOS that will show the time it took to run the command. Not sure if it's available in Windows Powershell.
--cloudprofile = the name of your profile in Virtualbox Cloud Profile Manager
If you don't specify the --cloudlaunchmode parameter it will default to EMULATED which has a performance impact. See below.
The actual run of the above command with the timing:
|
$ time VBoxManage export OL7 --output OCI:// \ real 35m25.498s |
The timing is obviously dependent on your internet connection. In my case I'm working from home using a 100/100 Mbps fiber.
Shrink the VM's disk file
This is optional but on a VM that's been in use for a while you can save many GB's of disk storage used and this save upload time. More on this topic here: https://www.ateam-oracle.com/methods-of-compacting-a-disk-image-file.
Test cases
I built a few VM's from scratch and I also tested a couple of our official vagrant VM's.
OracleAPEX (vagrant)
- NIC name: eth0
- Uploaded and worked without any preparation
OL7 (vagrant)
- NIC name: eth0
- Uploaded and worked without any preparation
OL7, server with GUI install
- NIC name: enp0s3, needs to be renamed to eth0.
Ubuntu Server, 18.04.3 LTS
- NIC name: enp0s3, needs to be renamed to eth0.
Ubuntu Server, 19.10
- NIC name: enp0s3, needs to be renamed to eth0
Ubuntu Desktop, 19.10
- NIC name: enp0s3, needs to be renamed to eth0
Steps to rename the NIC are basically the same on Oracle Linux and Ubuntu. You need to disable Consistent Network Device Naming and the biosdevname naming scheme. This is done by appending net.ifnames=0 biosdevname=0 to GRUB_CMDLINE_LINUX in the /etc/default/grub file. It's described in detail in the White Paper. The condensed version is:
Edit /etc/default/grub
- Change (or append) GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
- Rebuild grub with grub-mkconfig -o /boot/grub/grub.cfg
- Reboot
Emulated vs. Paravirtualized launch mode
When you export from Virtualbox (GUI or command line) the outcome will be a Compute instance that is launched in emulated mode. This has a downside when it comes to performance. I did some simple comparisons with an OL7 Vbox image (7.7) and compared it with the native Oracle Linux 7.7 image on OCI. Both used the same shape, VM.Standard2.1.
I ran the FIO IOPS test suite described here: https://support.binarylane.com.au/support/solutions/articles/1000055889-how-to-benchmark-disk-i-o
Below are the different test runs.
Disclaimer! This is by no means an official benchmark. Anyone can install these and other utilities and test for themselves. Different shapes will also impact the outcome.
Native - VM.Standard2.1
This is a default OCI compute instance
read : io=3073.3MB, bw=9042.6KB/s, iops=2260 , runt=348023msec
write: io=1022.8MB, bw=3009.3KB/s, iops=752 , runt=348023msec
For reference, the full FIO output.
| [root@vm1-fra fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.0.9 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m] [100.0% done] [8988K/2992K /s] [2247 /748 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=31296: Mon Oct 21 11:56:29 2019 read : io=3073.3MB, bw=9042.6KB/s, iops=2260 , runt=348023msec write: io=1022.8MB, bw=3009.3KB/s, iops=752 , runt=348023msec cpu : usr=2.05%, sys=9.00%, ctx=942181, majf=0, minf=5 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=786756/w=261820/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=3073.3MB, aggrb=9042KB/s, minb=9042KB/s, maxb=9042KB/s, mint=348023msec, maxt=348023msec WRITE: io=1022.8MB, aggrb=3009KB/s, minb=3009KB/s, maxb=3009KB/s, mint=348023msec, maxt=348023msec Disk stats (read/write): sda: ios=786245/261788, merge=0/15, ticks=19716155/2511774, in_queue=21836737, util=99.77% [root@vm1-fra fio-2.0.9]# |
Emulated - VM.Standard2.1
This compute instance is exported from Virtualbox to OCI and it runs in emulated mode
read : io=3070.2MB, bw=4222.1KB/s, iops=1055 , runt=744475msec
write: io=1025.9MB, bw=1411.2KB/s, iops=352 , runt=744475msec
As you can see it's a %50 drop in IOPS when you run in emulated mode.
For reference, the full FIO output.
| [root@vboxol7mac9 fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.0.9 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m] [100.0% done] [4307K/1466K /s] [1076 /366 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=14291: Mon Oct 21 13:50:09 2019 read : io=3070.2MB, bw=4222.1KB/s, iops=1055 , runt=744475msec write: io=1025.9MB, bw=1411.2KB/s, iops=352 , runt=744475msec cpu : usr=1.06%, sys=4.03%, ctx=1045079, majf=0, minf=5 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=785962/w=262614/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=3070.2MB, aggrb=4222KB/s, minb=4222KB/s, maxb=4222KB/s, mint=744475msec, maxt=744475msec WRITE: io=1025.9MB, aggrb=1411KB/s, minb=1411KB/s, maxb=1411KB/s, mint=744475msec, maxt=744475msec Disk stats (read/write): dm-0: ios=785700/263051, merge=0/0, ticks=40256234/7254145, in_queue=47512199, util=100.00%, aggrios=785512/263096, aggrmerge=450/93, aggrticks=40249304/7255532, aggrin_queue=47504404, aggrutil=100.00% sda: ios=785512/263096, merge=450/93, ticks=40249304/7255532, in_queue=47504404, util=100.00% You have new mail in /var/spool/mail/root [root@vboxol7mac9 fio-2.0.9]# |
Restoring performance by importing a Virtualbox VM as a Custom image
Instead of exporting the VM from Virtualbox to OCI you can convert the Virtualbox VM disk from the native vdi format to vmdk. Upload it to your Object Storage bucket and import it as a custom image and select Paravirtualized Mode.
Overview of the process
Convert the Virtualbox disk image file from .vdi to .vmdk format with:
- VBoxManage clonehd myoldfile.vdi mynewfile.vmdk --format VMDK
Upload to your Object Storage Bucket, three alternative methods:
- Using the OCI Console
- Using OCI CLI
- Using CyberDuck. Free, supports Windows and Mac. Simple drag & drop UI. Support for OCI.
Import the image to OCI, https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/imageimportexport.htm#URLs
- Make sure to import it as a PV image
- Create a Compute instance using the imported custom image
PV - VM.Standard2.1
This is the exact same Virtualbox VM but the disk is converted to vmdk format, uploaded and imported as described above.
read : io=3072.5MB, bw=9065.6KB/s, iops=2266 , runt=347046msec
write: io=1023.6MB, bw=3020.2KB/s, iops=755 , runt=347046msec
Performance is restored. The whole process can be scripted.
For reference, the full FIO output.
| [root@vm-standard2-1-vbox fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.0.9 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m] [100.0% done] [8795K/3032K /s] [2198 /758 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=13927: Thu Oct 24 13:09:51 2019 read : io=3072.5MB, bw=9065.6KB/s, iops=2266 , runt=347046msec write: io=1023.6MB, bw=3020.2KB/s, iops=755 , runt=347046msec cpu : usr=2.14%, sys=11.20%, ctx=901820, majf=0, minf=5 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=786541/w=262035/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=3072.5MB, aggrb=9065KB/s, minb=9065KB/s, maxb=9065KB/s, mint=347046msec, maxt=347046msec WRITE: io=1023.6MB, aggrb=3020KB/s, minb=3020KB/s, maxb=3020KB/s, mint=347046msec, maxt=347046msec Disk stats (read/write): dm-0: ios=786336/262117, merge=0/0, ticks=19230667/2856610, in_queue=22089741, util=100.00%, aggrios=786541/262183, aggrmerge=0/12, aggrticks=19238447/2856909, aggrin_queue=21755296, aggrutil=99.85% sda: ios=786541/262183, merge=0/12, ticks=19238447/2856909, in_queue=21755296, util=99.85% You have new mail in /var/spool/mail/root [root@vm-standard2-1-vbox fio-2.0.9]# |
New features in 6.1
Some of the new features are:- Support for importing a virtual machine from Oracle Cloud Infrastructure
- Extended support for exporting a virtual machine to Oracle Cloud Infrastructure, allowing the creation of multiple virtual machines without re-uploading.
- Also added option to export a VM to the cloud using the more efficient variant "paravirtualized"
- Virtualization core: support for nested hardware-virtualization on Intel CPUs (starting with 5th generation Core i, codename Broadwell)
- All details here: https://forums.virtualbox.org/viewtopic.php?f=15&t=94724
Export directly in PV mode for better performance
I tested this with the exact same Virtualbox VM as used in the tests above but now using the new launch mode option on export in Beta 2.
The full export syntax
| VBoxManage export <machines> --output|-o <name>.<ovf/ova/tar.gz> [--legacy09|--ovf09|--ovf10|--ovf20|--opc10] [--manifest] [--iso] [--options manifest|iso|nomacs|nomacsbutnat] [--vsys <number of virtual system>] [--vmname <name>] [--product <product name>] [--producturl <product url>] [--vendor <vendor name>] [--vendorurl <vendor url>] [--version <version info>] [--description <description info>] [--eula <license text>] [--eulafile <filename>] [--cloud <number of virtual system>] [--vmname <name>] [--cloudprofile <cloud profile name>] [--cloudbucket <bucket name>] [--cloudkeepobject <true/false>] [--cloudlaunchmode EMULATED|PARAVIRTUALIZED] [--cloudlaunchinstance <true/false>] [--clouddomain <domain>] [--cloudshape <shape>] [--clouddisksize <disk size in GB>] [--cloudocivcn <OCI vcn id>] [--cloudocisubnet <OCI subnet id>] [--cloudpublicip <true/false>] [--cloudprivateip <ip>] |
The command used
| time VBoxManage export OL7 --output OCI:// \ --cloud 0 \ --vmname VboxOL7mac9 \ --cloudprofile myprofile \ --cloudbucket myBucket1 \ --cloudkeepobject true \ --cloudlaunchmode PARAVIRTUALIZED \ --cloudshape VM.Standard2.1 \ --clouddomain uFjs:EU-FRANKFURT-1-AD-1 \ --clouddisksize 60 \ --cloudocivcn ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaawy3... \ --cloudocisubnet ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaalpb... \ --cloudkeepobject true \ --cloudlaunchinstance true \ --cloudpublicip true |
The execution and its timing
|
$ time VBoxManage export OL7 --output OCI:// \ |
Running the same FIO test as above
read : io=3070.9MB, bw=10851KB/s, iops=2712 , runt=289793msec
write: io=1025.2MB, bw=3622.4KB/s, iops=905 , runt=289793msec
IOPS actually a little better than both NATIVE and earlier tests in PV mode
For reference, the full FIO output.
| [root@vboxol7mac9 fio-2.0.9]# ./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.0.9 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m] [100.0% done] [10741K/3684K /s] [2685 /921 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=30069: Sat Oct 26 12:24:18 2019 read : io=3070.9MB, bw=10851KB/s, iops=2712 , runt=289793msec write: io=1025.2MB, bw=3622.4KB/s, iops=905 , runt=289793msec cpu : usr=2.56%, sys=12.02%, ctx=924372, majf=0, minf=4 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=786142/w=262434/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=3070.9MB, aggrb=10851KB/s, minb=10851KB/s, maxb=10851KB/s, mint=289793msec, maxt=289793msec WRITE: io=1025.2MB, aggrb=3622KB/s, minb=3622KB/s, maxb=3622KB/s, mint=289793msec, maxt=289793msec Disk stats (read/write): dm-0: ios=785991/262508, merge=0/0, ticks=16163236/2316169, in_queue=18481330, util=100.00%, aggrios=786142/262550, aggrmerge=0/9, aggrticks=16168156/2316216, aggrin_queue=18201200, aggrutil=99.85% sda: ios=786142/262550, merge=0/9, ticks=16168156/2316216, in_queue=18201200, util=99.85% You have new mail in /var/spool/mail/root [root@vboxol7mac9 fio-2.0.9]# |
Import from OCI
The import now works for my test case which is a default OL 7.7 image using the VM.Standard2.1 Shape.
Screenshots of the actual process from Virtualbox Manager:From the File menu select Import Appliance... The first screen is where you select what Source and and Account (Tenancy) to use. It will then list available instances that you can import.
The next screen allows you to change Type, Version, Memory etc. Double click on the Value fields to change their value. Make sure to navigate away from the fields as the selected value might not stick otherwise.
The next screen will now show the progress. The time it takes depends very much on the bandwidth speed you have access to.
Using the command line instead.
Once you have the VM on your local machine in Virtualbox you might have issues to login since you don't know the opc or root password. The solution is to do a port mapping on the default NAT adapter in the Virtualbox VM Settings. Map the guest OS port 22 to a free port on the host, maybe 2222. Then you can login with ssh -i ~/.ssh/id_rsa opc@localhost -p 2222.
|
$ VBoxManage import OCI:// --cloud --vmname vm1-public --memory 4000 --cpus 2 --ostype Oracle_64 --cloudprofile intvravipati-eu-frankfurt-1 --cloudinstanceid ocid1.instance.oc1.eu-frankfurt-1.antheljtc3adhhqcp5qsfjtn4wfplx4u4czsog2xxde3u5f6vbsqytrqsszq --cloudbucket mfranssoBucket1 real 23m11.117s
|
References
Journey to Oracle Cloud Infrastructure with Oracle VM VirtualBox
https://blogs.oracle.com/virtualization/journey-to-oracle-cloud-infrastructure-with-oracle-vm-virtualbox
White Paper referenced in the blog post above
http://www.oracle.com/us/technologies/virtualization/oracle-vm-vb-oci-export-20190502-5480003.pdf
Image Import/Export
https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/imageimportexport.htm
Mikael Fransson
Consulting Cloud Solutions Architect
Consulting Cloud Solutions Architect - A-Team
Next Post