How to Run 10 nested VMs in one EC2 VM (Linux containers vs hypervisors)

Published by H2SO4 on

 

In a world of scalable distributed infrastructures and systems, there is a great need for cost effective virtual development environments to build and test these scalable applications.  This is a short post to debunk a common misconception about running virtual machines within Amazon’s EC2 environment.

Yes,it it possible to run multiple virtual machines within a single EC2 virtual machine (instance).…you just need to know the right technology to use and why.

In this post I’ll explain what OpenVZ is and the benefits of Linux containers vs hypervisors.  I’ll explain my one liner script installation process for the OpenVZ virtualization system on an EC2 server running CentOS.

Please note that this post is not a deep dive into the technologies mentioned – Use this as a primer to probe deeper into OpenVZ and it’s full set of features

 

What is OpenVZ?

OpenVZ is operating system-level virtualization based on a modified Linux kernel that allows a physical server to run multiple isolated instances known as containers, virtual private servers (VPS), or virtual environments (VE). The preferred term these days is container. Containers are sometimes compared to chroot or jail type environments but containers are really much better in terms of isolation, security, functionality, and resource management.

OpenVZ consists of a custom Linux kernel (available from the OpenVZ Project) and some user-level tools. OpenVZ is very portable, does not rely on VT support in the CPU, and as a result it is available for a number of CPU families including x86, x86-64, IA-64, PowerPC and SPARC.

OS-level virtualization is quite different from machine / hardware virtualization products such as VMware Server, Parallels Workstation, VirtualBox, QEMU, KVM, and Xen in that with OpenVZ you can only do Linux on Linux virtualization.

OpenVZ modifies the Linux kernel to add advanced containerization features which allow for isolated groups of processes under a parent init along with about twenty dynamic resource management parameters for controlling container resource usage. The OpenVZ Project maintains three stable kernel branches:

RHEL4 / CentOS4 2.6.9 based
RHEL5 / CentOS 5 2.6.18 based
Vanilla 2.6.18 based
There are a number of unstable branches based on newer versions of the Linux kernel that may eventually reach stable status.

For more info on checkpointing, live migration and backups check the Full set of OpenVZ features

 

Use cases

  • Development environment for running distributed and/or scalable applications and systems
  • Chef cookbook development testing using Vagrant (with the OpenVZ provider)
  • Isolating parts of an application for fine grained resource allocation and management
  • Dev/Test servers that run multiple disposable environments
  • Running multiple application servers on a single Linux server with better server density than most other VM types.

Performance

Operating system-level virtualization solutions have little to no overhead when running virtual machines/virtual environments because all processes run directly on the host hardware.  All system resource isolation is handled by the kernel and all virtual machines run in user space.

Popular Technology Options for OS virtualization

Installation Assumptions:

  • You are comfortable on the linux commandline
  • Your server has access to the internet
  • You are running CentOS 6.3, 6.4 or 6.5 (May work for others but these are the ones I’ve had great success with)
  • Fairly proficient with Linux networking
  • You know how to navigate the Amazon marketplace
  • You know how to create EC2 instances from AMIs in the Amazon marketplace
  • You are an Amazon Web Services registered user

Installing OpenVZ in EC2 (Verified with CentOS 6.3, 6.4, 6.5)

Accessing the VMs

After setting up the server there are two ways to create VMs and manage them.

  1. Web console (create/destroy/run commands/backup VMs and more)
  2. command line (ssh to VMs, configure VMs, check status and much much much more)
  3. port forwarding or reverse proxy (Access internal VMs externally)

:: Web console

This is the easiest way to get started with OpenVZ, however when you need to go into Ninja mode and do real work – you need the command line tools.

Web Panel: http://[EC2-PUBLIC-DNS-NAME]:3000 (e.g. http://ec2-174-129-128-25.compute-1.amazonaws.com:3000)

  • Username: admin
  • Password: admin

:: Command line tools

The following command line tools need to be run as root.  I’ll go over a few common commands.

vzlist -a

This command lists all VMs you have created including some properties of the VMs like name, ip and current state.

vzctl

This is a simple high-level command-line tool to manage a VE.

vzctl create VEID [–ostemplate <name>] [–config <name>]
This command will create a new Virtual Environment with numeric ID of VEID, which will be based on a specified OS template (a Linux distro) and having resourse management parameters taken from a specified config sample. Both –ostemplate and –config parameters are optional, defaults for them are given in a global configuration file.

vzctl start VEID
Starts a given VE. Start means creating a Virtual Environment context within the kernel, setting all the resource management parameters and running VE’s /sbin/init in that context.

vzctl stop VEID
Stops a given VE. A VE can also be stopped (or rebooted) by its owner using standard /sbin/halt or /sbin/reboot commands.

vzctl exec VEID <command>
Execute a command inside a gived VE. Say, to see a list of processes inside VE 102, use vzctl exec 102 ps ax.

vzctl enter VEID
Open a VE shell. This is useful if, say, sshd is dead for this VE and you want to troubleshoot the case.

vzctl set VEID –parameter <value> […] [–save]
Set a parameter for VE. There are a lot of different parameters. Say, to add an IP address to a VE, use vzctl set VEID –ipadd x.x.x.x –save. To set VE disk quota, use vzctl set VEID –diskspace soft:hard –save. To set VE kernel memory barrier and limit, use vzctl set VEID –kmemsize barrier:limit –save.

Other tools you can look up and learn more about:

vzcalc         vzcpucheck     vzdqdump       vzifup-post    vzmigrate      vzoversell     vzsplit        vzubc          
vzcfgvalidate  vzctl          vzdqload       vzlist         vznetaddbr     vzpid          vzstats        
vzcptcheck     vzdqcheck      vzeventd       vzmemcheck     vznetcfg       vzquota        vztmpl-dl

:: Port forwarding and reverse proxy

This can be done using IP forwarding or by setting up a reverse proxy web server on the OpenVZ host server to route traffic to internal VMs.  This setup is beyond the scope of this post – but it’s a fun exercise.

Why XEN, VMWare, VirtualBox don’t work

XEN, VMWare, VirtualBox, HyperV or any other hypervisor based virtualization technology will not work because EC2 instances are all XEN virtual machines.  For a server to work as a hypervisor, the CPU must have instruction sets that are capable of segregating and handling the load from the server and from each VMs it hosts.  In the case of Amazon, the Amazon EC2 instances that are spun up from XEN hypervisor systems that expose  CPU(s) and other system resources to operating system, however with limited CPU instruction sets that don’t include the ones necessary to run a nested hypervisor.

Though it is “technically” possible to make these instruction sets available to VM hosts, amazon would never allow this for security and sanity reasons because this gives the VM host much higher level of access to the underlying CPU and in turn the whole system.  Hackers will have a field day crippling the underlying hosts.  And to be honest, I wouldn’t even bother nesting hypervisor based VMs due to the performance impact of instructions traversing two or more hypervisors before getting executed by the CPU.  And if you disagree with me you are stupid and ugly 🙂

Disclaimer:  Using this method in a production environment requires PROPER planning and preparation to account for load balancing and disaster recovery to avoid downtime in the event of server failures.  Experiment and architect in a way that works for your environment.

Hope this was informative.  If you have any odd ways you have squeezed juice out of your linux servers, please share in the comments section below.

References:

  • http://en.wikipedia.org/wiki/OpenVZ
  • http://wiki.centos.org/HowTos/Virtualization/OpenVZ

H2SO4

Manager, Infrastructure Engineering in a Seattle Advertising company Interests: Configuration management (Chef, Ansible) Application development (Python, Ruby) Cloud platforms (AWS, Google compute engine) Operating system virtualization (LXC, OpenVZ, Docker)

5 Comments

Okezie E · June 2, 2015 at 3:23 AM

Me like

Sheldon Hearn · November 19, 2015 at 3:58 AM

“…will not work because EC2 instances are all KVM virtual machines.”

Are you sure? I think it’s Xen: https://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud

    Okezie E · November 19, 2015 at 10:02 AM

    Thanks Sheldon you are right. I’ll update this post.

    I even checked one of my EC2 machines to confirm.
    System Information
    Manufacturer: Xen
    Product Name: HVM domU
    Version: 4.2.amazon

Ajay · August 23, 2016 at 12:22 AM

can you do the same on Google Cloud Instance ?

Zachary · June 30, 2014 at 4:31 PM

.

good.

Comments are closed.