36-651/751: Virtualization and Containers

– Spring 2019, mini 3 (last updated February 14, 2019) all courses · refsmmat.com

As our software systems have gotten more complex, it’s become more difficult to manage all the pieces. Your package depends on certain versions of certain libraries, and breaks when some of those libraries are updated; it’s built with a certain compiler version and doesn’t work with others; you’ve set various environment variables and modified a bunch of configuration files; notebook servers have to be connected to the Internet on the right ports with the right keys; and so on and so on. You sometimes find that if you have to set up your software on a different computer, it can take days of work just to get everything working right again.

The tech industry – all the companies using The Cloud to run their complicated software – have hit similar problems. Their website uses PostgreSQL with an ElasticSearch index and user events go through RabbitMQ to Celery workers running tasks with PyPy. Orchestrating all this complexity is hard work. Assigning services (like PostgreSQL and ElasticSearch) to different physical computers is also work: if we suddenly need more ElasticSearch capacity, we need to install it on a bunch more servers quickly, then uninstall it when no longer needed and reuse those servers for something else.

Cloud server providers need to solve all these problems and more. They rent servers to customers. Back in the 2000s, this mostly meant renting physical servers to customers, but that was wasteful; why not let customers share servers and pay for how much RAM and CPU time they want allocated to them? But if they share servers, separate customers want to be able to install their own software on the servers without any other customer being able to see or touch it.

There are two related solutions to these problems: virtualization and containers. Virtualization allows us to set up independent virtual machines on one physical computer, with their own operating systems, file systems, and resources (like RAM and CPU time). We can install anything we want on a virtual machine and run many virtual machines on one server. Containers isolate individual software or services with the configuration, libraries, and files they need; multiple containers can run on one computer and are isolated from each other, though they are all managed by the same operating system kernel.

Virtualization and containers are related ideas, so let’s start with virtualization and see how containers differ later.

Virtualization

Suppose you’ve bought a great big expensive server with 32 CPU cores and 128GB of RAM. You’d like to sell your customers pieces of this: a customer could, say, buy access to 2 cores and 4GB of RAM, plus some hard drive space, and use this to install their own software and run their own stuff. But how do you split up a system like this?

Think back to our discussion of computer architecture. Your operating system must interact with the physical hardware in various ways, from getting input from the keyboard to coordinating with the Memory Management Unit to map virtual addresses to physical locations in memory. This all requires executing privileged CPU instructions.

If I try to run two different operating systems simultaneously on one computer, they’ll interfere with each other. One will try to set up the MMU one way and the other will set it up its own way; they’d fight over who gets keyboard input and who has control of the hard drive.

The solution is virtualization. Virtualization software, like VirtualBox, operates as a host. You can install VirtualBox on Windows, Mac, or Linux – the host operating system – and ask VirtualBox to start up a virtual machine. A virtual machine has its own operating system, the guest operating system, plus a file system and USB ports and everything, and VirtualBox intercepts its privileged instructions – say, pretending to provide a keyboard but only providing its input when the user has opened the VM window, or pretending to allow MMU operations but rewriting them to coordinate between the different operating systems. (Modern CPUs let the host ask the CPU to handle this, making the guest run faster.)

Virtual machine hosts often let you allocate specific amounts of memory and CPU to the guest operating systems, and can control whether they get Internet access at what speed. The host can even intercept their network traffic and act as a firewall.

Another kind of virtual machine software has no host operating system: Xen, for example, runs directly on the CPU and hosts all the guest systems, managing all the hardware. You don’t install Xen on Windows or Linux – it runs directly on the system.

Since the host controls all access to the hardware, it can access the file systems of the guests. Those file systems can be saved to a file on the host operating system called an image: one big file representing the entire file system contents of the guest. Images can be duplicated and used to start new guests on different computers.

Some virtualization use cases:

Running another operating system

You have a Mac but have video games that only run on Windows. You have Windows but want to try out Linux without partitioning your hard drive. You have Linux but want to try out a different version of Linux. With software like VirtualBox, you can run another operating system as a guest, install software on it, and often share files back and forth.

There are websites like OSBoxes with premade images of many operating systems ready for download. Just be careful that someone hasn’t sneaked malicious software into the images they distribute.
Deploying software
I could create a virtual machine and install software on it, then save the disk image. If I need to run many copies of that software on different servers – say, to make a Spark cluster – I could copy that disk image to multiple servers and run it on all of them.
Sharing hardware
A company might have many teams developing different software that has to run on their servers. Rather than coordinate all the configuration and setup so all the software works perfectly together, each team can create virtual machine images. The servers all run a system like Xen, and teams can say “I need 4 CPUs and 16GB of RAM for this thing” and provide a virtual machine image to run. Need more resources? The host can easily allocate more, or the virtual machine can be migrated to another host server with more resources.
Isolation
If I sell cloud computing services, I want to isolate customers from each other. No two customers should be able to access each other’s files and data. I can simply rent them virtual machines and each customer will have root access to their own servers, without being able to use that against anyone else.
Emulation

If I want to run classic GameBoy games on my laptop, I have a problem: those games were compiled for an ancient 8-bit CPU with an instruction set completely different from the instructions used by modern CPUs. Some virtual machine software, like QEMU, can translate instructions for one CPU to a different CPU, letting you run virtual machines with ancient software or software meant for a different system. Apple, for example, introduced Rosetta so old software written for PowerPC-based Macs could work on Intel-based Macs after the transition in 2006.

Q: What use cases can you imagine for virtual machines in statistics and data science?

Automating virtual machines

We can start to imagine a virtual machine workflow:

  1. Download a virtual machine image for the operating system you want – say, the latest version of Ubuntu.
  2. Open that virtual machine in VirtualBox and set up the software you want.
  3. Save the new virtual machine image.
  4. Distribute that image to people who need it, load it on servers, or otherwise do whatever you want to do with the VM.

But we are software nerds. Why should we do manually what we can do with code? And we’re also statisticians – why should we do manually what we can do reproducibly with code?

This is where tools like Vagrant come in. Vagrant’s intended users are software developers working on complicated projects involving lots of dependencies and packages and libraries and configuration. These developers don’t want to have to install all sorts of crud on their computer just to work on the software; they want that part to be automated.

Suppose you install vagrant on your computer and have your complicated software project in ~/project. You can create ~/project/Vagrantfile and fill it with something like

(Example adapted from here. If this looks suspiciously like Ruby, that’s because it’s Ruby.)

Then, at the shell, we can just run vagrant up and a new virtual machine will automatically be created and booted following our specification. Run vagrant ssh and we can SSH into it and do stuff; run vagrant halt and we turn it off.

If you need other packages or software configuration, add it to the Vagrantfile and build a brand-new VM with the right configuration. The Vagrantfile is, essentially, a reproducible description of the software environment you need.

Instead of distributing large virtual machine images to people, they can just run Vagrant and get the Vagrantfile to create a new VM with all the right software. Working on multiple projects? You can switch between their VMs, without worrying about the software conflicting or packages becoming incompatible.

Containers

Virtual machines are cool, but they’re sometimes overkill. If we have ten virtual machines running on a computer, that means ten operating system kernels plus the host system: ten kernels each trying to manage their own filesystems (which are then managed by the host), handle their own networking, run cronjobs and system services, and everything.

If I want to run one operating system within a different one, this is unavoidable. I need a VM.

But what if all I want to do is isolate different software packages from each other? To install complex software but not have it conflict with other software? To distribute my setup to different servers and have it run on them regardless of what else is running on them?

This is when containers are useful. Containers take advantage of isolation features provided by operating systems. Linux has several features:

Control groups (cgroups)
Ways to group processes together and limit their resource use (in CPU, memory, disk, and so on), so e.g. I can start a program but limit it to 1 CPU and 2GB of RAM.
Namespaces
Ways of splitting up the file system, process IDs, network interfaces, and other shared resources. A process belongs to a namespace, and can see all the resources attached to that namespace, but no others, and can’t tell that other resources exist. For example, a container can be given access to a certain part of the file system and will not know that other files exist, and may only be able to see processes within its own namespace.
Union file systems
A file system that is the sum of file systems. A container is given access to specific parts of the file system; any new files it creates are stored in a layer on top of that file system. Only the container sees that layer. This isolates the container from the rest of the system.

Container software combines all these features together into one easy-to-use system. A program running in a container is still managed by the host operating system kernel, and so it isn’t a full virtual machine, but it is isolated from the rest of the system.

Note one consequence of this: the container is managed by the host operating system, so you can’t run a Linux container on Windows, for example. You couldn’t make a container that runs on the department Linux servers and then use it unmodified on your Mac or Windows laptops. (Microsoft is working on making this possible on Windows.) You could use a virtual machine to run the containers, though that starts to become painful.

The most popular container software is Docker, which is open source.

Working with Docker

Rather like Vagrant’s Vagrantfile, Docker has the Dockerfile. Here’s an example Dockerfile from the official documentation:

(If this doesn’t look like Ruby, that’s because it’s not Ruby.)

Suppose we have a Python file named app.py in the same directory as the Dockerfile, as well as requirements.txt. (You should be using requirements.txt for your Python projects even if you don’t use Docker; check out the documentation for examples.)

Next we run

That builds a Docker image, an overlay (union) file system with the extra things run. All the Python packages were installed by the RUN step and all the files in the current directory were copied into the image’s /app directory.

If we then run

the container starts up and app.py runs. The output of app.py is printed to our console.

Suppose app.py starts up a webserver, like a Jupyter Notebook. That webserver is isolated inside the container, so we can’t access it – unless we use a command like

to tell Docker to make localhost:4000 be forwarded to port 80 on the container. Then we can access the notebook at http://localhost:4000.

Similarly, we can use a command like

to share ~/data/bigdata.txt with the Docker container as a volume. You can share individual files or whole directories.

Warning: In the Dockerfile, commands run with CMD run as root inside the container. But the container is completely isolated from the host, so that’s no problem, right? Wrong-o, buffalo chips breath! If it’s root inside the container, it has access to any files in the shared volumes, regardless of permissions; if you share too large a volume or there are other programs inside the volume, the container may be able to use them to do bad things. This matters if your container is exposed to the Internet or processes untrusted data from users. Best practice is to create a new unprivileged user inside the Dockerfile and use the USER option to run scripts in the container as that user.

Anyway, in the above Dockerfile we saw a FROM command declaring a “parent” image – you can build containers atop other containers. Docker Hub has a registry of public images you can use, including basic Linux distributions like Debian and images with specific software preinstalled. An instructive example is to view the Dockerfile of an image on the Hub, like this R image.

Containers for tech companies

For a tech company or anyone building a computing infrastructure, containers are great ways to deploy services. For each service you can specify exactly what files it needs access to, how it should be set up, and how it should be run. When you need to add new Hadoop nodes or set up a new webserver to replace one whose hard disk crashed, just run Docker to build the container and bam! the service is running.

Cloud computing services often build tools to make it easy to use containers in the cloud. Amazon has the Elastic Container Service that allows you to make your own registry of Docker images and then automatically start new servers running different containers. DigitalOcean lets you upload a Dockerfile and get a running server in a minute or two. Kubernetes is “container orchestration” software that you can install on your own server cluster to make it easy to automatically run containers, control how many are running, decide how much RAM and CPU they get, and monitor how all the containers are running.

One interesting service is Travis CI. CI stands for Continuous Integration, the idea that as you develop a software project, the tests should be running automatically at every commit, every pull request should be tested automatically, and developers should see reports on the results. Travis is a hosted CI service – free for open source projects and paid for commercial projects – that lets you provide a Docker image for the setup your software needs, and runs the Docker image to run all your tests and report results.

Containers for scientists

Scientists and statisticians can benefit from containers in several ways.

Complicated pipelines
We, unfortunately, often build complicated analysis pipelines with lots of different software. (My ADA stored its data in PostgreSQL with the PostGIS extension, used Python with many packages and parallel code, and relied on big data files I kept losing track of.) Just like for software engineers setting up web services, containers can make it easier for us to set up our analysis on new computers.
Reproducibility
In our Dockerfile we specify all the dependencies and packages needed by our analysis. We can also specify their versions, if we want, installing specific R and Python packages with the versions we used on our own computer. We can distribute the Dockerfile to other people who want to replicate our analysis so they don’t struggle to run our code.
Deployment
If you want to run your big analysis on AWS or set up an interactive Shiny app somewhere, containers make it easy to set up all the dependencies on a server.

Reproducibility in particular can be achieved with containers and several useful tools. Python’s pip, for example, lets you specify version numbers in requirements.txt; the pip freeze command does this automatically and will ensure that anyone else who runs your code will use the same version of everything.

R doesn’t have something similar, but Microsoft does host the CRAN Time Machine, and rocker archives Docker images with specific versions of R. You can put these together to get a reproducible R setup in a Dockerfile:

(Be sure to add a Docker volume if you want a way for the container to save output that remains on the host after the container is shut down.)

For Jupyter users, there are automated tools to build Docker images for your notebooks: Binder will scan your repository for requirements.txt and other files declaring dependencies, build a Docker image, and deploy it to a JupyterHub server so other people can use the notebook.