Securing Docker’s Remote API

One piece to docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it.

One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker!

Generate The SSL Certificates

We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx.

Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security.

We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs.

Setup Nginx

The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default.

One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else!

Hack It Up!

With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one.

For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot!

But wait there’s more… let’s build a container that can call the service to launch other containers!

For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting).

First, the node.js application that will connect and display information:

And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it!

That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. :)

I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

Parameterized Docker Containers

I’ve been hacking a lot on docker at Zapier lately and one of the things I found to be somewhat cumbersome with docker containers is that it seemed to be a little difficult to customize published containers without extending them and modifying files within them or some other mechanism. What I have come to discover is that you can publish containers that are customizable without modification from the end user by utilizing one of the most important concepts from 12 factor application development to Store Configuration in the Environment.

Let’s use a really good example of this, the docker-registry application used to host docker images internally. When docker first came out I whipped up a puppet manifest to configure this bad boy but then realized that the right way would be to run this as a container (which was published). Unfortunately the Dockerfile as it was didn’t fit my needs.

The gunicorn setup was hardcoded and to make matters more complicated the configuration defaulted strictly to the development based configuration that stored images in /tmp vs. the recommended production setting that stored images in S3 (where I wanted them).

The solution was easy, create a couple bash script that utilized environment variables that could be set when calling `docker run`.

First we generate the configuration file:

And wrap the gunicorn run call:

Finally the Dockerfile is modified to call these scripts with CMD, meaning that they are called when the container starts.

Since we use puppet-docker, the manifest for our dockerregistry server role simply sets these environment variables when it runs the container to configure it to our liking.

I’m really a big fan of this concept. This means people can publish docker containers that can be used as standalone application appliances with users tweaking to their liking via environment variables.

EDIT: Although I used puppet in this example to run docker, you don’t need to. You can easily do the following as well.

Handy Hub Alias

I’ve recently become a big fan of Hub and use a lot of the commands to interact with github from the comfort of my commandline. One of my personal favorites is pull-request as we use PRs often as a form of both code reviews and code promotion. Here’s a handy alias I have for the common task of issuing a PR for promotion.

Now I just need to figure out how to make it open the URL for the pull request that it dumps to the console. :)

Immutable Servers With Packer and Puppet

Lately I’ve been becoming more and more of a fan of is the concept of Immutable Servers while automating our infrastructure at Zapier. The concept is simple: never do server upgrades or changes on live servers, instead just build out new servers with applied updates and throw away the old ones. You basically get all the benefits of immutability in programming at the infrastructure level plus you never have to worry about configuration drift. And even better, I no longer have to have the fear that despite extensive tests someone might push a puppet manifest change that out the blue breaks our front web servers (sure we can rollback the changes and recover, but there is still a small potential outage to worry about).

Obviously you need some good tooling to make this happen. Some recent fooling around with packer has allowed me to put together a setup that I’ve been a little pleased with so far.

The Nodes

In our infrastructure project we have a nodes.yaml that defines node names and the AWS security groups they belong to. This is pretty straightforward and used for a variety of other tools (for example, vagrant).

The Rakefile

We use this nodes.yaml file with rake to produce packer templates to build out new AMIs. This keeps me from having to manage a ton of packer templates as they mostly have the same features.

This is used in conjunction with a simple erb template that simply injects the nodename into it.

This will generate a packer template for each node that will

  • Create an AMI in us-east-1
  • Uses an Ubuntu Server 13.04 AMI to start with
  • Sets the security group to packer in EC2. We create this and allow it access to puppetmaster’s security group. Otherwise packer will create a random temporary security group that won’t have access to any other groups (if you follow best practices at least)!
  • installs puppet
  • Runs puppet once to configure the system

We also never enable puppet agent (it defaults to not starting) so that it never polls for updates. We could also remove puppet from the server after it completes so the AMI doesn’t have it baked in.

The Script

Packer has a nice feature of enabling the user to specify shell commands and shell files to run. This is fine for bootstrapping but not so fine for doing the level of configuration management that puppet is more suited for. So our packer templates call a shell script that makes sure we don’t use the age old version of ruby linux distros love to default to and installs puppet. As part of the installation it also specifies the puppet master server name (if you’re using VPC instead of EC2 classic, you don’t need this as you can just assign the internal dns “puppet” to puppetmaster).

Building It

Now all we need to do to build out a new AMI for redis is run packer build packs/redis.json and boom! A server is created, configured, imaged and terminated. Now just set up a few jobs in jenkins to generate these based on certain triggers and you’re one step closer to automating your immutable infrastructure.

Cleaning Up

Of course, each AMI you generate is going to cost you a penny a day or some such. This might seem small, but once you have 100 revisions of each AMI it’s going to cost you! So as a final step I whipped up a simple fabfile script to cleanup the old images. This proved to be a simple task because we include a unix timestamp in the AMI name.

Set this up as a post-build job to the jenkins job that generates the AMI and you always ensure you have only the latest one. You could probably also tweak this to keep the last 5 AMIs around too for archiving purposes.

What’s Next?

I admit I’m still a little fresh with this concept. Ideally I’d be happy as hell to get our infrastructure to the point where each month (or week!) servers get recycled with fresh copies. Servers that are more transient like web servers or queue works this is easy. With data stores this can be a little more trickier as you need an effective strategy to boot up replicas of primary instances, promote replicas to primaries and retire the old primaries.

A final challenge is deciding what level of mutability is allowed. Deployments are obviously fine as they don’t tweak the server configuration but what about adding / removing users? Do we take an all or nothing approach or allow tiny details like SSH public keys to be updated without complete server rebuilds?

Puppet Provisioning in Docker

Lately I’ve been messing quite a bit with Docker and my mind is pretty blown about how awesome it is. Remember that feeling you had the first time you launched a vagrant instance and had a virtual machine running with your web application in just a few minutes? That’s the feeling I have again playing with docker… pure awesomeness.

Docker is essentially a wrapper around Linux Containers that makes them usable for mere mortals. With docker I can essentially take my application and all of it’s dependencies, bundle them up into a re-usable image and say “Here, run this!” This has a lot of implications for testing, for repeatability and for system provisioning. I must say it is quite impressive to be able to launch a vagrant box and then run multiple isolated instances of services like rabbitmq or redis inside of it.

It’s easy to try out… if you don’t already run linux you can clone the docker repository and try it out in a vagrant virtual instance. There is also a docker puppet module that you can use to install docker on your own machines if you use puppet. Walking through the examples is a good place to start, once you get a good feel for things you can check out the index for containers that others have shared (there’s some good ones out there, especially the ones published by heroku that emulate their stacks).

For fun, I created an image of a rabbitMQ server. After some trial and error I created the following Dockerfile to build a new image from.

Now to generate a new image I just run “docker build < Dockerfile" and wait. Once it is done I am given an image id for the generated image which I can use to tag the image (for those of you following along, "docker tag jamescarr/rabbitmq). Now to run rabbitmq in it’s own isolated environment I can just run “docker run jamescarr/rabbitmq” and see the magic happen. The final piece of the puzzle is you’ll need to run some docker commands to find out what port the ports 5672 and 15672 are exposed as on the host machine, once you have that figured out you should be able to navigate to http://localhost:(WHATEVER PORT 15672 IS) and be greeted by the rabbitmq management console. The next thing I want to figure out along these lines? Run multiple docker instances to emulate a rabbitmq cluster. ;)

Key Based Substitution in Python

Once in awhile I like to write a nice simple post about a simple language feature. Today is one of those days. :)

In languages like ruby and coffeescript, I am used to doing something like this:

In python I was slightly dismayed that this kind of variable embedding within strings isn’t quite supported. Like I said, not quite. You can still get away with it using keyword arguments to str’s format function.

Nothing earth shattering but I’m sure this will be useful for those getting started! Cheers!

How to Manage Puppet Modules Better

One of the things I’ve learned over the past few months is that many git submodules in a project can be a little bit of a nightmare to maintain. I’ve found that removing a git submodule can be a REAL chore. But it comes down to common sense: you typically don’t bundle jars with a java project, gems with a ruby project or pip dependencies with a python project, so why distribute your dependencies with a puppet project?

I’ve found the ruby gem librarian-puppet to be a good solution here as you can simply define a Puppetfile that describes the modules that you depend on and have it install them for you. It’s nice as it can pull modules both from pupeptforge (or an internal forge you run yourself) and from git.

Unfortunately I have found it is not uncommon to create your own modules for a project that may not feel like they warrant their own stand alone module (for example, I typically create a module representing each physical server configuration). The r10k project tries to remedy this, supporting librarian-puppet’s Puppetfile format while also kind of putting the training wheels on to develop using the philosophy of roles and profiles. This is a really good approach if (like me) you already have a module directory that consists of both git submodules of third party modules and some one off modules specific to your infrastructure. :)

So how do you manage your third party modules for your configuration management tool of choice? I’d be interested to know the strategies others use not just for Puppet but also Chef, Salt, Ansiable, or perhaps another tool I am unaware of.

Edit: Someone rightfully pointed out (and also mentioned in the stack overflow post I linked which wasn’t there before) is that as of git 1.8.3 you can now use git submodule deinit to remove a submodule. This doesn’t change the fact however that you should stop linking 3rd party modules in your project this way though. ;)