Time is more valuable than money.
I don’t actually remember but once I’ve read that way to reduce lead time is to eliminate ‘waste’, which is ‘waiting time’. Humans are a good example of the wave of frustration when they’re forced to wait for something. We should have written banner on our foreheads: “don’t come any closer”. Moreover, during the cooperation with customers we’ve noticed that the most common deployment strategy was to provision new nodes from top to bottom as the nodes were being launched. As you’re probably aware, running system updates, downloading packages and setting up configurations can take quite a long time. Everything looks nice unless some fuckups occur during provision the package updates or a new release of your app, which in scenarios with rolling-updates (canary or blue/green are different story) might prolong the whole process dramatically. In CHaos Team we’re hiring believers in automation and the ‘everything fails’ principle and therefore, we pay strong attention to simplicity and exclude ourselves from being part of each repeatable task.
Having all this information in our minds we considered a different approach for one of our latest customers.
Here are some thoughts…
Immutable = do it once and do it right.
According to 12factor app rules, immutable components are replaced for every deployment rather than being updated in-place. Following this method, you treat your instances as deployable, ephemeral artifacts. They should be standalone and self-executable among environments.
NOTE: Do not tag images. In accordance with 10th rule of 12factor app, ‘Dev/Prod parity’, make images consistent across your dev, qa, staging, prod or any other cloud environment. Believe me, you’re going to save a lot of time.
We all agreed that the application instances should have no dependency on external services which could be package repositories or configuration services. Literally, the AMI must be discrete and hermetic.
What kind of cake do you want?
Honestly, there is no “best” answer to the question regarding AMI baking but you can check which way works best for you:
- Would you like to bake the software, configuration and your code into the AMI (Netflix-style)?
- Would you like to bake only the software and configuration and then download the app code during the instance launch time?
- Would you like to use a clean OS AMI then do everything on boot with usage of Ansible, Chef, Salt or simple ‘user-data’ feature in AWS?
We’ve chosen method number one. The reason was trivial – do it once and do it right. We were obligated to decrease the time of each release to the minimum (haven’t reached the final goal yet – containers needed to go in under 5 minutes). Our AMI has to go through a candidate test and release process every two weeks. We’ve divided the whole mechanism into several steps:
- Foundation AMI – AMI prepared by AWS.
- Base AMI – the one we’re talking about in this article with all necessary OS packages and tools.
- Base App AMI – image with all necessary packages required for proper app launch. It can be Tomcat, Apache, Python or anything else.
- Last step – mount the volume with app files and update them in terms of new release.
Packer, Aminator or maybe… something else?
After making the decision how to ease our pain, we came across the next obstacle: which was the simplest tool to maintain (remember I was writing that we don’t necessarily want to participate in each process – it’s called ‘rule of autonomy’) and which gave us the opportunity to bake AMI preparation.
Generally, we decided to take 3 “bakers” under investigation:
Written in python and designed by Netflix Team after long period of using the Bakery (which was the predecessor of current Aminator). Bakery had some drawbacks as it had been customized for CentOS base AMI and allowed to experiment with other Linux distributions just out-of-the-box. This is why the Netflix Team had rewritten Bakery into Aminator. It supports Debian, Linux, RedHat and many other distributions and, what is more important, because of it is structured using a plugin architecture it can be extended for other cloud providers, operating systems, or packaging formats.
So, it works like this:
- Create a volume from the snapshot of the base AMI.
- Attach and mount the volume.
- Chroot into the mounted volume.
- Provision application onto mounted volume using rpm or deb package.
- Unmount the volume and create a snapshot.
- Register the snapshot as an AMI.
Allows you to use a framework such as Chef, Ansible or Puppet, to install and configure the software within your Packer-made images. An important aspect of Packer is that it creates identical images for multiple platforms, you can run production in AWS, staging/QA or even in a private cloud. After a machine image is built, that machine image can be quickly launched, and smoke tested to verify that things are working.
AWS Systems Manager
AWS Systems Manager is quite a different tool, combining several features, like built-in safety controls, allowing you to incrementally roll out new changes and automatically halt the roll-out when errors occur. Additionally, it presents a detailed view of system configurations, operating system patch levels, software installations, application configurations. In terms of security, AWS SSM maintains security and compliance by scanning your instances against your patch, configuration and custom policies. Last but not least, it supports hybrid environments containing Windows and Linux instances. Of course, we’re talking about AMI Baking but having some extra feature in one place especially in terms of patching or maintenance for me sounds at least encouraging.
We have experience with Packer and Aminator as well but… the winner was AWS Systems Manager.
Key benefits, which helped us to select the most suitable solution:
- Out of the box solution = don’t need to maintain it, easy to understand and configure.
- Free of charge.
- Feature of stocktaking.
- Integration with Ansible (we wanted to integrate SSM with tools we’d had an experience with).
- Patching feature in AWS SSM which our customer is willing to use.
- Supported and developed by AWS (our customer’s infrastructure is fully designed on AWS).
Next part coming soon! Stay tuned.