Using Ansible to deploy new servers
I’ve turned my manual server setup playbook into an Ansible playbook.
Although it was in fact October 2021 when I migrated this server to Debian 11, it seems like it was less than a year ago. And now with Debian 12 just around the corner, I’m eagerly looking forward to upgrading again.
Looking forward? Absolutely. The process went smoothly last time. And I delight in having a server that’s running the latest software.
Besides the Debian Bookworm release, another reason looms over me. My hosting provider has shown signs of instability.
I’ve been with Linode since 2006 – 17 years! Linode has been a pleasure, with fast, reliable servers at a good price. I’ve only made about six support requests in those 17 years, and they’ve been responsive and a pleasure to deal with.
However, Linode was acquired by Akamai last year. Akamai seems to be following the standard playbook: 1. Big company acquires little company. 2. Big company commodifies little company, removing everything its customers loved about it. 3. Big company raises prices to squeeze revenue from its new acquisition. 4. Little company loses its customer base. 5. Big company shuts down little company.
I haven’t left Linode yet, but many others are. Akamai has already started raising prices. I feel it’s only prudent to be prepared for the worst.
For the last upgrade, I wrote everything down in a playbook, which is just a big text file. That worked fine, and I could keep that same method, but I decided to try and formalize things. That original text has gotten sketchy. I’ve done some minor upgrades to the server over the past year and a half – things like adding web stats, a mail agent, and a comments system. Each time I would try to document it in my playbook, but that mean crossing some things out and adding others in random places and putting in comments like “this no longer applies since I’m doing something else”.
I wanted to formalize my playbook. Instead of a plain text file of informal notes, I wanted an automated process that could build my server from scratch. After doing some research, I decided to use Anisble for configuration and deployment.
Ansible versus the alternatives
Ansible and Docker manage large server clusters, but differently. Ansible tasks are basically a thin wrappers around unix commands. These commands, when run on a target server, configure that server to its final state. Docker files focus less on the build process. The main idea when using docker is to compose Docker images together to create your final image.
Docker has about three times the user base of Ansible, and I think it’s also seen as “cooler”, but for my needs, it’s not ideal.
First of all, Docker obscures contents. This can be seen as a benefit by some. If you use the Debian image as a base for your container, you don’t have to think at all about what’s in it. But then, your final image is also just that, an image. The build process matters less than the result.
I wanted to retain as much of the process as possible. I like managing and configuring my server. I was afraid that if I containerized everything, I would start to forget how I had built it in the first place. Ansible gives me a nice blend between a text narrative of how I’m building everything and a set of wrapper scripts which do what I want.
Docker also seems to be turning evil. Dealing with Docker right now feels like dealing with the devil. J.B. Crawfold on Computers Are Bad has written an excellent summary of docker in general, and I agree with much of what he’s written there. Docker’s underlying technology is great, but there isn’t really much there, as it’s mostly a wrapper around Linux cgroups. Docker’s success comes more from social and commercial forces than from the superiority of their technology.
Lastly, Docker lacks Mac support. I use a MacBook as my main work driver and I need a solution that works well there. While you certainly can use Docker on a Mac, it feels like a second-class citizen.
Reviewing Ansible
Building an Ansible playbook was an experience, taking longer than expected for several reasons. First, converting my notes to a structured format was non-trivial. As noted, I tweaked my setup for two years, each tweak adding or changing lines in my playbook notes. All these changes had to be consolidated when I turned it into an Ansible playbook. This is a good thing! Servers tend to accumulate cruft over the years, which is why I like to rebuild everything whenever Debian does a new release. Similarly, documentation accumulates cruft in the form of patch notes, and a clean rebuild keeps everything organized and clear.
Having spent some time with Ansible, I’ve come across several points that make it less than ideal.
YAML
This is a big one. Anisble uses YAML as its file format. I imagine when Ansible was first being developed, YAML looked promising because it’s fast and easy and can handle anything you throw at it. It’s a good way to prototype something out. But as a final file format for a tool like Ansible, it’s just too awkward to use. YAML is designed to document static, structured data. An Ansible playbook is a list of commands that you execute to bring a server into a desired state. These are not the same thing. It’s possible to shoehorn a list of Linux commands into a YAML document, but it’s painful and awkward and error-prone.
When you’re done, you end up with a list of dictionaries with list-type entities, and none of it is typed. In some places you need variables, and in others you need attributes, and it’s confusing to know what goes where. YAML structures its children with indentation, so you have to carefully indent all your lists otherwise something that should be a parameter to a child element ends up being assigned to its parent.
On top of all that, Ansible uses a templating framework called Jinja, where you have to surround elements to be expanded with curly braces. Of course, YAML doesn’t like this, and so you have to enclose that in double quotes. Except of course, if you’re in a magic context where Ansible simply assumes you’re writing templated code and then you don’t need any of that.
What Ansible should have done is create their own Domain-Specific Language. This is a perfect use case for it. Then you can create your own structures where everything is neatly labelled, and there’s no confusion as to what goes where. When I studied computer science in college one of the first projects assigned was to create a parser. I’m afraid kids these days don’t have any experience with that. They just throw everything into YAML and let some third-party package handle everything. While there’s something to be said for not reinventing the wheel, sometimes you need to write your own code too.
Dynamic Host Inventory
Ansible doesn’t quite fit my particular use-case. What Ansible really wants is a big “inventory” list of hosts, and then it wants to run its playbook on every host on the inventory, making them all the same. What I’m trying to do is create one specific host, and simply document that procedure. I don’t even know what the host IP will be before I run the script. There’s an Ansible package for Linode that lets me create hosts on the fly. So I put everything into the playbook:
- Spin up a new Linode
- Install Debian and all the packages I need
- Snapshot my server config and databases
- Copy everything over to the new host
It’s possible to add new hosts to the inventory list, but again it’s kind of a hack and you have to do a bunch of workarounds to get it to work. So I create my new Linode, get its IP address, and tell Ansible to run everything there.
Secrets
Setting up a server of course requires passwords, which means you have to manage secrets. The built-in way to do this with Ansible is with something called Vault files. You encrypt all your secrets, and then when you run your playbook you enter a password to decrypt it on the fly. I went a different route. I use 1Password as a password manager on my MacBook, and since all my secrets are in there already I didn’t want to copy them to a another file, encrypted or no. Fortunately there’s an Ansible plugin for 1Password and it works pretty well. My playbook looks like this:
- name: Set PostgreSQL account password
ansible.builtin.user:
name: postgres
state: present
password: "{{ lookup('onepassword', 'Reiterate postgres account', field='password') | password_hash }}"
When I run the playbook, the first time it does a lookup it asks for my biometric authentication. I touch my thumb to the Touch ID, and the script proceeds from there. Very nice.
Recommendations
Would I recommend Ansible? Sure. For me, the downsides weren’t so bad that I wouldn’t recommend it. I realize it’s not perfectly suited to my own particular use case. The file syntax is awkward, but not completely unusable.
In the end, I have a script that deploys a fresh new server and configures it exactly how I need it. When it spins up, it’s running an up-to-date copy of my web server and backend. I’m ready for Debian 12 when it comes.
Rat Troupe
