For me, the best (and most fun!) way to learn is to find a problem with a new set of tools you want to learn. I’ve documented my process of learning Ansible below, I hope it’s interesting to others!

Motivation

I built an application with Elixir and Phoenix and deployed it using Gigalixir. Gigalixir worked well, but after a couple of weeks the site shut down due to a lack of updates (I was on the free tier). Since this project is strictly for learning, I figured it would be fun to learn Ansible and save a couple bucks by signing up for a free VPS service.

I initially chose Vultr because they offered $50 of free credit towards a $3.50/month VPS, which should be more than enough for a year. This ended up now working out and I switched to AWS (detailed below).

I have some experience with Ansible-like technologies. Long ago, I used Puppet to configure and manage configuration on a single VPS which hosted a Spree Commerce application. It also had a Solr and MySQL server (this was before managed services were a thing and you had to host things yourself). It was interesting to set up, but a pain to manage. Making changes was always scary and created surprising and hard-to-debug errors. Puppet has a unique DSL and both the client and the server have to have Puppet installed for the configuration to work properly. It felt better than configuring Apache & Ubuntu by hand in the PHP days, but it wasn’t that much better.

I keep hearing about Ansible, let’s learn it and see how things have improved!

What I’m building

Here’s what I’d like to build:

• An Ansible configuration that will bootstrap a bare VPS with Dokku.
• Setup the Dokku application with an SSL certificate using Lets Encrypt.
• Elixir + Phoenix running using the community buildpacks.
• Ideally, I don’t want to do any manual configuration on the VPS. I want my entire production setup to be built via Ansible.

Learning Ansible

Here’s my "liveblog" of my thinking and learnings as I built my ansible config:

• Awhile back, I used Dokku to manage ~5 different microservices on a single (small) AWS VPS (via Lightsail). It worked amazingly well and was very stable. Before I move forward with Dokku, I took a look at the project on GitHub and it’s still (very) active, which is amazing! Let’s use that to manage our Elixir deployment.
• Ansible is a Python-based replacement for puppet/chef. Looks like it consumes yml files and configures servers via ssh.
• You only need Ansible installed on the "controller machine". This sounds like I can just install it on my laptop and avoid having to install anything on the target/remote server. This is a huge improvement over Chef/Puppet.
• MacOS install: sudo easy_install pip && sudo pip install ansible && ansible --version
  • A brew command I ran in the meantime ended up breaking my easy_install version. There was a library conflict. I ended up installing via brew instead and this fixed the issue.
• Setup a ansible.cfg in your project directory. You’ll also need an inventory file to specify where your servers are.
• You may need to add your SSH key to the VPS you spun up ssh-copy-id -i ~/.ssh/id_rsa.pub root@123.123.123.123.
  • Alternatively you can specify a SSH key in your inventory. Put ansible_ssh_private_key_file=~/.ssh/yourkey.pem after your IP address.
• I have ansible all -m ping working. Now to try to whip up a Ansible playbook that will install Dokku.
• Playbooks are a separate yml file that describes how you want to setup the server. Let’s call ours playbook.yml. We’ll run it using ansible-playbook playbook.yml.
• An Ansible "role" is a bundle of tasks. You can then layer on additional tasks on top of the role. I’m guessing you can also run multiple roles (confirmed this later on).
• My main goal is to use https://github.com/dokku/ansible-dokku to bootstrap a server. I cloned this to my local to more easily poke around at the code.
  • It look like the variable defaults are specified in defaults/main.yml
  • At least in this repo, each task contained in the ansible-dokku repo is a separate py file which defines an interface to Ansible using a AnsibleModule
• A "lookup plugin" can pull data from a URL, file, etc for a variable. This will be handy for setting up SSH keys, etc. Here’s an example: "{{lookup('file', '~/.ssh/id_rsa.pub')}}"
• Looks like roles don’t auto install when you run Ansible. "Galaxy" is the package registry for roles. You need to run a separate command to install packages.
• Best way to manage roles is to setup a requirements.yml and then run ansible-galaxy install -r requirements.yml. Docs are straightforward: https://galaxy.ansible.com/docs/using/installing.html
• Think of "modules" as a library. An abstraction around some common system task so you can call it via yml. A module can contain roles and tasks.
• You’ll see name everywhere in the yml files. This is optional and is only metadata used for logging & debugging.
• {{ }} are used for variable substitution. Does not need to be inside a string. You can call lookups from inside the brackets. I’m not a yml expert, but this seems like a custom layer on top of the core yml spec.
• become: true at the top of your playbook tells Ansible to use sudo for everything. Think of it as root: true.
• Each task has a default state. You can override the state by adding state=thestate to your task options. Each task defines a method to extract the current state from the system Ansible is operated on. Here’s an example. State is mostly extracted by reading configuration files or running a command to read the status of various systems (it’s not as magical as you might expect).
• Ansible has a vault feature which can encrypt an entire file or an inline variable. Rails introduced something similar where it would encrypt your production secrets into a local file so you could edit/manage them all in a single place.
  • You can also inline encrypt a string using ansible-vault encrypt_string the_thing_to_encrypt --name the_yml_key. You can then copy/paste the resulting string into a var.
  • Add vault_password_file = ./vault_password to your ansible.cfg and hide the file via .gitignore. This eliminates the need to enter the password each time you deploy via Ansible. You can then store the password in 1Password for safekeeping.
  • Encrypted variables need to be stored in vars. I wanted to use encrypted variables for secret definitions passed to dokku config, but I couldn’t use the encrypted string directly in the ENV var config. In vars define your secret app_database_url: !vault |..., then reference the secret in your ENV config DATABASE_URL: "{{ app_database_url }}".
• Use -vvvv as a CLI option to enable verbose logging. I ran into an issue where a subcommand was hanging waiting around a reply from stdin. However, verbose logging didn’t help me here. I’m guessing the subprocess called didn’t redirect output to the parent stdout/stderr so I couldn’t see any helpful debugging output.
  • This issue ended up being a bit interesting. ansible-dokku used the python3 subprocess module to run dokku commands on the machine. check_call was used, which doesn’t redirect stdin or stdout but subprocess data didn’t pipe it’s way to the ansible stdout or stdin even after I switched to using run. I’m guessing there’s a layer of abstraction in the ansible library which overrides all process pipes and prevents output from making its way to the user without a specific flag passed to AnsibleModule.
• Alright! I finally have my playbook running properly. Note that most ansible roles seem to work with Ubuntu, but not CentOS which was the default on the VPS provider I was testing out (Vultr).
• To modify a role that you are using, clone the repo, remove the repo from ~/.ansible/roles and then symlink the directory you removed from the directory. This will allow you to edit role code locally and test it on a live server (obviously, a horrible idea for a real product, fine for a side project).
• If you see a plain killed message in your deployment log, it’s probably because the server is running out of memory. Let’s add some swap to fix this! There’s got to be a role for adding swap memory to a server. There is: geerlingguy.swap. Added that to requirements.yml and added configuration options to my vars and boom, it works! Nice.
• I tried to add my own task dokku_lets_encrypt to the dokku-ansible role, but I ran into strange permission issues. Also, the development loop was pretty poor: make a change on my local and rerun the change on the server. Not fun. I ended up just giving up and running the letsencrypt setup manually on the server, so I failed in my goal to fully automate the server configuration.
• If you just want to run a single task use the --tags option https://stackoverflow.com/questions/23945201/how-to-run-only-one-task-in-ansible-playbook.

Here’s the template I based my config off of. Here’s the playbook configuration I ended up with, which demonstrates how to configure specific dokku module versions and uses encrypted strings:

---
- hosts: all
  become: true
  roles:
    - dokku_bot.ansible_dokku
    - geerlingguy.swap
  vars:
    swap_file_size_mb: '2048'

    dokku_version: 0.21.4

    herokuish_version: 0.5.14
    plugn_version: 0.5.0
    sshcommand_version: 0.11.0

    dokku_users:
      - name: mbianco
        username: mbianco
        ssh_key: "{{lookup('file', '~/.ssh/id_rsa.pub')}}"
    dokku_plugins:
      - name: clone
        url: https://github.com/crisward/dokku-clone.git
      - name: letsencrypt
        url: https://github.com/dokku/dokku-letsencrypt.git

  tasks:
    - name: create app
      dokku_app:
        # change this name in your template!
        app: &appname the_app
    - name: environment configuration
      dokku_config:
        app: *appname
        config:
          MIX_ENV: prod
          DATABASE_URL: "{{ app_database_url }}"
          SECRET_KEY_BASE: "{{ app_secret_key_base }}"
          DOKKU_LETSENCRYPT_EMAIL: hello@domain.com
          # specify port so `domains` can setup the port mapping properly
          PORT: "5000"
      vars:
        # encrypted variables need to be in `vars` and then pulled into `config` via
        app_database_url: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          abc123
        app_secret_key_base: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          abc123
    - name: add domain
      dokku_domains:
        app: *appname
        domains:
          - domain.com
          - www.domain.com
    - name: add domain
      dokku_domains:
        app: *appname
        global: True
        domains: []

    # this command doesn't work via ansible, but always works when run locally...
    # https://github.com/dokku/ansible-dokku/pull/49
    # - name: letsencrypt
    #   dokku_lets_encrypt:
    #     app: *appname

    # you'll need to `git push` once this is all setup

Here are key commands to manage your servers:

# can we reach our inventory?
ansible all -m ping

# encrypt secret keys in playbook
ansible-vault encrypt_string 'the_value' --name the_key

# install dependencies
ansible-galaxy install -r requirements.yml --force-with-deps --force

# run playbook
ansible-playbook playbook.yml

Deploying Elixir & Phoenix on Dokku

I’ve used dokku for projects in the past, and blogged about some of the edge cases I ran into. It took some fighting to get Elixir + Phoenix running on the Dokku side of things:

• I needed to create a Procfile with an elixir web worker definition web: elixir --sname server -S mix phx.server. Things aren’t as out of the box compared with rails. I think this is mostly because there’s two separate buildpacks required that aren’t officially maintained.
• Dokku plugins are just git repos. There’s no registry. Best place to find plugins is the dokku documentation. There’s an install command that pulls them from GitHub.
• The dokku-ansible role handles many common plugins, but you need to add them to your vars => dokku_plugins config to get them to autoinstall.
• dokku clone needs you to add the generated key to GitHub. ssh dokku@45.77.156.135 clone:key to get the public key, then add it as a deploy key in the GitHub repo. It may not be worth it to set this up. Easier to just git-push deploy manually.
• Dokku (apparently, just like Heroku) allows you set a .buildpacks file in the root directory. Just add a list of git repo URLs. Use a # to specify an exact git repo SHA to use.
• If you keep messing around with deploys you may exit the shell while there is a lock on the deploy. dokku apps:unlock to the rescue. This has never happened to me on Heroku, although I have always been much more careful with my production applications. Curious how Heroku handles this.
• If the build is failing, instead of continuing to run builds via git push you can find the failing build container and jump in.
  • docker ps -a | grep build. The second ID, which is either a short SHA or a string (dokku/yourapp:latest), is what you want to plug into docker run -ti 077581956a92 /bin/bash. From there you can experiment and tinker with the build.
  • Most buildpacks modify the PATH to point to executables like npm, node, etc that are pulled locally for bundling web assets. Helpful for debugging issues with buildpacks.
  • If you want to jump into a running container: docker exec -it CONTAINER_ID /bin/bash.
• herokish (the set of scripts which creates the heroku experience on dokku) builds things in the /tmp/build directory. https://github.com/gliderlabs/herokuish/blob/master/include/herokuish.bash and https://github.com/gliderlabs/herokuish#paths
• It looks like the cache dir is actually stored in /home/APPNAME/cache. This is linked to the build container during a git-push. I ran into issues with node_modules cache that required some manually debugging.
• dokku run does not enter into the same container that’s running your app. Use dokku enter app_name process_type the_command for that. If you are generating a sitemap, using dokku run won’t work because it doesn’t persist the files to the same container that is serving your static assets. Using S3 for static asset hosting would eliminate this problem.

Here’s what my buildpack config looks like:

# .buildpacks
https://github.com/HashNuke/heroku-buildpack-elixir.git#1251439227711cf28bbfbafc101f9c9ff7f9345a
https://github.com/gjaldon/heroku-buildpack-phoenix-static.git#b44e094c9da48483af5e221ff11f954a8b85479b

# pheonix_static_buildpack.config

# the pheonix buildpack does not specify recent versions of node & npm, which causes webpack issues
node_version=12.14.1
npm_version=6.14.4

# elixir_buildpack.config

elixir_version=1.10.4

# https://erlang.org/download/otp_versions_tree.html
erlang_version=22.3.4

Configuring AWS EC2 using Ansible

Vultr’s free credits ended up expiring after a couple of months (as opposed to a year). I wasn’t thrilled with the service and was curious to learn more about AWS by using additional services in the future, so I decided to move the server over to AWS:

• Looks like amazon linux isn’t supported on Ansible. Use the ubuntu image instead. https://github.com/geerlingguy/ansible-role-docker/issues/141
• "Amazon Linux" root user is ec2-user, ubuntu’s root is ubuntu. Amazon Linux is not compatible with many ansible packages, so use ubuntu.
• become: true (sudo mode) is required on Amazon.
• The local disk space of EC2 instances is tiny by default. You can expand the local disk space, which is a EBS instance, but navigating to the elastic block store and adjusting the instance. You’ll probably need to restart shutdown -h now
• I forgot about this: ports for http and https not exposed by default. If you run through the one-click EC2 wizard, only ssh will be exposed. Use the longer wizard to generate a "security group" exposing the proper ports.
• You’ll also want to setup an elastic IP. This is an IP that you can assign, and then reassign, to another EC2 instance.
• I’ve always been annoyed by AWS. It’s incredibly powerful, but hard to understand. You have to think of every little configuration option as a separate object with state that needs to be configured just right. Designing infra with code via https://github.com/aws/aws-cdk makes a ton of sense. I bet once you load the entire AWS data model in your head things make a lot more sense.

Learning Resources

Ansible

• https://medium.com/@mitesh_shamra/introduction-to-ansible-e5b56ee76b8c
• https://blog.morizyun.com/blog/dokku-isntall-vultr-pass-mini-heroku/index.html
• https://www.digitalocean.com/community/tutorials/configuration-management-101-writing-ansible-playbooks
• https://lebenplusplus.de/2017/06/09/how-secure-are-ansible-vaults/
• https://medium.com/@burakkarakan/what-exactly-is-docker-1dd62e1fde38
• https://opensource.com/article/16/12/devops-security-ansible-vault

Dokku

• https://www.petekeen.net/introduction-to-heroku-buildpacks
• https://github.com/jeffrafter/howto/blob/master/unformatted/elixir-phoenix-dokku.md
• https://www.petekeen.net/introduction-to-heroku-buildpacks
• https://dokku.github.io/general/automating-dokku-setup

Yaml

Interestingly, there’s not great canonical documentation for yaml. There’s a spec, but not docs on the official homepage.

• http://lzone.de/cheat-sheet/YAML
• https://github.com/darvid/trine/wiki/YAML-Primer