Using Ansible to Deploy Elixir Applications on Dokku

For me, the best (and most fun!) way to learn is to find a problem with a new set of tools you want to learn. I’ve documented my process of learning Ansible below, I hope it’s interesting to others!


I built an application with Elixir and Phoenix and deployed it using Gigalixir. Gigalixir worked well, but after a couple of weeks the site shut down due to a lack of updates (I was on the free tier). Since this project is strictly for learning, I figured it would be fun to learn Ansible and save a couple bucks by signing up for a free VPS service.

I initially chose Vultr because they offered $50 of free credit towards a $3.50/month VPS, which should be more than enough for a year. This ended up now working out and I switched to AWS (detailed below).

I have some experience with Ansible-like technologies. Long ago, I used Puppet to configure and manage configuration on a single VPS which hosted a Spree Commerce application. It also had a Solr and MySQL server (this was before managed services were a thing and you had to host things yourself). It was interesting to set up, but a pain to manage. Making changes was always scary and created surprising and hard-to-debug errors. Puppet has a unique DSL and both the client and the server have to have Puppet installed for the configuration to work properly. It felt better than configuring Apache & Ubuntu by hand in the PHP days, but it wasn’t that much better.

I keep hearing about Ansible, let’s learn it and see how things have improved!

What I’m building

Here’s what I’d like to build:

An Ansible configuration that will bootstrap a bare VPS with Dokku. Setup the Dokku application with an SSL certificate using Lets Encrypt. Elixir + Phoenix running using the community buildpacks. Ideally, I don’t want to do any manual configuration on the VPS. I want my entire production setup to be built via Ansible. Learning Ansible

Here’s my "liveblog" of my thinking and learnings as I built my ansible config:

Awhile back, I used Dokku to manage ~5 different microservices on a single (small) AWS VPS (via Lightsail). It worked amazingly well and was very stable. Before I move forward with Dokku, I took a look at the project on GitHub and it’s still (very) active, which is amazing! Let’s use that to manage our Elixir deployment. Ansible is a Python-based replacement for puppet/chef. Looks like it consumes yml files and configures servers via ssh. You only need Ansible installed on the "controller machine". This sounds like I can just install it on my laptop and avoid having to install anything on the target/remote server. This is a huge improvement over Chef/Puppet. MacOS install: sudo easy_install pip && sudo pip install ansible && ansible --version A brew command I ran in the meantime ended up breaking my easy_install version. There was a library conflict. I ended up installing via brew instead and this fixed the issue. Setup a ansible.cfg in your project directory. You’ll also need an inventory file to specify where your servers are. You may need to add your SSH key to the VPS you spun up ssh-copy-id -i ~/.ssh/ root@ Alternatively you can specify a SSH key in your inventory. Put ansible_ssh_private_key_file=~/.ssh/yourkey.pem after your IP address. I have ansible all -m ping working. Now to try to whip up a Ansible playbook that will install Dokku. Playbooks are a separate yml file that describes how you want to setup the server. Let’s call ours playbook.yml. We’ll run it using ansible-playbook playbook.yml. An Ansible "role" is a bundle of tasks. You can then layer on additional tasks on top of the role. I’m guessing you can also run multiple roles (confirmed this later on). My main goal is to use to bootstrap a server. I cloned this to my local to more easily poke around at the code. It look like the variable defaults are specified in defaults/main.yml At least in this repo, each task contained in the ansible-dokku repo is a separate py file which defines an interface to Ansible using a AnsibleModule A "lookup plugin" can pull data from a URL, file, etc for a variable. This will be handy for setting up SSH keys, etc. Here’s an example: "{{lookup('file', '~/.ssh/')}}" Looks like roles don’t auto install when you run Ansible. "Galaxy" is the package registry for roles. You need to run a separate command to install packages. Best way to manage roles is to setup a requirements.yml and then run ansible-galaxy install -r requirements.yml. Docs are straightforward: Think of "modules" as a library. An abstraction around some common system task so you can call it via yml. A module can contain roles and tasks. You’ll see name everywhere in the yml files. This is optional and is only metadata used for logging & debugging. {{ }} are used for variable substitution. Does not need to be inside a string. You can call lookups from inside the brackets. I’m not a yml expert, but this seems like a custom layer on top of the core yml spec. become: true at the top of your playbook tells Ansible to use sudo for everything. Think of it as root: true. Each task has a default state. You can override the state by adding state=thestate to your task options. Each task defines a method to extract the current state from the system Ansible is operated on. Here’s an example. State is mostly extracted by reading configuration files or running a command to read the status of various systems (it’s not as magical as you might expect). Ansible has a vault feature which can encrypt an entire file or an inline variable. Rails introduced something similar where it would encrypt your production secrets into a local file so you could edit/manage them all in a single place. You can also inline encrypt a string using ansible-vault encrypt_string the_thing_to_encrypt --name the_yml_key. You can then copy/paste the resulting string into a var. Add vault_password_file = ./vault_password to your ansible.cfg and hide the file via .gitignore. This eliminates the need to enter the password each time you deploy via Ansible. You can then store the password in 1Password for safekeeping. Encrypted variables need to be stored in vars. I wanted to use encrypted variables for secret definitions passed to dokku config, but I couldn’t use the encrypted string directly in the ENV var config. In vars define your secret app_database_url: !vault |..., then reference the secret in your ENV config DATABASE_URL: "{{ app_database_url }}". Use -vvvv as a CLI option to enable verbose logging. I ran into an issue where a subcommand was hanging waiting around a reply from stdin. However, verbose logging didn’t help me here. I’m guessing the subprocess called didn’t redirect output to the parent stdout/stderr so I couldn’t see any helpful debugging output. This issue ended up being a bit interesting. ansible-dokku used the python3 subprocess module to run dokku commands on the machine. check_call was used, which doesn’t redirect stdin or stdout but subprocess data didn’t pipe it’s way to the ansible stdout or stdin even after I switched to using run. I’m guessing there’s a layer of abstraction in the ansible library which overrides all process pipes and prevents output from making its way to the user without a specific flag passed to AnsibleModule. Alright! I finally have my playbook running properly. Note that most ansible roles seem to work with Ubuntu, but not CentOS which was the default on the VPS provider I was testing out (Vultr). To modify a role that you are using, clone the repo, remove the repo from ~/.ansible/roles and then symlink the directory you removed from the directory. This will allow you to edit role code locally and test it on a live server (obviously, a horrible idea for a real product, fine for a side project). If you see a plain killed message in your deployment log, it’s probably because the server is running out of memory. Let’s add some swap to fix this! There’s got to be a role for adding swap memory to a server. There is: geerlingguy.swap. Added that to requirements.yml and added configuration options to my vars and boom, it works! Nice. I tried to add my own task dokku_lets_encrypt to the dokku-ansible role, but I ran into strange permission issues. Also, the development loop was pretty poor: make a change on my local and rerun the change on the server. Not fun. I ended up just giving up and running the letsencrypt setup manually on the server, so I failed in my goal to fully automate the server configuration. If you just want to run a single task use the --tags option

Here’s the template I based my config off of. Here’s the playbook configuration I ended up with, which demonstrates how to configure specific dokku module versions and uses encrypted strings:

--- - hosts: all become: true roles: - dokku_bot.ansible_dokku - geerlingguy.swap vars: swap_file_size_mb: '2048' dokku_version: 0.21.4 herokuish_version: 0.5.14 plugn_version: 0.5.0 sshcommand_version: 0.11.0 dokku_users: - name: mbianco username: mbianco ssh_key: "{{lookup('file', '~/.ssh/')}}" dokku_plugins: - name: clone url: - name: letsencrypt url: tasks: - name: create app dokku_app: # change this name in your template! app: &appname the_app - name: environment configuration dokku_config: app: *appname config: MIX_ENV: prod DATABASE_URL: "{{ app_database_url }}" SECRET_KEY_BASE: "{{ app_secret_key_base }}" DOKKU_LETSENCRYPT_EMAIL: # specify port so `domains` can setup the port mapping properly PORT: "5000" vars: # encrypted variables need to be in `vars` and then pulled into `config` via app_database_url: !vault | $ANSIBLE_VAULT;1.1;AES256 abc123 app_secret_key_base: !vault | $ANSIBLE_VAULT;1.1;AES256 abc123 - name: add domain dokku_domains: app: *appname domains: - - - name: add domain dokku_domains: app: *appname global: True domains: [] # this command doesn't work via ansible, but always works when run locally... # # - name: letsencrypt # dokku_lets_encrypt: # app: *appname # you'll need to `git push` once this is all setup

Here are key commands to manage your servers:

# can we reach our inventory? ansible all -m ping # encrypt secret keys in playbook ansible-vault encrypt_string 'the_value' --name the_key # install dependencies ansible-galaxy install -r requirements.yml --force-with-deps --force # run playbook ansible-playbook playbook.yml Deploying Elixir & Phoenix on Dokku

I’ve used dokku for projects in the past, and blogged about some of the edge cases I ran into. It took some fighting to get Elixir + Phoenix running on the Dokku side of things:

I needed to create a Procfile with an elixir web worker definition web: elixir --sname server -S mix phx.server. Things aren’t as out of the box compared with rails. I think this is mostly because there’s two separate buildpacks required that aren’t officially maintained. Dokku plugins are just git repos. There’s no registry. Best place to find plugins is the dokku documentation. There’s an install command that pulls them from GitHub. The dokku-ansible role handles many common plugins, but you need to add them to your vars => dokku_plugins config to get them to autoinstall. dokku clone needs you to add the generated key to GitHub. ssh dokku@ clone:key to get the public key, then add it as a deploy key in the GitHub repo. It may not be worth it to set this up. Easier to just git-push deploy manually. Dokku (apparently, just like Heroku) allows you set a .buildpacks file in the root directory. Just add a list of git repo URLs. Use a # to specify an exact git repo SHA to use. If you keep messing around with deploys you may exit the shell while there is a lock on the deploy. dokku apps:unlock to the rescue. This has never happened to me on Heroku, although I have always been much more careful with my production applications. Curious how Heroku handles this. If the build is failing, instead of continuing to run builds via git push you can find the failing build container and jump in. docker ps -a | grep build. The second ID, which is either a short SHA or a string (dokku/yourapp:latest), is what you want to plug into docker run -ti 077581956a92 /bin/bash. From there you can experiment and tinker with the build. Most buildpacks modify the PATH to point to executables like npm, node, etc that are pulled locally for bundling web assets. Helpful for debugging issues with buildpacks. If you want to jump into a running container: docker exec -it CONTAINER_ID /bin/bash. herokish (the set of scripts which creates the heroku experience on dokku) builds things in the /tmp/build directory. and It looks like the cache dir is actually stored in /home/APPNAME/cache. This is linked to the build container during a git-push. I ran into issues with node_modules cache that required some manually debugging. dokku run does not enter into the same container that’s running your app. Use dokku enter app_name process_type the_command for that. If you are generating a sitemap, using dokku run won’t work because it doesn’t persist the files to the same container that is serving your static assets. Using S3 for static asset hosting would eliminate this problem.

Here’s what my buildpack config looks like:

# .buildpacks # pheonix_static_buildpack.config # the pheonix buildpack does not specify recent versions of node & npm, which causes webpack issues node_version=12.14.1 npm_version=6.14.4 # elixir_buildpack.config elixir_version=1.10.4 # erlang_version=22.3.4 Configuring AWS EC2 using Ansible

Vultr’s free credits ended up expiring after a couple of months (as opposed to a year). I wasn’t thrilled with the service and was curious to learn more about AWS by using additional services in the future, so I decided to move the server over to AWS:

Looks like amazon linux isn’t supported on Ansible. Use the ubuntu image instead. "Amazon Linux" root user is ec2-user, ubuntu’s root is ubuntu. Amazon Linux is not compatible with many ansible packages, so use ubuntu. become: true (sudo mode) is required on Amazon. The local disk space of EC2 instances is tiny by default. You can expand the local disk space, which is a EBS instance, but navigating to the elastic block store and adjusting the instance. You’ll probably need to restart shutdown -h now I forgot about this: ports for http and https not exposed by default. If you run through the one-click EC2 wizard, only ssh will be exposed. Use the longer wizard to generate a "security group" exposing the proper ports. You’ll also want to setup an elastic IP. This is an IP that you can assign, and then reassign, to another EC2 instance. I’ve always been annoyed by AWS. It’s incredibly powerful, but hard to understand. You have to think of every little configuration option as a separate object with state that needs to be configured just right. Designing infra with code via makes a ton of sense. I bet once you load the entire AWS data model in your head things make a lot more sense. Learning Resources Ansible Dokku Yaml

Interestingly, there’s not great canonical documentation for yaml. There’s a spec, but not docs on the official homepage.

Continue Reading

Running Tests Against Multiple Ruby Versions Using CircleCI

I’ve been a long-term maintainer of the NetSuite ruby gem. Part of maintaining any library is automated tests against multiple versions of various dependencies. Most of the time, this is limited to the language version, but can include other dependencies as well.

Recently my build config stopped working as CircleCI upgraded to V2 of their infrastructure. I found it challenging to find an example CircleCI V2 config with the following characteristics:

No Gemfile.lock and therefore no caching of gems. When you are testing across ruby versions you can’t use a single Gemfile.lock. No rails, no databases, just plain ruby

Here’s an heavily documented CircleCI config that tests multiple ruby versions:

version: 2.1 orbs: # orbs are basically bundles of pre-written build scripts that work for common cases # ruby: circleci/ruby@1.1 jobs: # skipping build step because Gemfile.lock is not included in the source # this makes the bundler caching step a noop test: parameters: ruby-version: type: string docker: - image: cimg/ruby:<< parameters.ruby-version >> steps: - checkout - ruby/install-deps: bundler-version: '1.17.2' with-cache: false - ruby/rspec-test # strangely, there seems to be very little documentation about exactly how martix builds work. # By defining a param inside your job definition, Circle CI will automatically spawn a job for # unique param value passed via `matrix`. Neat! # workflows: build_and_test: jobs: - test: matrix: parameters: # # only supports the last three ruby versions ruby-version: ["2.5", "2.6", "2.7"]

Continue Reading

Time Machine Backups with a Raspberry Pi and External Drives

As I was reviewing my backup strategy, I realized I hadn’t completed a Time Machine backup on my machines in a long time. Plugging in the drive was just enough friction to forget doing it completely.

The Airport Express has a USB port to plug hard drives, printers, etc into. These devices would be magically broadcasted to the network. It was awesome, and then Apple killed the device. The Eero I upgraded to is great, but the USB port is useless.

But, there’s silver lining! I’ve been looking for a good excuse to buy a Raspberry Pi and mounting external hard drives on the network fit the bill! $35 for a tiny computer more powerful that anything I had growing up and more powerful than a $5 DigitalOcean or AWS VPS. What’s not to like?

Purchasing the Hardware Raspberry PI 4 2GB. $45. I didn’t end up using the USB-C => micro USB connector and the eBook was useless. HDMI connector was helpful. Case, fan, and power supply. $12. The 5V 3A power supply required isn’t common, so you’ll most likely need to buy one. Having a case is really nice. You’ll also need a micro SD card, but I had an extra 16GB card.

So not exactly the $35 sticker price that is advertised, but still cheap.

Setting up Raspberry Pi for Remote VNC & SSH Access

My goal was to run the Pi headless. Here’s how I got the Pi setup for VNC access over the network that works across reboots:

Download Unzip and put it on the SD card. Make sure the SD card is FAT formatted. Make sure you don’t put the unzipped folder on the root directory, but rather the contents of the unzipped folder. Startup the Pi. You’ll want a monitor connected via HDMI and a (wired) keyboard to complete the setup process. You don’t need a mouse. Setup VNC & ssh. Open up a terminal and run sudo raspi-config. Navigate to "Interfacing Options", enable VNC & SSH. Set boot options to desktop for easy VNC usage. Here’s more info You also want to set the default resolution via raspi-config or VNC won’t work when you reboot without a monitor. On your mac brew cask install vnc-viewer. Username: pi, password is what you used during the on-screen setup. You should be able to manage the device right from your mac.

At this point, you’ll have access to the PI without a keyboard and mouse. Let’s setup the Pi to serve up the hard drives over the network!

Setting Up External Hard Drives as Network Attached Storage (NAS)

Here’s a couple articles I found that were helpful:

This one is the most recent and complete:

None of the articles seemed to completely match by setup. Here’s what I wanted to setup:

I have two external drives. I wanted to use one as a networked time machine drive and the other as general storage. One of the drives had a power supply and the other did not.

Here’s how I ended up serving my two hard drives on the network:

Install the packages we’ll need: sudo apt-get --assume-yes install netatalk A quick note on HFS+ formatted drives: I ended up corrupting the drives on HFS+ mode, most likely because I aggressively turned the power on/off without unmounting the drives. I’d recommend against using HFS+ formatted drives and instead format to the linux-native Ext4. I’ve documented this below. Run netatalk -v to make sure you have a recent version and get the location to the config file. Latest version is indicated here: sudo nano /etc/netatalk/afp.conf to edit the config file. This is the location for version 3.1. Pull the location of the config from the output of the previous netatalk command we ran if you run into issues finding this file. The instructions inlined in the config file are pretty straightforward. In Global add mimic model = TimeCapsule6,106. This broadcasts the time machine drive to look like a ‘real’ time machine device. Here’s a list of other options you can use. Neat! You’ll also want to edit sudo nano /etc/nsswitch.conf and append mdns4 mdns to the line with dns. This broadcasts the drive availability on the network. Get a list of all services running on your Pi with sudo service --status-all sudo shutdown -r now or sudo reboot to restart the system from the command line Change your AFP config? sudo service netatalk restart I did find the MacOS finder was pretty glitchy when I restarted services on the Pi. I ended up force quitting the finder a couple times to pick up new drive configurations. sudo chown -R pi:pi /media/pi/MikeExternalStorage to fix strange permission issues when accessing the drive. This may have had to do with attempting to use HFS+ formatting at first, so you most likely don’t need to do this.

Here’s the final /etc/nsswitch.conf:

# /etc/nsswitch.conf # # Example configuration of GNU Name Service Switch functionality. # If you have the `glibc-doc-reference' and `info' packages installed, try: # `info libc "Name Service Switch"' for information about this file. passwd: files group: files shadow: files gshadow: files hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 mdns networks: files protocols: db files services: db files ethers: db files rpc: db files netgroup: nis

And /etc/netatalk/afp.conf:

; ; Netatalk 3.x configuration file ; [Global] mimic model = TimeCapsule6,106 ; [Homes] ; basedir regex = /xxxx [ExternalStorage] path = /media/pi/ExternalStorage [TimeMachine] path = /media/pi/TimeMachine time machine = yes A Warning on Filesystem Types

I had everything working, and then I accidentally restarted the Pi and everything was mounted as a read-only drive. I ran mount and saw type hfsplus (ro... in the info line. RO = read only.

After some googling I found that this seemed to be caused by the HFS+ filesystem (i.e. macos’s default) that I attempted to support by installing hfsprogs hfsplus hfsutils. Don’t try to host a HFS drive via the Pi. The drivers are not great and I believe this is why I ran intro trouble here.

Here’s what I tried to fix the problem:

After some googling I found: sudo fsck.hfsplus -f /dev/sda1. This didn’t do anything for me. I tried This didn’t seem to work for me because I had partitions lsblk, blkid, sudo fdisk -l, ls -l /dev/disk/by-uuid/ are useful tools for inspecting what disks/devices are mounted. sudo cp fstab fstab.bak, edit fstab with nano, UUID=1-1-1-1 /media/pi/ExternalStorage hfsplus force,rw. No quotes around uuid. Tried sudo umount /dev/sda2 && sudo mount /dev/sda2. No luck. Determine how large a folder is: sudo du -sh Tried disabling sudo service netatalk stop && sudo service avahi-daemon stop and restarting the computer. No dice.

Uh oh. Not good.

I plugged the drive into my MacOS computer and told me the drive was corrupted. I tried to repair the disk and it gave me esoteric error. I think accidentally turning off the power may have corrupted the disk. HFS+ isn’t supported natively and seems to be prone to corruption issues if drives are not unmounted properly. Ext4 is the recommended file system format.

Some reference articles:

It’s surprising to me, that in 2020, file system formats still matter across operating systems. I feel like I’m transported back to the 90s. Why isn’t this a solved problem yet?

Here’s what I did to fix the issue:

Luckily, although I can’t write to my drives, I can read from them and was able duplicate the data other devices. I pulled the data off to various drives and computers using rsync. It was a huge pain: I needed to spread the data across a couple of different devices. Here’s the commands I used: sudo rsync -avh --progress pi@ ~/Desktop/MikeiTunes After the data was moved off I reformatted the drive: Unmount the drive: sudo umount /dev/sba2 Wipe the drive and format sudo mkfs.ext4 /dev/sba Create a mount point sudo mkdir /media/TimeMachine && sudo chmod 777 /media/TimeMachine Add the table entry: sudo nano /etc/fstab, then /dev/sda /media/TimeMachine auto defaults 0 2. fstab => "File System Table" Add a label to the drive sudo e2label /dev/sda TimeMachine Mount the drive sudo mount /dev/sda After this was done, I rsync’d the data back to the drive. In some cases I needed to use sudo on the Pi to avoid any permission issues. sudo rsync -avh --progress --rsync-path="sudo rsync" ~/Desktop/EmilyBackup pi@ I found it helpful to use screen to manage long running sessions on the PI: sudo apt install screen screen to start a new session screen ls to list all sessions. Attach to a session like screen -r THE_ID tmux is also great for this (and probably better) I also needed a way to unzip some folders. There’s not a built-in util that provides unzipping with a progress indicator. I found 7z which fits the bill: 7z x -o./.

Summary: don’t use HFS/mac formatted hard drives on linux!

A Note on Using Old Drives for Storage

I couldn’t get my 2TB drive to pass the SMART monitoring (details in a future blog post!) ‘long’ test. I did a bit of research and external drives tend to only last ~5 years. The 2TB drive was ~10 years old and had exhibiting some glitchy behavior. I ended up replacing it with a much smaller 2TB drive for $60.

This is all to say: it’s worth replacing drives every 5 years or so and ensure they are monitoring by SMART to catch any failures early on.

Fixing Raspberry Pi’s Emergency Mode

I connected the new drive to replace my old one, formatted it, and setup the fstab config just like the other drives. The UI started glitching out and I had two mounts setup for my old drive. I figured restarting the Pi would fix the issue, but that was a bad idea.

The Pi wouldn’t connect to the network and appeared dead. I tried unplugging the drives, but that didn’t help.

I ended up having to plug it back into a monitor and found the following message:

You are in emergency mode. After logging in, "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.

Here’s what I found:

If any of the devices in fstab cannot be found, it will hang the boot process and you’ll be kicked into emergency mode. This was surprising to me: looks like linux is not too forgiving with bad configuration. Here’s how to fix the issue: Plug your SD card into another computer, edit cmdline.txt on the root of the card, and add init=/bin/sh to the end of it. Looks like the Pi reads that txt file to determine how to boot. I believe this is a Pi-specific config file. Plug the SD card back into the pi, run mount -o remount,rw / when the prompt appears and comment out custom lines in /etc/fstab. Reboot the Pi and you’ll be back in action. I ran journalctl -xb but couldn’t find any errors specifically identifying the drive. /var/log/syslog is also a good place to look. sudo findmnt --verify --verbose is a way to verify your fstab config If you specify default,nofail in fstab it looks like you can avoid this problem. I’m not sure what the side effects of this approach is. I don’t understand why fstab definitions are necessary if the default drive config is working fine. All drives automount when connected. I ended up removing all fstab entries and using the autogenerated mount points at /media/pi.

Resources: Under Voltage & USB-Powered Devices

When I replaced my old hard drive, I grabbed a USB powered one off of Amazon. However, the Pi can only support powering a single external drive drive.

You can determine if this is happening by searching the syslogs:

cat /var/log/kern.log | grep -i 'voltage'

Some references:

The solution is buying a USB hub that is externally powered, like this one.

Spotlight Indexing on NAS

It’s not possible:

As an aside, I’ve also learned it’s not possible to exclude folders with a specific pattern (such as node_modules) or with a specific dot file within the folder (.metadata-no-index). You can only control what’s indexed via the control panel.

Wow, this seems like a lot of work? Was this even a good idea?

Yes, it was. Took way more time than I expected. Probably not a great idea! If you just want to get a networked time machine up and running quickly, I wouldn’t do this.

But…I learned a bunch, which was the fun part for me.

Why is Linux still hard to use?

Way back before Heroku & AWS were a thing, I used to manage server config for various apps I developed. It was a massive pain. I remember clearly staring blankly into my terminal editing files in /etc/* as instructed by obscure blog posts across the internet and hoping things worked. Once I had things working, I left them alone.

Now, to be sure, things have gotten better. Ansible, Terraform, CDK, etc all allow you to configure servers and cloud services with code rather than manually editing files. However, these abstractions are simply that—abstractions. Many times you’ll run into issues with the underlying system config that you need to correct.

The Pi experience, which I’m assuming mirrors the general state of Linux config in general, is really bad. I forgot how incredibly valuable it is to have sane, smart defaults configured on MacOS that is tailored to the hardware which it’s running on. A Given the slow decay of Mac devices (high hopes for Apple Silicon, but overall Apple machines have gotten worse over the years), I’ve thought about moving to Linux, but this experience has eliminated that thought from my mind.

Maybe some of this pain can be chalked up to the Pi OS, but I can’t imagine things are many orders-of-magnitude better on other Linux variants. I hope I’m wrong, and I hope Linux desktops can eventually get to the ‘just works’ state that MacOS maintains.

Continue Reading

My Process for Intentional Learning

Lately, I’ve been able to carve out dedicated to learning new skills. What I’ve learned has been random, from programming languages to how to build a tiny house. I’ve found a lot of joy in learning new skills, slowly becoming a generalist.

Over the last year, I’ve found you can optimize your "learning time" by thinking through the process of learning before you start. In my experience, picking a learning project, and creating a "learning log" for each skill is hugely helpful.

Identify a Learning Project

Learning in a vacuum doesn’t work for me.

I love reading fiction, but reading a topic that I have no immediate need to understand makes it much harder to comprehend. When I’m motivated by a problem I’m trying to solve, I can plow through books and other information quickly. Without an immediate need, I’ll read the same page many times or fall asleep with the book in my hand.

In other words, learning something Just in Case doesn’t work for me. It has to be Just in Time.

This is why a ‘learning project’ is really important. A small, useful, and preferably time-bound project that requires new skills to complete. The project is a forcing function for learning new skills. You want a project where the pain of leaving it half-done is painful.

For example, when our second daughter was born, I knew she would need the room in our house that I was using as an office (I work remotely). I could move into a room in our basement, but I loved having a large window in the room and for some reason, I didn’t want to work in a basement. So, I decided to build a tiny house to work in.

I’d never built any physical thing in my life before.

I knew I’d lose motivation once I started it (especially as the Colorado summer heat ramped up). I ordered a massive truckload of wood and dumped it in my driveway and built the initial foundation. I knew our new daughter would need my room at the end of the summer and it would become too cold to make real progress on it by October.

These factors created enough motivation to force me to finish the project when I didn’t want to. I’m glad I did! By building a mini house I learned most of the handyman skills I’ve been wanting to learn for years—the perfect learning project.

Before jumping into learning something new, take some time in picking your learning project.

For instance, let’s say you wanted to learn software programming. You could take a bunch of online courses or start reading random tutorials online. You could spend a bunch of money on a coding bootcamp, or join something like Lambda School.

However, you could also find a a simple job on UpWork that feels simple & small enough for you to figure out. This provides a context and specific application for your learnings and the extrinsic motivation to finish the work (there’s someone on the internet trusting you to get this thing done for their business).

Structure Your Learning

After you’ve picked a project, I’ve found it to be helpful to structure your learning process by asking some questions (here’s a post that roughly follows this structure):

What’s your learning project? Example: build a tiny house or automatically mark RSS articles as read What does success look like? This prevents you from following rabbit holes and forces you to finish the project. Example: build an insulated tiny house (not painted, not drywalled) or a script which marks articles more than two weeks old as read. What ‘open questions’ do you have? What are the gaps in your knowledge that would prevent you from completing the project? Write these down at the top of the document. What tools are you missing? This won’t be apparent to you at the outset, but as you start learning you’ll find friction in your process that you’ll want to eliminate. For instance, I found that the hammer I had was hard to use. I noted this down and found that $10 bought me a much better hammer. Or, in the context of programming, your IDE autocomplete may not be working in the language you are learning. What are some of the top books, tutorials, YouTube channels, etc that align most closely with what you are trying to do? What completed pieces of work are similar to what you are trying to do? For digital projects, this could be open source projects or raw asset files for a media project. Is there a community (online or otherwise) around the thing you are learning? Documenting the places where friendly people on the internet, who are obsessed with what you are learning, is super helpful. You’ll remember to ask them a question when you get stuck!

With this information in place, I start working on the project. As questions come to mind I write them down in a "learning log"—bullets in a document. If there’s a large piece of knowledge or tool that’s missing I’ll add it to the top of the document and handle it later.

I’ve found that this live-blogging style learning log helpful, even if no one reads it. By writing down questions and problems that are coming to mind as I’m learning, it forces me to clarify and refine my thinking. This often helps me solve a problem quickly. Writing down the question helps prompt my mind to provide better & unique answers.

As a meta-point, by writing down this little guide it helped me better structure my learning process for my next project!

Continue Reading

Securing and Backing Up Your Data

I’m sure you’ve had this experience: you flip open your laptop and it doesn’t boot up. You drop your phone and it won’t turn on.

You pause for a second… “is any of my stuff backed up?”

It wasn’t until this happened to me (my hard drive almost fried) that I started taking backups seriously. Security and digital backups are one of those “important not urgent” things it’s easy to forget about until it’s too late.

Here’s my take on how to think about backing up your digital world and making sure others can’t get access to it.

Security & Privacy

Assume all of your private information—SSN, address, password, phone, etc—will be public at some point. It’s only going to get easier to hack into large systems.

Why? Here’s one example.

Each website requires 100s or 1,000s of software packages downloaded from across the internet in order to run. If just one of them gets hacked or has a vulnerability, it’s relatively easy for a hacker to scan the internet for all websites using that package and attempt to break-in. This sort of hack happens often. 

Still think I’m crazy? Plug your email into this site.

The good news is, if you setup your digital world correctly you can easily be immune to these hacks:

Make sure any single password isn’t that important Make it challenging or impossible for anyone to login with just a password Setup automated alerts when someone else is acting on your behalf Backing up Your Digital World

Thanks to Time Machine and WiFi iPhone backups, it’s gotten much easier to backup your data. For most folks, a time machine backup against an external disk is probably enough. 

However, my computer contained everything to run my business for many years so I took my backup game up a notch.

Here’s how I think about backups:

You should assume something bad is going to happen. Your computer will break, your external hard drives will die, a virus will wipe your disks, or someone will steal your computer. Hopefully, none of these things ever happen, but planning as if they will happen is the safest approach.  The time it would take to recreate a single important file is more costly than ~decade of storage costs. In other words, don’t try to save a couple bucks by not backing up your files.  Disk space is cheap. Wether cloud or external drive space, it’s worth paying another $100/year to have enough space to never worry or think about losing data. Eliminate a single point of failure. A time machine backup is great, but what if someone comes in to your house and steals your electronics? Or a baby spills water all over your laptop and hard drive? I want my data to be safe if my entire house burns down.  Make it easy and automated. I want to set my backup strategy up once and never think about it again. My Toolbox

So, how do we get this done? First, here are the tools you’ll need:

1Password. This is, by far, the most important part of this toolbox. I’ve used 1Password for many years and it’s awesome. Gets better without fail year. It’s a paid product, but it’s worth every penny. I store passwords, credit cards, personal data, etc in this. Yes, it’s safe to store all of your data in your 1Password vault—lots of articles out there explaining why.  Arq. Backups data on your computer (and other attached devices) to cloud storage. Supports lots of different storage options, including Amazon Drive. It’s a paid product, but it’s not a subscription service. Amazon Drive. My prime membership has free photo storage, and their storage tiers beyond photos are very cheap ($60/year for 1TB). I don’t use it for document storage/sharing, but it’s great for backing up data. 1TB external hard drives for my wife and I’s computer for Time Machine. Those that don’t have an external power source are more convenient.  Google Drive. I use this for any non-media documents.  GitHub. For code. I store all of my projects in ~/Projects. For the technically inclined, documenting your configuration in a dotfiles repo is a great way to backup config or preferences that may be skipped by Arq. Building Your Backup System

Next, you’ll want to setup your backup system:

Setup the Amazon Photos app on your phone. Setup the app to automatically backup all photos and movies.  Setup WiFi iCloud backups on your phone. Disable photo backups via iCloud to avoid running out of space super quickly.  Don’t try to organize old files, and keep your workspace (i.e. Desktop, Download, Document, etc folders) organized. If you find old files are piling up, group them into a “May 2020 Documents” folder and throw them on an Archive folder on your external hard drive (if you need more space on your primary machine) Setup DropBox & Google Drive sync apps. If you have multiple Google Drives, use InSync to sync them to different folders. Setup Time Machine against an external hard drive.  Setup Arq: Add your Google Drive & DropBox folders. Add places you put files outside of cloud storage tools. For me, that’s Desktop & Downloads. I put all of my GitHub code in ~/Projects. This approach works for any non-media “project” files—put them all in a common directly and back the whole thing up. Add any external drive folders that contain files that aren’t on your machine’s drive. Connect to a backup destination. I use Amazon Drive. Setup a daily automatic backup. 

Here’s what my Arq configuration looks like:


Some notes & caveats:

I don’t have any on-disk music or video files. This approach should still work if you have a lot of media, but you’ll want to think about what videos & audio you really need to backup to Amazon Drive to limit your backup costs. Keeping an external drive connected to your computer isn’t really practical. I haven’t connected my external drive to my laptop in a couple months, which isn’t great. The easy way around this is buying a drive that connects to the network. Securing Your Data

Now it’s time to secure your data! Here’s what to do:

Setup 1Password on your phone and computer. Store every password you ever use here. 1Password will magically identify passwords that are insecure or have been leaked out to hackers. This makes it really easy to incrementally eliminate insecure passwords.  Use a unique password for every login. Do not use the same password everywhere. 1Password will generate a beautifully random password for you automatically if you install the Chrome/Safari extensions (which you should do)! Setup Two Factor Authentication (2FA) on all important sites. This includes Google, Amazon, banks, investment accounts, etc. Yes, it takes a couple minutes to setup, but it makes it much more challenging for someone to hack your account. 1Password can actually be your 2FA device (as opposed to using your phone). If you add a “one time password” field to the 1Password entry, a scanner will pop-up on screen. You can put that scanner over the QR code and boom—your 2FA codes will exist in 1Password.  Review your Google, GitHub, DropBox, etc account connections each year (don’t forget!). It’s very easy to grant permissions to a 3rd party app which are larger in scope than you thought when you clicked “ok”. Printout your 1Password password each year and put it somewhere super safe. This protects against 1Password going down, or something else horrible happening to your computer or the internet. Monitor & Lock Your Credit

It doesn’t hurt to lock your credit and setup some simple monitoring. Here are the three credit unions where you’ll need to lock your credit:

TransUnion Equifax Experian

I’ve found that CreditKarma is a great service for monitoring your credit score & activity.

As with most of these posts, they are mainly written so I can document and improve my thinking. If you have any critiques or ideas, I’d love to hear them.

Continue Reading

How to Block Distracting Websites on Your Laptop

"What exactly did I do the last 30 minutes?"

I’m sure you’ve been there, asking that same question, staring blankly into your computer screen.

I’ve written about how I’m working to minimized distraction. For me, a big component of that is blocking distraction on the device I spend the most time: my laptop.

Here’s what I’m looking to do:

Automatically block distracting websites, but allow an easy way to temporarily unblock them. Example: I want to block Amazon by default, but sometimes I want to jump on and buy something quickly. I don’t want to have to manage a schedule. Creating exceptions to schedules and then remembering to re-enable the schedule never works well. I don’t want crappy software that is going to slow down my computer or cause weird networking issues. I want it to be hard, but not impossible, to disable. One or two clicks to disable is too easy. The Easy Way

For most folks, you’ll want to use one of the couple apps out there that do this for you. Here are some that I’ve tried:

RescueTime Focus Freedom Cold Turkey

Focus is the best option I’ve found. It’s a simple and nicely designed app. Check it out!

The Hard Way

If you like tinkering with your system setup, read on.

The pre-built applications always seemed to do strange things to the networking stack on my computer or hog lots of resources (GBs of memory in some cases). This is probably due to how much I customize my computer.

Also, I found that if I disabled my "blocking schedule" it didn’t automatically re-enable. I would then find myself down the Twitter rabbit-hole with 20m wasted. That was a big issue for me.

Eventually, I got frustrated and built a solution which works surprisingly well:

Maintain a simple file listing every host is distracting. Run a script every time the computer wakes up. I used sleepwatcher for this. The script consumes a list of distracting hosts and adds them to /etc/hosts with a reference to a non-existent server. After trying a couple of tools, a node package hostile worked best. 1. Build a List of Distracting Websites

First, create a simple text file. Back it up on Gist or somewhere where it won’t be lost. Version tracking the file allows you to view a history of what websites are distracting over time.

I keep this file in my dotfiles repo. Here’s what it looks like:

(yes, that’s Amazon Smile since I have a browser extension to redirect me there)

Then you’ll want to clean the file, add www variants of each host, and point them to

sed '/^$/d' ./distracting_websites.txt | sed $'s/\(.*\)/ \\1\\\n127.0.0.1 www.\\1/' > ~/distracting_sites.txt

I put this script in the setup process of my dotfiles for easy ad-hoc execution (you’ll want to continually update your distracting_websites.txt as new things distract you).

2. Block all Distracting Websites with a Script

Below is a script that is run every time I wake my computer. Here’s what it does:

Updates /etc/hosts using hostile and the distracting_sites.txt file Clears system DNS cache Clears Safari cache, which seems to have its own DNS cache. Chrome does not. # asdf is a node version management tool I use. Your exact execution paths will probably be different /Users/mike/.asdf/installs/nodejs/12.14.1/bin/node \ /Users/mike/.asdf/installs/nodejs/12.14.1/.npm/bin/hostile \ load /Users/mike/distracting_sites.txt # clear system cache # sudo killall -HUP mDNSResponder # clear safari cache osascript << EOF tell application "Safari" activate end tell tell application "System Events" tell process "Safari" tell menu bar 1 to tell menu bar item "Develop" to tell menu 1 to tell menu item "Empty Caches" to click end tell end tell EOF

You can test this script by running it with sudo:

sudo /usr/local/sbin/sleepwatcher --verbose --wakeup .wakeup 3. Run Website Blocking Script When your Computer Wakes from Sleep

First, install sleepwatcher:

brew install sleepwatcher

Then, you’ll want to find the location of the plist file which starts up sleepwatcher as a daemon process:

$ brew services Name Status User Plist sleepwatcher started root /Library/LaunchDaemons/homebrew.mxcl.sleepwatcher.plist

You’ll want to edit this plist:

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" ""> <plist version="1.0"> <dict> <key>Label</key> <string>homebrew.mxcl.sleepwatcher</string> <key>ProgramArguments</key> <array> <string>/usr/local/sbin/sleepwatcher</string> <string>-V</string> <string>-w /Users/mike/.wakeup</string> </array> <key>RunAtLoad</key> <true/> <key>KeepAlive</key> <true/> <key>StandardOutPath</key> <string>/usr/local/var/log/sleepwatcher.log</string> <key>StandardErrorPath</key> <string>/usr/local/var/log/sleepwatcher.log</string> </dict> </plist>

Then, you’ll want to ensure the process runs as root and the script you created is executable:

chmod + /Users/mike/.wakeup brew services stop sleepwatcher sudo brew services start sleepwatcher

And… you’re done! Depending on your OS configuration you may need to grant some permissions on first run.

Was this overkill? Definitely. Does it prevent me from wasting any time on distracting websites? Absolutely.

Continue Reading

How I Broke My Phone Addiction

The launch of Neuralink started a conversation across the web about the “merge”. The day when you can plug your brain into a computer and communicate with it through your thoughts. No keyboard, mouse, or touch screen. Something out of a sci-fi film.

I think Sam Altman has an interesting take:

I believe the merge has already started, and we are a few years in. Our phones control us and tell us what to do when; social media feeds determine how we feel; search engines decide what we think.

This resonated very strongly with me. My phone does control me to a certain extent and I feel uncomfortable if I hop in the car without it.

I’ve been on a kick this year of being intentional about how I use technology. A big part of that is my phone. It’s the most distracting—and the most useful—thing I own.

“What’s wrong with your phone? Is it broken?” has been a common refrain when I show a friend a photo or map on my phone. Friends often complain about how slow I am to respond to texts. My phone isn’t broken and I know how to use the messages app, but I have taken some “extreme” measures to disconnect from my phone.

Below is the list of things I’ve done to disconnect from my phone. They work. Not that I’m close to perfect, but I can easily leave my phone in another room now and forget to check it for nearly a day. That’s huge.

If they seem extreme, I encourage you try one or two and see what happens.

Turn off Text Notifications

Turn off all your text notifications. Settings > Notifications > Messages > Allow Notifications.

Yes, it’s hard and annoying for a week. But man, it is amazing not to hear or see that ding from incoming messages. Once you get over the week or two of withdrawal you’ll love it and never be able to go back. I’ve had my text notifications disabled for months at this point and it is the single biggest change you can make on your phone.

“Communication for work comes through text and I’m expected to respond instantly!”

You win. This won’t work for you. Here are two ideas for you:

Can you block texts from non-work numbers? If your work will pay for a separate phone, you could setup that phone with text notifications and disable them on your device.

“What if you are meeting someone i person and need to communicate in real-time?”

This happens to me often. Just open up your message app.

“What if you miss an emergency text from your spouse/friend/whatever?”

Tell your spouse and close friends you’ve disabled text notifications. If they need you right away, they can call you.

1. Setup an Incoming Call Whitelist

The idea here is only allowing calls from people you know. Any other calls can go to voicemail. If it’s important, they’ll leave a message and you can listen to it later.

Here’s how to setup this whitelist:

Navigate to Settings > Do Not Disturb. Setup a schedule and make it nearly the whole day. Allow Calls From: Favorites (or whatever list includes the people you need to be responsive to). This is your whitelist of people you want to hear from. Install Hiya. I’ve found this to be a helpful tool for identifying spam callers when reviewing my missed calls and voicemail.

“I may receive a callback from a customer service department or other unknown numbers”

Just disable do not disturb. Because you’ve setup a schedule, it’ll automatically be enabled the next day.

2. Enable Grayscale Mode

This was a trick I pulled from Make Time. Makes your phone less addicting, but no less functional:

Settings > General > Accessibility >Display Accommodations >Color Filters. Switch Color Filters on and select Grayscale. 3. Remove Distracting Applications

Remove any apps which you find yourself pull-to-refresh’ing. Some examples:

Social: Twitter, Facebook, Pinterest, whatever Dedicated news apps Email. Truth be told, I still have this on my phone for work communication. Move any apps that you need to keep, but are distracting, off the home screen. 4. Block Nearly All Notifications

Go through every app on your phone (Settings > Notifications) and turn off notifications. Think hard about the couple apps you really need notifications from and enable them.

Here’s my list:

Google Calendar FaceTime Phone Airline apps Google Maps Scooter apps, Lyft, Uber DMs on Slack within working hours Todoist 5. Block Distracting Websites

It’s not intuitive but you can block distracting websites on chrome/safari on your phone.

Navigate to ‘Screen Time > Content & Privacy Restrictions > Content Restrictions > Web Content > Limit Adult Websites’ and enter in all distracting sites under “Never allow”.

For example, here’s some of my list: reddit.con

Setup a monthly reminder on your todo list to add any new distracting news sites that you’ve started looking at on your phone.

Other Tips & Tricks

If you’ve made it this far, I challenge you to stick with the setting changes for two weeks. That’s about how long it took for me to stop being annoyed by the changes.

Below are some other configuration tips & tricks I’ve written down over the years for when I get a new iPhone.

Other Misc Tips & Tricks Move mail off of the home screen General > Display & Brightness > Night Shift. Enable, 9am-7pm General > Accessibility > Home Button > Reset Finger to Open Delete default apps I’ll never use: Home, Books, iTunes Store, Watch, Tips, TV, Apple Mail, News, Stocks. Messages: Enable send as SMS, disable send read receipts, enable text message forwarding. iCloud: disable photos (use Amazon photos instead), enable contacts, disable calendar, enable Messages, disable Stocks, enable iCloud backup, disable Keychain (use 1Password instead) Phone > Call Blocking & Identification: Hiya Spam & Block Enable password autofill for 1Password. Password & Accounts > Autofill Passwords > 1Password. Disable keychain passwords. Amazon Music: download some music you like, enable automatic downloads of offline music, and disable cellular streaming. Download offline google maps for your local area. Automatic Backup Configuration If you have Amazon Prime. Install Prime Photos and use it to backup all of your photos. Settings > iCloud > Storage > Manage Storage > Backups > Disable Photo Backup . 5gb is not enough room for anything, and Amazon gives you unlimited photo storage for free. Plus, there additional storage tiers (if you take a lot of videos on your iphone) is really cheap. Settings > iCloud > Photos > Disable Photo Stream. Manually initiate a backup to ensure everything goes smoothly. Without photos + videos, your iPhone backup should be able to fit into the 5gb default iCloud storage. Warranty & Documentation

Some notes on warranty replacement:

An IMIE number is a unique identifier for your phone. Document this number in a 1Password note. Settings > General > About > IMIE Your ICC number is the unique identifier for your SIM card. Document this as well. If something is going wrong with your iPhone, try backing up the phone to iTunes and then doing a fresh restore. If that doesn’t work it’s a hardware issue. Try this before wasting your time with Apple/your cell provider. If you get an “Invalid SIM” error when switching cell phone providers or phones your IMIE and ICC numbers may not be “paired’. You can often pair these numbers yourself through the settings area of your cell provider. The support reps often do not check this or understand it fully.

Continue Reading

Learning Clojure by Automating an RSS Reader

I’ve been working on revamping how I consume information. Most of my information consumption has been moved to RSS feeds, but I can’t keep up with the number of articles in my feeds. When I take a look at my reader I tend to get overwhelmed and spend more time than I’d like to trying to "catch up" on information I generally was consuming out of curiosity.

Not good.

I want articles to be automatically marked as read after they are a month old to eliminate the feeling of being "behind". This is a perfect little project to learn a programming language that’s looked interesting for a while!

Building a small project in a new language or technology is the best way to learn. While I was building this tool, I documented what questions I was asking, answers to these questions, and what articles and resources I found helpful.

Posts like this have been interesting to me, hopefully this is a fun read for others!

What do I want to build?

I want to build a Clojure script for FeedBin that will:

Inspect all unread articles If the publish date is more than two weeks in the past, mark the article as unread Automatically run every day

Let’s get started!


Here are some helpful blogs & tutorials I used while learning:

Also, I always try to grab a couple of large open-source repos to look at when I’m learning a new language. Here are some places I searched:

Some repos I found interesting: This is probably the largest full-blown open-source Clojure application out there. Most other projects I found were libraries, not applications. Syntax & Structure

Now that I have some browser tabs open with documentation, let’s start learning!

How do I install this thing? => brew install clojure/tools/clojure Going through the "Learn X in Y" guide, some interesting takeaways: Clojure is built on the JVM and uses Java classes for things like arrays. Code in Clojure is essentially a list-of-lists. A list is how you execute code: the first element is the method name, and then arguments separated by spaces. This feels very weird at first, but it’s a really powerful concept. Simple made Easy explains the philosophy behind this a bit. "Quoting" (prefacing a list with a single quote) prevents the list from executing. This is helpful for defining a list, passing code as a data structure that can be mutated later on. Sequences (Arrays/Lists) seem to have some important different properties from vectors. I need to understand this a bit more. When you define a function it doesn’t get a name. You need to assign it (def) to a variable to give it a name. The [] in a function definition is the list of arguments. There are lots of ways to create functions: fn, defn, def ... #() multi-variadic function is a new word for me! It’s a function with a variable number of arguments. Looks like you can define different execution paths depending on the arguments, kind-of like Elixir’s pattern matching. [& args] is equivalent to (*args) in ruby The beginner (me!) can treat ArrayMap and HashMap as the same. Keywords == ruby symbols The language looks to execute from the inside out, and the composition of functions is done via spaces not commas, parens, etc. Looks like everything is immutable in Clojure. Everything is a function. So much so, that even basic control flow is managed the same way as a standard function. Looks like "STM" is an escape hatch if you need to store state. Similar to Elixir’s process state. The Clojure community is big on "repl driven development", but what exactly do they mean? How is that different from binding.pry in a ruby process to play around with code? Looks like it’s not that different. Some nice editor integrations make things a bit more clean, but more or less the same as opening up rails console with pry enabled. I’ve always disliked the ability to alias a module or function to a custom name. It makes it much harder for newcomers to the codebase to navigate what is going on. Looks like this is a pretty common pattern in Clojure, the require at the top of a file can setup custom aliases for all functions. "forms" have been mentioned a couple of times, but I still don’t get it. What is a form? I’ve heard that Clojure is a Lisp. What is a "lisp"? There was an original LISP programming language, but "a lisp" is a language patterned after the original LISP Seems like the unique property of a lisp-style language is code is essentially is a linked list data structure. Since all code is a data structure, you can define really interesting macros to modify your source code. Another property is the parentheses-based syntax. It’s interesting to look at the different lisp styles available. I feel like the only language that is popular today is Clojure. Sounds like immutability is unique to Clojure and isn’t a core structure other lisps.

I think I know just enough to start coding.

Coding in Clojure

Here’s the learning process which generated the final source code:

Let’s define the namespace and get a "Hello World" to make sure I have the runtime executing locally without an issue. 184408626bb41b87d53f9b0bb5485a8e9201d8d5 Ok, now let’s outline the logic we’ll need to implement. 7e018b05ff8ad925ef2bfe9c56c4a702dce4c3d0 Now, let’s pick a HTTP library and figure out how to add it as a dependency. looks like the most popular package repository. It doesn’t seem like there’s any download/popularity indicator that you can sort by. Bummer. Hard to figure out what sort of HTTP library I should use. Looks like project.clj is a gemspec type definition file. Metabase’s http library is clj-http. Let’s use that. We’ll also need to figure out how to setup this dependency file. is linked in the project.clj files I’ve seen. It’s listed as a dependency manager on the clj-http library: Let’s install it via brew install leiningen. lein new feedbin and mv ./feedbin./ ./ to setup the project structure. Looks like lein will help us with dependencies and deployment. b0b4022618abac840af6679f900584d04de510c1 There’s this skip-aot thing in the main: definition which I don’t understand. In any case, if I stuff a defn -main in the file for the namespace defined in main lein run works! 764d7a1e2a537d61b036df4229a2c96671725dd8 It looks like this ^: syntax is used often. What is it? Ok, let’s copy our logic outline from the other file we were working on over to the src/feedbin/core.clj and try to add our HTTP dependency. Added [clj-http "3.10.0"] to the dependency list in project.clj, lein run seemed to pull down a bunch of files and run successfully. Now, let’s pull the FeedBin variable from the ENV and store it to a var. Looks like you have to wrap let in parens, and include commands that rely on the var within the scope of the parens. I could see how this would force you to keep methods relatively short. 6f1f8099ffd0ed5f997be93685d18d1c574efb6b Let’s hit the API and get all unread entries and store them in a var. Looks like cheshire is a popular JSON decoder, let’s use that. It looks like let is only when you want temporary bindings within a specific scope. Otherwise, you should use def to setup a variable. 5b63cd289052d9fcebec2cb2965d598927b0616a Convention is - for word separation, not _ or camel case. Let’s refactor the getenv to use def. Much better! a6a95a1e4703c07e76ecce32b56b6b0f1903acca Time to select entries that are two months old. A debugger is going to be helpful here to poke at the API responses. Looks like debugger is the pry equivalent. I had trouble getting this to work and deep-dived on this a bit: (pst) displays the stacktrace associated with the last exception. This is not dependent on clj-debugger Looking closer at clj-debugger it has ~no documentation and hasn’t been updated in nearly two years. Is there a better option? Doesn’t look like it (require 'feedbin.core :reload-all) seems like the best way to hot reload the code in a repl. Then you can rerun (feedbin.core/-main) Ah, figured it out! (break) on it’s own won’t do anything. It needs an input to debug. (break true) works. You need to run this in lien repl for it to work. As a side note, I’ve found the REPL/debugging aspect of learning a new programming language to be really important. Languages that don’t have great tooling and accessible documentation around this make it much harder for newcomers to come up to speed. The REPL feedback loop is just so much faster and in developer tooling speed matters. I was able to extract the published date, now I just need to do some date comparison to figure out which entries are over a month old. ca16f54f66a39753933168c3f8deac636144ca47 Now to mark the entries as "read" (in feedbin this is deleting the entries). Should be able to just iterate through the ID list and POST to the delete endpoint. I started running into rate limiting errors as I was testing this. # turns a string into a regex, but appears to do much more. Looks like it’s a shorthand for creating lambda. macroexpansion is an interesting command to help with debugging. With the rate limit errors gone, I can finally get this working for good. I tried passing in the article IDs as a comma-separated list as a query string and it didn’t work. I need to send this data in as a JSON blob. 166ea49439ed690ff08c8fd987530b170b9bb80e Got the delete call working. You can pass a hash directly to clj-http and it’ll convert it into JSON. Nice. 63ac8bf1d4fd969326fffa9ad7b50ad1f0a4b56d

Great! We have the script working. Now, let’s deploy it.

Clojure Deployment Using AWS Serverless

I have a friend who is constantly talking about how awesome serverless is (i.e. AWS Lambda). I also remember hearing that you can setup cron-like jobs in AWS that hit a lambda. Let’s see if that’s the case and if we can get this script working on lambda.

Some things we’ll need to figure out:

How/where do I specify that an endpoint should be hit every X hours? How do I specify where the entrypoint is for the lambda function? How do we specify environment variables?


I jumped into AWS lambda dashboard and created a function named "Mark-Feedbin-Entries-As-Read" with Java 11. It looks like the crazy AWS permission structure is generated for me. I added the com.amazonaws/aws-lambda-java-core package and it looks like I need to run gen-class to expose my handler. What is gen-class? It generates a .class file when compiling, which I vaguely remember is a file which is bundled into the .jar executable. Looks like aot compilation needs to be enabled as well. Still need to understand what aot is. I ran lein uberjar and specified feedbin.core::handler as my handler. Created a test event with "testing" as the input. Used the -standalone jar version that was generated. Looks like environment variables can be setup directly in the Lambda GUI. "Cron jobs" are setup via CloudWatch events. What is CloudWatch? It’s AWS’s monitoring stack. Strange that this is the recommended way to setup cron jobs. I would have thought there was a dedicated service for recurring job schedules. "Serverless" (looks like a CDK-like YML configuration syntax for AWS serverless) makes it look easy to deploy a lambda which executes on a schedule, but doesn’t indicate how it’s actually managed in AWS in the blog post. Aside: It’s interesting the more you dig into AWS, the more it feels like a programming language. Each of the services is a library and the interface to configure them in yaml. It looks like "Amazon EventBridge" is the new "CloudWatch Events". Looks like we can setup a rule which triggers a lambda function at a particular rate. Neat, you can setup a rule directly with the AWS Lambda GUI. Use a EventBridge trigger with rate(1 day) to trigger the function every day. Really easy! I checked on it the next day and it’s failing. How can we inspect the request? It’s probably failing due to the input data being some sort of JSON object vs a simple string that I tested with. Here’s what I found: you can inspect the logs, use CloudTrail to view an event, enable X-Ray tracing, and send failed events to a dead letter queue. I enabled all of this stuff: my end goal to inspect the event JSON passed the lambda to determine how to fix it. Ah! After a bit more digging, if you find the event in CloudTrail there’s a "View event" button that will give you the JSON output. I can then copy the JSON into the test event in the configuration for the lambda and run it there to get helpful debugging information. Feels a bit primitive, but it works. I wonder how you would run the function and locally and write integration tests using example AWS JSON? Looks like the function signature for my handler is incorrect. When handling events, the handler accepts two arguments [Object]. This fixed the issue! 8520e8a319bd5d41a67a01f9517ce4cf559ab381

Resources: Open Questions

Here’s a list of questions that I wasn’t able to answer during my learning process:

How can you parallelize operations in Clojure? How easy is deployment? How does interop with Java work? Is there a rails-like web stack? Is there a style guide?

Continue Reading

Reclaiming Your Mind: Creating an Information Diet

There’s been a lot of areas of my life that I’ve been ‘auditing’ and attempting to tweak the habits that have intentionally or accidentally fallen into place. One of these is my information diet: how I find, consume, and process information.I’ve been tracking my time spent on reading/time on the internet and I’m not liking the trend. I’ve felt more addicted to information this year and I want to eliminate that feeling. Revamping my information intake is one way I’m going to do that. It’s worth thinking about why it’s worth spending time consuming information, how I consume information, and how I want to change my information consumption.

Categories Stories. I’ve been almost exclusively consuming non-fiction for the last decade and rarely read any non-fiction. At the suggestion of my ever-wise wife and the promptings of a great podcast on story I’ve reprioritized non-fiction as something worth spending time on. Great stories can change our perspective on our life and increase our creative thinking.Curiosity & Exploration. Investing in discovering new and interesting ideas has always paid off for me. For me, this generally looks like browsing community sites like hacker news, reading a random newsletter, of following interesting people on Twitter. Learning about random, interesting topics has always been really enjoyable for me—it sparks creativity (and joy) and is useful later Work. Learning specific to a work-related problem. Entertainment & Social. Twitter, Facebook, news, etc. When you look back at your time spent here you always feel like it was a waste. I’ve been convinced that keeping up with news is largely a waste of time (you’ll hear anything worth knowing about through friends), and time spend communicating with friends over social is better spent with friends in person. This doesn’t mean that there isn’t a place for this category, but for me it means I need to bias towards eliminating any time spent on this category. Infrequent Personal-ish Updates. There’s a group of organizations or people I follow that I want to keep tabs on, but don’t send emails often. A friend running a non-profit, bands announcing a new album, etc. Deals/Promotions. Transaction/Service Emails. Mediums Podcasts.BooksBlogs.Community News. Hacker News, Lobsters, Reddit, Product Hunt.News Sites. Google News, Bloomberg, TechCrunch, etc. Social. Twitter, FaceBook.Movies. NetFlix, Amazon Prime, YouTube.Email.Personal communication. Texts, voicemails, etc.

Thoughts on Consumption.

Continually improve the system. Set aside time every month to quickly audit what I’m consuming and what tools I’m using. Make consumption a choice rather than a reaction. Right now, I randomly visit Hacker News or see an article come through my email. Instead I want to centralize information in one category into a single place that I can go. Optimize for pull vs push consumption. A great example here is email newsletters. They are push, not pull, and are often messy to read and pile up in my inbox. I want to separate “conversations with people” over “updates from companies/interesting news”.Categorizing information is critical. Email newsletters can’t be categorized easily. I want to put feeds into separate buckets that I can prioritize and triage separately. I should use RSS again. Way back when I read everything via NetNewsWire. Email newsletters took over seemingly overnight and I forgot that RSS existed. Most sites I care about still support RSS (even if it’s not advertised explicitly).Use a RSS Reader. Specifically, one supported by paid subscriptions. Free is great, but most free things (without a huge market) die or have negative externalities over time. I don’t want to have to mess with this part of my toolkit much and deal with a killed product. Paid subscriptions mean it’s a real business that will continue to improve over time. Limit consumption. I want to enforce a limit on the numbers of things I’m consuming. I wonder if there is a way to automatically reset the read count of various feeds so it doesn’t look like there are too many articles to read when I use a reader.Prefer books over articles. For most business/technology problems, blogs and Q&A sites are the main source of data, but work aside, books are generally higher-quality information compared to blogs. The time it took someone to create the content is a good indicator of the quality. Books > Blogs > Twitter. (this gets a bit tricky with low-cost kindle books: skip these). Optimize for highest impact & quality information at the beginning of the day. This means reading books and long-form articles at the beginning of the day while my mind is clear, instead of consuming blogs, tweets, texts, etc. Treat books like a blog archive. I really like this concept, can’t remember where I first heard it. Reading books from cover to cover doesn’t make a ton of sense, although it’s definitely how I’m trained to read books. Skimming through a chapter (or skipping it entirely) if you find it boring or too verbose shouldn’t feel ‘wrong’. If the writer can’t keep your attention, that’s their fault. Additionally, books are generally longer than they need to be in order to hit page quotas.Don’t switch contexts. If you are reading a book, don’t stop and read a blog article. Cultivating sustained laser-focus attention on a single thing is critically important. I’ve found this to be more challenging as the years go by, and it’s something I need to be even more intentional about.Focus on managing written internet media. I don’t over-consume podcasts or books. I struggle most with interesting, distracting news sources like Hacker News or What I’m going to change Limit number of news feeds to 30. I suspect this number will change as I continue to slowly improve how I’m processing information, but this is a good start.Convert email newsletters to RSS. Most newsletters (like Ruby Weekly) have an RSS feed. For those that don’t, FeedBin has a service to convert email newsletters into a feed, and I imagine there are standalone services that will do this for you automatically.Mass-unsubscribe from email newsletters. I’ve been using for years (I don’t love the privacy component, but it’s a useful tool). It looks like their unsubscribe option will actually click through the unsubscribe links for me. I should go through my daily summary and remove newsletters I’m not interested in, and convert the others into a feed.Setup two aliases and Forward all updates to FeedBin and auto-archive. Auto-archive & tag all promotions.Subscribe to weekly summaries on community news site. Mass unfollow everyone on Twitter, and limit the people I follow to 50.Update my website blocking strategy, including blocking of all news & social sites. More on this in a separate post.Stop using Apple Podcasts. I find it hard to keep things organized and Apple seems to randomly reverse the listing of certain podcasts. I should trim which Podcasts I subscribe to and find another Podcast application. Categories: engineering, techReview all gmail filtersReview and trim all YouTube subscriptions. Review all Twitter app connectionsReview any compromised passwords via 1Password

After a bunch of investigation, I settled on using FeedBin. doesn’t look like it is updated often went open source and hasn’t been touched in over a year podcast and RSS reader, open-source, commercially supported and recently updated. Standalone paid product, has an API, doesn’t look too complex. 64 feeds for free. Free and paid is the most popular, but looks to be overdone. Standalone paid product. Not updated frequently. Doesn’t look like a great design. Free and paid tiers. I found this reader most commonly referenced by HN and Lobsters. ties into various services to create a great reading experience. another macos reader. web-based reader with a low-cost paid subscription. well-designed feed reader. Supports podcasts and receiving email newsletters via a special email address. Also has a Twitter reader as well. Bootstrapped business. Also open source, very cool. Pete Cooper is involved in this one. Open source. Looks like a zombie product. new reader from HN with some fancy “AI” grouping. Leaf. Looks dead. Hasn’t been updated in two years.

Other interesting finds: RSS feed APIhttps://throttlehq.com and

Continue Reading

2019 Goal Retrospective

I’ve been doing retrospective’s on my yearly goals for a couple years now. Although it’s a little late, I wouldn’t want to break the habit (plus, I’m trying to open source my thinking).

Let’s go!

What Worked Creating a distinction between habits and goals. I have a separate “habit document” where I document habits that are important to me. Setting a goal to kickstart a habit. Habit-goals shouldn’t be all, or even most, of your goals for the year but having one or two habit-goals can be really effective at changing behavior. It was useful to commit to an action (like hiring a personal trainer) to force building momentum for a specific habit. If you’ve set a goal for the last couple years and haven’t been able to make it happen, consider doing something drastic. How can you up the ante and put something on the line associated with the goal? Maybe it’s hiring a coach, tying money to it, making a commitment that you can’t back out of without causing issues for someone else, etc. Figuring out how to raise the stakes has been hugely helpful. Including a just-for-fun goal: vacation, hobbies, etc that you’ll be really motivated to accomplish. This has helped me be excited about the year and maintain motivation for the important but not exciting goals. Setting aside project time as a married couple. It was fun to work on our goals together, and we got some important and run things done during this time. Looking forward to more of this. Joint goals or projects with my wife was really fun and motivating. For instance, we built a garden together this year. Zero-targets. Setting a non-action goal was a great way to break some behaviors I wanted to change. What Didn’t Goals that weren’t exciting or specific enough fell to wayside. We didn’t do the quarterly review at all. This is the second year that this wasn’t an effective practice. In the season of life that we are in (little kids), we just don’t have the time to really set aside the time to do a proper quarterly review. We need to rethink this. This may be obvious, but having a kid is a goal in and of itself. We knew we were going to be growing our family, but I didn’t account for this in my list of goals. Make sure to a in that year and you need to plan for that. We didn’t create “project time” that often individually or as a couple. I wish we blocked off time for projects 2-3x more than we did. What Needs to Change? Remove the quarterly review. We haven’t stuck to it for the last two years and with two young kids carving out that amount of time just isn’t practical. Next iteration on this is adding reminders to our monthly review to ask a couple of the questions that we wanted to incorporate into the monthly review. More project time. This is super fun if you set goals at a couple and helps create focus around making progress on goals that are slipping.

Here are some other reflections I had about the year:

Many of the exciting life changes have come and gone (moving, buying a house, etc) and we are in a season where family (kids) take up the majority of our time. This means that most of our goals are less exciting, and that’s ok. We have to remember that raising amazing kids and being present to them is our top priority. What that requires shifts and changes throughout the year. Some things in our life which need to change are hard to tie to a specific and measurable goal. Mostly because we don’t know exactly what needs to change. With two young kids “improving our family balance” is a thing we need to improve, but what that exactly means isn’t clear. What we decided to do was pick a specific thing that represented the best forcing function we could think of for improving on the vauge state that we are marching towards, and then adjust the specifics of that goal as we move through the year. Most of your goals shouldn’t be actions that you naturally motivated to take. I tend towards this mode and need to think hard about what goals work against things I don’t

Continue Reading