Learning Swift Development for macOS by Building a Website Blocker

I loved Focus App. It blocked websites and apps on a schedule. But, years ago it started glitching out: sucking up tons of ram and freezing my computer. They didn’t fix the bug and I abandoned using it and instead switched to a host-based blocking system which has served me well.

However, there are some issues with the host-based approach:

I can’t block specific URLs, only hosts (focus app couldn’t do this either) I can’t set a schedule I can’t block apps If I remove a host it will not automatically get blocked unless I sleep and wake the computer Sleepwatcher (cli tool) is dead and requires some manual set up to get working.

My goal is to layer on top of the existing host-based system that has been working great and add another layer of focus tooling:

CLI-first tool Allow configuration to be easily set using a JSON file Allow different blocking configuration to be scheduled Replace sleepwatcher by configuring script execution on wake Add a ‘first wake of the day’ trigger that I can tie into clean browsers and todoist scheduler Allow both hosts and partial match urls to be blocked ‘Partial match’ means (a) anchors are excluded and (b) the configured block url must only be a subset of the url on the browser in order to be blocked. This will enable things like blocking news or shopping search on google. Support blocking urls in google chrome and safari No UI, maybe build a simple REST API that could be tied into my beloved Raycast Run CLI tool as privileged (in order to mutate /etc/hosts)

With a clear goal in mind for this learning project, I was able to get started and build this out. Here are the two repos with the resulting code:

hyper-focus CLI source code hyper-focus GUI via Raycast extension

I haven’t touched macOS development in years and hadn’t done any Swift development before. Below are my notes from learning swift and macOS development.

Swift Language The guard statement is explicitly used to return early. It’s like unless in ruby with some special scoping properties. More info. Specifically guard is useful for unwrapping an optional and assigning the unwrapped variable to something that can be used in the outer scope. There’s a community built package manager, but it requires that you (a) have a Package.swift and (b) use a specific source code structure. Both of which are a pain for a simple utility. I found later on that it’s better to just set up your application using Package.swift, even if it’s small. You’ll end up needing a community package and using the swift CLI tooling is nice. There’s a built-in JSON decoder, but it requires you to describe the incoming JSON payload as a struct. This makes sense since swift is strictly typed, but makes fiddling with data structures a PITA. There’s no built-in logging library with levels. There’s an open-source package out there, but not having it included with the stdlib is crazy to me. Here’s a < 50 line implementation of a simple stdout log. @objc exposes the swift function/class to the objective-c side of the world. You don’t have to worry too much about this, the compiler will warn you and enforce that you put these attributes in the right places. You can extend existing classes via extension String and add whatever methods you’d like onto them. I’m surprised by this for what seems an otherwise very structured language. This was a great compromise. One of the guys who works on the Swift language built Rust. I don’t know Rust (it’s on my learning list!) but from what I’ve heard—and the adoption it’s gotten across the new CLI tooling that has been emerging—it’s an amazing language. Probably part of the reason Swift seems so well-designed. Doesn’t seem like there are union types in Swift. You have to define an enum and then unwrap the enum using a switch statement. This seems insane to be and makes for very ugly code, I must be missing something here. You can nest struct definitions, which is nice. You can’t add a trailing comma to arrays or dicts, which drives me nuts. Makes it harder to refactor code and adds additional mental overhead to editing anything. It’s puzzling to me why more languages don’t allow this (one of the things I love about Ruby). You can typecast an object to a specific type with as! SafariWindow I imagine, since Swift is strongly typed, this has some limitations + compile errors, but I don’t know what they are and didn’t bother to learn. You only need an import to pull in a framework, not individual files. All files in the project are automatically compiled. Anything marked with public is available to everything in the project. This seems to indicate something otherwise, still some more investigation needed here. Argument order matters even when using keyword arguments. Bummer. Crash reports are still nearly useless. They have a stack trace, but no line numbers. You need to convert the crash report into a stack trace which is usable, which requires symbol-mapping file (dSYM) generated at the same time as the binary that generated the crash report. PLCrashReporter does a lot of this for you, but for a simple single-file swift script this is a massive pain. There are no stack traces on the command line, even in debug mode. ! asserts that the optional is not nil. If it is, your app will crash. You can use as? to define a default value if a non-nil value does not exist Method overloads exist, so you can define a method multiple times with different params. I really like this pattern, wish Swift had method guards like Elixir (one of my favorite things about Elixir). You have to explicitly indicate that a func could throw an exception with throws in the method signature. This is interesting, I think I like it, makes the design of the function more explicit. Empty dictionary is [:], and you can inline-type Any to a dictionary via varName: [String: Any]. I think Swift dictionaries are the same as an NSDictionary under the hood. dispatchMain() is not the same as RunLoop.main.run() despite what some blog articles say. let == const in JavaScript, var is roughly equivalent to JavaScript. Multiple let statements in an if can be separated by a comma. If any of the let statements results in a nil value, then the if statement fails. I don’t understand the value of this syntax above &&. I don’t like this language design choice. There are some magic variables. For instance, if you are in a catch block the error variable represents the exception. If you have a global function named error it is not accessible and overwritten by the local error variable. I didn’t read up on Swift’s memory allocation strategy, but my assumption is if a var isn’t referenced any longer (i.e. out of scope) it’s removed/garbage collected. The foot gun here is you have a class which subscribed to a notification (NSWorkspace.shared.notificationCenter.addObserver) but that class is not assigned to a var that will continue to persist after the caller completes (i.e. a class or global variable) the object will be garbage collected and you’ll never receive that notification and an error will not be thrown. However, if a function creates a Task which creates its own run loop, that task will continue to run as long as the loop is created even after the caller that created the Task has completed. I would imagine this is a bad design pattern. This also applies to other systems which receive ‘notifications’. I use this word very vaguely because I don’t understand macos subsystems very well/at all. It seems like there are ‘grand central dispatch’ queues which feel similar to a SQS queue, and those seem to be impacted as well. Any async pub/sub type interface would be impacted by the subscriber being garbage collected and you will not receive an error. It puzzles me why errors are not thrown. Hosting a localhost server

This is simple as long as you do bind to a local IP: localhost, 127.0.0.1, etc. If you bind to your router’s IP address you’ll run into all sorts of permissioning issues:

The default permissioning is different depending on what macos version you are on. Here’s an example of how to check an application’s default permissioning You cannot change your entitlements/permissions if you are just building a simple binary or cli app. You need an app with a Info.plist to set the proper security config. This is because of new security stuff that apple has introduced. This means you need to use xcode to setup and build your application. I couldn’t find any good examples of an app that is built without using XCode. The alternative to this is using another layer of indirection, like tuist. This is bringing back memories of all of the stuff I hated about desktop application development. Don’t bind to the device IP (i.e. the wifi- or ethernet-assigned address) unless you need to. Bind to localhost so the server is only accessible on the device. Swift server package options https://criollo.io https://github.com/httpswift/swifter https://github.com/Building42/Telegraph https://github.com/envoy/Ambassador Packaging

Not using a Package.swift for anything even slightly complex will bring a world of pain:

The VS Code tooling doesn’t work as well (no error highlights and LSP stuff) You can’t use a package manager and therefore can’t easily pull in community packages Anything that uses swift build doesn’t work

You’ll want to use a Package.swift in your project. Generating a Package.swift is pretty easy:

swift package init --type executable

When running swift build I ran into:

no such module 'PackageDescription

This post describes the issue and the following command fixes it for me:

sudo xcode-select --reset

If you run into issues with compilation errors due to some features not being available on older macos versions, you’ll need to add a platform requirement to your Package.swift:

platforms: [ .macOS(.v13) ],

Here’s an example Package.swift for the CLI tool.

Cleaning All Cache

I ran into a very weird build error:

❯ swift run Building for debugging... Build complete! (0.25s) dyld[21481]: Symbol not found: (_$s10Foundation11JSONDecoderC6decode_4fromxxm_AA4DataVtKSeRzlFTj) Referenced from: '/Users/mike/Projects/focus-app/.build/x86_64-apple-macosx/debug/focus-app' Expected in: '/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation' [1] 21481 abort swift run

Even after resetting the project to a state where I knew it compiled, it still errored out. After walking away for a while, I found this post and tried updating the min macos version. It magically fixed the issue.

Here’s what I used to clear all build caches:

rm -Rf .build/ rm Package.resolved rm -Rf ~/Library/Developer/Xcode/DerivedData rm -Rf /Users/mike/Library/Caches/org.swift.swiftpm Open Questions Is there a way to open a repl with your application’s code imported? It was nice that a compiled language had a recent repl, but ideally, I want to open a repl and be able to import/use my applications code. How is the debugger? I just did caveman debugging for this project and didn’t bother understanding the GUI debug tooling. It’s unclear how good the package ecosystem is. It seems better than my Cocoa days, but there weren’t that many options and the package activity seems pretty dead. It doesn’t seem like you can build a .app without an xcode project. This is annoying, especially if you are building a small tool and don’t want to learn and understand the xcode toolchain (it still seems terrible). I wonder if I’m missing something here and if there’s some good tooling to support a CLI-based application build? I was surprised at how many errors were not reported. If you’ve subscribed an object as an observer to a notification center, the object was GC’d, that should give you an error. It seems like there were a good number of silent failures which made it harder to discover unexpected failures, especially to someone who is not a desktop developer. I wonder if there’s some env flags that change this behavior. I never understood/learned exactly what the @ does in Swift. It looks like a JS/Python decorator, but it’s unclear if all of the annotations are owned by Swift or if developers can write their own. Where is the documentation for all of the magic variables? i.e. error in a catch block? Open Source https://github.com/Ranchero-Software/NetNewsWire https://github.com/rxhanson/Rectangle Has automated some of the release process https://github.com/exelban/stats https://github.com/kean/PulsePro https://github.com/piemonte/Player https://github.com/cirruslabs/tart https://github.com/signalapp/Signal-iOS https://github.com/onevcat/Rainbow https://github.com/Sequel-Ace/Sequel-Ace https://github.com/HedvigInsurance/ugglan https://github.com/lvillani/chai https://github.com/halo/LinkLiar Thoughts on Swift

Swift is a really nice language. I like how it is strongly typed, but the typing system is good at inferring types when it can, so you don’t have to specify that many types. The type inference seems very good—better than TypeScript, Sorbet, and python from what I can tell.

I don’t like how there are not any imports, and how anything marked as public can clutter the global namespace. I hate this about ruby, and it’s something I think python gets very right. I wish there would be explicit imports and any package-level functions would be forced to be called with their package name. I can understand how this would get very messy with the objc stuff, but that could have been special-cased in some way.

Some of the objc interface stuff is strange, but I think the language designers did a very good job of dealing with it in a simple way.

The tooling isn’t bad but there are some strange gaps in the stdlib, largely because of the legacy cocoa infrastructure you can leverage. I found this annoying: there’s not a simple logger, there’s no built-in yaml parser, etc. The Cocoa apis have a lot of legacy decisions to deal with and they are generally a pain to use. I wish the stdlib was more expansive and designed without thinking about the legacy APIs too much.

The package manager requires you to build your application in a specific way, which is annoying, but if you follow the golden path things work in a pretty clean way. It’s nice that there is an official package manager that Apple is committed to maintaining.

After writing something simple in Swift, I found myself wishing JavaScript was Swift. It feels like JavaScript in many ways, but has less foot guns and is more simple. The language designers did a great job, and it felt fun to work in.

Continue Reading

My Experience With GitHub Codespaces

I have an older intel MacBook (2016, 2.9ghz) that I use for personal projects. My corporate machine is an M1 Macbook Pro and I love it, but I’ve been holding off on replacing my personal machine until the pro M2 comes out (hopefully soon!).

I love playing with new technology, especially developer tools, and when I got accepted to the codespace beta I couldn’t resist tinkering with it. To speed up my ancient MacBook, try some new tech, and have the ability to learn more ML/AI tooling in the future.

Summary

I largely agree with this analysis.

Codespaces are very cool. They work better than I expected—it felt like I was developing on a local machine. Given how expensive the sticker pricing is, I don’t get why you wouldn’t just buy a more powerful local machine in a corporate setting (codespaces is free for open source work). I can’t see devs being ok with a Chromebook vs MacBook pro, so the cost savings aren’t there (i.e. buy a cheaper machine and put the savings into rented codespace).

You could run a similar dockerized setup locally on the MacBook if you wanted to normalize the dev environment (which is a big benefit, esp in larger orgs). I think this is one of the best benefits of codespaces—completely documented and normalizing your development environment so it’s portable across machines.

Notes

Here are some notes & thoughts on my experience with codespaces:

Codespace is essentially a docker image running on a VM in the cloud wired up to your VS code local installation in a way that makes your experience feel like you aren’t using a remote machine. Amazingly, code, gh pr view --web, etc all work (i.e. opens a local browser) and integrate with macOS. They’ve done a decent job integrating codespaces into the native experience so you forgot If you are curious, this is done by a magic environment variable: BROWSER=/vscode/bin/linux-x64/e7f30e38c5a4efafeec8ad52861eb772a9ee4dfb/bin/helpers/browser.sh Add Development Container Configuration is the command you need to run to autogen the default .devcontainer/ config for your codespace. Your dotfiles are magically cloned to /workspaces/.codespaces/.persistedshare/dotfiles File system changes are not instantly updated in the file explorer. There is a slight delay, which is frustrating. It looks like there is a reference that has emerged after the initial beta. Lots of examples/open source code still references some of the old stuff, so you’ll have to be careful not to cargo-cult everything if you want to build things in the latest style that will be resilient to changes. /workspaces/.codespaces/shared/.env has a bunch of tokens and context about the environment. You can have multiple windows/editors against multiple folders. You can do this by cloning additional folders to /workspaces and then run code . when cd‘d in that folder. Terminal state is not restored when a codespace is paused Codespace logs are persisted to /workspaces/.codespaces/.persistedshare/EnvironmentLogbackup.txt. You can also access them via the cli gh codespace logs Some of the utilities used to communicate with your local installation of vscode are located in ~/.vscode-remote/bin/[unique sha]/bin/. It’s interesting to poke around and understand how client communication works. /workspaces/.codespaces/shared/.env-secrets contains github credentials, and other important secrets. CODESPACE_VSCODE_FOLDER is not setup in /etc/profile.d. This is injected into the environment via VS Code extension JavaScript. Therefore, this variable is not available during postCreateCommand execution. If you’ve used the remote SSH development, much of the magic that makes that work is used in a codespace. There’s a hidden .vscode folder installed on the remote machine and some binaries which run there to make VS Code work properly. Load order

I couldn’t find clear documentation on the load order: when does your code get copied to the container, when do all of the VS code tools startup on the machine, etc. https://containers.dev/implementors/spec/ for the general devcontainer specification, but it’s not too helpful.

Dockerfile. Your application code does not exist, features are not installed. Features (like brew). Each feature is effectively a bundle of shell scripts that are executed serially. Application code does not exist at this point. Post Install. Dockerfile is built, features are installed, application code exists, dotfiles are not installed. Dotfiles. At this step (and all previous steps), code (vs code cli) does not exist and has not yet been installed. Sometime after this the code binary is installed and some of the daemon-like processes that run on the remote machine are started up. From what I can tell, there’s not a single-run lifecycle hook that you can use at this stage. ASDF: Version Manager for Everything

I really like asdf conceptually: one version manager to rule them all. Consistent versions and installation methods across machines and languages. Simple and beautiful. I’ve been using it for years on Elixir, Ruby, JavaScript, and Python projects and have had a great experience.

The devcontainer image examples had a completely different runtime for each major language. What if you use multiple languages? What if your environment is more custom?

I thought it would make sense to try to use asdf across all projects, as opposed to language-specific builds.

Some notes:

If you install asdf via homebrew it will throw asdf installation files in /home/linuxbrew/.linuxbrew/Cellar/asdf/0.10.2/libexec/asdf.sh. Many tools, including ElixirLS assume that the full installation exists in ~/.asdf . This caused issues on the codespace, it seems as though the shell script to start ElixirLS was not using the default shell and did not seem to be sourcing standard environment variables. I’m guessing depending on how the extension is built it does not properly run in I ran into weird issues with pyright: poetry run pyright . returned zero errors, while running pyright. inside of poetry shell triggered a lot of errors relating to missing imports (related issue). Erlang uses devcontainers and asdf, which is a good place to look for examples.

Here’s the image I ended up building and it’s been working great across a couple of projects.

Docker Compose & Docker-in-Docker

Using docker compose (to run postgres, redis, etc) is super helpful but is not straightforward. Here’s how I got it working:

You can specify a docker-compose.yml file to be used in your devcontainer.json. This seems like a great idea until you realize that you can’t manage the other services that are started through the compose definition at all. You are "trapped" inside your application container and cannot inspect or manage the other processes at all. Most of the documentation + content out there recommends using dockerComposeFile in your devcontainer.json. This is not the best way. The more flexible approach is to install docker inside a single container. This requires a bit more setup, specifically passing additional flags to the parent docker container in order to be able to run docker. Dotfiles transformation

My dotfiles are very well documented, but were not ready for codespaces. I needed to do some work to separate out the macos specific stuff from the cross-platform compatible tools.

Here’s a great guide on how to get your dotfiles setup Thankfully, brew works on linux and has a really easy integration within codespaces. This made my life easier since my dotfiles are built around brew. Pull out packages that are system-agnostic and stick them in a Brewfile. Here’s mine. Create an install script specifically for codespaces. Here’s what mine looks like. VS Code Extensions

Sync Settings extensions are not installed automatically. You have to specify which extensions you want installed on the codespace through a separate configuration github.codespaces.defaultExtensions.

Homebrew Installation Failure

Due to old packages (or old apt-get state, not sure which) installed on the image. If you use a raw base image for your codespace, you need to ensure you run apt-get update in order for homebrew install to work properly.

Another alternative is using the dev- variant of many of the base images (here’s an example).

GPG Signing

It looks like the codespace machine calls some sort of GH API to power the GPG signing. If you have a .gitconfig in your dotfiles, it will overwrite the custom settings GitHub creates when generating the codespace machine. You’ll run into errors writing commits in this scenario.

Here’s what you need to do to fix the issue:

git config --global credential.helper /.codespaces/bin/gitcredential_github.sh git config --global gpg.program /.codespaces/bin/gh-gpgsign

You’ll also want to ensure that GPG signing is enabled for the repository you are working in. If it’s not, you’ll get the following error:

error: gpg failed to sign the data fatal: failed to write commit object

You can ensure you’ve allowed GPG access by going to your codespace settings and looking at the "GPG Verification" header.

As an aside, this was an interesting post detailing out how to debug git & gpg errors.

Awk, and other tools

The version of awk on some of the base machines seems old or significantly different than the macOS version. It wouldn’t even respond to awk --version. I installed the latest version via homebrew and it fixed an issue I was having with git fuzzy log where no commit found on line would be displayed when viewing the commit history.

I imagine other packages are old or have strange versions installed too. If you run into issues with tooling in your dotfiles that work locally, try updating underlying packages.

Shell Snippets

Here are some useful shell commands to make integrating cs with your local dev environment more simple.

# gh cli does not provide an easy way to pull the codespace machine name when inside a repo targetMachine=$(gh codespace list --repo iloveitaly/$(gh repo view --json name | jq -r ".name") --json name | jq -r '.[0].name') # copy files from local to remote machine. Note that `$RepositoryName` is a magic variable that is substituted by the gh cli gh codespace cp -e -c $targetMachine ./local_file 'remote:/workspaces/$RepositoryName/remote_file' # create a new codespace for the current repo in the pwd gh alias set cs-create --shell 'gh cs create --repo $(gh repo view --json nameWithOwner | jq -r .nameWithOwner)' Unsupported CLI Tooling

Here are some gotchas I ran into with my tooling:

zsh-notify. Macos popup when a command completes won’t work anymore. pbcopy/pbpaste doesn’t work in the terminal. You lose all of your existing shell history. There are some neat tools out there to sync shell history across machines, might be a way to fix this. Open Questions Is there more control available for codespaces generated by a pull request? Ideally, you could have a script that would run to generate sample data, spin up a web server, etc and make that web server available to the public internet in some secure way. I think vercel does this in some way, but it would be neat if this was built into GitHub, tied into VS Code, and allowed for a high level of control. I’m still in the process of learning/mastering tmux, there seemed to be some incompatibilities that I’ll need to work around. cmd+f within the integrated shell doesn’t search through scroll buffer clipboard integration doesn’t work (main reason for using tmux is keyboard scroll-buffer search and copy/paste support) pbcopy/pbpaste, which I use pretty often, doesn’t work. A good option is using something like Uniclip, but this will require some additional effort to get working. Other alternatives that might be worth investigating: https://github.com/jedisct1/piknik https://gist.github.com/dergachev/8259104 I had trouble with some specific VS Code tasks not working properly. This was due to how some tasks build the shell environment. Can you run github actions locally within the codespace? This would be super cool. Looks like it’s not possible right now, but there’s some open source tooling around this which looks interesting. There’s got to be a cleaner way to sharing a consistent ssh key with a codespace for deploys. This post had some notes around this. I’m not sure how the timeout works. What if I’m running a long-running test or some other terminal process? Will it be terminated? Is there a way to keepalive the session in some other side process? Can you mount the remote drive locally and have it available in the finder? scping files to view and manipulate locally is going to get tired fast.

Continue Reading

Learning TypeScript by Migrating Mint Transactions

Years ago, I built a chrome extension to import transactions into Mint. Mint hasn’t been updated in nearly a decade at this point, and once it stopped connecting to my bank for over two months I decided to call it quits and switch to LunchMoney which is improved frequently and has a lot of neat developer-focused features.

However, I had years of historical data in Mint and I didn’t want to lose it when I transitioned. Luckily, Mint allows you to export all of your transaction data as a CSV and LunchMoney has an API.

I’ve spent some time brushing up on my JavaScript knowledge in the past, and have since used Flow (a TypeScript competitor) in my work, but I’ve heard great things about TypeScript and wanted to see how it compared. Building a simple importer tool like this in TypeScript seems like a great learning project, especially since the official bindings for the API is written in TypeScript.

Deno vs Node

Deno looks cool. It uses V8, Rust, supports TypeScript natively, and seems to have an improved REPL experience.

I started playing around with it, but it is not backwards compatible with Node/npm packages which is a non-starter for me. It still looks pretty early in its development and adoption. I hope Deno matures and is more backwards compatible in the future!

Learning TypeScript You can’t run TypeScript directly via node (this is one of the big benefits of Deno). There are some workarounds, although they all add another layer of indirection, which is the primary downfall of the JavaScript ecosystem in my opinion. ts-node looks like the easiest solution to run TypeScript without a compilation step. npm i ts-node will enable you to execute TypeScript directly using npx ts-node the_script.ts. However, if you use ESM you can’t use ts-node. This is a known issue, and although there’s a workaround it’s ugly and it feels easier just to have a watcher compile in the background and execute the raw JS. .d.ts within repos define types on top of raw JS. This reason this is done is to allow a single package to support both standard JavaScript and TypeScript: when you are using TypeScript the .js and .d.ts files are included in the TypeScript compilation process. Use npx tsc --init to setup an initial tsconfig.json. I turned off strict mode; it’s easier to learn a new typing system without hard mode enabled. Under the hood, typescript transpiles TypeScript into JavaScript. If you attempt to debug a TypeScript file with node inspect -r ts-node/register the code will look different and it’ll be challenging to copy/paste snippets to debug your application interactively. Same applies to debugging in a GUI like VS Code. You can enable sourcemaps, but the debugger is not smart enough to map variables dynamically for you when inputting strings into the console session. This is massive bummer for me: I’m a big fan of REPL-driven development. I can’t copy/paste snippets of code between my editor + REPL, it really slows me down. Similar to most other languages with gradual typing (python, ruby, etc), there are ‘community types’ for each package. TypeScript is very popular, so many/most packages includes types within the package itself. The typing packages need to be added to package.json. There’s a nice utility to do this automatically for you. If you want to be really fancy you can overload npm i and run typesync automatically. VS Code has great support for TypeScript: you can start a watcher process which emits errors directly into your VS Code status bar via cmd+shift+b. If you make any changes to tsconfig.json you’ll need to restart your watcher process. You can define a function signature that dynamically changes based on the input. For instance, if you have a configuration object, you can change the output of the function based on the structure of that object. Additionally, you can inline-assign an object to a type, which is a nice change from other languages (ruby, python). Example of inline type assignment: {download: true} as papaparse.ParseConfig<Object>. In this case, Object is an argument into the ParseConfig type and changes the type of the resulting return value. Very neat! I ran into Element implicitly has an 'any' type because expression of type 'string' can't be used to index type 'Object'. No index signature with a parameter of type 'string' was found on type 'Object. The solution was typing a map/object/hash with theVariable: { [key: string]: any } . I couldn’t change the any type of the value without causing additional typing errors since the returning function was typed as a simple Object return. There’s a great, free, extensive book on TypeScript development.

One of the most interesting pieces of TypeScript is how fast it’s improving. Just take a look at the changelog. Even though JavaScript isn’t the most well-designed language, "One by one, they are fixing the issues, and now it is an excellent product." A language that has wide adoption will iterate it’s way to greatness. There’s a polish that only high throughput can bring to a product and it’s clear that after a very long time JavaScript is finally getting a high level of polish.

Linting with ESLint, Code Formatting with Prettier ESLint looks like the most popular JavaScript linting tool. It has lots of plugins and huge community support. You can integrate prettier with eslint, which looks like the most popular code formatting tool. VS code couldn’t run ESLint after setting it up. Had trouble loading /node_modules/espree/dist/espree.cjs. Restarting VS Code fixed the problem.

Here’s the VS Code settings.json that auto-fixed ESLint issues on save:

{ "[typescript]": { "editor.formatOnSave": true, "editor.codeActionsOnSave": { "source.fixAll.eslint": true }, }, "eslint.validate": ["javascript"] }

And here’s the .eslintrc.json which allowed ESLint, prettier, and ESM to play well together:

{ "env": { "browser": false, "es2020": true }, "extends": [ "standard", "plugin:prettier/recommended" ], "parser": "@typescript-eslint/parser", "parserOptions": { "ecmaVersion": 2020, "sourceType": "module" }, "plugins": [ "@typescript-eslint" ], "rules": { } } Module Loading

As with most things in JavaScript-land, the module definition ecosystem has a bunch of different community implementation/conventions. It’s challenging to determine what the latest-and-best way to handle module definitions is. This was a great overview and I’ve summarized my learnings below.

require() == commonjs == CJS. You can spot modules in this format by module.exports in their package. This was originally designed for backend JavaScript code. AMD == Asynchronous Module Definition. You can spot packages in this style by define(['dep1', 'dep2'], function (dep1, dep2) { at the header of the index package. Designed for frontend components. UMD == Universal Module Definition. Designed to unify AMD + CJS so both backend and frontend code could import a package. The signature at the top of the UMD-packaged module is messy and basically checks for define, module.exports, etc. import == ESM == ES Modules. This is the latest-and-greatest module system officially baked into ES6. It has wide browser adoption at this point. This is most likely what you want to use. import requires module mode in TypeScript (or other compilers) not set to commonjs. If you use ESM, your transpiled JS code will look a lot less garbled, and you’ll still be able to use the VS Code debugger. The big win here is your import variable names will be consistent with your original source, which it makes it much easier to work with a REPL. There are certain compatibility issues between ESM and the rest of the older package types. I didn’t dig into this, but buyer beware. It looks like experimental support for loading modules from a URL exist. I hope this gets baked in to the runtime. There are downsides (major security risks), but it’s great for getting the initial version of something done. This was one of the features I thought was neat about Deno: you could write a script with a single JavaScript file without creating a mess of package*, tsconfig.json, etc files in a new folder. https://unpkg.com is a great tool for loading a JS file from any repo on GitHub. You’ll get Cannot use import statement inside the Node.js REPL, alternatively use dynamic import if you try to import inside of a repl. This is a known limitation.. The workaround (when in es2020 mode) is to use await import("./out/util.js"). When importing a commonjs formatted package, you’ll probably need to import specific exports via import {SpecificExport} from 'library'. However, if the thing you want to import is just the default export you’ll run into issues and probably need to modify the core library. Here’s an example commit which fixed the issue in the LunchMoney library When importing a local file, you need to specify the .js (not the ts) in the import statement import { readCSV, prettyPrintJSON } from "./util.js"; Package Management You can install a package directly from a GitHub reference npm i lunch-money/lunch-money-js You can’t put comments in package.json, which is terrible. Lots of situations where you want to document why you are importing a specific dependency, or a specific forked version of a dependency. npm install -g npm to update to the latest npm version. By default, npm update only updates packages to the latest minor semver. Use npx npm-check-updates -u && npm i to update all packages to the latest version. This is dangerous, and only makes sense if there are a small number of packages https://openbase.com is a great tool for helping decide which package to use. JavaScript Learnings You’ll want to install underscore and use chain for data manipulation: _.chain(arr).map(...).uniq().value(). Lots of great tools you are missing from ruby or python. ES6 introduced computed property names so you can use a variable as an object key { [variableKey]: variableValue } I had trouble getting papaparse to read a local file without using a callback. I hate callbacks; here’s a promise wrapper that cleaned this up for me. Merge objects with _.extend. The dotenv package didn’t seem to parse .env with exports in the file. Got tripped up on this for a bit. require can be used to load a JSON file, not just a javascript file. Neat! There are nice iterators now! for(const i in list) There’s array destruction too const [a, b] = [1,2] Underscore JS has a nice memoize method. I hate the pattern of having a package-level variable for memoization. Just feels so ugly. There’s a in keyword that can be used with objects, but not arrays (at least in the way you’d expect). There’s a null-safe operator now. For instance, if you want to safely check a JSON blob for a field and set a default you can now do something like const accounts = json_blob?.accounts || [] You are iterate over the keys and values of an object using for (const [key, value] of Object.entries(object)) https://github.com/ccxt/ccxt is a neat project which transpiles JavaScript code into multiple languages. Hacking & Debugging The most disappointing part of the node ecosystem is the REPL experience. There are some tools that (very) slightly improve it, but there’s nothing like iPython or Pry. nbd is dead and hasn’t been updated in years node-help is dead as well and just made it slightly easier to view documentaiton. node-inspector is now included in node and basically enables you to use Chrome devtools local-repl looks neat, but also hasn’t been updated in ~year. The updated repl project wouldn’t load for me on v16. The debugging happy path seems to be using the GUI debugger. You can use toString() on a function to get the source code. Helpful alternative to show-source from ruby or ll from python. However, it has some gotchas: It’s specifically discouraged since it’s been removed from the standard Arguments and argument defaults are not specified It’s not obvious how to list local variables in the CLI debugger. There’s a seemingly undocumented exec .scope that you can run from the debugger context (but not from a repl!). You can change the target to ES6 to avoid some of the weird JS transpiling stuff, Run your script with node inspect and then before conting type breakOnUncaught to ensure that you can inspect any exceptions. I prefer terminal-based debugging, if you want to connect to a GUI (chrome or VS Code) use --inspect. There’s not a way I could find to add your own aliases to the debugging (i.e. c == continue == cont). It’s worth writing your own console.ts to generate a helpful repl environment to play with imports and some aliases defined. Unfortunately, this needs to be done on a per-project basis. You can’t redefine const variables in a repl, which makes it annoying to copy/paste code into a console. It looks like there are some hacks you can use to strip out the const and replace with a let before the copy/pasted code gets eval’d. This seems like a terrible hack and should just be a native flag added to node. In more recent versions of node (at least 16 or greater), you can use await within a repl session. If you are in a debugger session await does not work, unlike when you are a standard node repl. You cannot resolve promises and therefore cannot interact with async code. This is a known bug, will not be changed, and makes debugging async code interactively extremely hard. Very surprised this is still a limitation. console.dir is the easiest way to inspect all properties of an object within a REPL. This uses util.inspect under the hood, so you don’t need to import this package and remember the function arguments. There’s a set of functions only available in the web console. Most of these seem to model after jQuery functions. Open Questions How can I run commands without npx? Is there some shim I can add to my zsh config to conditionally load all npx-enabled bins when node_modules exists? Is there anything that can done to make the repl experience better? This is my biggest gripe with JavaScript development. https://github.com/11ways/janeway looks interesting but seems dead (no commits in over a year) This code looks like an interesting starting point to removing all const that are pasted into a repl. The number of configuration files you need to get started in a repo just to get started is insane (tsconfig.json, package*.json, .eslintc.json). Is there a better want to handle this? Some sort of single configuration file to rule them all?

Continue Reading

Building a SouthWest Price Monitor and Learning Server Side JavaScript

I originally wrote a draft of this post in early 2019. I’m spending some time learning TypeScript, so I wanted to finally get my JavaScript-related posts out of draft. Some notes and learnings here are out of date.

Both sides of our family live out of state. Over the last couple years, we’ve turned them on to credit card hacking to make visiting cheap (free). SouthWest has some awesome point bonuses on credit cards, but you can’t watch for price drops on Kayak and other flight aggregators.

After a bit of digging, I found a basic version of a tool to do just this. It’s a self-hosted bot to watch for flight cost drops so you can book (or rebook for free). I’ve been wanting to dig into server side JavaScript development, and this is the perfect excuse.

Here’s what I’d like to do:

Get the tool running somewhere simple: Heroku, Raspberry Pi, etc Convert the use of redis to mongodb. Redis isn’t a database, it’s a key-value store. But this project is using it for persistence. Why switch to MongoDB? I’ve been wanting to understand document databases a bit more. Postgres would have been easier for me, but this project is all about learning. Possibly add the option of searching for the best flight deal on a particular month

Below is a ‘learning log’ of what I discovered along the way. Let’s get started!

Learning JavaScript

As I mentioned in an earlier post, my JavaScript knowledge was very out of date (pre ES6). Some findings and musings below will be obvious to a seasoned JavaScript developer, but to someone more experienced in Ruby/Python/etc they’ll be some interesting tidbits.

Looks like express is the dominant HTTP router + server. It’s equivalent to the routing engine of Rails combined with rack and unicorn. It doesn’t seem like there are strong conventions to how you setup an express-based app. You bring your own ODM/ORM, testing library, etc. There is a consistent template/folder structure. However, express doesn’t make any assumptions about a database library, although it does support a couple of different templating languages and has a preferred default (pug). app.use adds additional middleware to the stack. Middleware is simply a function with three arguments. Very similar to rack in ruby-land or plugs in Elixir-land. There’s a part of me that loves the micro-modularity of the node/npm ecosystem, but the lack of declarative programming like DateTime.now + 1.day starts to feel really messy. The equivalent in node is (new Date()).setDate((new Date()).getDate() + 1);. Another example: there’s no built-in sortBy and sort mutates the original array. Some popular packages that solve this (moment, datefuncs, underscore, etc) and the popular choice is to just pull in these packages and use them heavily. I always find including many external decencies adds a lot of maintenance risk to your code. These packages can die, cause strange performance issues, cause weird compatibility issues with future iterations of the language, etc. The good news is the JavaScript ecosystem is so massive, the most popular packages have a very low risk of abandonment. Variable scoping is weird in debugger mode. If the variable isn’t referenced in the function, it’s not available to inspect in the debugger repl. Make sure you reference the variable to inspect/play with it in real time. Node, express, etc are not billed as full-stack web frameworks like rails. I find this super frustrating: not being able spin up a console (rails console) with your entire app’s environment loaded up is annoying. For this particular problem, it looks like the best alternative is to write your own console.js (here’s another guide) with the things you need and startup a repl. The annoying thing here is you need to manually connect to your DB and trigger the REPL after the DB connection is successful. Blitz and Redwood are solving these problems, although these didn’t exist when this post was written. It seems like node inspect + a debugger line doesn’t run the code ‘completely’. For instance, if the code runs past a mongodb.connection line it doesn’t connect. I wonder if this is because the .connection call runs async and doesn’t get a chance to execute before the debugger line is called? Is there a way to instruct the repl to execute anything in the async queue? I found that starting up a vanilla node console and requiring what you needed works better. There are some interesting utility libraries that convert all methods on an object to be promises (async). http://bluebirdjs.com/docs/api/promise.promisifyall.html Languages with declarative convenience methods are just so much nicer. args.priceHistory[args.priceHistory.length - 1] is just ugly compared to args.priceHistory.last. My time at a BigCo has helped me understand the value of typing. I still find the highest velocity developer experience is type-hinting (i.e. types are not required) combined with a linter. This lets you play with code without getting all the details hardened, but still enforces guardrails to avoid a class of production errors. I’m not seeing the value in the event-loop programming paradigm. I get how it allows you to handle more concurrent connections, but isn’t that something that should be handled by the language or in some lower level abstraction? It’s much easier to reason about code when it runs sequentially. For instance, not having object.save throw an exception right away is really annoying: I need to either use callbacks to act when the code has executed OR use async and await everywhere. I do not understand why this pattern has become so popular. https://repl.it is very cool. The idea of sending out links with a console running your code is very handy. This is used a lot in the JavaScript community. It’s fascinating to me how there’s always the 10x-er that becomes a hero of the community. https://github.com/substack has created a ridiculous number of npm packages. Think about let r = await promise as let r = null; promise.then(rr => r = rr) which is executed synchronously. Instead of hash.merge(h2) you write Object.assign({}, h2, hash). There are many unintuitive sharp edges to the language, as you learning, just googling "how to do X with JavaScript" is the best way to determine the JavaScript equivalent. http://jsnice.org is great at parsing obfuscated JS. It tries to rename variables based on the context. Very cool. ... is the splat operator used on objects It’s called the ‘rest’ operator. constructor is the magic method for class initialization Looks like function definitions within a class don’t need the function keyword Puppeteer, Proxies, and Scraping

Part of this project involved scraping information the web. Here’s some tidbits about scraping that I learned:

The node ecosystem is great for web scraping. Puppeteer is a well maintained chrome-controller package and there’s lot of sample code you can leverage to hack things together quickly. Websites have gotten very good at detecting scrapers. There are some workarounds to try to block bot detection, but if you are using a popular site, you will most likely be detected if you are using the default puppeteer installation. A common (and easy) detection method is IP address. If you are scraping from an AWS/cloud IP, you’ll be easily blocked. The way around this is a proxy to a residential IP address. Another option is to host your scraper locally on a Raspberry Pi or on your local computer. https://chrome.browserless.io cool way to test puppeteer scripts I learned a bit about web proxies. Firstly, there are a bunch of proxy protocols (SOCKS, HTTP with basic auth, etc). Different systems support different type of proxies. Package Management You can’t effectively use npm and yarn in the same project. Pick one or the other. Yarn is a more stable, more secure version of npm (but doesn’t have as many features / as much active development) module.exports lets a file expose constants to others which import the file, similar to python’s import system (but with default exports). I like this compared with ruby’s "everything is global" approach. It allows the other author to explicitly define what it wants other users to access. Npm will run pre & post scripts simply based on the name of the scripts. import Section, {SectionGroup} assigns Section to the default export of the file, and imports the SectionGroup explicitly. If you try to import something that isn’t defined in the module.exports of a file you will not get an error and will instead get an undefined value for that import. Testing tape is the test runner that this particular project used. It doesn’t look like it’s possible to run just a single test in a file without changing the test code to use test.only instead of test. The "Test Anything Protocol" is interesting http://testanything.org. Haven’t run into this before. I like consistent test output across languages. I do like how tape tests list out the status of each individual assertion. It becomes a bit verbose, but it’s helpful to see what assertions after the failing assertion succeeded or failed. VS Code + node debugging is very cool when you get it configured. You need to modify your VS Code launch.json in order to get it to work with test files. https://gist.github.com/dchowitz/83bdd807b5fa016775f98065b381ca4e#gistcomment-2204588 Debugging & Hacking

I’m a big fan of REPL driven development and I always put effort into understanding the repl environment in a language to increase development speed. Here are some tips & tricks I learned:

Tab twice (after inputting ob.) in a repl exposes everything that is available on the object under inspection. node inspect THE_FILE.js allows debugger statements to work. You can also debug remotely with chrome or with VS Code. Visual debugging is the happy path with node development, the CLI experience is poor. You don’t need to setup variables properly in the node repl. Nice! You can just a = 1 instead of let a = 1 I’ll often copy code into a live console to play around with it, but if it’s defined as const I need to restart the console and make sure I don’t copy the const part of the variable definition. That’s annoying. There’s a lot of sharp edges to the developer ergonomics. console.dir to output the entire javascript object Unlike pry you need to explicitly call repl after you hit a breakpoint when running node inspect. Also, debugger causes all promises not to resolve when testing puppeteer. https://github.com/berstend/puppeteer-extra/wiki/How-to-debug-puppeteer Cool! Navigating to about:inspect in Chrome allows you to inspect a node/puppeteer process. list is equivalent to whereami. You need to execute it explicitly with params list(5) _ exists like in ruby, but it doesn’t seem to work in a repl triggered by a debugger statement. _error is a neat feature which keeps the last exception that was thrown. .help while in a repl will output a list of "dot commands" you can use in the repl. I had a lot of trouble getting puppeteer to execute within a script executed with node inspect and paused with debugger. I’m not sure why, but I suspect it has something to do with how promises are resolved in inspect mode. You can enable await in your node console via --experimental-repl-await. This is really helpful to avoid having to write let r; promise.then(o => r) all of the time. Mongo & ODMs You’ll want to install mongo and the compass tool (brew install mongodb-compass) for GUI inspection. Running into startup problems? tail -f ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist If you had an old version of mongo install long ago, you may need to brew sevices stop mongodb-community && rm -rf /usr/local/var/mongodb && mkdir /usr/local/var/mongodb && brew services start mongodb-community -dv The connection string defaults to mongodb://localhost:27017 Mongoose looks like a well-liked JavaScript ODM for Mongo. You can think of each "row" (called a "document") as a JSON blob. You can nest things (arrays, objects, etc) in the blob. The blob is named using a UID, which is like a primary key but alphanumeric. You can do some fancy filtering that’s not possible with SQL and index specific keys on the blob. Looks like you define classes that map to tables ("schemas") but it doesn’t look like you can easily extend them. You can add individual methods to a class but you can’t extend a mongoose model class. It looks like a mongoose.connection call creates an event loop. Without closing the event loop, the process will hang. Use process.exit() to kill all event loops. Relatedly, all mongo DB calls are run async, so you’ll want to await them if you expect results synchronously. brew install mongodb-compass-community gives you a GUI to explore your mongo DB. Similar to Postico for Postgres. Open Questions How are event loops, like the one mongoose uses implemented? Is the node event loop built in Javascript or are there C-level hooks used for performance? There are lots of gaps in the default REPL experience. Is there an improved repl experience for hacking? Do Blitz/RedwoodJS/others materially improve the server side JS experience? What killer features does mongodb have? How does it compare to other document databases? Is there a real reason to use document databases now that most SQL databases have a jsonb column type with an array of json operators built in?

Continue Reading

Building a Chrome Extension to Import Transactions into Mint

I originally wrote a draft of this post in early 2019. I’ve since stopped using Mint and switched to LunchMoney. However, I’m spending some time learning TypeScript so I wanted to finally get my JavaScript-related posts out of draft.

I use Mint (although it’s rotting on the vine after being acquired by Intuit), and want to import a list of transactions from a bank account that isn’t supported. However, there’s not a way to do this through the mint UI, but there is a hack someone documented.

I know old-school JavaScript but haven’t learned ES6, and I’ve never built a Chrome extension. Building a Chrome extension to use the private mint API to import transactions from a CSV is a perfect learning project.

As I built this extension, I ‘liveblogged’ my learnings which I’ve included below. Here’s the final source code of the project.

Reverse Engineering the Mint API

The first question I needed to answer is "Can I batch import transactions using a Mint API?". There is no public API, so I wasn’t sure; I had to attempt to reverse engineer how manual transactions were added.

The hacky blog post explaining how to batch import transactions is really old, so I wanted to validate the approach myself.

First, let’s ensure that the request hitting mint’s servers look about the same as the linked blog post. I pulled this curl command from the web console when adding a transaction manually in the Mint UI:

curl 'https://mint.intuit.com/updateTransaction.xevent' \ -H 'cookie: ...' \ -H 'origin: https://mint.intuit.com' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' \ -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \ -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=REDACTED' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' \ -H 'adrum: isAjax:true' \ --data ' cashTxnType=on& mtCashSplit=on& mtCheckNo=& tag806772=0& tag897426=0& tag1109697=0& tag975947=0& tag806773=0& tag806774=0& task=txnadd& txnId=%3A0& mtType=cash& mtAccount=3444067& symbol=& note=& isInvestment=false& catId=7& category=Food%20%26%20Dining& merchant=TEST& date=04%2F12%2F2019& amount=1& mtIsExpense=true& mtCashSplitPref=1& token=REDACTED' --compressed

It looks similar, but different enough from the old blog post I found. Some notes:

%3A0 url decoded is :0 mtAccount is not the same as the redacted accountId in the referrer I wonder if there is a query that can dump the catId list. Or if you can submit without a catId and category will auto match. Do we need a token? That would be a bummer.

I tried running this exact command again locally to see if it works with the tokens embedded in the command. I’d be surprised if it did, since some of those tokens look like a server-side generated CSRF token.

But, to my surprise, it worked! Here was the result.

{"task":"txnAdd","mtType":"CASH"}

I refreshed the Mint account and the transaction appears there as well. Great! At least we know it’s possible to push the data into mint.

Now, let’s see what parameters we can eliminate to make the request as simple as possible. the tag* and token params seem like the lowest hanging fruit…

<error><code>1</code><description>Session has expired.</description><name></name><type></type></error>

Hmm, let’s try adding in the token param. That’s probably tied to the session:

{"task":"txnAdd","mtType":"CASH"}

It worked! I’m guessing the token is embedded in the page source somewhere, or it could be pulled via another HTTP call (which would be a bummer). I poked around the

<input type="hidden" id="javascript-token" value="REDACTED_HASH"/>

I ran another request from the mint UI and the token used in the request matches up. We’ll have to parse the source for #javascript-token and extract that from the page. A pain, but doable.

Digging around the console a bit more it does look like there is a category API that is called!

curl 'https://mint.intuit.com/app/getJsonData.xevent?task=categories&rnd=1555119027945' \ -H 'cookie: REDACTED' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=3871381' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' --compressed

This returns a nice JSON blob:

{ "set": [ { "data": [ { "children": [ { "isStandard": true, "id": 1405, "value": "Auto Insurance", "isL1": false } ... ], "id": 14, "value": "Auto & Transport", "isL1": true } ] } ] }

Which we can use to set the catId in the previous API which added a transaction.

After discovering each category has an ID, I wanted to test if we could remove the text representation of the category in the previous API call (this might break future iterations of the mint internal API, so may not be worth doing). Additionally, this reminds me that we also will need to set the mtAccount ID dynamically. Let’s see if this ID exists in the page source.

It looks like when you initially load the transaction view the accountId query parameter is set. However, when you click on a different account the URL fragment (everything after the #) changes to include the account that was chosen:

https://mint.intuit.com/transaction.event?accountId=3444067#location:%7B%22accountId%22%3A3436098%2C%22offset%22%3A0%2C%22typeSort%22%3A8%7D

Searching through the dynamic page source (via the elements tab) I was able to find this reference to the ID:

<a id="transactionExport" href="https://mint.intuit.com/transactionDownload.event?accountId=3436098&offset=0&comparableType=8">Export all 1797 transactions</a>

Looks like we can either pull the accountId off of the URL of the #transactionExport element or parse the URL fragment. The fragment may be a bit more work given the weird format (URL encoded params look to be prefixed by location:). It looks like mint is using very old javascript technology, so we can’t make too many assumptions about the URL fragment structure.

Now we know we can build a prototype of a mint importing tool. Here’s what we need to do:

Pull the session cookie after the user logs in Pull the #javascript-token from the page source Pull the accountId either from the URL fragment or the transactionExport Hit the https://mint.intuit.com/updateTransaction.xevent endpoint to add transactions Optionally use the category API

Now to learn how google chrome extensions work!

Building a Chrome Extension

Firstly, how can we iterate on a Chrome extension while it’s in development? I google’d "best practices building a Chrome extension":

https://usersnap.com/blog/develop-chrome-extension

https://thoughtbot.com/blog/how-to-make-a-chrome-extension

https://github.com/kippt/kippt-chrome

https://github.com/yeoman/generator-chrome-extension

@@extension_id can be used to reference your extension ID in CSS. Most likely this works for JS and HTML as well.

To publish the extension, you need to register with Google and follow some guidelines.

"If you’re building a Chrome extension which needs to interact with web pages that are loaded by users, you definitely need a content script." This is our scenario.

"And luckily testing your new extension is pretty straightforward. Once you’ve activated the “developer mode”… You can simply add your unpacked extension to your Chrome browser to test it." I want to make sure I can test the Chrome extension quickly by editing some JS and reloading the page.

"When you change or add code in your extension, just come back to this page and reload the page." sounds like we need to reload the chrome://extensions extensions page?

Looks like we can load external libraries via a CDN using the manifest JSON. Great! This will make things easier for initial development.

You can limit which domains your extension activates on. We should do this and scope it down to Mint

Looks like all we need is a folder with a JS file and a manifest. Looks easy enough.

Yeoman scaffold looks cool. Uses babel which allows you to write ES6 (which I want to learn) and implements best practices. Looks kind of updated (last commit <1yr ago).

Ok, I think I have enough information. Let’s try that scaffold out:

npm install --global yo gulp-cli bower npm WARN deprecated bower@1.8.8: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/

Eek, not good. Scaffold looks to be too old. Let’s try to search for another newer scaffold…

https://github.com/edrpls/chrome-extension-template

This one looks newer. It uses webpack which I’ve heard good things about and been wanting to understand better.

git clone git@github.com:edrpls/chrome-extension-template.git mintporter cd mintporter/ npm install

The dependency graph that npm generated was massive, but it worked. Let’s try to run the start command.

yarn start -bash: yarn: command not found

Yarn is an npm alternative. I needed to install it via brew:

brew install yarn yarn start

With yarn running, I edited the manifest.json to use the "Mintporter" name, enable dev mode on chrome://extensions/, and loaded the extension via the dist/ folder. Now it’s loading on Chrome.

Now to understand how the template is structured:

Looks like webpack is similar to bower and other JS bundling tools. entry defines the output files. Interesting, looks like there is a library to specifically help with bundling chrome extensions. https://github.com/johnagan/crx-webpack-plugin Looks like mostly babel (transpiler) packages in package.json. I was confused by the rimraf in the template. It just adds the rm command in node.

Time to learn the latest JavaScript syntax!

Learning ES6

Googled "learn latest javascript":

https://javascript.info https://developer.mozilla.org/en-US/docs/Web/JavaScript/A_re-introduction_to_JavaScript https://derickbailey.com/2017/06/06/3-features-of-es7-and-beyond-that-you-should-be-using-now/ https://www.youtube.com/watch?v=cCOL7MC4Pl0 How the javascript event loop works. I’ve always found the javascript runtime a bit strange. https://www.smashingmagazine.com/2016/07/how-to-use-arguments-and-parameters-in-ecmascript-6/

Learnings:

let is the new var, but scoped to the block. You should use this instead of var const prevents the var from being reassigned. null/undefined/typeof seem to be unchanged We finally have default parameter values! Same syntax as ruby: function fn(arg="default") The => is just a shortcut for defining a new function. Left side is a list of function arguments, the right side is the code (normally a oner-liner) to be executed. data => port.postMessage(data) is the same as function(data) { return port.postMessage(data); } () => { console.log('in-content.js - disconnected from popup'); } is similar to the above syntax, but the {} does not default to returning the evaluated value. Brackets with arrow-defined functions allow multi-line functions but require an explicit return. => without brackets will return the last value in the one-liner function by default. Arrow-functions inherit the this value of the callee. async/await is the native way to write asynchronous code. Promises are now JavaScript-native and async/await use these under the hood. You can think of them as background jobs you can easily run in the browser. Template literals now exist! Use backticks: string text ${expression} string text. You can now define classes without prototypes! Yay. Use class as the keyword, constructor as the initializer. There’s no native Class object; it’s basically syntactic sugar on top of the old prototype model. Feels a lot like CoffeeScript. get and set language keywords exist inside a class to define custom getters and setters. Protected class variables are still convention-only. Still no private properties and methods, although this is in progress. Class-methods are defined using static Mixins are still messy. No language keywords to make this simple: you need to define objects and use low-level javascript calls to copy the methods of the object in. for...of loops are an easy way to iterate through all objects in an array. ...args is the "rest" syntax which allows you to represent multiple args passed to a function as an array. Similar to the splat argument in ruby *args. import is the newer version of require. I didn’t get a good sense of when import isn’t supported in various JavaScript versions. Destructuring is a mix between pattern matching and keyword arguments. function boom({ keyword }) => function boom(obj) { keyword = obj.keyword }. When looking at a function call: boom(a, b: c) is equivilent to boom(a: a, b: c)

Now, with enough new JavaScript knowledge under my belt, I can start hacking away at the extension!

Building the Prototype You need to manually click the reload button in chrome://extensions/ to pull in a new version of the javascript. Bummer. https://www.npmjs.com/package/webpack-chrome-extension-reloader looks like it may fix the issue. npm install webpack-chrome-extension-reloader --save-dev and NODE_ENV=development yarn start. https://stackoverflow.com/questions/2963260/how-do-i-auto-reload-a-chrome-extension-im-developing and https://github.com/arikw/chrome-extensions-reloader are also interesting. Use import $ from "jquery"; to pull jQuery in using the new import syntax. Babel will automatically backport this to be supported on an old JS version. Hmm, my extension is being automatically disabled. Weird. I’m getting an error on the extension page relating to inter-page communication. Cutting out that code from the in-content file. I’m going to start with two main classes: MintContext (to pull auth) and MintIntegrator to add transactions to mint. It would be nice to have an "Import Transactions" button. Looks like we can add it to #controls-top. I’ll need a way to watch for an element to appear on the page since transactions are loaded async. This page indicates I could just jQuery’s ajaxStop, but that isn’t working. DOMSubtreeModified event looks like the old-school way of watching for changes. #product-view-root looks like the best object to observe and MutationObserver looks like the API we want to use. Also, the chrome extension reloader isn’t working, I think I need to modify how webpack is interacting with yarn. To add that to that, my extension keeps getting disabled and I need to restart chrome to allow me to re-enable it again. https://www.ghacks.net/2017/07/04/hide-chromes-disable-developer-mode-extensions-warning/ let’s try using chrome canary and see if that fixes the chrome reloading issue. Looks like we don’t need the chrome reloader extension. The webpack plugin creates a websocket and listens for changes. Sidenote: I know some folks don’t like global namespaces, but I do find it annoying that you can mutate the name of any library into whatever global object you like via require or import. Maybe I’ll discover the benefits of this later on and change my mind. After more playing around, I got a better understanding of yarn. It’s just an improved npm and works off of the commands defined in package.json => scripts. It’s meant to be more stable than NPM. webpack-chrome-extension-reloader asks you to use --watch in webpack. However, the chrome extension template is setup to use nodemon with yarn build which I’m guessing monitors the filesystem and circumvents webpack‘s --watch. I wonder if there is a way to reload the extensions outside the --watch lifecycle. You can learn which process is using a port using lsof -i tcp:3000. Discovered this while trying to get the extension reload working. https://stackoverflow.com/questions/4075287/node-express-eaddrinuse-address-already-in-use-kill-server Hmm, I can’t get the reloader to work. Going to give up and manually reload. Ugh. #body-mint is the only element that exists on $(document).ready. Watching for changes in the DOM is going to kill performance (lots of events to comb through). Let’s just use a setTimeout based approach. https://gist.github.com/chrisjhoughton/7890303 Weird. My jQuery version is really old (1.x.x) but the npm package is the latest version. There is some sort of namespace conflict happening. Reverting back to the simple import $ from 'jquery'; to eliminate the issue. Working in the console uses the mint version of jQuery as opposed to the version you bundle with the app. This makes things tricky: you can’t run code live in the console. Hmm, the #javascript-token is empty. Mint.getToken() seems to work though. I’ll try using this instead. Looks like class vars are a new-ish feature. They are causing a compilation error in the version of babel I’m using. Looking into a bit deeper, they aren’t supported in babel yet. Bummer. Running into weird scoping issues when accessing the top-level Mint object. I’m guessing this has to do with my babel config. Man, the stack of javascript transforms on top of the raw browser is still such a pain. Actually, it’s not a babel config issue: chrome extensions can access the entire DOM of the page they are on, but they can’t access the javascript runtime of that page. All javascript runs in a separate sandboxed environment. However, you can access localStorage. This means Mint.getToken() won’t work. Luckily the CSFR token is stored in the session so we can just pull it from there. Now that we have a single transaction pushing into Mint when the "Import Transactions" button that we added to the page is pressed, we want to allow a CSV to be imported. From what I could tell, there’s not any great API to allow the user to select a file and read it into a string. You need to create a hidden file input, and "click" the input during a user-initiate click event, and then when the file input value has changed use the FileReader class to read the file into a string. Bummer. https://stackoverflow.com/questions/32490959/filereader-on-input-change-jquery and https://mariusschulz.com/blog/programmatically-opening-a-file-dialog-with-javascript. Ok, I have the file dialog opening within a click event triggered by a button (so the user doesn’t need to see the file input element). I wonder if I can wrap the entire "read a file" logic into a single function which returns a call back… Cool! We can insert the file input during the click call and the dialog still opens. This allows us to wrap the file input in a single method. Frustrating: you can’t execute code in the console referencing top-level constants defined via ES6. You need to use the webpack converted versions: __WEBPACK_IMPORTED_MODULE_1_papaparse___default. This makes it hard to fiddle around with code in the console. Ideally, we could run the import task (iterating over the CSV) async. We do want to run the mint request sync so we can easily aggregate errors and blow up if something breaks mid-way through. https://petetasker.com/using-async-await-jquerys-ajax/the I was curious what the difference is between TypeScript and the latest JS spec. Looks like TypeScript is just a typed version of the latest JS implementation. https://www.quora.com/What-is-the-difference-between-TypeScript-and-JavaScript Bah! You can’t add an account in mint that isn’t tied to a bank account login. Bummer. I can use an old credit card account as a hack, but this is a unfortunate limitation. I’m curious if someone else has reverse engineered mint’s API. https://webapps.stackexchange.com/questions/11398/does-mint-com-have-an-api-to-download-data-if-not-are-any-scraping-tools-avail and https://github.com/dhleong/pepper-mint. Interesting! Let’s see if there’s any interesting parameters that we can use. After digging a bit, it doesn’t look like there’s anything we didn’t find. Although, it does look like the accountId isn’t used in the API call. Transactions imported will always just hit the global transaction list. Bummer! Passing over a category name that doesn’t match a name in Mint doesn’t do anything. I could implement fuzzy matching on inputs from the CSV to the category API, but that wouldn’t help me learn anything new (I’ve worked with string distance algorithms before). Time to clean this up and call it done! Lessons Learned

This was fun! Great to way to learn Chrome Extension Development, refresh my JavaScript toolchain knowledge, and learn the new ES6 JavaScript standard.

When doing a "hack" project, spend more time fiddling with the system to make sure it can solve the use-cases you are trying to add functionality for. In this case, it looked like transactions could be added to a specific account but if you refreshed the page everything pushed to the "global" transaction list. Would have been helpful to know this ahead of time. The layers of indirection present in JavaScript development cause issues. It would have eliminated some frustration and wasted time to remove babel and develop directly against the browser’s JS engine as opposed to layering in babel all at once. JavaScript development is powerful (adding in a custom script to a webpage that inherits cookies and localStorage allows for some interesting and powerful hacks). There’s a lot of quirks (browser differences, nuances in what extensions can and can’t do, etc) but it’s the #1 language that allows you to manipulate a common interface that everyone has easy access to. This was my first time using VS Code. There was a lot of interesting contextual information VS code was able to provide while editing files. It seems very powerful and is a lot more snappy compared to Atom. However, I felt like an infant struggling with the keyboard shortcuts and lack of some nice Atom features I’m used to. I should spend some more time learning VS Code (especially now that MS owns GitHub, I can’t imagine they’ll invest in two similar editors for that much longer). It’s super useful to have a simple sample project to work with when attempting to learn new technologies. It’s worth spending some time to think about an interesting and genuinely useful project you can work on that involves new technologies you are interested in.

Continue Reading

Building a Docker image for a Python Django application

After building a crypto index fund bot I wanted to host the application so the purchase routines would run automatically. In addition to this bot, there were a couple of other smaller applications I’ve been wanting to see if I could self-host (Monica, Storj, Duplicati).

In addition to what I’ve already been doing with my Raspberry Pi, I wanted to see if I could host a couple small utilities/applications on it, and wanted to explore docker more. A perfect learning project!

Open Source Docker Files

As with any learning project, I find it incredibly helpful to clone a bunch of repos with working code into a ~/Projects/docker so I can easily ripgrep my way through them.

https://github.com/schickling/dockerfiles/ Older, but simple Dockerfiles. Helpful to understand the basics of how to solve various problems in Docker. https://github.com/linuxserver/docker-duplicati Example of how to build a Docker image compatible with the raspberry pi. The @linuxserver group on GitHub has a lot of interesting Dockerfiles to learn from. https://github.com/monicahq/docker Docker images for a classic LAMP application. https://github.com/getsentry/sentry Image for a Python Django application https://github.com/mdn/kuma Another Python Django application example

And here’s my resulting Dockerfile for hosting the crypto index fund bot I’ve been playing with.

Learning Docker

I first ran into Docker at a Spree conference way before it was widely adopted. I remember thinking the technology sounded neat, but it was hard to imagine why you’d want to build a docker container.

It takes time for new technologies to make sense. Now docker containers are everywhere, and you can’t imagine living without them. Although I’ve used docker indirectly through Heroku, Dokku, or blindly running docker compose up on an open source project, I’ve never dug in and actually created my own docker image.

Here’s what I learned while writing my first image:

Docker has great install instructions. The repository-based install instructions did not work for me. I went the sh install script route. This guide was helpful Run sudo docker run hello-world to verify docker is working Each command in a Dockerfile generates a new ‘layer’ (intermediate container image). These layers are incrementally built upon to generate your final docker image. ENTRYPOINT always has a default of a shell, CMD is not set by default. ENTRYPOINT cannot be overwritten, CMD can when specified with a docker run command The base images are generally pretty bare. You’ll need to install the packages that you need using something like RUN apt-get update && apt-get install -y --no-install-recommends bash You’ll see set -eux at the beginning of most RUN or other shell commands executed by docker. This ensures that when one shell command fails, the failure bubbles up and the docker build fails as well. Look at the manpage for set to learn more about the specific failure codes. docker exec runs a command within an existing container, docker run creates a new container and executes the command. .dockerignore is like .gitignore but for the COPY command, which is generally used to grab your source code and stuff it in the container. This is important because each command that is run in a Dockerfile attempts to create a cache of the image at that state. If you include files in COPY that are not core to your application, and they are modified often, it will cause longer docker build times, which will slow down your development loop. If a docker command fails, you’ll get a image SHA that you can use to jump into teh container and debug its state: docker image inspect b01352c2271a dive is a really neat tool to inspect each layer of an image. Helpful for debugging container issues. It’s not possible to map a layer SHA to a Dockerfile. When the layers are pulled into your local, they aren’t tagged. Your best bet is using the FROM commands in your Dockerfile and attempting to find the source Dockerfile the tagged images were created from. However, you can publish a docker image to Docker Hub without linking it to an open source Dockerfile (this seems to be rare in practice). What are the differences between all of these base image types? The most popular ones I’ve seen are Debian (buster, stretch, etc) and Alpine. This is a good explanation. Bottom line is most likely you want debian’s latest release (right now, it’s ‘buster’). You may see ‘busybox’ referenced in Dockerfiles. For a while, alpine linux was popular. It was a slimmed-down linux base layer designed to be small (I don’t fully understand why folks are so concerned with image size). The downside is it doesn’t include important utils—like cron. This is where busybox comes in, it’s a space-efficient GNU-toolset replacement. Most likely, you should just use the full debian image and forget about busybox. However, there are cases where the busybox implementation is better and designed to play well with containerized environments. For instance, if you are running a cron (on debian, alpine makes it easier), it’s challenging to get stdout redirected to the parent process without busybox. Build your image with docker build -t your-image-name . and then run it with docker run --env-file .env -it your-image-name You’ll see rm -rf /some/cache/folder in Dockerfiles. This is to eliminate package management cache, which increases the file size of the image. apt-get clean can be used instead of rm -rf /cache/folder. I’m not sure why this is more commonly used in Dockerfiles. By default, COPY requires the source file to exist. However, you can use a glob to safely optionally copy a file COPY *external_portfolio.json ./ You can have multiple FROM statements in your file. This is helpful if you need to install two runtimes (rust and python, for example). Running Cron in a Docker Debian Container

At some point, you’ll need to run a specific command on some sort of schedule without installing a full-blown job scheduler like Resque or Celery.

The ‘easiest’ way to do that is via a simple cron entry. However, cron is not plug-n-play on docker images as I painfully discovered.

Cron is not installed by default in debian base layers. This is done to save space. Installing busybox does not install the cron component when using debian. This is probably because it’s available via the standard cron package. Here’s how to install cron on a debian-based image apt-get update && apt-get install -y --no-install-recommends cron && apt-get clean You may be wondering: why use debian? This all seems so difficult, right? In my specific scenario, I’m using the python docker image which defaults to debian. From what I understand, alpine could cause other dependency issues with python and python extensions which contain C-based extensions. You don’t need to install rsyslog in order to get stdout routed to the parent process (and therefore displayed in the docker logs). To get stdout routed to the parent process, add > /proc/1/fd/1 2>&1 at the end of your cron job definition. By default, cron uses sh not bash and does not pick up on any of the environment variables passed into the docker container. To pick up ENV vars, some people recommend executing a bash script with a login flag. This didn’t work for me. Some recommended storing ENV variables in a file and sourcing it within the cron job script. Similarly, others recommend modifying BASH_ENV in your crontab Neither of these solutions worked perfectly for me. What worked was exporting the current environment variables (being careful to handle special characters) into /etc/profile which is automatically sourced by the cron process.

Here’s my cron.sh to setup the cron schedule and execute it:

#!/bin/bash -l set -eu printenv | awk -F= '{print "export " "\""$1"\"""=""\""$2"\"" }' >> /etc/profile echo "$SCHEDULE root sh -lc '/pull/path/to/executable' > /proc/1/fd/1 2>&1" >> /etc/crontab cron -L 8 -f

It’s insane to me this isn’t more simple. Another argument for keeping docker containers as simple as possible and moving as much execution logic into your application.

Building a Dockerfile

In many cases, a repo will have multiple different dockerfiles. For instance, the Monica repo has a couple different dockerfiles for various purposes. You can specify which file to build using -f:

docker build -t monicahq/monicahq -f scripts/docker/Dockerfile

The -f argument is important, as opposed to cding into the directory with the Dockerfile, since we want many of the commands (notably COPY) to run from a specific directory on the host.

As build is running, it outputs a hash (e.g. c1861cb1ff7f) at each step. When the build fails, you can use that hash to debug the container by shelling in and poking around:

docker run -it c1861cb1ff7f bash

Note that run takes a single command. You cannot pass a shell command with arguments.

In my specific situation, my build was failing due to javascript compilation errors on the Pi. After digging into it, I realized it was going to be a major pain to build the web assets on the Raspberry Pi. I just built them locally and scp‘d them over:

cd public && scp -r css/ js/ fonts/ mix-manifest.json monica@raspberrypi.local:~/monica-source/public/

After the build is complete locally, you can use it in your docker-compose.yml:

image: monicahq/monicahq

This is helpful if you are using a docker-compose.yml with a pre-existing reference to a named/tagged (with -t) Dockerfile, but you need to patch that Dockerfile to work properly. If you can edit docker-compose.yml, a better approach is to just reference the sub-Dockerfile directly:

services: worker: build: context: . dockerfile: Dockerfile

After you’ve rebuilt your docker image (or simply edited the component Dockerfile if you are using build), here’s how to apply the changes:

docker-compose up -d --remove-orphans

I’ll detail some learnings about docker-compose in a separate blog post in the future.

Hosting on a Raspberry Pi

Raspberry Pi’s architecture (32bit ARM by default) is supported by docker. However, some software isn’t packaged to run on the Pi’s ARM architecture. Additionally, the running images on the Pi generally isn’t tested as well as a traditional EC2 instance.

I ran into lots of weird and interesting bugs hosting images on the Pi. I wouldn’t recommend it if you just want to get something working quickly.

Modifying a Dockerfile to work with Raspberry Pi

If you do choose to host an application on the Pi, you’ll inevitably run into weird execution issues. Here’s one that I ran into and how I debugged it.

There’s a great dockerfile for backing up a mysqlsql database, but it was failing for me on the Pi with the following error:

exec user process caused: exec format error

It looks like this error was caused by a missing shebang at the top of the sh files.

git clone https://github.com/schickling/dockerfiles.git schickling-dockerfiles cd schickling-dockerfiles/mysql-backup-s3/

Both install.sh and run.sh had an extra space in their shebang line.I removed the spaces and built the docker image:

docker build -t iloveitaly/mysql-backup-s3 .

I got a build error:

fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/main: temporary error (try again later) fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/community: temporary error (try again later) WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory

I jumped into the last successful build step (note that sh needed to be used instead of bash, I’m assuming this is because alpine is used as the base image and doesn’t contain bash by default):

docker run -it 186581f43b48 sh

It looks like the error is caused by a raspberrypi issue that requires updating a specific library:

wget http://ftp.de.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1_armhf.deb sudo dpkg -i libseccomp2_2.5.1-1_armhf.deb

This fixed the particular build error I was running into, but caused another one: the apk install command was referencing an old python package. I bumped the apk command and that was fixed.

At this point, docker build was running but executing the image caused a different error! This time python was complaining:

ModuleNotFoundError: No module named 'six'

With some googling it looks like that can happen if the pypi project is removed, which is what was happening in the Dockerfile script. I updated the docker file to stop removing pypi which fixed the issue.

However, when I tried to run the image with a SCHEDULE: '@daily' (in the yaml file above) I ran into a go-cron failure. The package hasn’t been updated in many years, so I’m guessing it was an incompatibility with the latest alpine version.

Instead of using that package, I opted to modify the run.sh script to use the native cron functionality. I found conflicting information about using native cron functionality:

Some claimed you needed to use complex workarounds or use some sort of wrapper (similar to the workaround described earlier in the post). I found that (a) running cron in the foreground and (b) using -d 8 (option available via busybox cron) routes all cron logs to the parent stdout so you’ll see it in the docker logs.

I rebuilt the container (docker build -t iloveitaly/mysql-backup-s3 .) and applied the modifications; finally everything was working.

I ended up trying out Storj, which is a decentralized s3-compatible storage service. It comes with a generous (150gb) free tier, and it gave me an excuse to tinker around with some dweb stuff. It worked surprisingly well.

Moral of the story: if something does go wrong (high likelihood when using a system with relatively low adoption like raspberrypi) it’s a pain to debug, the feedback loop is painful.

Thoughts on Docker

It was fun playing with Docker images and getting a feel for the ecosystem. I’ll write about docker-compose separately, but it’s a very nice abstraction on top of a raw Dockerfile. The ecosystem has consistently improved over the years and Docker has been hugely helpful in eliminating differences between development, CI, staging, and production environments.

That being said, it was surprising to me how brittle Dockerfiles were (broke easily on the Pi) and how slow it was to debug them. They also take up a ton of ram on macOS. I’m due for a new MacBook, but I have 16gb of RAM, and Docker ate up my free RAM and slowed down my computer to a halt. I can see the value in using Docker to quickly spin-up a local Redis, Postgres, etc but the speed cost for local development was too high for me.

I find it fun to play around with lower level linux system stuff, but I don’t have much patience for tinkering with it when I’m just trying to get something deployed for an application I’m building. I’m a big fan of Heroku for this reason—they build the container image(s) for you automatically with basically zero configuration on your part. If you want more control over your infrastructure, you can use the open source alternative Dokku. Or, if you still want to run Docker images manually, you can use BuildPacks to generate the docker image for you.

This is all to say, I don’t see the value in managing Dockerfiles directly unless you are a very large company who needs nuanced control over your application’s runtime environment. Definitely helpful to understand how this technology works under the hood, but I can’t see myself managing these Dockerfiles directly instead of using a Heroku-like system.

Continue Reading

Using GitHub Actions With Python, Django, Pytest, and More

GitHub actions is a powerful tool. When GitHub was first released, it felt magical. Clean, simple, extensible, and adds so much value that it felt like you should be paying for it. GitHub actions feel similarly powerful and positively affected the package ecosystem of many languages.

I finally had a chance to play around with it as part of building a crypto index fund bot. I wanted to setup a robust CI run which included linting, type checking, etc.

Here’s what I learned:

It’s not possible to test changes to GitHub actions locally. You can use the GH CLI locally to run them, but GH will use the latest version of the workflow that exists in your repo. The best workflow I found is working on a branch and then squashing the changes. You can use GitHub actions to run arbitrary scripts on a schedule. This may sound obvious, but it can be used in really interesting ways, like updating a repo everyday with the results of a script. You can setup dependabot to submit automatic package update PRs using a .github/dependabot.yml file. The action/package ecosystem seems relatively weak. The GitHub-owned actions are great and work well, but even very popular flows outside of the default action set do not seem widely used and seem to have quirks. There are some nice linting tools available with VS Code so you don’t need to remember the exact key structure of the GitHub actions yaml. Unlike docker’s depends_on, containers running in the services key, are not linked to the CI jobs in a similar way to docker compose yaml files. By ‘linked’ I’m referring to exposing ports, host IP, etc to the other images that are running your jobs. You need to explicitly define ports to expose on these service images, and they are all bound to localhost. on: workflow_dispatch does not allow you to manually trigger a workflow to run with locally modified yaml. This will only run a job in your yaml already pushed to GitHub. Matrix builds are easy to setup to run parallelized builds across different runtime/dependency versions. Here’s an example. Some details about the postgres service: Doesn’t seem like you can create new databases using the default postgres/postgres username + password pair. You must use the default database, postgres. Unlike docker, the image does not resolve the domain postgres to an IP. Use 127.0.0.1 instead. You must expose the ports using ports: otherwise redis is inaccessible. You must set the password on the image, which felt very strange to me. You’ll run into errors if you don’t do this.

Here’s an example .github/workflows/ci.yml file with the following features:

Redis & postgres services for Django ORM, Django cache, and Celery queue store support. Django test configuration specification using DJANGO_SETTINGS_MODULE. This pattern is not standard to django, here’s more information about how this works and why you probably want to use it. Database migrations against postgres using Django Package installation via Poetry Caching package installation based on VM type and SHA of the poetry/package lock file Code formatting checks using black and isort Type checking using pyright Linting using pylint Test runs using pytest name: Django CI on: workflow_dispatch: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest # each step can define `env` vars, but it's easiest to define them on the build level # if you'll add additional jobs testing the same application later (which you probably will env: DJANGO_SECRET_KEY: django-insecure-@o-)qrym-cn6_*mx8dnmy#m4*$j%8wyy+l=)va&pe)9e7@o4i) DJANGO_SETTINGS_MODULE: botweb.settings.test REDIS_URL: redis://localhost:6379 TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres # port mapping for each of these services is required otherwise it's inaccessible to the rest of the jobs services: redis: image: redis # these options are recommended by GitHub to ensure the container is fully operational before moving options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 ports: - 6379:6379 postgres: image: postgres ports: - 5432:5432 env: POSTGRES_PASSWORD: postgres steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 with: python-version: 3.9.6 # install packages via poetry and cache result so future CI runs are fast # the result is only cached if the build is successful # https://stackoverflow.com/questions/62977821/how-to-cache-poetry-install-for-github-actions - name: Install poetry uses: snok/install-poetry@v1.2.0 with: version: 1.1.8 virtualenvs-create: true virtualenvs-in-project: true - name: Load cached venv id: cached-poetry-dependencies uses: actions/cache@v2 with: path: .venv key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }} - name: Install dependencies run: poetry install if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' - name: Linting run: | source .venv/bin/activate pylint **/*.py - name: Code Formatting run: | # it's unclear to me if `set` is required to ensure errors propagate, or if that's by default in some way # the examples I found did not consistently set these options or indicate that it wasn't required set -eax source .venv/bin/activate black --version black --check . isort **/*.py -c -v - name: Setup node.js (for pyright) uses: actions/setup-node@v2.4.0 with: node-version: "12" - name: Run type checking run: | npm install -g pyright source .venv/bin/activate pyright . - name: Run DB migrations run: | source .venv/bin/activate python manage.py migrate - name: Run Tests run: | source .venv/bin/activate pytest

Continue Reading

Lessons learned building with Django, Celery, and Pytest

As someone who writes ruby professionally, I recently learned python to build a bot which buys an index of crypto using binance.

The best thing about ruby is Rails, so I wanted an excuse to try out Django and see how it compared. Adding multi-user mode to the crypto bot felt like a good enough excuse. My goal was to:

Add a model for the user that persisted to a database Cron job to kick off a job for each user, preferably using a job management library Add some tests for primary application flows Docker-compose for the DB and app admin

I’ll detail learnings around Docker in a separate post. In this post, I walk through my raw notes as I dug into the django + python ecosystem further.

(I’ve written some other learning logs in this style if you are interested)

Open Source Django Projects

I found a bunch of mature, open-source django projects that were very helpful to grep (or, ripgrep) through. Clone these into a ~/Projects/django folder so you can easily search through them locally when learning:

https://github.com/getsentry/sentry https://github.com/arrobalytics/django-ledger https://github.com/intelowlproject/IntelOwl https://github.com/mdn/kuma – manages the MDN docs https://github.com/apache/airflow https://github.com/kiwicom/kiwi-structlog-config – Advanced structlog configuration examples. More python language learnings

I learned a bunch more about the core python language. I was using the most recent (3.9) version of python at the time.

You can setup imports in __init__ to make it more convenient for users to import from your package. As of python3, you don’t need a __init__ within a folder to make it importable. You can import multiple objects in a single statement from sentry.db.models import (one, two, three) iPython can be setup to automatically reload modified code. Somehow VS Code’s python.terminal.activateEnvironment got enabled again. This does not seem to play well with poetry’s venv. I disabled it and it eliminated some weird environment stuff I was running into. When using poetry, if you specify a dependency with path in your toml, even if it’s the dev section, it still is referenced and validated when running poetry install. This can cause issues when building dockerfiles for production when still referencing local copies of a package you are modifying. It doesn’t seem like there is a way to force a non-nil value in mypy. If you are getting typing errors due to nil values assert var is not None or t.cast are the best options I found. Inline return with a condition is possible: if not array_of_dicts: return None There doesn’t seem to be a one-command way to install pristine packages. poetry env remove python && poetry env use python && poetry install looks like the best approach. I ran into this when I switched a package to reference a github branch; the package was already installed and poetry wouldn’t reinstall it from the github repo. You can copy/paste functions into a REPL with iPython, but without iPython enabled it’s very hard to copy/paste multiline chunks of code. This is a good reason to install iPython in your production deployment: makes repl debugging in production much easier. By default all arguments can be either keyword or positional. However, you can define certain parameters to be positional-only using a / in the function definition. Variable names cannot start with numbers. This may seem obvious, but when you are switching from using dicts to TypedDict you may have keys which start with a number that will only cause issues when you start to construct TypedDict instances. There is not a clean way to update TypedDicts. Looks like the easiest way is to create a brand new one or type cast a raw updated dict. Cast a union list of types to a specific type with typing.cast. Convert a string to a enum via EnumClassName('input_string') as long as your enum has str as one of its subclasses. Disable typing for a specific line with # type: ignore as an inline comment Memoize a function by specifying a variable as global and setting a default value for that variable within the python file the function is in. There is also @functools.cache included with stdlib for this that should work in most situations. mypy is a popular type checker, but there’s also pyright which is installed by default with pylance (VS Code’s python extension). pylint seems like the best linter, although I was surprised at how many different options there were. This answer helped me get it working with VS Code. Magic methods (i.e. __xyz__) are also called dunder methods. A ‘sentinel value is used to distinguish between an intentional None value and a value that indicates a failure, cache miss, no object found, etc. Think undefined vs null in Javascript. First time I heard it used to describe this pattern. The yield keyword is interesting. It returns the value provided, but the state of the function is maintained and somehow wrapped in a returned iterator. Each subsequent next will return the value of the next yield in the logic Unlike ruby, it does not seem possible to add functions to the global namespace. This is a nice feature; less instances of ‘where is this method coming from. Black code formatting is really good. I thought I wouldn’t like it, but I was wrong. The cognitive load it takes off your mind when you are writing code is more than I would have expected. Structured logging with context & ENV-customized levels

structlog is a really powerful package, but the documentation is lacking and was hard to configure. Similar to my preferred ruby logger I wanted the ability to:

Set global logging context Easily pass key/value pairs into the logger Configure log level through environment variables

Here’s the configuration which worked for me:

# utils.py import structlog from decouple import config from structlog.threadlocal import wrap_dict def setLevel(level): level = getattr(logging, level.upper()) structlog.configure( # context_class enables thread-local logging to avoid passing a log instance around # https://www.structlog.org/en/21.1.0/thread-local.html context_class=wrap_dict(dict), wrapper_class=structlog.make_filtering_bound_logger(level), cache_logger_on_first_use=True, ) log_level = config("LOG_LEVEL", default="WARN") setLevel(log_level) log = structlog.get_logger()

To add context to the logger and log a key-value pair

from utils import log log.bind(user_id=user.id) log.info("something", amount=amount) Django poetry add django to your existing project to get started. Then, poetry shell and run django-admin startproject thename to setup the project Django has an interesting set of bundled apps: activeadmin-like Swap the DB connection information in settings.py to use PG and poetry add psycopg2. Django will not create the database for you, so you need to run CREATE DATABASE <dbname>; to add it before running your migrations. The default configuration does not pull from your ENV variables. I’ve written a section below about application configuration; it was tricky for me coming from rails. django-extensions is a popular package that includes a bunch of missing functionality from the core django project. Some highlights: shell_plus, reset_db, sqlcreate. It doesn’t look like there are any generators, unlike rails or phoenix. Asset management is not included. There’s a host of options you can pick from. There’s a full-featured ORM with adaptors to multiple DBs. Here are some tips and tricks: There’s a native JSONField type which is compatible with multiple databases. Uses jsonb under the hood when postgres is in place. After you’ve defined a model, you autogen the migration code and then run the migrations. python manage.py makemigrations Then, to migrate: python manage.py migrate To get everything: User.objects.all() or User.objects.iterator() to page through them. Getting a single object: User.objects.get(id=1) Use save() on an object to update or create it Create an object in a single line using User.objects.create(kwargs) You need a project (global config) and apps (actual code that makes up the core of your application) It looks like django apps (INSTALLED_APPS) are sort of like rails engines, but much more lightweight. Apps can each have their own migrations and they are not stored in a global folder. For instance, the built-in auth application has a bunch of migrations that will run but are not included in your application source code. I found this confusing. Table names are namespaced based on which app the model is in. If you have a user model in a users app the table will be named users_user. It looks like there is a unicorn equivalent, gunicorn, that is the preferred way for running web workers. It’s not included or configured by default. Flask is a framework similar to sinatra: simple routing and rendering web framework. The app scaffolding is very lightweight. Views, models, tests, and admin UI has a standard location. Everything else is up to the user. There’s a caching system built into django, but it doesn’t support redis by default. I already have redis in place, so I don’t want to use the default adapter (memcache). There’s a package django-redis that adds redis support to django cache. django-extensions has a nifty SHELL_PLUS_PRE_IMPORTS = [("decimal", "Decimal")] setting that will auto-import additional packages for you. It’s annoying to have to import various objects just to poke around in the REPL, and this setting eliminates this friction. Use decimal objects for floats when decoding JSON

In my case, I needed to use Decimals instead of floats everywhere to avoid floating point arithmetic inaccuracies. Even $0.01 difference could cause issues when submitting orders to the crypto exchange.

This is really easy when parsing JSON directly:

requests.get(endpoint).json(parse_float=decimal.Decimal),

If you are using a JSONField to store float values, it gets more complicated. You can’t just pass parse_float to the JSONField constructor. A custom decoder must be created:

class CustomJSONDecoder(json.JSONDecoder): def __init__(self, *args, **kwargs): from decimal import Decimal kwargs["parse_float"] = Decimal super().__init__(*args, **kwargs) class YourModel(models.Model): the_field = models.JSONField(default=dict, decoder=CustomJSONDecoder) Multiple django environments

There is not a standard way of managing different environments (staging, development, test, prod) in django. I found this very confusing and wasted time attempting to figure out what the best practice was here.

Here are some tips & recommendations:

Django doesn’t come with the ability to parse database URLs. There’s an extension, dj_database_url, for this. Poetry has a built-in dev category, which can be used for packages only required for development and test packages. There are no separate test or development groups. python-dotenv seems like the best package for loading a .env file into os.environ. However, if you are building an application with multiple entrypoints (i.e. web, cli, repl, worker, etc) this gets tricky as you need to ensure load_dotenv() is called before any code which looks at os.environ. After attempting to get python-dotenv working for me, I gave decouple a shot. It’s much better: you use it’s config function to extract variables from the environment. That function ensures that .env is loaded before looking at your local os.environ. Use this package instead. By default, Django does not setup your settings.py to pull from the environment. You need to do this manually. I included some snippets below. After getting decouple in place, you’ll probably want separate configurations for different environments. The best way to do this is to set DJANGO_SETTINGS_MODULE to point to a completely separate configuration file for each environment. In your toml you can set settings path [tool.pytest.ini_options] DJANGO_SETTINGS_MODULE = "app.settings.test" to force a different environment for testing. In production, you’ll set the DJANGO_SETTINGS_MODULE to app.settings.production in the docker or heroku environment For all other environments, you’ll set DJANGO_SETTINGS_MODULE to app.settings.development in your manage.py In each of these files (app/settings/development.py, app/settings/test.py, etc) you’ll from .application import * and store all common configuration in app/settings/application.py. Here’s a working example.

Here’s how to configure django cache and celery to work with redis:

CACHES = { "default": { "BACKEND": "django_redis.cache.RedisCache", "LOCATION": config("REDIS_URL"), "OPTIONS": { "CLIENT_CLASS": "django_redis.client.DefaultClient", }, } }

Here’s how to use dj_database_url with decouple:

DATABASES = {"default": dj_database_url.parse(config("DATABASE_URL"))} Job management using Celery

Django does not come with a job queue. Celery is the most popular job queue library out there and requires redis. It looks like it will require a decent amount of config, but I chose to use it anyway to understand how it compared to Sidekiq/Resque/ActiveJob/Oban/etc.

poetry add celery --allow-prereleases (I needed a prerelease to work with the version of click I was using) If you are using redis as the broker (easier for me, since I already had it installed + running) you’ll need to poetry add redis Celery does not use manage.py so it would not load the .env file. I needed to manually run dotenv_load() at the top of your celery config. I discovered that this needed to be conditionally loaded for prod, at which point I discovered that decouple is a much better package for managing configuration. I put my celery tasks within the users application as tasks.py. You can specify a dot-path to the celery config via the CLI: celery -A users.tasks worker --loglevel=INFO You can configure celery to store results. If you do, you are responsible for clearing out results. They do not expire automatically. Celery has a built-in cron scheduler. Very nice! There’s even a nice -B option for running the scheduler within a single worker process (not recommended for prod, but nice for development). When I tried to access django models, I got some weird errors. There’s a django-specific setup process you need to run through. DJANGO_SETTINGS_MODULE needs to be set, just like in manage.py. You can’t import django-specific modules at the top of the celery config file. Celery is threaded by default. If your code is not thread safe, you’ll need to set --concurrency=1. By default, tasks do not run inline. If you want to setup an integration test for your tasks, you need either (a) run tasks in eager mode (not recommended) or (b) setup a worker thread to run tasks for you during your tests. Eager mode is not recommended for testing, since it doesn’t simulate the production environment as closely. However, running a worker thread introduces another set of issues (like database cleanup not working properly). There’s no real downside to using @shared_task instead of @app.task. It’s easier to do this from the start: less refactoring to do when your application grows. Testing

Some more learnings about working with pytest & vcr in combination with django:

Database cleaning is done automatically for you via @pytest.mark.django_db at the top of your test class. This is great: no need to pull in a separate database cleaner. To be able to run pytest which relies on django models/configuration, you need the pytest-django extension. You can stick any config that would be in pytest.ini in your toml file under [tool.pytest.ini_options] You need to setup a separate config for your database to ensure it doesn’t use the same one as your development environment. The easiest way to do this is to add DJANGO_SETTINGS_MODULE = "yourapp.settings.test" to your toml file and then override the database setup in the yourapp/settings/test.py file. You can use pytest fixtures to implement ruby-style around functions. Redis/django cache is not cleared automatically between test runs. You can do this manually via django.core.cache.clear() In a scenario where you memoize/cache a global function that isn’t tied to a class, you may need to clear the cache to avoid global state causing indeterminate test results. You can do this for a single method via clear_cache() or identify all functions with lru cache and clear them. Django has a test runner (python manage.py test). It seems very different (doesn’t support fixtures), and I ran into strange compatibility issues when using it. Use pytest instead. My thoughts on Django

I continue to be impressed with the python ecosystem. The dev tooling (linting, repls, type checking, formatting, etc) is robust, there are reasonably well-written and maintained packages for everything I needed. It seems as though most packages are better maintained than the ruby equivalents. I only once had to dive into a package and hack a change I needed into the package. That’s pretty impressive, especially since the complexity of this application grew a lot more than I expected.

Working with python is just fun and fast (two things that are very important for me!). A similar level of fun to ruby, but the language is better designed and therefore easy to read. You can tell the ecosystem has more throughput: more developers are using various packages, and therefore more configuration options and bugs worked out. This increases dev velocity which matters a ton for a small side project and even more for a small startup. I don’t see a reason why I’d use ruby if I’m not building a rails-style web application.

Rails is ruby’s killer app. It’s better than Django across a couple of dimensions:

Better defaults. Multiple environments supported out of the box. Expansive batteries-included components. Job queuing, asset management, web workers, incoming/outgoing email processing, etc. This is the biggest gap in my mind: it takes a lot more effort & decisions to get all of these components working. Since django takes a ‘bring your own application components’ approach, you don’t get the benefit of large companies like Shopify, GitHub, etc using these and working out all of the bugs for you.

The Django way seems to be a very slim feature set that can be easily augmented by additional packages. Generally, I like the unix-style single responsibility tooling, but in my experience, the integration + maintenance cost of adding 10s of packages is very high. I want my web framework to do a lot for me. Yes, I’m biased, since I’m used to rails but I do think this approach is just better for rapid application development.

This was a super fun project. Definitely learned to love python and appreciate the Django ecosystem.

What I’m missing

There were some things I missed from other languages, although the list is pretty short and nitpicky:

Source code references within docs. I love this about the ruby/elixir documentation: as you are looking at the docs for a method, you can reveal the source code for that method. It was painful to (a) jump into a ipython session (b) import the module (c) ?? module.reference to view the source code. Package documentation in Dash More & better defaults in django setup. Improved stdlib map-reduce. If you can’t fit your data transformation into a comprehension, it’s painful to write and read. You end writing for loops and appending to arrays. Format code references in the path/to/file.py:line:col format for easy click-to-open support in various editors. This drove me nuts when debugging stack traces. Improved TypedDict support. It seems this is a relatively new feature, and it shows. They are frustrating to work with. Open Questions

I hope to find an excuse to dig a bit more into the python ecosystem, specifically to learn the ML side of things. Here are some questions I still had at the end of the project:

Does numpy/pandas eliminate data manipulation pain? My biggest gripe with python is the lack of chained data manipulation operators like ruby/elixir. How does the ML/AI/data science stuff work? This was one of my primary motivations for brushing up on my python skills and I’d love to deeply explore this. How does async/await work in python? How does asset management / frontend work in django? Debugging asdf plugin issues

Although unrelated to this post, I had to debug some issues with an asdf plugin. Here’s how to do this:

Clone the asdf plugin repo locally: git clone https://github.com/asdf-community/asdf-poetry ~/Projects/ Remove the existing version of the repo ~/.asdf/plugins && rm -rf poetry Symlink the repo you cloned: ln -s ~/Projects/asdf-poetry poetry

Now all commands hitting the poetry plugin will use your custom local copy.

Continue Reading

Blocking Ads & Monitoring External Drives with Raspberry Pi

I’ve written about how I setup my raspberry pi to host time machine backups. I took my pi a bit further and set it up as a local DNS server to block ad tracking systems and, as part of my digital minimalism kick/obsession, to block distracting websites network-wide on a schedule.

Pi-hole: block ads and trackers on your network

Pi-hole is a neat project: it hosts a local DNS server on your Pi which automatically pulls in a blacklist of domains used by advertisers. The interesting side effect is you can control the blacklist programmatically, enabling you to block distracting websites on a schedule. This is perfect for my digital minimalism toolkit.

Pi-hole has an active Discourse forum. I’ve come to love these project-specific forums instead of everything being centralized on StackOverflow. Really impressed with how simple and well designed in the install process is. Run curl -sSL https://install.pi-hole.net | bash and there’s a nice CLI wizard that walks you through the process. By the end, you can You’ll need to point your DNS resolution to your pi on your router, but you can manually override your router settings in your internet config in MacOS for testing. After you have DNS resolution setup to point to the Pi, you can access the admin via http://pi.hole/admin Upgrade your pi-hole via pihole -up There’s also an interesting project which bundles wireguard (vpn) into a docker image: https://github.com/IAmStoxe/wirehole Automatically blocking distracting websites

Now to automatically block distracting websites! I have a system for aggressively blocking distracting sites on my local machine, but I wanted to extend this network-wide.

First, we’ll need two scripts to block and allow websites. Let’s call our blocking script block.sh:

#!/bin/bash blockDomains=(facebook.com www.facebook.com pinterest.com www.pinterest.com amazon.com www.amazon.com smile.amazon.com) for domain in ${blockDomains[@]}; do pihole -b $domain done

For the allow.sh just switch the pihole command in the above script to include the -d option:

pihole -b -d $domain

You’ll need to chmod +x both allow.sh and block.sh. Put the scripts in ~/Documents/. Test them locally via ./allow.sh.

Now we need to add them to cron. Run crontab -e and add these two entries:

0 21 * * * bash -l -c '/home/pi/Documents/block.sh' | logger -p cron.info 0 6 * * * bash -l -c '/home/pi/Documents/allow.sh' | logger -p cron.info

Next, make the following changes to enable a dedicated cron log file and more verbose cron logging:

# uncomment line with #cron sudo nano /etc/rsyslog.conf # add EXTRA_OPTS='-L 15'. 15 is the *sum* of the logging options that you want to enable # I found this syntax very confusing and it wasn't until I read the manpage that I realized # why my logging levels were not taken into effect. sudo nano /etc/default/cron # restart relevant services sudo service rsyslog restart sudo service cron restart # follow the new log file tail -f /var/log/cron.log What’s all this extra stuff around our script?

I wanted to see the stdout of my cron jobs in cron.log. Here’s why the extra cruft around {block,allow}.sh enables this.

The bash -l -c is important: it ensures that the pi user’s env configuration is used, which ensures the script can find pihole and other commands you might use in the script. Sourcing the user’s environment is not recommended for a ‘real’ production system, but it’s ok for our home-based pi project.

By default, the stdout of the script run in your cron definition is not sent to the parent processes stdout. Instead, it’s emailed to you (if you don’t have email configured on your pi it will land in /var/mail/pi). To me, this is insane, but I imagine this is the result of a decision made long ago and any seasoned sysadmin has this drilled into his memory.

As an aside, it is unfortunate that many ancient decisions made on a whim continue to cause wasted hours and lots of frustration to newcomers. Think of all of the lost time, or people who give up continuing to learn, because of the unneeded barriers to entry in various technologies. Ok, back to the explanation I promised!

In order to avoid having your cron job output sent to mail you need to redirect the output. | logger does this for us and sends the stdout to syslog. the -p cron.info argument sets the facility.level of the log message. Facility is a weird word used for ‘process’ or ‘log category’ and is important because it maps the log entry to the cron.log file specified in the rsyslog.conf modification we made earlier. In other words, it sets the facility of the log message so syslog can run it through its internal ruleset engine to determine which file it should go in. man logger has more nitty-gritty details about how this works.

How long will it take for these block/allow changes to take effect?

Since pi-hole uses DNS for the blacklist, the TTL on the DNS entry matters. Luckily, it’s very very short (2m) by default. This means that it will take ~2m for websites to be blocked after the scripts above run on the Pi. You can check the local-ttl value by cat /etc/dnsmasq.d/01-pihole.conf. You can also see the TTL value on a specific DNS entry via the first number under then ANSWER SECTION response when running dig google.com.

If you want to test query response times (and the response content!) between your previous DNS server and your pi-hosted DNS server you can specify a DNS server to use: dig @raspberrypi.local facebook.com. However, something funky is going on with the query response times: code>@raspberrypi.local@192.168.1.2

Continue Reading

Building a Elixir & Phoenix Application

Learning Elixir

Ever since I ran into Elixir/Phoenix through a couple of popular Hacker News posts I’ve been interested in tinkering with the language. I have a little idea for an app that I’m just motivated enough to build that Elixir would work for. I’ve document my learning process below by logging my thoughts as I learned Elixir via a ‘learning project’.

What I’m building

Here’s what I’d like to build:

Web app which detects the user’s location using the built-in location service in the browser The zip code of that location is determined (server or client-side) The zip code is handed off to a server-side process which renders a page with the zip code.

Here’s what I’ll need to learn:

Elixir programming language Phoenix application framework Managing packages and dependencies Erlang runtime architecture How client-side assets are managed in phoenix How routing in Pheonix works

I’m not going to be worried about deploying the application in this project.

This is going to be fun, let’s get started!

Learning Elixir & Pheonix

I’ve worked with Rails for a while now, so most of the conceptual mapping is going to be from Ruby => Elixir and Rails => Phoenix.

First, let’s get a basic Pheonix dev environment up and running: https://hexdocs.pm/phoenix/installation.html Wow: "An Erlang system running over one million (Erlang) processes may run one operating system process". Processes are not OS processes but are instead similar to green threads with much less overhead. Some tooling equivalents: https://thoughtbot.com/blog/elixir-for-rubyists. asdf, exenv, kiex == rbenv. Looks like asdf is the most popular replacement. Reading this through, I can see why rubyists are so angry about the pipe operator (|>). The elixir version is much different (better, actually useful) than the proposed ruby version. It takes the output of a previous function and uses it as the first input to the next function in the chain. "Function declarations support guards and multiple clauses". What does that mean? It sounds like you can define a method multiple times by defining what the argument shape looks like. Instead of a bunch of if conditions at the top of a function to change logic based on inputs, you simply define the function multiple times. Makes control flow easier to reason about. There’s some great syntactical sugar for array iteration for document <- documents == documents.each { |document| ... } "I believe Elixir and Ruby are interchangeable for simple web applications with no high-traffic or that don’t require very short response times." This has been my assumption thus far: Elixir is only really helpful when performance (specifically concurrent connections) is a critical component. We will see if this plays out as I learn more. I’d recommend creating an elixir folder and cloning all of the open-source projects I reference below into it. Makes it very easy to grep (I’d recommend ripgrep, which is much better than grep) for various API usage patterns. To install elixir: brew install elixir; elixir -v verifies that we have the minimum required erlang and elixir versions. I ran this check, we are ready to go! mix is a task runner and package manager in one (rake + bundle + bin/*). It uses dot syntax instead of a colon for subcommands: bundle exec rake db:reset => mix ecto.reset When I ran the install command for pheonix it asked for hex. Looks like bundler/rubygems for elixir. https://hex.pm Webpack is used for frontend asset management and isn’t tied into Elixir at all (which I really like). Postgres is configured as the default DB. Now can I start running through the Phoenix hello world: https://hexdocs.pm/phoenix/up_and_running.html etco == ActiveRecord, kind-of. Seems a bit more light weight. Time to setup the database! config/dev.exs is the magic file. Looks like a very Rails-like folder structure at first glance. Interesting that they have a self-signed local https setup built in. That was a huge pain in ruby-land. Looks like lib/NAME_web => app/ eex === erb and has ~same templating language https://milligram.io looks like an interesting minimalist bootstrap. This was included in the default landing page. elixir atom == ruby symbol Erlang supports hot code updates: "We didn’t need to stop and re-start the server while we made these changes." Very cool. Later on, I learned that this isn’t as cool/easy as it sounds. Most folks don’t use this unless their applications have very specific requirements. Routing (routes.ex) looks to be very similar to rails. The biggest difference is the ability to define unique middleware stacks ("pipelines") that match against specific URL routes or content-types. Later on, I realized there aren’t nearly as many configuration options compared to rails. For example, I don’t believe you use regexes to define a URL param constraint. Huh, alias seems to be like include within modules. Nope! Got this one wrong: Looks like it just makes it easier to type in a module reference. Instead of Some.Path.Object, with alias you can just use Object (without specifying the namespace). use is similar to include in ruby. mix phx.server === bundle exec rails server mix deps.get === bundle Plugs seem similar to Rails engines. Nope! Plug is just a middleware stack. Umbrella applications are similar to Rails engines. Dots . instead of double colons :: for nested modules: MyApp.TheModule == MyApp::TheModule Huh, never ran into the HTTP HEAD method before https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD Examples seem to indicate that router pipelines should be used for before_filter type of logic. Looks like a plug can be a full-blown module, or simply a function on the controller that’s called before the action starts executing. https://stackoverflow.com/questions/30958446/rails-before-filter-equivalent-in-phoenix You can setup after_action-like functionality, but it’s not as intuitive: https://elixirforum.com/t/phoenix-controllers-post-action-plugs/18267 fn is a lambda function in Elixir. Doesn’t look like there are multiple ways to do lambdas. Yay! I hated the many ways of defining anonymous functions in ruby that all worked slightly differently (procs, blocks, and lambdas). There is shorthand syntax fn(arg) -> arg.something end == &(&1.something) There’s defp, def, and defmodule. What’s the difference? After a bit of digging, these are core elements to elixir which slightly change how the methods are defined. defp is a private method, for instance. Ahh func/2 references the implementation of func with two arguments. When referencing a function you must specify the number of arguments using this syntax. use Phoenix.Endpoint references a macro, how exactly do macros work? Macros are Elixir’s metaprogramming primitive. That’s all for now, I’ll read more later. I ended up not doing any metaprogramming in my application but learned it a bit about it. It sounds like you essentially specify code you want to inject into a module by quoteing it within the defmacro __using__ function in your module. This __using__ function is automagically called when you use the module. This enables you to dynamically write the elixir code you want to include (you can think of a quote as dynamically eval’d code). Live reload for the front and backend is installed by default and "just works" when running a development server. Yay! Hated all of the config in rails around this. "It is also possible for an application to have multiple endpoints, each with its own supervision tree" sounds very cool. I’m guessing this allows for multiple applications to be developed within one codebase but to run as essentially separate running processes? Something to investigate in another project. Interesting that the SSL config is passed directly to the core phoenix endpoint configuration. I wonder if there is something like unicorn/puma in the mix? It looks like there is, Cowboy is the unicorn/puma equivalent. Ecto is not bundled in the Phoenix framework. It’s a separate project. Looks like phoenix favors a layered vs all-in-one approach, but is opinionated about what packages which are installed by default (which I like). I don’t fully understand this yet, but it looks like there is an in-memory key-value store built into OTP, which is the elixir runtime (i.e. erlang). In other words, something like Redis is built-in. What are the trade-offs here? Why use this over Redis or another key/value store? Because you can define multiple variations of a method, things like action_fallback is possible. Define error handling farther up the chain and just think about the happy path in the content of the method you are writing. Neat. "EEx is the default template system in Phoenix…It is actually part of Elixir itself" Great, so this isn’t something specific to Phoenix. This made something click for me: "pattern matching is strong typing" https://news.ycombinator.com/item?id=18842123 It seems as though one of the goals behind pattern matching + function definitions is to eliminate nested conditionals. Elixir (and probably functional programming in general) seems to favor "flat" logic: I’m not seeing many nested if statements anywhere. As I learned later, if statements are generally discouraged and hard to use as they have their own scope (you can’t modify variables in the outer scope at all). ^ ‘pins’ a variable. const in node, but slightly different because of this "matching not assignment" concept (which I don’t fully get yet). This is used a lot in Ecto queries, but I’m not sure why. Gen prefix stands for Generic NOT Generate as I thought. i.e. GenServer == Generic Server. I still don’t understand this "Let it crash" philosophy. Like, if a sub-routine of some sort fails, it would corrupt the response of any downstream logic. I can see the benefits of this for some sort of async map-reduce process, but not a standard web stack. What am I missing? After many rabbit holes, I’m ready to tackle my initial goal! I’m having a blast, it all seems very well designed: I’m getting the same feeling as when I first started learning Rails via Spree Commerce year ago. What I’m missing from Ruby

Overall, I found the built-in Elixir tooling to be top-notch. There didn’t seem to be too many obvious gaps and things generally "just worked". However, there’s some tooling from the ruby ecosystem that I was missing as I went along.

Automatically open up a REPL when an exception is thrown. In ruby, this is done via pry-rescue. Super helpful for quickly diving into the exact context where the error occurs. In Phoenix, it would be amazing if the debugging plug (which displays a page when an exception is thrown) displays the variables bound in a specific scope so I can reproduce & fix errors quickly. It would be even better if a REPL could be opened and interacted with on the exception page. better-errors does this in ruby. Given that all code in Elixir is functional, simply knowing the local variables in a specific scope would be enough to reproduce most errors and would make for a very quick debugging loop. iex -S mix phx.server feels weird. It would feel a bit nicer if there was a mix phx.console which setup IEx for you. The Allow? [Yn] prompt is annoying when I’m debugging a piece of code. It would be great if you could auto-accept require IEx; IEx.pry requests. In a debugging session, I couldn’t figure out how to navigate up and down the call stack. Is there something like pry-nav available? Scan dependencies for security issues. In ruby, this is done via bundler-audit. I couldn’t find a VS Code extension with Phoenix snippets. Built-in Structured Logging. In my experience, using structured logs is incredibly helpful in effectively debugging non-trival production systems. I’ve always found it frustrating that it’s not built-in to the language (I built one for ruby). I think it would be amazing if this was provided as an optional feature in Elixir’s logger: Logger.info "something happened", user: user.id => something happened user=1 It doesn’t seem possible to run a mix task in production when using Elixir releases. There are many scenarios where you’d want to run a misc task on production data (a report, migration, etc). In Rails-land, this has been a great tool to have to solve a myriad of operational problems when running a large-ish application. Ability to add multiple owners/authors to a hex package. This makes it challenging to hand off ownership of a package when the original creator doesn’t have the time to maintain it anymore. Coming from Rails, phoenix_html feels very limited. There are many convenience methods I’m used to in Rails that I wasn’t excited about re-implementing. In ruby, if you are working on improvements to a gem (package) you can locally override the dependency using bundle config local.gem_name ~/the_gem_path. This is a nice feature for quickly debugging packages. There’s not a built-in way to do this.

I posted about this on the Elixir forums and got helpful workarounds along with confirmations about missing functionality.

Initial impressions

I enjoyed learning Elixir! It’s a well designed language with great tooling and a very supportive community. However, it still feels too early to use for a traditional SaaS product.

Although there are packages for most needs, they just don’t have as many users as the ruby/javascript ecosystem and there’s a lot of work you’ll need to do to get any given package working for you. Phoenix is great, but it’s nowhere close to rails in terms of feature coverage and you’ll find yourself having to solve problems the Rails community has already perfected over the years. The deployment story is really poor and is not natively supported on Lambda, Heroku, etc.

There are specific use-cases where Elixir is a great choice: applications that have high concurrency and/or performance demands (i.e. chat, real-time, etc) and IoT/embedded systems (via nerves) are both situations where Elixir will shine. The Elixir language has been more carefully curated compared to ruby and continues to improve at a great velocity. It’s cool to see the creator of Elixir very active in the forums an actively listening to the users and incorporating feedback. It very much reminds me of the early days of Rails.

This is all to say, in my experience, Ruby + Rails is still the fastest way to build web applications that don’t have intense concurrency/performance requirements on day one. The ecosystem, opinionated defaults, and hardened abstractions battle-tested by large companies (Shopify, GitHub, Stripe) are just too good. The dynamic nature of the language allows for tooling (better-errors, pry-rescue, byebug, etc) that materially increases development velocity.

Other Learnings Community matters

When I first started learning how to program, Kirupa (which still exists, amazingly) was an incredible resource. Random people from the internet answered by basic programming questions. All of my initial freelancing work came from the job board. The Flash/Actionscript tutorials on the site were incredibly helpful. It was a relatively small tightly-nit community that was ready to help.

I’ve feel like we’ve lost that with StackOverflow and googling for random blog posts.

The ElixirForum.com community is awesome and has that same kind, tight-nit, open-to-beginners feel that the forums of the 90s had. I was impressed and enjoyed participating in the community.

Confirmation bias is very real

I already liked Elixir before I dug into it. It looked cool, felt hot, etc. I was looking for reasons to like it as I did this example project.

It was interesting to compare this to my experience with node. I already didn’t like Javascript as a whole and was ready to find reasons I didn’t like node.

I found them, but would I have found just as many frustrating aspects of Elixir if I didn’t have a pre-existing positive bias towards Elixir?

Managing your psychology and biases is hard, but something to be aware of in any project.

Functional programming isn’t complicated

"Functional programming" is an overloaded concept. Languages are touted as "functional programming languages", there are dedicated FP conferences, and fancy terms (like "monads") all make it harder for an outsider to understand what’s going on.

I want to write up a deep-dive on functional programming at some point, but getting started with this style of programming is very easy:

You can program in a functional style in any language. Don’t store state (or store as little as possible) in objects. This forces you to declare all inputs needed for the function as arguments, instead of sourcing variables from an instance or class variable. Writing functions that don’t depend on external state are deterministic/idempotent by default. In other words, running the function against the same set of inputs yields the same results.

Boom! You are programming in a functional style. There’s more to it, but that’s the core.

Per-language folders for easy code search

Having a set of repositories is very helpful is understanding how various libraries are used in production. I’ve found it super helpful to have a folder with any great open source applications I can find in the language I’m learning. This makes it very easy to grep for various keywords or function names to quickly understand patterns and real-world usage.

For example:

cd ~/Projects mkdir elixir cd elixir git clone https://github.com/thechangelog/changelog.com # ripgrep is a faster and much easier to use version of grep rg -F 'Repo.'

Along these lines, grep.app is a great tool for quickly searching a subset of GitHub repositories (can’t wait until GitHub fixes their code search).

Open questions

There’s a bunch of concepts I didn’t get a chance to look into. Here’s some of the open questions I’d love to tackle via another learning project:

Processes/GenServer/GenStage. Although I did work with packages that create their own processes, I didn’t work with Gen{Server,Stage} from scratch. Macros / metaprogramming. Testing. Ecto/ORM. Callbacks (what does @behaviour do?). Clusters/Nodes (connecting multiple erlang VMs together to load balance) Functional programming concepts. These were referenced around the edges but I never dug into them in a deep way. Recommended Elixir style guide. I know there’s a built-in formatter/linter, but I wonder if there’s a community-driven opinionated style guide. Background jobs. Deployment. VS Code/language server/development environment optimizations. What’s up with @spec? Is there typing coming to elixir? Supervisor trees. Built-in ETS tables. Looks like a built-in key-value store similar to redis. Resources for learning Elixir & Phoenix General https://www.sihui.io/first-impression-of-elixir/ http://crevalle.io/mistakes-rails-developers-make-in-phoenix-pt-1-background-jobs.html https://dockyard.com/blog/2016/05/02/phoenix-tips-and-tricks http://blog.plataformatec.com.br/2018/04/elixir-processes-and-this-thing-called-otp/ https://www.scopelift.co/blog/2018/3/1/phoenix-on-heroku-our-experience-getting-coinrecapio-deployed https://davidlaprade.github.io/inserting-breakpoints-in-elixir https://github.com/h4cc/awesome-elixir https://github.com/thoughtbot/constable https://thoughtbot.com/services/elixir-phoenix http://digitalfreepen.com/2017/08/16/elixir-in-depth-notes.html https://elixirforum.com/t/elixir-blog-posts/150 https://howistart.org/posts/elixir/1/index.html https://til.hashrocket.com/elixir https://extips.blackode.in https://elixirschool.com/en/ https://gist.github.com/raviwu/2e128666ef7e7325c94753097f48c500 Specific Topics Elixir with a Rubyist: http://joaomdmoura.com/articles/learn-elixir-with-a-rubyist-episode-i Debugging: https://www.youtube.com/watch?v=w4xMarVUZQ4 String interpolation: https://thepugautomatic.com/2016/01/elixir-string-interpolation-for-the-rubyist/ Regex: https://thepugautomatic.com/2016/01/pattern-matching-complex-strings/ Opinions https://journal.dedasys.com/2015/04/23/elixir-vs-erlang-a-question-of-momentum/ https://news.ycombinator.com/item?id=18838115 https://github.com/dwyl/learn-elixir/issues/102 https://news.ycombinator.com/item?id=20357055 http://underjord.io/why-am-i-interested-in-elixir.html https://adrian-philipp.com/post/why-elixir-has-great-potential Videos https://www.youtube.com/channel/UC0l2QTnO1P2iph-86HHilMQ https://www.youtube.com/channel/UCIYiFWyuEytDzyju6uXW40Q https://www.youtube.com/channel/UCKrD_GYN3iDpG_uMmADPzJQ https://www.youtube.com/channel/UC47eUBNO8KBH_V8AfowOWOw https://www.youtube.com/watch?v=srQt1NAHYC0 https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=B4rOG9Bc65Q Example Applications

Clone these for easy local grepping.

https://github.com/thechangelog/changelog.com – actively managed https://github.com/rizafahmi/elixirjobs – dead project https://github.com/poanetwork/blockscout – active https://github.com/aviacommerce/avia – looks like a zombie project. No commits in 2+ months. https://github.com/edgurgel/httparrot https://github.com/hashrocket/tilex https://github.com/yodiaditya/gps-monitoring https://github.com/ComeBike/come.bike https://github.com/getsentry/sentry-elixir

Continue Reading