Building a SouthWest Price Monitor and Learning Server Side JavaScript

I originally wrote a draft of this post in early 2019. I’m spending some time learning TypeScript, so I wanted to finally get my JavaScript-related posts out of draft. Some notes and learnings here are out of date.

Both sides of our family live out of state. Over the last couple years, we’ve turned them on to credit card hacking to make visiting cheap (free). SouthWest has some awesome point bonuses on credit cards, but you can’t watch for price drops on Kayak and other flight aggregators.

After a bit of digging, I found a basic version of a tool to do just this. It’s a self-hosted bot to watch for flight cost drops so you can book (or rebook for free). I’ve been wanting to dig into server side JavaScript development, and this is the perfect excuse.

Here’s what I’d like to do:

Get the tool running somewhere simple: Heroku, Raspberry Pi, etc Convert the use of redis to mongodb. Redis isn’t a database, it’s a key-value store. But this project is using it for persistence. Why switch to MongoDB? I’ve been wanting to understand document databases a bit more. Postgres would have been easier for me, but this project is all about learning. Possibly add the option of searching for the best flight deal on a particular month

Below is a ‘learning log’ of what I discovered along the way. Let’s get started!

Learning JavaScript

As I mentioned in an earlier post, my JavaScript knowledge was very out of date (pre ES6). Some findings and musings below will be obvious to a seasoned JavaScript developer, but to someone more experienced in Ruby/Python/etc they’ll be some interesting tidbits.

Looks like express is the dominant HTTP router + server. It’s equivalent to the routing engine of Rails combined with rack and unicorn. It doesn’t seem like there are strong conventions to how you setup an express-based app. You bring your own ODM/ORM, testing library, etc. There is a consistent template/folder structure. However, express doesn’t make any assumptions about a database library, although it does support a couple of different templating languages and has a preferred default (pug). app.use adds additional middleware to the stack. Middleware is simply a function with three arguments. Very similar to rack in ruby-land or plugs in Elixir-land. There’s a part of me that loves the micro-modularity of the node/npm ecosystem, but the lack of declarative programming like DateTime.now + 1.day starts to feel really messy. The equivalent in node is (new Date()).setDate((new Date()).getDate() + 1);. Another example: there’s no built-in sortBy and sort mutates the original array. Some popular packages that solve this (moment, datefuncs, underscore, etc) and the popular choice is to just pull in these packages and use them heavily. I always find including many external decencies adds a lot of maintenance risk to your code. These packages can die, cause strange performance issues, cause weird compatibility issues with future iterations of the language, etc. The good news is the JavaScript ecosystem is so massive, the most popular packages have a very low risk of abandonment. Variable scoping is weird in debugger mode. If the variable isn’t referenced in the function, it’s not available to inspect in the debugger repl. Make sure you reference the variable to inspect/play with it in real time. Node, express, etc are not billed as full-stack web frameworks like rails. I find this super frustrating: not being able spin up a console (rails console) with your entire app’s environment loaded up is annoying. For this particular problem, it looks like the best alternative is to write your own console.js (here’s another guide) with the things you need and startup a repl. The annoying thing here is you need to manually connect to your DB and trigger the REPL after the DB connection is successful. Blitz and Redwood are solving these problems, although these didn’t exist when this post was written. It seems like node inspect + a debugger line doesn’t run the code ‘completely’. For instance, if the code runs past a mongodb.connection line it doesn’t connect. I wonder if this is because the .connection call runs async and doesn’t get a chance to execute before the debugger line is called? Is there a way to instruct the repl to execute anything in the async queue? I found that starting up a vanilla node console and requiring what you needed works better. There are some interesting utility libraries that convert all methods on an object to be promises (async). http://bluebirdjs.com/docs/api/promise.promisifyall.html Languages with declarative convenience methods are just so much nicer. args.priceHistory[args.priceHistory.length - 1] is just ugly compared to args.priceHistory.last. My time at a BigCo has helped me understand the value of typing. I still find the highest velocity developer experience is type-hinting (i.e. types are not required) combined with a linter. This lets you play with code without getting all the details hardened, but still enforces guardrails to avoid a class of production errors. I’m not seeing the value in the event-loop programming paradigm. I get how it allows you to handle more concurrent connections, but isn’t that something that should be handled by the language or in some lower level abstraction? It’s much easier to reason about code when it runs sequentially. For instance, not having object.save throw an exception right away is really annoying: I need to either use callbacks to act when the code has executed OR use async and await everywhere. I do not understand why this pattern has become so popular. https://repl.it is very cool. The idea of sending out links with a console running your code is very handy. This is used a lot in the JavaScript community. It’s fascinating to me how there’s always the 10x-er that becomes a hero of the community. https://github.com/substack has created a ridiculous number of npm packages. Think about let r = await promise as let r = null; promise.then(rr => r = rr) which is executed synchronously. Instead of hash.merge(h2) you write Object.assign({}, h2, hash). There are many unintuitive sharp edges to the language, as you learning, just googling "how to do X with JavaScript" is the best way to determine the JavaScript equivalent. http://jsnice.org is great at parsing obfuscated JS. It tries to rename variables based on the context. Very cool. ... is the splat operator used on objects It’s called the ‘rest’ operator. constructor is the magic method for class initialization Looks like function definitions within a class don’t need the function keyword Puppeteer, Proxies, and Scraping

Part of this project involved scraping information the web. Here’s some tidbits about scraping that I learned:

The node ecosystem is great for web scraping. Puppeteer is a well maintained chrome-controller package and there’s lot of sample code you can leverage to hack things together quickly. Websites have gotten very good at detecting scrapers. There are some workarounds to try to block bot detection, but if you are using a popular site, you will most likely be detected if you are using the default puppeteer installation. A common (and easy) detection method is IP address. If you are scraping from an AWS/cloud IP, you’ll be easily blocked. The way around this is a proxy to a residential IP address. Another option is to host your scraper locally on a Raspberry Pi or on your local computer. https://chrome.browserless.io cool way to test puppeteer scripts I learned a bit about web proxies. Firstly, there are a bunch of proxy protocols (SOCKS, HTTP with basic auth, etc). Different systems support different type of proxies. Package Management You can’t effectively use npm and yarn in the same project. Pick one or the other. Yarn is a more stable, more secure version of npm (but doesn’t have as many features / as much active development) module.exports lets a file expose constants to others which import the file, similar to python’s import system (but with default exports). I like this compared with ruby’s "everything is global" approach. It allows the other author to explicitly define what it wants other users to access. Npm will run pre & post scripts simply based on the name of the scripts. import Section, {SectionGroup} assigns Section to the default export of the file, and imports the SectionGroup explicitly. If you try to import something that isn’t defined in the module.exports of a file you will not get an error and will instead get an undefined value for that import. Testing tape is the test runner that this particular project used. It doesn’t look like it’s possible to run just a single test in a file without changing the test code to use test.only instead of test. The "Test Anything Protocol" is interesting http://testanything.org. Haven’t run into this before. I like consistent test output across languages. I do like how tape tests list out the status of each individual assertion. It becomes a bit verbose, but it’s helpful to see what assertions after the failing assertion succeeded or failed. VS Code + node debugging is very cool when you get it configured. You need to modify your VS Code launch.json in order to get it to work with test files. https://gist.github.com/dchowitz/83bdd807b5fa016775f98065b381ca4e#gistcomment-2204588 Debugging & Hacking

I’m a big fan of REPL driven development and I always put effort into understanding the repl environment in a language to increase development speed. Here are some tips & tricks I learned:

Tab twice (after inputting ob.) in a repl exposes everything that is available on the object under inspection. node inspect THE_FILE.js allows debugger statements to work. You can also debug remotely with chrome or with VS Code. Visual debugging is the happy path with node development, the CLI experience is poor. You don’t need to setup variables properly in the node repl. Nice! You can just a = 1 instead of let a = 1 I’ll often copy code into a live console to play around with it, but if it’s defined as const I need to restart the console and make sure I don’t copy the const part of the variable definition. That’s annoying. There’s a lot of sharp edges to the developer ergonomics. console.dir to output the entire javascript object Unlike pry you need to explicitly call repl after you hit a breakpoint when running node inspect. Also, debugger causes all promises not to resolve when testing puppeteer. https://github.com/berstend/puppeteer-extra/wiki/How-to-debug-puppeteer Cool! Navigating to about:inspect in Chrome allows you to inspect a node/puppeteer process. list is equivalent to whereami. You need to execute it explicitly with params list(5) _ exists like in ruby, but it doesn’t seem to work in a repl triggered by a debugger statement. _error is a neat feature which keeps the last exception that was thrown. .help while in a repl will output a list of "dot commands" you can use in the repl. I had a lot of trouble getting puppeteer to execute within a script executed with node inspect and paused with debugger. I’m not sure why, but I suspect it has something to do with how promises are resolved in inspect mode. You can enable await in your node console via --experimental-repl-await. This is really helpful to avoid having to write let r; promise.then(o => r) all of the time. Mongo & ODMs You’ll want to install mongo and the compass tool (brew install mongodb-compass) for GUI inspection. Running into startup problems? tail -f ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist If you had an old version of mongo install long ago, you may need to brew sevices stop mongodb-community && rm -rf /usr/local/var/mongodb && mkdir /usr/local/var/mongodb && brew services start mongodb-community -dv The connection string defaults to mongodb://localhost:27017 Mongoose looks like a well-liked JavaScript ODM for Mongo. You can think of each "row" (called a "document") as a JSON blob. You can nest things (arrays, objects, etc) in the blob. The blob is named using a UID, which is like a primary key but alphanumeric. You can do some fancy filtering that’s not possible with SQL and index specific keys on the blob. Looks like you define classes that map to tables ("schemas") but it doesn’t look like you can easily extend them. You can add individual methods to a class but you can’t extend a mongoose model class. It looks like a mongoose.connection call creates an event loop. Without closing the event loop, the process will hang. Use process.exit() to kill all event loops. Relatedly, all mongo DB calls are run async, so you’ll want to await them if you expect results synchronously. brew install mongodb-compass-community gives you a GUI to explore your mongo DB. Similar to Postico for Postgres. Open Questions How are event loops, like the one mongoose uses implemented? Is the node event loop built in Javascript or are there C-level hooks used for performance? There are lots of gaps in the default REPL experience. Is there an improved repl experience for hacking? Do Blitz/RedwoodJS/others materially improve the server side JS experience? What killer features does mongodb have? How does it compare to other document databases? Is there a real reason to use document databases now that most SQL databases have a jsonb column type with an array of json operators built in?

Continue Reading

Building a Chrome Extension to Import Transactions into Mint

I originally wrote a draft of this post in early 2019. I’ve since stopped using Mint and switched to LunchMoney. However, I’m spending some time learning TypeScript so I wanted to finally get my JavaScript-related posts out of draft.

I use Mint (although it’s rotting on the vine after being acquired by Intuit), and want to import a list of transactions from a bank account that isn’t supported. However, there’s not a way to do this through the mint UI, but there is a hack someone documented.

I know old-school JavaScript but haven’t learned ES6, and I’ve never built a Chrome extension. Building a Chrome extension to use the private mint API to import transactions from a CSV is a perfect learning project.

As I built this extension, I ‘liveblogged’ my learnings which I’ve included below. Here’s the final source code of the project.

Reverse Engineering the Mint API

The first question I needed to answer is "Can I batch import transactions using a Mint API?". There is no public API, so I wasn’t sure; I had to attempt to reverse engineer how manual transactions were added.

The hacky blog post explaining how to batch import transactions is really old, so I wanted to validate the approach myself.

First, let’s ensure that the request hitting mint’s servers look about the same as the linked blog post. I pulled this curl command from the web console when adding a transaction manually in the Mint UI:

curl 'https://mint.intuit.com/updateTransaction.xevent' \ -H 'cookie: ...' \ -H 'origin: https://mint.intuit.com' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' \ -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \ -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=REDACTED' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' \ -H 'adrum: isAjax:true' \ --data ' cashTxnType=on& mtCashSplit=on& mtCheckNo=& tag806772=0& tag897426=0& tag1109697=0& tag975947=0& tag806773=0& tag806774=0& task=txnadd& txnId=%3A0& mtType=cash& mtAccount=3444067& symbol=& note=& isInvestment=false& catId=7& category=Food%20%26%20Dining& merchant=TEST& date=04%2F12%2F2019& amount=1& mtIsExpense=true& mtCashSplitPref=1& token=REDACTED' --compressed

It looks similar, but different enough from the old blog post I found. Some notes:

%3A0 url decoded is :0 mtAccount is not the same as the redacted accountId in the referrer I wonder if there is a query that can dump the catId list. Or if you can submit without a catId and category will auto match. Do we need a token? That would be a bummer.

I tried running this exact command again locally to see if it works with the tokens embedded in the command. I’d be surprised if it did, since some of those tokens look like a server-side generated CSRF token.

But, to my surprise, it worked! Here was the result.

{"task":"txnAdd","mtType":"CASH"}

I refreshed the Mint account and the transaction appears there as well. Great! At least we know it’s possible to push the data into mint.

Now, let’s see what parameters we can eliminate to make the request as simple as possible. the tag* and token params seem like the lowest hanging fruit…

<error><code>1</code><description>Session has expired.</description><name></name><type></type></error>

Hmm, let’s try adding in the token param. That’s probably tied to the session:

{"task":"txnAdd","mtType":"CASH"}

It worked! I’m guessing the token is embedded in the page source somewhere, or it could be pulled via another HTTP call (which would be a bummer). I poked around the

<input type="hidden" id="javascript-token" value="REDACTED_HASH"/>

I ran another request from the mint UI and the token used in the request matches up. We’ll have to parse the source for #javascript-token and extract that from the page. A pain, but doable.

Digging around the console a bit more it does look like there is a category API that is called!

curl 'https://mint.intuit.com/app/getJsonData.xevent?task=categories&rnd=1555119027945' \ -H 'cookie: REDACTED' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=3871381' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' --compressed

This returns a nice JSON blob:

{ "set": [ { "data": [ { "children": [ { "isStandard": true, "id": 1405, "value": "Auto Insurance", "isL1": false } ... ], "id": 14, "value": "Auto & Transport", "isL1": true } ] } ] }

Which we can use to set the catId in the previous API which added a transaction.

After discovering each category has an ID, I wanted to test if we could remove the text representation of the category in the previous API call (this might break future iterations of the mint internal API, so may not be worth doing). Additionally, this reminds me that we also will need to set the mtAccount ID dynamically. Let’s see if this ID exists in the page source.

It looks like when you initially load the transaction view the accountId query parameter is set. However, when you click on a different account the URL fragment (everything after the #) changes to include the account that was chosen:

https://mint.intuit.com/transaction.event?accountId=3444067#location:%7B%22accountId%22%3A3436098%2C%22offset%22%3A0%2C%22typeSort%22%3A8%7D

Searching through the dynamic page source (via the elements tab) I was able to find this reference to the ID:

<a id="transactionExport" href="https://mint.intuit.com/transactionDownload.event?accountId=3436098&offset=0&comparableType=8">Export all 1797 transactions</a>

Looks like we can either pull the accountId off of the URL of the #transactionExport element or parse the URL fragment. The fragment may be a bit more work given the weird format (URL encoded params look to be prefixed by location:). It looks like mint is using very old javascript technology, so we can’t make too many assumptions about the URL fragment structure.

Now we know we can build a prototype of a mint importing tool. Here’s what we need to do:

Pull the session cookie after the user logs in Pull the #javascript-token from the page source Pull the accountId either from the URL fragment or the transactionExport Hit the https://mint.intuit.com/updateTransaction.xevent endpoint to add transactions Optionally use the category API

Now to learn how google chrome extensions work!

Building a Chrome Extension

Firstly, how can we iterate on a Chrome extension while it’s in development? I google’d "best practices building a Chrome extension":

https://usersnap.com/blog/develop-chrome-extension

https://thoughtbot.com/blog/how-to-make-a-chrome-extension

https://github.com/kippt/kippt-chrome

https://github.com/yeoman/generator-chrome-extension

@@extension_id can be used to reference your extension ID in CSS. Most likely this works for JS and HTML as well.

To publish the extension, you need to register with Google and follow some guidelines.

"If you’re building a Chrome extension which needs to interact with web pages that are loaded by users, you definitely need a content script." This is our scenario.

"And luckily testing your new extension is pretty straightforward. Once you’ve activated the “developer mode”… You can simply add your unpacked extension to your Chrome browser to test it." I want to make sure I can test the Chrome extension quickly by editing some JS and reloading the page.

"When you change or add code in your extension, just come back to this page and reload the page." sounds like we need to reload the chrome://extensions extensions page?

Looks like we can load external libraries via a CDN using the manifest JSON. Great! This will make things easier for initial development.

You can limit which domains your extension activates on. We should do this and scope it down to Mint

Looks like all we need is a folder with a JS file and a manifest. Looks easy enough.

Yeoman scaffold looks cool. Uses babel which allows you to write ES6 (which I want to learn) and implements best practices. Looks kind of updated (last commit <1yr ago).

Ok, I think I have enough information. Let’s try that scaffold out:

npm install --global yo gulp-cli bower npm WARN deprecated bower@1.8.8: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/

Eek, not good. Scaffold looks to be too old. Let’s try to search for another newer scaffold…

https://github.com/edrpls/chrome-extension-template

This one looks newer. It uses webpack which I’ve heard good things about and been wanting to understand better.

git clone git@github.com:edrpls/chrome-extension-template.git mintporter cd mintporter/ npm install

The dependency graph that npm generated was massive, but it worked. Let’s try to run the start command.

yarn start -bash: yarn: command not found

Yarn is an npm alternative. I needed to install it via brew:

brew install yarn yarn start

With yarn running, I edited the manifest.json to use the "Mintporter" name, enable dev mode on chrome://extensions/, and loaded the extension via the dist/ folder. Now it’s loading on Chrome.

Now to understand how the template is structured:

Looks like webpack is similar to bower and other JS bundling tools. entry defines the output files. Interesting, looks like there is a library to specifically help with bundling chrome extensions. https://github.com/johnagan/crx-webpack-plugin Looks like mostly babel (transpiler) packages in package.json. I was confused by the rimraf in the template. It just adds the rm command in node.

Time to learn the latest JavaScript syntax!

Learning ES6

Googled "learn latest javascript":

https://javascript.info https://developer.mozilla.org/en-US/docs/Web/JavaScript/A_re-introduction_to_JavaScript https://derickbailey.com/2017/06/06/3-features-of-es7-and-beyond-that-you-should-be-using-now/ https://www.youtube.com/watch?v=cCOL7MC4Pl0 How the javascript event loop works. I’ve always found the javascript runtime a bit strange. https://www.smashingmagazine.com/2016/07/how-to-use-arguments-and-parameters-in-ecmascript-6/

Learnings:

let is the new var, but scoped to the block. You should use this instead of var const prevents the var from being reassigned. null/undefined/typeof seem to be unchanged We finally have default parameter values! Same syntax as ruby: function fn(arg="default") The => is just a shortcut for defining a new function. Left side is a list of function arguments, the right side is the code (normally a oner-liner) to be executed. data => port.postMessage(data) is the same as function(data) { return port.postMessage(data); } () => { console.log('in-content.js - disconnected from popup'); } is similar to the above syntax, but the {} does not default to returning the evaluated value. Brackets with arrow-defined functions allow multi-line functions but require an explicit return. => without brackets will return the last value in the one-liner function by default. Arrow-functions inherit the this value of the callee. async/await is the native way to write asynchronous code. Promises are now JavaScript-native and async/await use these under the hood. You can think of them as background jobs you can easily run in the browser. Template literals now exist! Use backticks: string text ${expression} string text. You can now define classes without prototypes! Yay. Use class as the keyword, constructor as the initializer. There’s no native Class object; it’s basically syntactic sugar on top of the old prototype model. Feels a lot like CoffeeScript. get and set language keywords exist inside a class to define custom getters and setters. Protected class variables are still convention-only. Still no private properties and methods, although this is in progress. Class-methods are defined using static Mixins are still messy. No language keywords to make this simple: you need to define objects and use low-level javascript calls to copy the methods of the object in. for...of loops are an easy way to iterate through all objects in an array. ...args is the "rest" syntax which allows you to represent multiple args passed to a function as an array. Similar to the splat argument in ruby *args. import is the newer version of require. I didn’t get a good sense of when import isn’t supported in various JavaScript versions. Destructuring is a mix between pattern matching and keyword arguments. function boom({ keyword }) => function boom(obj) { keyword = obj.keyword }. When looking at a function call: boom(a, b: c) is equivilent to boom(a: a, b: c)

Now, with enough new JavaScript knowledge under my belt, I can start hacking away at the extension!

Building the Prototype You need to manually click the reload button in chrome://extensions/ to pull in a new version of the javascript. Bummer. https://www.npmjs.com/package/webpack-chrome-extension-reloader looks like it may fix the issue. npm install webpack-chrome-extension-reloader --save-dev and NODE_ENV=development yarn start. https://stackoverflow.com/questions/2963260/how-do-i-auto-reload-a-chrome-extension-im-developing and https://github.com/arikw/chrome-extensions-reloader are also interesting. Use import $ from "jquery"; to pull jQuery in using the new import syntax. Babel will automatically backport this to be supported on an old JS version. Hmm, my extension is being automatically disabled. Weird. I’m getting an error on the extension page relating to inter-page communication. Cutting out that code from the in-content file. I’m going to start with two main classes: MintContext (to pull auth) and MintIntegrator to add transactions to mint. It would be nice to have an "Import Transactions" button. Looks like we can add it to #controls-top. I’ll need a way to watch for an element to appear on the page since transactions are loaded async. This page indicates I could just jQuery’s ajaxStop, but that isn’t working. DOMSubtreeModified event looks like the old-school way of watching for changes. #product-view-root looks like the best object to observe and MutationObserver looks like the API we want to use. Also, the chrome extension reloader isn’t working, I think I need to modify how webpack is interacting with yarn. To add that to that, my extension keeps getting disabled and I need to restart chrome to allow me to re-enable it again. https://www.ghacks.net/2017/07/04/hide-chromes-disable-developer-mode-extensions-warning/ let’s try using chrome canary and see if that fixes the chrome reloading issue. Looks like we don’t need the chrome reloader extension. The webpack plugin creates a websocket and listens for changes. Sidenote: I know some folks don’t like global namespaces, but I do find it annoying that you can mutate the name of any library into whatever global object you like via require or import. Maybe I’ll discover the benefits of this later on and change my mind. After more playing around, I got a better understanding of yarn. It’s just an improved npm and works off of the commands defined in package.json => scripts. It’s meant to be more stable than NPM. webpack-chrome-extension-reloader asks you to use --watch in webpack. However, the chrome extension template is setup to use nodemon with yarn build which I’m guessing monitors the filesystem and circumvents webpack‘s --watch. I wonder if there is a way to reload the extensions outside the --watch lifecycle. You can learn which process is using a port using lsof -i tcp:3000. Discovered this while trying to get the extension reload working. https://stackoverflow.com/questions/4075287/node-express-eaddrinuse-address-already-in-use-kill-server Hmm, I can’t get the reloader to work. Going to give up and manually reload. Ugh. #body-mint is the only element that exists on $(document).ready. Watching for changes in the DOM is going to kill performance (lots of events to comb through). Let’s just use a setTimeout based approach. https://gist.github.com/chrisjhoughton/7890303 Weird. My jQuery version is really old (1.x.x) but the npm package is the latest version. There is some sort of namespace conflict happening. Reverting back to the simple import $ from 'jquery'; to eliminate the issue. Working in the console uses the mint version of jQuery as opposed to the version you bundle with the app. This makes things tricky: you can’t run code live in the console. Hmm, the #javascript-token is empty. Mint.getToken() seems to work though. I’ll try using this instead. Looks like class vars are a new-ish feature. They are causing a compilation error in the version of babel I’m using. Looking into a bit deeper, they aren’t supported in babel yet. Bummer. Running into weird scoping issues when accessing the top-level Mint object. I’m guessing this has to do with my babel config. Man, the stack of javascript transforms on top of the raw browser is still such a pain. Actually, it’s not a babel config issue: chrome extensions can access the entire DOM of the page they are on, but they can’t access the javascript runtime of that page. All javascript runs in a separate sandboxed environment. However, you can access localStorage. This means Mint.getToken() won’t work. Luckily the CSFR token is stored in the session so we can just pull it from there. Now that we have a single transaction pushing into Mint when the "Import Transactions" button that we added to the page is pressed, we want to allow a CSV to be imported. From what I could tell, there’s not any great API to allow the user to select a file and read it into a string. You need to create a hidden file input, and "click" the input during a user-initiate click event, and then when the file input value has changed use the FileReader class to read the file into a string. Bummer. https://stackoverflow.com/questions/32490959/filereader-on-input-change-jquery and https://mariusschulz.com/blog/programmatically-opening-a-file-dialog-with-javascript. Ok, I have the file dialog opening within a click event triggered by a button (so the user doesn’t need to see the file input element). I wonder if I can wrap the entire "read a file" logic into a single function which returns a call back… Cool! We can insert the file input during the click call and the dialog still opens. This allows us to wrap the file input in a single method. Frustrating: you can’t execute code in the console referencing top-level constants defined via ES6. You need to use the webpack converted versions: __WEBPACK_IMPORTED_MODULE_1_papaparse___default. This makes it hard to fiddle around with code in the console. Ideally, we could run the import task (iterating over the CSV) async. We do want to run the mint request sync so we can easily aggregate errors and blow up if something breaks mid-way through. https://petetasker.com/using-async-await-jquerys-ajax/the I was curious what the difference is between TypeScript and the latest JS spec. Looks like TypeScript is just a typed version of the latest JS implementation. https://www.quora.com/What-is-the-difference-between-TypeScript-and-JavaScript Bah! You can’t add an account in mint that isn’t tied to a bank account login. Bummer. I can use an old credit card account as a hack, but this is a unfortunate limitation. I’m curious if someone else has reverse engineered mint’s API. https://webapps.stackexchange.com/questions/11398/does-mint-com-have-an-api-to-download-data-if-not-are-any-scraping-tools-avail and https://github.com/dhleong/pepper-mint. Interesting! Let’s see if there’s any interesting parameters that we can use. After digging a bit, it doesn’t look like there’s anything we didn’t find. Although, it does look like the accountId isn’t used in the API call. Transactions imported will always just hit the global transaction list. Bummer! Passing over a category name that doesn’t match a name in Mint doesn’t do anything. I could implement fuzzy matching on inputs from the CSV to the category API, but that wouldn’t help me learn anything new (I’ve worked with string distance algorithms before). Time to clean this up and call it done! Lessons Learned

This was fun! Great to way to learn Chrome Extension Development, refresh my JavaScript toolchain knowledge, and learn the new ES6 JavaScript standard.

When doing a "hack" project, spend more time fiddling with the system to make sure it can solve the use-cases you are trying to add functionality for. In this case, it looked like transactions could be added to a specific account but if you refreshed the page everything pushed to the "global" transaction list. Would have been helpful to know this ahead of time. The layers of indirection present in JavaScript development cause issues. It would have eliminated some frustration and wasted time to remove babel and develop directly against the browser’s JS engine as opposed to layering in babel all at once. JavaScript development is powerful (adding in a custom script to a webpage that inherits cookies and localStorage allows for some interesting and powerful hacks). There’s a lot of quirks (browser differences, nuances in what extensions can and can’t do, etc) but it’s the #1 language that allows you to manipulate a common interface that everyone has easy access to. This was my first time using VS Code. There was a lot of interesting contextual information VS code was able to provide while editing files. It seems very powerful and is a lot more snappy compared to Atom. However, I felt like an infant struggling with the keyboard shortcuts and lack of some nice Atom features I’m used to. I should spend some more time learning VS Code (especially now that MS owns GitHub, I can’t imagine they’ll invest in two similar editors for that much longer). It’s super useful to have a simple sample project to work with when attempting to learn new technologies. It’s worth spending some time to think about an interesting and genuinely useful project you can work on that involves new technologies you are interested in.

Continue Reading

Building a Docker image for a Python Django application

After building a crypto index fund bot I wanted to host the application so the purchase routines would run automatically. In addition to this bot, there were a couple of other smaller applications I’ve been wanting to see if I could self-host (Monica, Storj, Duplicati).

In addition to what I’ve already been doing with my Raspberry Pi, I wanted to see if I could host a couple small utilities/applications on it, and wanted to explore docker more. A perfect learning project!

Open Source Docker Files

As with any learning project, I find it incredibly helpful to clone a bunch of repos with working code into a ~/Projects/docker so I can easily ripgrep my way through them.

https://github.com/schickling/dockerfiles/ Older, but simple Dockerfiles. Helpful to understand the basics of how to solve various problems in Docker. https://github.com/linuxserver/docker-duplicati Example of how to build a Docker image compatible with the raspberry pi. The @linuxserver group on GitHub has a lot of interesting Dockerfiles to learn from. https://github.com/monicahq/docker Docker images for a classic LAMP application. https://github.com/getsentry/sentry Image for a Python Django application https://github.com/mdn/kuma Another Python Django application example

And here’s my resulting Dockerfile for hosting the crypto index fund bot I’ve been playing with.

Learning Docker

I first ran into Docker at a Spree conference way before it was widely adopted. I remember thinking the technology sounded neat, but it was hard to imagine why you’d want to build a docker container.

It takes time for new technologies to make sense. Now docker containers are everywhere, and you can’t imagine living without them. Although I’ve used docker indirectly through Heroku, Dokku, or blindly running docker compose up on an open source project, I’ve never dug in and actually created my own docker image.

Here’s what I learned while writing my first image:

Docker has great install instructions. The repository-based install instructions did not work for me. I went the sh install script route. This guide was helpful Run sudo docker run hello-world to verify docker is working Each command in a Dockerfile generates a new ‘layer’ (intermediate container image). These layers are incrementally built upon to generate your final docker image. ENTRYPOINT always has a default of a shell, CMD is not set by default. ENTRYPOINT cannot be overwritten, CMD can when specified with a docker run command The base images are generally pretty bare. You’ll need to install the packages that you need using something like RUN apt-get update && apt-get install -y --no-install-recommends bash You’ll see set -eux at the beginning of most RUN or other shell commands executed by docker. This ensures that when one shell command fails, the failure bubbles up and the docker build fails as well. Look at the manpage for set to learn more about the specific failure codes. docker exec runs a command within an existing container, docker run creates a new container and executes the command. .dockerignore is like .gitignore but for the COPY command, which is generally used to grab your source code and stuff it in the container. This is important because each command that is run in a Dockerfile attempts to create a cache of the image at that state. If you include files in COPY that are not core to your application, and they are modified often, it will cause longer docker build times, which will slow down your development loop. If a docker command fails, you’ll get a image SHA that you can use to jump into teh container and debug its state: docker image inspect b01352c2271a dive is a really neat tool to inspect each layer of an image. Helpful for debugging container issues. It’s not possible to map a layer SHA to a Dockerfile. When the layers are pulled into your local, they aren’t tagged. Your best bet is using the FROM commands in your Dockerfile and attempting to find the source Dockerfile the tagged images were created from. However, you can publish a docker image to Docker Hub without linking it to an open source Dockerfile (this seems to be rare in practice). What are the differences between all of these base image types? The most popular ones I’ve seen are Debian (buster, stretch, etc) and Alpine. This is a good explanation. Bottom line is most likely you want debian’s latest release (right now, it’s ‘buster’). You may see ‘busybox’ referenced in Dockerfiles. For a while, alpine linux was popular. It was a slimmed-down linux base layer designed to be small (I don’t fully understand why folks are so concerned with image size). The downside is it doesn’t include important utils—like cron. This is where busybox comes in, it’s a space-efficient GNU-toolset replacement. Most likely, you should just use the full debian image and forget about busybox. However, there are cases where the busybox implementation is better and designed to play well with containerized environments. For instance, if you are running a cron (on debian, alpine makes it easier), it’s challenging to get stdout redirected to the parent process without busybox. Build your image with docker build -t your-image-name . and then run it with docker run --env-file .env -it your-image-name You’ll see rm -rf /some/cache/folder in Dockerfiles. This is to eliminate package management cache, which increases the file size of the image. apt-get clean can be used instead of rm -rf /cache/folder. I’m not sure why this is more commonly used in Dockerfiles. By default, COPY requires the source file to exist. However, you can use a glob to safely optionally copy a file COPY *external_portfolio.json ./ You can have multiple FROM statements in your file. This is helpful if you need to install two runtimes (rust and python, for example). Running Cron in a Docker Debian Container

At some point, you’ll need to run a specific command on some sort of schedule without installing a full-blown job scheduler like Resque or Celery.

The ‘easiest’ way to do that is via a simple cron entry. However, cron is not plug-n-play on docker images as I painfully discovered.

Cron is not installed by default in debian base layers. This is done to save space. Installing busybox does not install the cron component when using debian. This is probably because it’s available via the standard cron package. Here’s how to install cron on a debian-based image apt-get update && apt-get install -y --no-install-recommends cron && apt-get clean You may be wondering: why use debian? This all seems so difficult, right? In my specific scenario, I’m using the python docker image which defaults to debian. From what I understand, alpine could cause other dependency issues with python and python extensions which contain C-based extensions. You don’t need to install rsyslog in order to get stdout routed to the parent process (and therefore displayed in the docker logs). To get stdout routed to the parent process, add > /proc/1/fd/1 2>&1 at the end of your cron job definition. By default, cron uses sh not bash and does not pick up on any of the environment variables passed into the docker container. To pick up ENV vars, some people recommend executing a bash script with a login flag. This didn’t work for me. Some recommended storing ENV variables in a file and sourcing it within the cron job script. Similarly, others recommend modifying BASH_ENV in your crontab Neither of these solutions worked perfectly for me. What worked was exporting the current environment variables (being careful to handle special characters) into /etc/profile which is automatically sourced by the cron process.

Here’s my cron.sh to setup the cron schedule and execute it:

#!/bin/bash -l set -eu printenv | awk -F= '{print "export " "\""$1"\"""=""\""$2"\"" }' >> /etc/profile echo "$SCHEDULE root sh -lc '/pull/path/to/executable' > /proc/1/fd/1 2>&1" >> /etc/crontab cron -L 8 -f

It’s insane to me this isn’t more simple. Another argument for keeping docker containers as simple as possible and moving as much execution logic into your application.

Building a Dockerfile

In many cases, a repo will have multiple different dockerfiles. For instance, the Monica repo has a couple different dockerfiles for various purposes. You can specify which file to build using -f:

docker build -t monicahq/monicahq -f scripts/docker/Dockerfile

The -f argument is important, as opposed to cding into the directory with the Dockerfile, since we want many of the commands (notably COPY) to run from a specific directory on the host.

As build is running, it outputs a hash (e.g. c1861cb1ff7f) at each step. When the build fails, you can use that hash to debug the container by shelling in and poking around:

docker run -it c1861cb1ff7f bash

Note that run takes a single command. You cannot pass a shell command with arguments.

In my specific situation, my build was failing due to javascript compilation errors on the Pi. After digging into it, I realized it was going to be a major pain to build the web assets on the Raspberry Pi. I just built them locally and scp‘d them over:

cd public && scp -r css/ js/ fonts/ mix-manifest.json monica@raspberrypi.local:~/monica-source/public/

After the build is complete locally, you can use it in your docker-compose.yml:

image: monicahq/monicahq

This is helpful if you are using a docker-compose.yml with a pre-existing reference to a named/tagged (with -t) Dockerfile, but you need to patch that Dockerfile to work properly. If you can edit docker-compose.yml, a better approach is to just reference the sub-Dockerfile directly:

services: worker: build: context: . dockerfile: Dockerfile

After you’ve rebuilt your docker image (or simply edited the component Dockerfile if you are using build), here’s how to apply the changes:

docker-compose up -d --remove-orphans

I’ll detail some learnings about docker-compose in a separate blog post in the future.

Hosting on a Raspberry Pi

Raspberry Pi’s architecture (32bit ARM by default) is supported by docker. However, some software isn’t packaged to run on the Pi’s ARM architecture. Additionally, the running images on the Pi generally isn’t tested as well as a traditional EC2 instance.

I ran into lots of weird and interesting bugs hosting images on the Pi. I wouldn’t recommend it if you just want to get something working quickly.

Modifying a Dockerfile to work with Raspberry Pi

If you do choose to host an application on the Pi, you’ll inevitably run into weird execution issues. Here’s one that I ran into and how I debugged it.

There’s a great dockerfile for backing up a mysqlsql database, but it was failing for me on the Pi with the following error:

exec user process caused: exec format error

It looks like this error was caused by a missing shebang at the top of the sh files.

git clone https://github.com/schickling/dockerfiles.git schickling-dockerfiles cd schickling-dockerfiles/mysql-backup-s3/

Both install.sh and run.sh had an extra space in their shebang line.I removed the spaces and built the docker image:

docker build -t iloveitaly/mysql-backup-s3 .

I got a build error:

fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/main: temporary error (try again later) fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/community: temporary error (try again later) WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory

I jumped into the last successful build step (note that sh needed to be used instead of bash, I’m assuming this is because alpine is used as the base image and doesn’t contain bash by default):

docker run -it 186581f43b48 sh

It looks like the error is caused by a raspberrypi issue that requires updating a specific library:

wget http://ftp.de.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1_armhf.deb sudo dpkg -i libseccomp2_2.5.1-1_armhf.deb

This fixed the particular build error I was running into, but caused another one: the apk install command was referencing an old python package. I bumped the apk command and that was fixed.

At this point, docker build was running but executing the image caused a different error! This time python was complaining:

ModuleNotFoundError: No module named 'six'

With some googling it looks like that can happen if the pypi project is removed, which is what was happening in the Dockerfile script. I updated the docker file to stop removing pypi which fixed the issue.

However, when I tried to run the image with a SCHEDULE: '@daily' (in the yaml file above) I ran into a go-cron failure. The package hasn’t been updated in many years, so I’m guessing it was an incompatibility with the latest alpine version.

Instead of using that package, I opted to modify the run.sh script to use the native cron functionality. I found conflicting information about using native cron functionality:

Some claimed you needed to use complex workarounds or use some sort of wrapper (similar to the workaround described earlier in the post). I found that (a) running cron in the foreground and (b) using -d 8 (option available via busybox cron) routes all cron logs to the parent stdout so you’ll see it in the docker logs.

I rebuilt the container (docker build -t iloveitaly/mysql-backup-s3 .) and applied the modifications; finally everything was working.

I ended up trying out Storj, which is a decentralized s3-compatible storage service. It comes with a generous (150gb) free tier, and it gave me an excuse to tinker around with some dweb stuff. It worked surprisingly well.

Moral of the story: if something does go wrong (high likelihood when using a system with relatively low adoption like raspberrypi) it’s a pain to debug, the feedback loop is painful.

Thoughts on Docker

It was fun playing with Docker images and getting a feel for the ecosystem. I’ll write about docker-compose separately, but it’s a very nice abstraction on top of a raw Dockerfile. The ecosystem has consistently improved over the years and Docker has been hugely helpful in eliminating differences between development, CI, staging, and production environments.

That being said, it was surprising to me how brittle Dockerfiles were (broke easily on the Pi) and how slow it was to debug them. They also take up a ton of ram on macOS. I’m due for a new MacBook, but I have 16gb of RAM, and Docker ate up my free RAM and slowed down my computer to a halt. I can see the value in using Docker to quickly spin-up a local Redis, Postgres, etc but the speed cost for local development was too high for me.

I find it fun to play around with lower level linux system stuff, but I don’t have much patience for tinkering with it when I’m just trying to get something deployed for an application I’m building. I’m a big fan of Heroku for this reason—they build the container image(s) for you automatically with basically zero configuration on your part. If you want more control over your infrastructure, you can use the open source alternative Dokku. Or, if you still want to run Docker images manually, you can use BuildPacks to generate the docker image for you.

This is all to say, I don’t see the value in managing Dockerfiles directly unless you are a very large company who needs nuanced control over your application’s runtime environment. Definitely helpful to understand how this technology works under the hood, but I can’t see myself managing these Dockerfiles directly instead of using a Heroku-like system.

Continue Reading

Using GitHub Actions With Python, Django, Pytest, and More

GitHub actions is a powerful tool. When GitHub was first released, it felt magical. Clean, simple, extensible, and adds so much value that it felt like you should be paying for it. GitHub actions feel similarly powerful and positively affected the package ecosystem of many languages.

I finally had a chance to play around with it as part of building a crypto index fund bot. I wanted to setup a robust CI run which included linting, type checking, etc.

Here’s what I learned:

It’s not possible to test changes to GitHub actions locally. You can use the GH CLI locally to run them, but GH will use the latest version of the workflow that exists in your repo. The best workflow I found is working on a branch and then squashing the changes. You can use GitHub actions to run arbitrary scripts on a schedule. This may sound obvious, but it can be used in really interesting ways, like updating a repo everyday with the results of a script. You can setup dependabot to submit automatic package update PRs using a .github/dependabot.yml file. The action/package ecosystem seems relatively weak. The GitHub-owned actions are great and work well, but even very popular flows outside of the default action set do not seem widely used and seem to have quirks. There are some nice linting tools available with VS Code so you don’t need to remember the exact key structure of the GitHub actions yaml. Unlike docker’s depends_on, containers running in the services key, are not linked to the CI jobs in a similar way to docker compose yaml files. By ‘linked’ I’m referring to exposing ports, host IP, etc to the other images that are running your jobs. You need to explicitly define ports to expose on these service images, and they are all bound to localhost. on: workflow_dispatch does not allow you to manually trigger a workflow to run with locally modified yaml. This will only run a job in your yaml already pushed to GitHub. Matrix builds are easy to setup to run parallelized builds across different runtime/dependency versions. Here’s an example. Some details about the postgres service: Doesn’t seem like you can create new databases using the default postgres/postgres username + password pair. You must use the default database, postgres. Unlike docker, the image does not resolve the domain postgres to an IP. Use 127.0.0.1 instead. You must expose the ports using ports: otherwise redis is inaccessible. You must set the password on the image, which felt very strange to me. You’ll run into errors if you don’t do this.

Here’s an example .github/workflows/ci.yml file with the following features:

Redis & postgres services for Django ORM, Django cache, and Celery queue store support. Django test configuration specification using DJANGO_SETTINGS_MODULE. This pattern is not standard to django, here’s more information about how this works and why you probably want to use it. Database migrations against postgres using Django Package installation via Poetry Caching package installation based on VM type and SHA of the poetry/package lock file Code formatting checks using black and isort Type checking using pyright Linting using pylint Test runs using pytest name: Django CI on: workflow_dispatch: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest # each step can define `env` vars, but it's easiest to define them on the build level # if you'll add additional jobs testing the same application later (which you probably will env: DJANGO_SECRET_KEY: django-insecure-@o-)qrym-cn6_*mx8dnmy#m4*$j%8wyy+l=)va&pe)9e7@o4i) DJANGO_SETTINGS_MODULE: botweb.settings.test REDIS_URL: redis://localhost:6379 TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres # port mapping for each of these services is required otherwise it's inaccessible to the rest of the jobs services: redis: image: redis # these options are recommended by GitHub to ensure the container is fully operational before moving options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 ports: - 6379:6379 postgres: image: postgres ports: - 5432:5432 env: POSTGRES_PASSWORD: postgres steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 with: python-version: 3.9.6 # install packages via poetry and cache result so future CI runs are fast # the result is only cached if the build is successful # https://stackoverflow.com/questions/62977821/how-to-cache-poetry-install-for-github-actions - name: Install poetry uses: snok/install-poetry@v1.2.0 with: version: 1.1.8 virtualenvs-create: true virtualenvs-in-project: true - name: Load cached venv id: cached-poetry-dependencies uses: actions/cache@v2 with: path: .venv key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }} - name: Install dependencies run: poetry install if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true' - name: Linting run: | source .venv/bin/activate pylint **/*.py - name: Code Formatting run: | # it's unclear to me if `set` is required to ensure errors propagate, or if that's by default in some way # the examples I found did not consistently set these options or indicate that it wasn't required set -eax source .venv/bin/activate black --version black --check . isort **/*.py -c -v - name: Setup node.js (for pyright) uses: actions/setup-node@v2.4.0 with: node-version: "12" - name: Run type checking run: | npm install -g pyright source .venv/bin/activate pyright . - name: Run DB migrations run: | source .venv/bin/activate python manage.py migrate - name: Run Tests run: | source .venv/bin/activate pytest

Continue Reading

Lessons learned building with Django, Celery, and Pytest

As someone who writes ruby professionally, I recently learned python to build a bot which buys an index of crypto using binance.

The best thing about ruby is Rails, so I wanted an excuse to try out Django and see how it compared. Adding multi-user mode to the crypto bot felt like a good enough excuse. My goal was to:

Add a model for the user that persisted to a database Cron job to kick off a job for each user, preferably using a job management library Add some tests for primary application flows Docker-compose for the DB and app admin

I’ll detail learnings around Docker in a separate post. In this post, I walk through my raw notes as I dug into the django + python ecosystem further.

(I’ve written some other learning logs in this style if you are interested)

Open Source Django Projects

I found a bunch of mature, open-source django projects that were very helpful to grep (or, ripgrep) through. Clone these into a ~/Projects/django folder so you can easily search through them locally when learning:

https://github.com/getsentry/sentry https://github.com/arrobalytics/django-ledger https://github.com/intelowlproject/IntelOwl https://github.com/mdn/kuma – manages the MDN docs https://github.com/apache/airflow https://github.com/kiwicom/kiwi-structlog-config – Advanced structlog configuration examples. More python language learnings

I learned a bunch more about the core python language. I was using the most recent (3.9) version of python at the time.

You can setup imports in __init__ to make it more convenient for users to import from your package. As of python3, you don’t need a __init__ within a folder to make it importable. You can import multiple objects in a single statement from sentry.db.models import (one, two, three) iPython can be setup to automatically reload modified code. Somehow VS Code’s python.terminal.activateEnvironment got enabled again. This does not seem to play well with poetry’s venv. I disabled it and it eliminated some weird environment stuff I was running into. When using poetry, if you specify a dependency with path in your toml, even if it’s the dev section, it still is referenced and validated when running poetry install. This can cause issues when building dockerfiles for production when still referencing local copies of a package you are modifying. It doesn’t seem like there is a way to force a non-nil value in mypy. If you are getting typing errors due to nil values assert var is not None or t.cast are the best options I found. Inline return with a condition is possible: if not array_of_dicts: return None There doesn’t seem to be a one-command way to install pristine packages. poetry env remove python && poetry env use python && poetry install looks like the best approach. I ran into this when I switched a package to reference a github branch; the package was already installed and poetry wouldn’t reinstall it from the github repo. You can copy/paste functions into a REPL with iPython, but without iPython enabled it’s very hard to copy/paste multiline chunks of code. This is a good reason to install iPython in your production deployment: makes repl debugging in production much easier. By default all arguments can be either keyword or positional. However, you can define certain parameters to be positional-only using a / in the function definition. Variable names cannot start with numbers. This may seem obvious, but when you are switching from using dicts to TypedDict you may have keys which start with a number that will only cause issues when you start to construct TypedDict instances. There is not a clean way to update TypedDicts. Looks like the easiest way is to create a brand new one or type cast a raw updated dict. Cast a union list of types to a specific type with typing.cast. Convert a string to a enum via EnumClassName('input_string') as long as your enum has str as one of its subclasses. Disable typing for a specific line with # type: ignore as an inline comment Memoize a function by specifying a variable as global and setting a default value for that variable within the python file the function is in. There is also @functools.cache included with stdlib for this that should work in most situations. mypy is a popular type checker, but there’s also pyright which is installed by default with pylance (VS Code’s python extension). pylint seems like the best linter, although I was surprised at how many different options there were. This answer helped me get it working with VS Code. Magic methods (i.e. __xyz__) are also called dunder methods. A ‘sentinel value is used to distinguish between an intentional None value and a value that indicates a failure, cache miss, no object found, etc. Think undefined vs null in Javascript. First time I heard it used to describe this pattern. The yield keyword is interesting. It returns the value provided, but the state of the function is maintained and somehow wrapped in a returned iterator. Each subsequent next will return the value of the next yield in the logic Unlike ruby, it does not seem possible to add functions to the global namespace. This is a nice feature; less instances of ‘where is this method coming from. Black code formatting is really good. I thought I wouldn’t like it, but I was wrong. The cognitive load it takes off your mind when you are writing code is more than I would have expected. Structured logging with context & ENV-customized levels

structlog is a really powerful package, but the documentation is lacking and was hard to configure. Similar to my preferred ruby logger I wanted the ability to:

Set global logging context Easily pass key/value pairs into the logger Configure log level through environment variables

Here’s the configuration which worked for me:

# utils.py import structlog from decouple import config from structlog.threadlocal import wrap_dict def setLevel(level): level = getattr(logging, level.upper()) structlog.configure( # context_class enables thread-local logging to avoid passing a log instance around # https://www.structlog.org/en/21.1.0/thread-local.html context_class=wrap_dict(dict), wrapper_class=structlog.make_filtering_bound_logger(level), cache_logger_on_first_use=True, ) log_level = config("LOG_LEVEL", default="WARN") setLevel(log_level) log = structlog.get_logger()

To add context to the logger and log a key-value pair

from utils import log log.bind(user_id=user.id) log.info("something", amount=amount) Django poetry add django to your existing project to get started. Then, poetry shell and run django-admin startproject thename to setup the project Django has an interesting set of bundled apps: activeadmin-like Swap the DB connection information in settings.py to use PG and poetry add psycopg2. Django will not create the database for you, so you need to run CREATE DATABASE <dbname>; to add it before running your migrations. The default configuration does not pull from your ENV variables. I’ve written a section below about application configuration; it was tricky for me coming from rails. django-extensions is a popular package that includes a bunch of missing functionality from the core django project. Some highlights: shell_plus, reset_db, sqlcreate. It doesn’t look like there are any generators, unlike rails or phoenix. Asset management is not included. There’s a host of options you can pick from. There’s a full-featured ORM with adaptors to multiple DBs. Here are some tips and tricks: There’s a native JSONField type which is compatible with multiple databases. Uses jsonb under the hood when postgres is in place. After you’ve defined a model, you autogen the migration code and then run the migrations. python manage.py makemigrations Then, to migrate: python manage.py migrate To get everything: User.objects.all() or User.objects.iterator() to page through them. Getting a single object: User.objects.get(id=1) Use save() on an object to update or create it Create an object in a single line using User.objects.create(kwargs) You need a project (global config) and apps (actual code that makes up the core of your application) It looks like django apps (INSTALLED_APPS) are sort of like rails engines, but much more lightweight. Apps can each have their own migrations and they are not stored in a global folder. For instance, the built-in auth application has a bunch of migrations that will run but are not included in your application source code. I found this confusing. Table names are namespaced based on which app the model is in. If you have a user model in a users app the table will be named users_user. It looks like there is a unicorn equivalent, gunicorn, that is the preferred way for running web workers. It’s not included or configured by default. Flask is a framework similar to sinatra: simple routing and rendering web framework. The app scaffolding is very lightweight. Views, models, tests, and admin UI has a standard location. Everything else is up to the user. There’s a caching system built into django, but it doesn’t support redis by default. I already have redis in place, so I don’t want to use the default adapter (memcache). There’s a package django-redis that adds redis support to django cache. django-extensions has a nifty SHELL_PLUS_PRE_IMPORTS = [("decimal", "Decimal")] setting that will auto-import additional packages for you. It’s annoying to have to import various objects just to poke around in the REPL, and this setting eliminates this friction. Use decimal objects for floats when decoding JSON

In my case, I needed to use Decimals instead of floats everywhere to avoid floating point arithmetic inaccuracies. Even $0.01 difference could cause issues when submitting orders to the crypto exchange.

This is really easy when parsing JSON directly:

requests.get(endpoint).json(parse_float=decimal.Decimal),

If you are using a JSONField to store float values, it gets more complicated. You can’t just pass parse_float to the JSONField constructor. A custom decoder must be created:

class CustomJSONDecoder(json.JSONDecoder): def __init__(self, *args, **kwargs): from decimal import Decimal kwargs["parse_float"] = Decimal super().__init__(*args, **kwargs) class YourModel(models.Model): the_field = models.JSONField(default=dict, decoder=CustomJSONDecoder) Multiple django environments

There is not a standard way of managing different environments (staging, development, test, prod) in django. I found this very confusing and wasted time attempting to figure out what the best practice was here.

Here are some tips & recommendations:

Django doesn’t come with the ability to parse database URLs. There’s an extension, dj_database_url, for this. Poetry has a built-in dev category, which can be used for packages only required for development and test packages. There are no separate test or development groups. python-dotenv seems like the best package for loading a .env file into os.environ. However, if you are building an application with multiple entrypoints (i.e. web, cli, repl, worker, etc) this gets tricky as you need to ensure load_dotenv() is called before any code which looks at os.environ. After attempting to get python-dotenv working for me, I gave decouple a shot. It’s much better: you use it’s config function to extract variables from the environment. That function ensures that .env is loaded before looking at your local os.environ. Use this package instead. By default, Django does not setup your settings.py to pull from the environment. You need to do this manually. I included some snippets below. After getting decouple in place, you’ll probably want separate configurations for different environments. The best way to do this is to set DJANGO_SETTINGS_MODULE to point to a completely separate configuration file for each environment. In your toml you can set settings path [tool.pytest.ini_options] DJANGO_SETTINGS_MODULE = "app.settings.test" to force a different environment for testing. In production, you’ll set the DJANGO_SETTINGS_MODULE to app.settings.production in the docker or heroku environment For all other environments, you’ll set DJANGO_SETTINGS_MODULE to app.settings.development in your manage.py In each of these files (app/settings/development.py, app/settings/test.py, etc) you’ll from .application import * and store all common configuration in app/settings/application.py. Here’s a working example.

Here’s how to configure django cache and celery to work with redis:

CACHES = { "default": { "BACKEND": "django_redis.cache.RedisCache", "LOCATION": config("REDIS_URL"), "OPTIONS": { "CLIENT_CLASS": "django_redis.client.DefaultClient", }, } }

Here’s how to use dj_database_url with decouple:

DATABASES = {"default": dj_database_url.parse(config("DATABASE_URL"))} Job management using Celery

Django does not come with a job queue. Celery is the most popular job queue library out there and requires redis. It looks like it will require a decent amount of config, but I chose to use it anyway to understand how it compared to Sidekiq/Resque/ActiveJob/Oban/etc.

poetry add celery --allow-prereleases (I needed a prerelease to work with the version of click I was using) If you are using redis as the broker (easier for me, since I already had it installed + running) you’ll need to poetry add redis Celery does not use manage.py so it would not load the .env file. I needed to manually run dotenv_load() at the top of your celery config. I discovered that this needed to be conditionally loaded for prod, at which point I discovered that decouple is a much better package for managing configuration. I put my celery tasks within the users application as tasks.py. You can specify a dot-path to the celery config via the CLI: celery -A users.tasks worker --loglevel=INFO You can configure celery to store results. If you do, you are responsible for clearing out results. They do not expire automatically. Celery has a built-in cron scheduler. Very nice! There’s even a nice -B option for running the scheduler within a single worker process (not recommended for prod, but nice for development). When I tried to access django models, I got some weird errors. There’s a django-specific setup process you need to run through. DJANGO_SETTINGS_MODULE needs to be set, just like in manage.py. You can’t import django-specific modules at the top of the celery config file. Celery is threaded by default. If your code is not thread safe, you’ll need to set --concurrency=1. By default, tasks do not run inline. If you want to setup an integration test for your tasks, you need either (a) run tasks in eager mode (not recommended) or (b) setup a worker thread to run tasks for you during your tests. Eager mode is not recommended for testing, since it doesn’t simulate the production environment as closely. However, running a worker thread introduces another set of issues (like database cleanup not working properly). There’s no real downside to using @shared_task instead of @app.task. It’s easier to do this from the start: less refactoring to do when your application grows. Testing

Some more learnings about working with pytest & vcr in combination with django:

Database cleaning is done automatically for you via @pytest.mark.django_db at the top of your test class. This is great: no need to pull in a separate database cleaner. To be able to run pytest which relies on django models/configuration, you need the pytest-django extension. You can stick any config that would be in pytest.ini in your toml file under [tool.pytest.ini_options] You need to setup a separate config for your database to ensure it doesn’t use the same one as your development environment. The easiest way to do this is to add DJANGO_SETTINGS_MODULE = "yourapp.settings.test" to your toml file and then override the database setup in the yourapp/settings/test.py file. You can use pytest fixtures to implement ruby-style around functions. Redis/django cache is not cleared automatically between test runs. You can do this manually via django.core.cache.clear() In a scenario where you memoize/cache a global function that isn’t tied to a class, you may need to clear the cache to avoid global state causing indeterminate test results. You can do this for a single method via clear_cache() or identify all functions with lru cache and clear them. Django has a test runner (python manage.py test). It seems very different (doesn’t support fixtures), and I ran into strange compatibility issues when using it. Use pytest instead. My thoughts on Django

I continue to be impressed with the python ecosystem. The dev tooling (linting, repls, type checking, formatting, etc) is robust, there are reasonably well-written and maintained packages for everything I needed. It seems as though most packages are better maintained than the ruby equivalents. I only once had to dive into a package and hack a change I needed into the package. That’s pretty impressive, especially since the complexity of this application grew a lot more than I expected.

Working with python is just fun and fast (two things that are very important for me!). A similar level of fun to ruby, but the language is better designed and therefore easy to read. You can tell the ecosystem has more throughput: more developers are using various packages, and therefore more configuration options and bugs worked out. This increases dev velocity which matters a ton for a small side project and even more for a small startup. I don’t see a reason why I’d use ruby if I’m not building a rails-style web application.

Rails is ruby’s killer app. It’s better than Django across a couple of dimensions:

Better defaults. Multiple environments supported out of the box. Expansive batteries-included components. Job queuing, asset management, web workers, incoming/outgoing email processing, etc. This is the biggest gap in my mind: it takes a lot more effort & decisions to get all of these components working. Since django takes a ‘bring your own application components’ approach, you don’t get the benefit of large companies like Shopify, GitHub, etc using these and working out all of the bugs for you.

The Django way seems to be a very slim feature set that can be easily augmented by additional packages. Generally, I like the unix-style single responsibility tooling, but in my experience, the integration + maintenance cost of adding 10s of packages is very high. I want my web framework to do a lot for me. Yes, I’m biased, since I’m used to rails but I do think this approach is just better for rapid application development.

This was a super fun project. Definitely learned to love python and appreciate the Django ecosystem.

What I’m missing

There were some things I missed from other languages, although the list is pretty short and nitpicky:

Source code references within docs. I love this about the ruby/elixir documentation: as you are looking at the docs for a method, you can reveal the source code for that method. It was painful to (a) jump into a ipython session (b) import the module (c) ?? module.reference to view the source code. Package documentation in Dash More & better defaults in django setup. Improved stdlib map-reduce. If you can’t fit your data transformation into a comprehension, it’s painful to write and read. You end writing for loops and appending to arrays. Format code references in the path/to/file.py:line:col format for easy click-to-open support in various editors. This drove me nuts when debugging stack traces. Improved TypedDict support. It seems this is a relatively new feature, and it shows. They are frustrating to work with. Open Questions

I hope to find an excuse to dig a bit more into the python ecosystem, specifically to learn the ML side of things. Here are some questions I still had at the end of the project:

Does numpy/pandas eliminate data manipulation pain? My biggest gripe with python is the lack of chained data manipulation operators like ruby/elixir. How does the ML/AI/data science stuff work? This was one of my primary motivations for brushing up on my python skills and I’d love to deeply explore this. How does async/await work in python? How does asset management / frontend work in django? Debugging asdf plugin issues

Although unrelated to this post, I had to debug some issues with an asdf plugin. Here’s how to do this:

Clone the asdf plugin repo locally: git clone https://github.com/asdf-community/asdf-poetry ~/Projects/ Remove the existing version of the repo ~/.asdf/plugins && rm -rf poetry Symlink the repo you cloned: ln -s ~/Projects/asdf-poetry poetry

Now all commands hitting the poetry plugin will use your custom local copy.

Continue Reading

Building a Crypto Index Bot and Learning Python

A long time ago, I was contracted to build a MacOS application using PyObjc. It was a neat little app that controlled the background music at high-end bars around London. That was the last time I used python (early 2.0 days if I remember properly). Since then, python has become the language of choice for ML/AI/data science and has grown to be the 2nd most popular language.

I’ve been wanting to brush up on my python knowledge and explore the language and community. Building a bot to buy a cryptocurrency index was the perfect learning project, especially since there was a bunch of existing code on GitHub doing similar things.

You can view the final crypto index bot project here. The notes from this learning project are below. These are mainly written for me to map my knowledge in other languages to python. Hopefully, it’s also helpful for others looking to get started quickly in the language!

Tooling & Package Management

I work primarily in ruby (and still enjoy the language after years of writing professionally in it). Some of the comparisons below are to the equivalent tooling in ruby-land.

pip == bundle Package versions are specified in a requirements.txt file if you are using pip. https://rubygems.org/ = https://pypi.org/ There’s not really a rake equivalent that’s been adopted by the community. Poetry is an alternative to pip that seems to be the most popular choice for new projects. virtualenv = rbenv, but just for packages, not for the core python version, and is specific to each project. Poetry will autogen a virtualenv for you. There are dev and non-dev categories in poetry, but not a test category by default. Here’s how to add a dev dependency poetry add -D pytest If you are using the VS Code terminal, certain extensions will automatically source your virtualenv. I found this annoying and disabled this extension (can’t remember which extension was causing me issues). pyproject.toml alternative to requirements.txt, but also includes gemspec-like metadata about the package. It looks like poetry update consumes the .toml file and generates a poetry.lock. I’m guessing that other build tools also consume the .toml config and it’s not used just for poetry. The python community seems to be into toml configuration. This is used for poetry package specifications and project-specific variables. I don’t get it: it’s slightly nicer looking than JSON, but you can’t specify arrays or nested hash/dictionaries. Why not just use yaml instead? Or just keep it simple and use JSON? I ran into this issue where poetry was using the global ~/Library/Caches/pypoetry cache directory and I thought this was causing some package installation issues. I don’t think that ended up being the isweu poetry debug poetry config -vvv to see what configuration files are being loaded poetry config --list indicated that a global cache directory was being used. Tried upgrading pip, didn’t work: python3 -m pip install --upgrade pip I can’t remember how I fixed the issue, but these commands were helpful in understanding where poetry throws various code. If you want to hack on a package locally and use it in your project: vcrpy = { path = "/full/path/to/project", develop = true } in your toml file Note that you cannot use ~ in the path definition After adding this to your pyproject.toml run poetry lock && poetry install This will be easier in poetry 1.2 Want to make sure your project is pulling from your locally defined project? You can inspect the path that a module was pulled from via packagename.__file__ i.e. import vcr; print(vcr.__file__) I had trouble with a corrupted poetry env, I had to run poetry env use python to pick up my local package definition Working on a project not using poetry? Create a venv python -m venv venv && source ./venv/bin/activate If there’s a setup.py then run python setup.py install However, you can’t install ‘extra’ dependencies (like development/testing) via setup.py. It looks like pip install -e '.[dev]' It sounds like setup.py and requirements.txt do not define dev dependencies. You’ll probably need to install these manually. Look at the CI definition in the project to determine what dev dependencies need to be installed. There’s a .spec file that seems to be used with pyinstaller, a python package, when packaging a python application for distribution. Pyinstaller is primarily aimed at distributing packages for execution locally on someone’s computer. This use-case is one of the areas where python shines: there’s decent tooling for building a multi-platform desktop application. You’ll see readme-like documents written in rst (restructure text format) instead of md. I have no idea why markdown just isn’t used. A ‘wheel’ is an architecture-specific package bundle that contained compiled binaries. This is helpful if a python package contains non-python code that needs to be compiled since it eliminates the compile step and reduces the change of any library compatibility issues (this is a major problem in PHP-land). black looks like the most popular python code formatter. Language Multiline strings (""") at the beginning of a class or function definition isn’t just a python idiom. They are ‘docstrings’ and get automatically pulled into the autogen’d python documentation. Similar to ruby, camelCase is used for class names, snake_case is used for function/variable names. Calling a function requires parens, unlike ruby or elixir. Like javascript, return values need to explicitly be defined by return val. Conditionals do not return values, which means you need to assign variables inside the block (unlike the ability to assign a variable to the return value of a block in ruby, a feature that I love). Each folder in a python project is transformed into a package that can you import. the __init__ file in the folder is automatically imported when you import the folder name. Imports have to be explicitly defined, like javascript, to use any functions outside the set of global/built-in functions. Speaking of built-in functions, python provides a pretty random group of global functions available to you without any imports. I found this confusing: round() is a built-in but ceil() is not. When you import with a . it looks at the local directory for matching imports first. Import everything in package with from math import *. This is not good practice, but helpful for debugging/hacking. Although you can import individual functions from a package, this is not good practice. Import modules or classes, not individual functions. You have to from package.path import ClassName to pull a classname from a module. You can’t import package.path.ClassName None is nil and capitalization matters True and False are the bool values; capitalization matters. Hashes are called dicts in python Arrays are called lists in python You can check the existence of an element in a list with element in list. Super handy! Triple-quoted strings are like heredocs in other languages. They can be used for long comments or multi-line strings. Substring extraction ranges are specified by the_string[0:-1]. If you omit a starting range, 0 is used: the_string[:-1]. The traditional boolean operators && and || aren’t used. Natural language and and or is what you use instead. Keyword arguments are grouped together using **kwargs in the method definition. You can splat a dict into keyword arguments using function_call(**dict) All arguments are keyword arguments in python. More info. You can lazy-evaluate a comprehension using () instead of [] When playing with comprehensions inside of a ipython session variable scoping will not act the same as if you weren’t executing within a breakpoint(). I don’t understand the reasons for this, but beware! In addition to list comprehensions, there are dictionary comprehensions. Use {...} for these. When logic gets complex for a list comprehension, you’ll need to use a for loop instead (even if you want to do basic log debugging within a comprehension). I miss ruby’s multi-line blocks and chained maps. List comprehensions are neat, but there doesn’t seem to be a way to do complex data transformations cleanly. I hate having to define an array, append to it, and then return it. The filter/map/etc functions can’t be easily chained like ruby or javascript. I wonder what I’m missing here? I’ve heard of pandas/numpy, maybe this is what those libraries solve? There are strange gaps in the stdlib, especially around manipulating data structures. For instance, there’s no dead-simple way to flatten an array-of-arrays. import operator; from functools import reduce; reduce(operator.concat, array_of_arrays) Similarly, there’s no easy way to get unique values from a list. Get all of the string values of an enum [choice.value for choice in MarketIndexStrategy] By subclassing str and enum.Enum (ex: class MarketIndexStrategy(str, enum.Enum):) you can use == to compare strings to enums. There’s no ? tertiary operator, instead you can do a one-liner if-else: assignment = result if condition else alternative To enable string interpolation that references variable names you need to use f"string {variable}". Otherwise you’ll need to run format on the string to get it interpolated: "string {}".format(variable) Python has built-in tuples (1, 2, 3). I’ve always found it annoying when languages just have arrays and don’t support tuples. Unlike ruby, not all code has a return value. You have to explicitly return from a function and you can’t assign the result of a code block to a variable. There’s some really neat python packages: natural language processing, pandas, numpy. Python has gained a lot of traction in the deep learning/AI space because of the high-quality packages available. is is NOT the same as ==. is tests if the variable references the same object, not if the objects are equal in value You can’t do an inline try/catch. Many bad patterns that ruby and other languages really shouldn’t let you do are blocked. In a lot of ways, python is a simpler language that forces you to be more explicit and write simpler code. I like this aspect of the language a lot. Sets are denoted with {}, which is also used for dictionaries/hashes. Here’s how decorators work: The @decorator on top of a method is like an elixir macro or ruby metaprogramming. It transforms the method beneath the decorator. The @ syntax ("pie" operator) calls the decorator function, passing the function below the decorator as an argument to the decorator function, and reassigning the passed function to the transformed function definition. The decorator function must return a function. There is no special syntax to designate a function as a ‘decorator function’. As long it accepts a function as an argument and returns a function, it can be used as a decorator. Referencing an unspecified key in a dict raises an exception. You need to specify a default: h.get(key, None) to safely grab a value from a dict. An empty array will evaluate to false. You don’t need to if len(l) == 0:. Instead you can if !l:. Same goes with empty dicts and sets. Lambdas can only be single-line. This is a bummer, and forces you to write code in a different style. := allows you to assign and test a value within a conditional. Interesting that there’s a completely separate syntax for ‘assign & test’. __init__.py in a folder defines what happens when you import a folder reference. Here’s how classes work: class newClass(superClass): for defining a new class __init__ is the magic initialization method self.i_var within __init__ defines a new instance variable for a class. This is a good breakdown of instance and class variables. you can execute code within a class outside of a method definition for class-level variables and logic, new instances of a class are created via newClass(). Instance methods of a class are always passed self as the first argument Class variables are available on the instance as well, which is a bit strange. You can use class variables as default values for instance variables. This doesn’t seem like a great idea. newClass.__dict__ will give you a breakdown of everything on the class. Kind of like prototype in javascript. Python has multiple inheritance. class newClass(superClass1, superClass2). Inherited classes are searched left-to-right. There are not private variables built into the language, but the convention for indicating a variable is private is using a _ like self._private = value There’s a javascript-like async/await pattern (coroutines). I didn’t dig into it, but seems very similar to Javascript’s pattern. Debugging & Hacking

One of the important aspects of a language for me is the REPL and tinkering/hacking environment. If I can’t open up a REPL and interactively write/debug code, I’m a much slower developer. Thus far, ruby has the best interactive development environment that I’ve encountered:

binding.pry and binding.pry_remote when your console isn’t running your code directly to open a repl Automatic breakpoints on unhandled exceptions, in tests or when running the application locally Display code context in terminal when a breakpoint is hit Print and inspect local variables within a breakpoint Navigate up and down the callstack and inspect variables and state within each frame Overwrite/monkeypatch existing runtime code and rerun it with the new implementation within a repl Define new functions within the repl Inspect function implementation within the repl

I’d say that python is the first language that matches ruby’s debugging/hacking environment that I’ve used. It’s great, and better than ruby in many ways.

inspect is a very helpful stdlib package for poking at an object in a repl and figuring out the method, variables, etc available to it. traceback provides some great tools for inspecting the current stack. How you drop an interactive console at any point in your code? There are a couple ways: Uses the ipython enhanced repl in combination with the built in debugger import ipdb; ipdb.set_trace(). Requires you to install a separate package. There’s a breakpoint() builtin that launches the standard pdb debugger. You can configure breakpoint() to use ipdb via export PYTHONBREAKPOINT=ipdb.set_trace. All of the standard pdb functions work with ipdb import code; code.interact(local=dict(globals(), **locals())) can be used without any additional packages installed. bpython is a great improvement to the default python. You need to install this within your venv otherwise the packages within your projects venv won’t be available to it: pip install bpython && asdf reshim ipython is a bpython alternative that looks to be better maintained and integrates directly with ipdb. python -m ipdb script.py to automatically open up ipython when an exception is raised when running script.py Some misc ipython tips and tricks: If something is throwing an exception and you want to debug it: from ipdb import launch_ipdb_on_exception; with launch_ipdb_on_exception(): thing_causing_exception() who / whos in whereami %psource or source like show-source pp to pretty print an object ipython --pdb script.py to break on unhandled exceptions Great grab bag of interesting tips %quickref for detailed help exit gets you out of the repl entirely All of the pypi information is pulled from a PKG-INFO file in the root of a package rich-powered tracebacks are neat, especially with locals=True The ruby-like metaprogramming/monkeypatching stuff happens via the __*__ functions which are mostly contained within the base object definitions. For instance, logging.__getattribute__('WARN') is equivalent to logging.WARN You can reload code in a REPL via from importlib import reload; reload(module_name). Super helpful for hacking on a module (definitely not as nice as Elixir’s recompile). Monkeypatching in python isn’t as clean as ruby, which in some ways is better since monkeypatching is really an antipattern and shouldn’t be used often. Making it harder and more ugly helps to dissuade folks from using it. To monkeypatch, you reassign the function/method to another method: ClassName.method_name = new_method. Here’s an example. Typing

I’ve become a huge fan of gradual types in dynamic languages. I never use them right away, but once the code hardens and I’m relatively sure I won’t need to iterate on the code design, I add some types in to improve self-documentation and make it safer to refactor in the future.

Python has a great gradual type system built-in. Way better than Ruby’s.

mypy . on the command line to test all python files within a folder. If your project fails to pass mypy, it won’t cause any runtime errors by default. There’s a VS Code extension. This extension is included in Pylance, which you should probably be using instead, but you need to set the typing mode to ‘basic’. Return value types are set with -> before the : at the end of the method definition. Otherwise, typing works very similar to other languages with gradular typing (TypeScript, Ruby, etc). A common pattern is importing types via import types as t t.Union[str, float] for union/any types, You can’t merge dictionaries if you are using a TypedDict (dict | dict_to_merge). Massive PITA when mutating API data. Verbose types can be assigned to a variable, and that variable can be used in type definintions. Handy way to make your code a bit cleaner. Enums defined with enum.Enum can be types. Testing Like Elixir, there are doctests that execute python within docstrings to ensure they work. Neat! There are built-in test libraries that look comparable to ruby’s testunit. pytest is similar to minitest: provides easy plugins, some better standard functionality, and builds on top of unittest. You probably want to use pytest for your testing framework. setup.cfg is parsed by pytest automatically and can change how tests work. conftest.py is another magic file autoloaded by pytest which sets up hooks for various plugins. You can put this in the root of your project, or in test/ Test files must follow a naming convention test_*.py or *_test.py. If you don’t follow this convention, they won’t be picked up by pytest by default. breakpoint()s won’t work by default, you need to pass the -s param to pytest Like ruby, there are some great plugins for recording and replaying HTTP requests. Checkout pytest-recording and vcrpy. To record HTTP request run pytest --record-mode=once If you want to be able to inspect & modify the API responses that are saved, use the VCR configuration option "decode_compressed_response": True There’s a mocking library in stdlib, which is comprehensive. I’m not sure why other languages don’t do this—everyone needs a mocking library. It looks like you set expectations on a mock after it runs, not before. Here’s how mocking works: The @patch decorator is a clean way to manage mocking if you don’t have too many methods or objects to mock in a single test. If you add multiple patch decorators to a method, the mocks for those methods are passed in as additional arguments. The last patch applied is the first argument. mock.call_count, mock.mock_calls, mock.mock_calls[0].kwargs are the main methods you’ll want for assertions asset without parens is used in tests. This confused me, until I looked it up in the stdlib docs and realized assert is a language construct not a method. tox is much more complex that pytest. It’s not a replacement for pytest, but seems to run on top of it, adding a bunch of functionality like running against multiple environments and installing additional packages. It feels confusing—almost like GitHub actions running locally. If you want to just run a single test file, you need to specify an environment identifier and test file tox -epy38-requests -- -x tests/unit/test_persist.py My thoughts on Python

Overall, I’m impressed with how python is improved over the years. Here are some things I enjoyed:

Gradual typing included in the core language Comprehensions are natural to write Syntax simplicity: there are not too many ways to do things, which makes code more straightforward to read. Mature, well-designed libraries Virtual environments out of the box Robust, well-maintained developer tooling (ibpd, ipython, etc) with a advanced REPL Great built-in testing libraries Lots of example code to grep through for usage examples Explicit imports and local-by-default logic (unlike ruby, where it’s much easier to modify global state) Easy to understand runtime environment (in comparison to JavaScript & Elixir/BEAM)

The big question is if Django is a good alternative to Rails. I love Rails: it’s expansive, well-maintained, thoughtfully designed and constantly improving. It provides a massive increase in development velocity and I haven’t found a framework that’s as complete as Rails. If Django is close to rails, I don’t see a strong argument for not using anything python over ruby for a web product.

Open Questions

Some questions I didn’t have time to answer. If I end up working on this project further, this is a list of questions I’d love to answer:

How good is django? Does it compare to Rails, or is it less batteries-included and more similar to phoenix/JS in that sense. Does numpy/pandas solve the data manipulation issue? My biggest gripe with python is the lack of chained data manipulation operators like ruby. How does the ML/AI/data science stuff work? This was one of my primary motivations for brushing up on my python skills and I’d love to deeply explore this. How does async/await work in python? Learning Resources

General guides:

https://python-patterns.guide/python/module-globals/ https://book.pythontips.com/en/latest/ternary_operators.html https://realpython.com/python-lambda/#anonymous-functions https://google.github.io/styleguide/pyguide.html

Monkeypatching:

https://sharmapacific.in/monkey-patching-in-python/ https://github.com/ytdl-org/youtube-dl/commit/00fcc17aeeab11ce694699bf183d33a3af75aab6 https://filippo.io/instance-monkey-patching-in-python/ https://tryolabs.com/blog/2013/07/05/run-time-method-patching-python/ Open Source Example Code

There are some great, large open source python projects to learn from:

https://github.com/getsentry/sentry https://github.com/arachnys/cabot – opens source APM https://github.com/vitorfs/bootcamp https://github.com/rafalp/Misago

Download these in a folder on your local to easily grep through.

Continue Reading

How to install an old HomeBrew package

I accidentally ran brew upgrade on a set of packages which caused brew to bump most library packages that were installed on my computer. This caused issues compiling PHP via asdf.

I thought it would be easy to install an older version of the offending package, icu4c, but I was very wrong. In recent (2021?) versions of brew. I ended up learning a bit about the homebrew internals and solving

Installing an old version of icu4c to fix PHP compilation on macOS

I discovered the offending package was icu4c, but the PHP compile script did not indicate that I was using an unsupported package version. make emitted an error with an ‘international’ (intl) identifier in the offending file, which after some googling indicated that icu4c was the offending package:

3 errors generated. make: *** [ext/intl/dateformat/dateformat_attr.lo] Error 1

Brew used to support installing packages via a formula URL. Unfortunately, this method of installation was removed in a recent version of brew. Instead of something like brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/284ffd99cb862671202e685d7eced806bbc30dc4/Formula/icu4c.rb you need to execute (found this snippet through this blog post):

brew tap-new $USER/local-tap brew extract --version=68.2 icu4c $USER/local-tap brew install icu4c@68.2

You can determine the last supported version in homebrew using git history via the URL exposed when running brew info icu4c.

However, extracting a specific version through this method does not use the same package name. The package name is icu4c@68.2 with version 68.2 which means any scripts which check for a package via brew list --version icu4c will fail. If it wasn’t for this nuance, this StackOverflow answer would have had all of the information I needed.

Additionally, even if the package name was the same, brew removed the switch command. Now, the preferred way to switch versions is to relink a specific version:

brew link --overwrite --force icu4c@68.2

However, this doesn’t work:

Warning: Refusing to link macOS provided/shadowed software: icu4c@68.2 If you need to have icu4c first in your PATH, run: echo 'export PATH="/usr/local/opt/icu4c@68.2/bin:$PATH"' >> ~/.zshrc echo 'export PATH="/usr/local/opt/icu4c@68.2/sbin:$PATH"' >> ~/.zshrc For compilers to find icu4c you may need to set: export LDFLAGS="-L/usr/local/opt/icu4c@68.2/lib" export CPPFLAGS="-I/usr/local/opt/icu4c@68.2/include" For pkg-config to find icu4c you may need to set: export PKG_CONFIG_PATH="/usr/local/opt/icu4c@68.2/lib/pkgconfig"

From what I can tell, the only real workaround for this error is to manually specify the environment variables indicated in the installation post-install message. The downside with this approach is if you aren’t installing the package directly you’d have to modify the underlying script which could be a major pain.

If you don’t want to go that route, here’s the easiest alternative I found:

# uninstall the newer version of the package that you accidentally installed brew uninstall --ignore-dependencies icu4c # `extract` the version you'd like to install into a custom tap brew tap-new $USER/local-tap brew extract --version=68.2 icu4c $USER/local-tap # jump into the new tap you created cd $(brew --repository $USER/local-tap)/Formula # rename the formula mv icu4c@68.2.rb icu4c.rb # change the name of the formula by removing "AT682" from the `class` definition nano icu4c.rb # then, install this specific formula directly brew install $(brew --repository $USER/local-tap)/Formula/icu4c.rb

PHP 7.4.x & 8.x works with icu4c 68.2. PHP 7.2 did not compile properly with this version. To my surprise the PHP compilation script will not notify you that you are using an unsupported version of a package. php-build is a great project, which is probably worth using instead of asdf, and is a look place to look for PHP compilation tips & tricks.

Worth noting that the only reason I couldn’t use the simple extract and install route is I needed brew list --version $package_name to split out a valid package reference. If you don’t need that, you should be able to get away with the extract route and setting export PKG_CONFIG_PATH directly in the shell.

I’m very surprised installing an older version of a package was this messy. I can see why using a docker container is a better way to go for PHP development. The build story for PHP isn’t much better than what it was 10 years ago.

Installing the PHP ImageMagick Pecl extension

I also ran into issues installing the imagemagick extension with PHP8 (pecl install imagick) . Here’s what I found as far as version compatibility goes:

You need to brew install imagemagick to install the pecl extension Imagick pecl extension works with the latest ImageMagick from homebrew 7 Imagick doesn’t work with PHP8 yet You need to enable the extension "extension=imagick.so" > $(asdf where php)/conf.d/php.ini Upgrading an old brew-powered MySQL installation

I had MySQL data from an older installation on my machine that was causing issues. MySQL would not start properly and brew services was not listing the correct status.

Here’s what worked for me to fix it, although you might need to do more:

# this will wipe all mysql data # if you want to keep data from your old mysql installation, do not do this sudo rm -rf /usr/local/var/mysql/ brew remove mysql brew cleanup brew install mysql

I was then able to mysql.server start without an issue. If you want MySQL to start automatically on boot, run brew services start mysql. Quick pro-tip: take a look Sequel ace, a great GUI MySQL client.

Continue Reading

Scripting macOS with Javascript Automation

I’ve been playing with ActivityWatch, a really neat open-source application to track what you are doing when you are on your computer. Similar to rescue time RescueTime, but open source, with some more advanced features. I’ve been using it for a couple of months as part of my digital minimalism toolkit and it’s worked great to give me an idea of what’s taking up my time.

There’s been a couple of things that have bugged me about the application, and it’s written in a couple of languages I’ve wanted to learn (Rust, Vue), so I decided to make a couple of changes as part of a learning project.

I ended up needing to modify an AppleScript and ran into macOS’s Javascript Automation for the first time. It’s a really powerful system but horribly documented, with very little open source code to learn from.

Retrieving the Active Application, Window, and URL using AppleScript

I wanted to extract the active application, title of the main window, and the URL of that window (if the active application is a browser). I found this AppleScript, which was close to what I wanted, but I also wanted to identify the main window if a non-browser was in use:

global frontApp, frontAppName, windowTitle set windowTitle to "" tell application "System Events" set frontApp to first application process whose frontmost is true set frontAppName to name of frontApp tell process frontAppName try tell (1st window whose value of attribute "AXMain" is true) set windowTitle to value of attribute "AXTitle" end tell end try end tell end tell do shell script "echo " & "\"\\\"" & frontAppName & "\\\",\\\"" & windowTitle & "\\\"\""

Here’s what combining these two scripts looks like in Javascript for Automation:

var seApp = Application("System Events"); var oProcess = seApp.processes.whose({frontmost: true})[0]; var appName = oProcess.displayedName(); // these are set to undefined for a specific reason, read more below! var url = undefined, incognito = undefined, title = undefined; switch(appName) { case "Safari": url = Application(appName).documents[0].url(); title = Application(appName).documents[0].name(); break; case "Google Chrome": case "Google Chrome Canary": case "Chromium": case "Brave Browser": const activeWindow = Application(appName).windows[0]; const activeTab = activeWindow.activeTab(); url = activeTab.url(); title = activeTab.name(); break; default: mainWindow = oProcess. windows(). find(w => w.attributes.byName("AXMain").value() === true) // in some cases, the primary window of an application may not be found // this occurs rarely and seems to be triggered by switching to a different application if(mainWindow) { title = mainWindow. attributes. byName("AXTitle"). value() } } JSON.stringify({ app: appName, url, title, incognito });

Some notes & learnings that help explain the above code:

You can write & test JXA from the "Script Editor" application. You can connect the script editor to Safari for a full-blown debugger experience, which is neat. Open up a REPL via osascript -il JavaScript There’s not really an API reference anywhere. The best alternative is Script Editor -> File -> Open Dictionary. The javascript runtime is definitely heavily modified: Object.getOwnPropertyNames returns __private__ for all of the system events-related objects. This makes it much harder to poke around in a repl to determine what methods are available to you. Use #!/usr/bin/osascript -l JavaScript at the top of your jxa to run a script directly in your terminal. whose only seems to work with properties, not attributes. If you want to filter on attributes you need to iterate over each element: windows().find(w => w.attributes.byName("AXMain").value() === true) Application objects seem to use some sort of query ORM-type model underneath the hood. The application only seems to execute queries when value() or another value is requested, otherwise you’ll just get a reference to the query that could retrieve the object. This makes it harder to poke at the objects in a repl. If you compile a script once and rerun it, you must reset your variables to undefined otherwise the values they were set to will stick around. This is why all var declarations above are set to undefined. You can import objc libraries and use them in your JXA.

It’s worth noting that some folks online mention that JXA is dead, although not deprecated. I think this is a general state on macOS scripting (including AppleScript): Apple has built some very neat technologies but has done a horrible job at continuing to develop and evangelize them so they have many sharp edges and there is sparse documentation out there.

Executing Javascript Automation Scripts from Python

A powerful aspect of the python ecosystem is PyObjc which enables you to reach into the macOS Objective-C APIs within a python script. In this case, this allows you to compile & run applescript/javascript from within a python script without shelling out to osascript. This improves performance, but also makes it much easier to detect errors and parse output from the script.

The snippet below was adapter from this StackOverflow post and requires that you pip install pyobjc-framework-OSAKit :

script = None def compileScript(): from OSAKit import OSAScript, OSALanguage scriptPath = "path/to/file.jxa" scriptContents = open(scriptPath, mode="r").read() javascriptLanguage = OSALanguage.languageForName_("JavaScript") script = OSAScript.alloc().initWithSource_language_(scriptContents, javascriptLanguage) (success, err) = script.compileAndReturnError_(None) # should only occur if jxa is incorrectly written if not success: raise Exception("error compiling jxa script") return script def execute(): # use a global variable to cache the compiled script for performance global script if not script: script = compileScript() (result, err) = script.executeAndReturnError_(None) if err: raise Exception("jxa error: {}".format(err["NSLocalizedDescription"])) # assumes your jxa script returns JSON as described in the above example return json.loads(result.stringValue())

Here’s the structure of an AppleScript err after executing the script:

{ NSLocalizedDescription = "Error: Error: Can't get object."; NSLocalizedFailureReason = "Error: Error: Can't get object."; OSAScriptErrorBriefMessageKey = "Error: Error: Can't get object."; OSAScriptErrorMessageKey = "Error: Error: Can't get object."; OSAScriptErrorNumberKey = "-1728"; OSAScriptErrorRangeKey = "NSRange: {0, 0}"; }

Here are some tips and tricks for working with pyobjc in python:

Always pass None for objc reference arguments. References are returned in a tuple instead. You can see this in the above code ((result, err) = script.executeAndReturnError_(None)): result is the return value of the method, while err is reference argument passed as None in : is replaced by _ in the method signatures There’s a separate package for each objc framework. Import only what you need to avoid application bloat. Objc keyword arguments are transformed into positional arguments, not python keyword arguments. I ran into weird initialization errors if I had pyobj calls in the global namespace (for instance, caching the script immediately as opposed to setting script = None). I’m not sure if this was specific to how the rest of the application I was working in was structured. Resources

Here are some helpful resources I ran into when

Best group open source example scripts I could find: https://github.com/voostindie/vincents-productivity-suite-for-alfred Not sure why, but this forum has a lot of good sample code to copy from. https://forum.keyboardmaestro.com https://apple-dev.groups.io/g/jxa/wiki/3202 Some helpful snippets & usage examples https://gist.github.com/heckj/5b7bb332463a762639e179a37ea3a216 Official Apple release notes which a nice group of snippets. A great technical deep dive with links to many interesting resources

Continue Reading

Migrating from bash to zsh

I love productivity tools. Anyone who works with me knows I love my keyboard shortcuts and tiny productivity hacks. Small, incremental productivity improvements add up over time: feeling fast makes you fast. Plus, I just enjoy tinkering and making things more productive.

One of the rabbit holes I love to go down is optimizing my development environment. I spend a lot of time in a terminal, so it’s a fun place to optimize my setup. Ever since hearing of Oh My ZSH I wanted to try out zsh, so I set aside some time to update my dotfiles to use zsh as the default shell.

Below are some notes & learnings from the transition.

What’s new in zsh? There are lots of small packages out there for neat things like autocomplete, async prompts, etc. This is the best part about zsh and the main reason I put the effort into switching. There’s a bunch of configuration managers out there. Oh My ZSH, zplug, antigen, antibody, zinit, etc. These managers pull various bundles of zsh scripts together and source them for you. Antibody was the best manager that I could find (when I originally wrote this post in 2020). Allows you to pull directly from GitHub repositories, and load shell scripts that aren’t packaged as a "plugin". However, in less than a year it died out and is unmaintained. Here’s my plugin list with antibody Zinit looks like the best package manager nowadays (2021). Here’s how I moved from antibody to zinit and the change that enabled turbo mode. The syntax is strange. ice is a command that modifies the next command (why not just add a modifier to the command itself? Who knows.) for allows you to execute a command as a loop (like you’d expect) without having to separate ice from the actual command. Helpful if you don’t need separate ice modifiers for each command lucid eliminates the loading messages. Not sure why this isn’t enabled by default. I found this example setup to be the most helpful in decoding the zinit syntax. zi update to update all plugins Packaging something as a plugin is super simple. Create a name.plugin.zsh file in your repo. This file is autoloaded by plugin managers. I’ve always struggled to understand where I can map key pressed to the strange double-bracket definitions I see (e.g. ^[[A] is equivalent to the up arrow key). Run /bin/cat -v and when you press a key it’ll output the key definition you can use in key bindings. There are many options for up/down history matching. I like the substring search package, but there are great builtins for this as well There are many little changes to the shell which make life easier. For instance, mv something.{js,ts} will rename a file. zsh variables have different types. Run type var_name to inspect types of various variables. zsh line editor is zle. zle -N widget-name adds the widget to the line editor so you can bindkey the widget. bindkey lists out all of your keyboard shortcuts zle -la lists out all ‘widgets’ (zsh commands, not sure why they are called widgets). You can bind keyboard sequences to these widgets. The edit-command-line widget ‘parks’ the current command until the next command you type is done executing. Here’s how to bind to ctrl-e (the default ctrl-q binding wasn’t working for me). Function path is fpath, the list of paths to search for the definition of a function definition. This is distinct from $PATH in zsh. A big improvement with zsh is the ability to async run commands. For instance, you can display your prompt and then run various git commands and update your prompt status. This is critical for large repos (where git commands can take seconds to run) and is the main reason I switched to zsh. <<< is a here string. Useful for passing a string to stdin (echo 'hi' | cat is equal to cat <<< 'hi'). zsh also has here docs with the standard <<EOL syntax. Nifty command to list out all autocompletions. zinit also has a similar (cleaner) command zi clist. Snippet to list aliases, functions, and variables. Globs support regex-like syntax. It’s worth spending some time reading about this and getting familiar with it. There’s a neat trend of folks rewritten common utilities (cd, cat, find, etc) in rust. Here’s a great writeup of improved utilities you can use. You can find my set of tools here. Plugins

Some notes on my plugin configuration:

Here’s my list of zsh plugins. It took some extra bindkey config to get substring history search working zsh-autosuggestions caused weird formatting issues when deleting and pasting text (the autocomplete text wouldn’t use a different color and I couldn’t tell what was actually deleted). Modifying ZSH_AUTOSUGGEST_IGNORE_WIDGETS fixed the issue for me. I tried to get larkery/zsh-histdb working (really neat project) but it doesn’t play well with the fzf reverse-i search, which I really love. Hoping to give this another go in a year or so to see if the integration with fzf and other standard tooling is improved. Being able to filter out failed commands from your zsh history search would be neat. zsh-autosuggest and bracketed paste don’t play well together. This snippet fixed it for me. fasd is a really neat tool, but I wanted to customize the j shortcut to automatically pick the first result. Here’s how I did it. Resources

Some helpful posts and guides I ran into:

Really awesome guide to fancy zsh features & syntax https://reasoniamhere.com/2014/01/11/outrageously-useful-tips-to-master-your-z-shell/ https://remysharp.com/2018/08/23/cli-improved https://github.com/unixorn/awesome-zsh-plugins https://scriptingosx.com/zsh/ https://sourabhbajaj.com/mac-setup/iTerm/zsh.html http://zpalexander.com/switching-to-zsh/ https://chenhuijing.com/blog/bash-to-zsh https://medium.com/rootpath/replacing-bash-with-zs… http://jeromedalbert.com/migrate-from-oh-my-zsh-to-prezto/ https://terminalsare.sexy/#tools-and-plugins https://callstack.com/blog/supercharge-your-terminal-with-zsh/

Continue Reading

2020 Goal Retrospective

Another year, another yearly goal retrospective. This year included a grab bag of curveballs, most notably COVID. Although there was a lot of loss this year, I’m blessed to be able to say this year was really good for me and my family.

Without further ado, here’s the retro!

What Worked Not doing the quarterly reviews and focusing on the monthly reviews. In this season of life (young kids) quiet/focused time is precious and it’s not possible to spend too much time planning together (or individually) for that matter. Small, specific goals that created a habit or helped figure out a workflow worked well. We should continue to pick key habits and work on them through a focused goal. It’s important to have only one or two of these per year to prevent your goals from becoming too boring. The simple habit tracking sheet (gsheet with the number of times per week I did my target list of habits) provided a nice weekly reminder of the habits I want to build. I started this year reconnecting with a group of friends focused on changing a handful of specific habits. It’s been a great motivator to ‘flip the defaults’ on some behaviors at the beginning of the year. I spent a lot of time over the last year being more intentional about my screen time usage. This has paid off: I feel more focused and less distracted than in the past even if it means I’m the "horrible texter" in group chats. It’s worth continuing to improve my systems & disciplines around controlling screen time, it pays a handsome dividend. What Didn’t Goals that required lots of communication/coordination with my wife and weren’t essential to this year, didn’t get done. Getting time alone to work on common projects is challenging with young kids. I don’t think there is a great solution to this other than being very careful about committing to goals that fall into this category. Goals that weren’t impactful to get done this year were hard to prioritize. Be thoughtful about goals that are ‘nice to haves’, or something that is very exciting/an important long-term goal, but not something that can be tied to real progress this year. If the goal isn’t really important to get done by the end of the year, don’t include it. For example, one of my goals was completing a list of house projects. Most of these were not essential and I made progress on these without intentionally prioritizing them. I enjoy learning new skills and doing things with my hands, so I’d made certain improvements a priority without any additional willpower. We didn’t hold each other accountable for goals that didn’t make any progress by default. In our monthly review, we spent time reviewing the month and what we could improve, but not tracking against the goals we committed to. We didn’t adjust our goals and revisit some of the things that were just impractical after covid hit. Historically, we’ve been bad at adjusting goals after setting them. It feels like admitting defeat, which is something I hate doing. I need to get better at just accepting that life is dynamic and the focus of a year could change on a dime. I naively thought we had the parenting thing down. Kids pushed the limits of our parenting skills this year. My wife and I have spent a lot of time in the second half of the year talking, reading books, implementing new ideas, etc relating to our parenting. This took a lot of time and was the right place to put our efforts, but it was not reflected in our goals (either explicitly or by reducing the number of additional goals). I don’t expect this year to be too different as our oldest continues to get… well… older and we continue to attempt to figure out how to parent well. What Should Change? Don’t include goals that impact us more than a year out. Don’t include goals that aren’t critical and will partially get completed by default. Think about which goals require dedicated willpower to change behavior or make significant progress and focus on those. Make reviewing our goals and keeping each other accountable to them part of our monthly review. Either have goals tied to parenting or leave lots of margin to include time for parenting over the next year.

Continue Reading