Learning TypeScript by Migrating Mint Transactions

Years ago, I built a chrome extension to import transactions into Mint. Mint hasn’t been updated in nearly a decade at this point, and once it stopped connecting to my bank for over two months I decided to call it quits and switch to LunchMoney which is improved frequently and has a lot of neat developer-focused features.

However, I had years of historical data in Mint and I didn’t want to lose it when I transitioned. Luckily, Mint allows you to export all of your transaction data as a CSV and LunchMoney has an API.

I’ve spent some time brushing up on my JavaScript knowledge in the past, and have since used Flow (a TypeScript competitor) in my work, but I’ve heard great things about TypeScript and wanted to see how it compared. Building a simple importer tool like this in TypeScript seems like a great learning project, especially since the official bindings for the API is written in TypeScript.

Deno vs Node

Deno looks cool. It uses V8, Rust, supports TypeScript natively, and seems to have an improved REPL experience.

I started playing around with it, but it is not backwards compatible with Node/npm packages which is a non-starter for me. It still looks pretty early in its development and adoption. I hope Deno matures and is more backwards compatible in the future!

Learning TypeScript You can’t run TypeScript directly via node (this is one of the big benefits of Deno). There are some workarounds, although they all add another layer of indirection, which is the primary downfall of the JavaScript ecosystem in my opinion. ts-node looks like the easiest solution to run TypeScript without a compilation step. npm i ts-node will enable you to execute TypeScript directly using npx ts-node the_script.ts. However, if you use ESM you can’t use ts-node. This is a known issue, and although there’s a workaround it’s ugly and it feels easier just to have a watcher compile in the background and execute the raw JS. .d.ts within repos define types on top of raw JS. This reason this is done is to allow a single package to support both standard JavaScript and TypeScript: when you are using TypeScript the .js and .d.ts files are included in the TypeScript compilation process. Use npx tsc --init to setup an initial tsconfig.json. I turned off strict mode; it’s easier to learn a new typing system without hard mode enabled. Under the hood, typescript transpiles TypeScript into JavaScript. If you attempt to debug a TypeScript file with node inspect -r ts-node/register the code will look different and it’ll be challenging to copy/paste snippets to debug your application interactively. Same applies to debugging in a GUI like VS Code. You can enable sourcemaps, but the debugger is not smart enough to map variables dynamically for you when inputting strings into the console session. This is massive bummer for me: I’m a big fan of REPL-driven development. I can’t copy/paste snippets of code between my editor + REPL, it really slows me down. Similar to most other languages with gradual typing (python, ruby, etc), there are ‘community types’ for each package. TypeScript is very popular, so many/most packages includes types within the package itself. The typing packages need to be added to package.json. There’s a nice utility to do this automatically for you. If you want to be really fancy you can overload npm i and run typesync automatically. VS Code has great support for TypeScript: you can start a watcher process which emits errors directly into your VS Code status bar via cmd+shift+b. If you make any changes to tsconfig.json you’ll need to restart your watcher process. You can define a function signature that dynamically changes based on the input. For instance, if you have a configuration object, you can change the output of the function based on the structure of that object. Additionally, you can inline-assign an object to a type, which is a nice change from other languages (ruby, python). Example of inline type assignment: {download: true} as papaparse.ParseConfig<Object>. In this case, Object is an argument into the ParseConfig type and changes the type of the resulting return value. Very neat! I ran into Element implicitly has an 'any' type because expression of type 'string' can't be used to index type 'Object'. No index signature with a parameter of type 'string' was found on type 'Object. The solution was typing a map/object/hash with theVariable: { [key: string]: any } . I couldn’t change the any type of the value without causing additional typing errors since the returning function was typed as a simple Object return. There’s a great, free, extensive book on TypeScript development.

One of the most interesting pieces of TypeScript is how fast it’s improving. Just take a look at the changelog. Even though JavaScript isn’t the most well-designed language, "One by one, they are fixing the issues, and now it is an excellent product." A language that has wide adoption will iterate it’s way to greatness. There’s a polish that only high throughput can bring to a product and it’s clear that after a very long time JavaScript is finally getting a high level of polish.

Linting with ESLint, Code Formatting with Prettier ESLint looks like the most popular JavaScript linting tool. It has lots of plugins and huge community support. You can integrate prettier with eslint, which looks like the most popular code formatting tool. VS code couldn’t run ESLint after setting it up. Had trouble loading /node_modules/espree/dist/espree.cjs. Restarting VS Code fixed the problem.

Here’s the VS Code settings.json that auto-fixed ESLint issues on save:

{ "[typescript]": { "editor.formatOnSave": true, "editor.codeActionsOnSave": { "source.fixAll.eslint": true }, }, "eslint.validate": ["javascript"] }

And here’s the .eslintrc.json which allowed ESLint, prettier, and ESM to play well together:

{ "env": { "browser": false, "es2020": true }, "extends": [ "standard", "plugin:prettier/recommended" ], "parser": "@typescript-eslint/parser", "parserOptions": { "ecmaVersion": 2020, "sourceType": "module" }, "plugins": [ "@typescript-eslint" ], "rules": { } } Module Loading

As with most things in JavaScript-land, the module definition ecosystem has a bunch of different community implementation/conventions. It’s challenging to determine what the latest-and-best way to handle module definitions is. This was a great overview and I’ve summarized my learnings below.

require() == commonjs == CJS. You can spot modules in this format by module.exports in their package. This was originally designed for backend JavaScript code. AMD == Asynchronous Module Definition. You can spot packages in this style by define(['dep1', 'dep2'], function (dep1, dep2) { at the header of the index package. Designed for frontend components. UMD == Universal Module Definition. Designed to unify AMD + CJS so both backend and frontend code could import a package. The signature at the top of the UMD-packaged module is messy and basically checks for define, module.exports, etc. import == ESM == ES Modules. This is the latest-and-greatest module system officially baked into ES6. It has wide browser adoption at this point. This is most likely what you want to use. import requires module mode in TypeScript (or other compilers) not set to commonjs. If you use ESM, your transpiled JS code will look a lot less garbled, and you’ll still be able to use the VS Code debugger. The big win here is your import variable names will be consistent with your original source, which it makes it much easier to work with a REPL. There are certain compatibility issues between ESM and the rest of the older package types. I didn’t dig into this, but buyer beware. It looks like experimental support for loading modules from a URL exist. I hope this gets baked in to the runtime. There are downsides (major security risks), but it’s great for getting the initial version of something done. This was one of the features I thought was neat about Deno: you could write a script with a single JavaScript file without creating a mess of package*, tsconfig.json, etc files in a new folder. https://unpkg.com is a great tool for loading a JS file from any repo on GitHub. You’ll get Cannot use import statement inside the Node.js REPL, alternatively use dynamic import if you try to import inside of a repl. This is a known limitation.. The workaround (when in es2020 mode) is to use await import("./out/util.js"). When importing a commonjs formatted package, you’ll probably need to import specific exports via import {SpecificExport} from 'library'. However, if the thing you want to import is just the default export you’ll run into issues and probably need to modify the core library. Here’s an example commit which fixed the issue in the LunchMoney library When importing a local file, you need to specify the .js (not the ts) in the import statement import { readCSV, prettyPrintJSON } from "./util.js"; Package Management You can install a package directly from a GitHub reference npm i lunch-money/lunch-money-js You can’t put comments in package.json, which is terrible. Lots of situations where you want to document why you are importing a specific dependency, or a specific forked version of a dependency. npm install -g npm to update to the latest npm version. By default, npm update only updates packages to the latest minor semver. Use npx npm-check-updates -u && npm i to update all packages to the latest version. This is dangerous, and only makes sense if there are a small number of packages https://openbase.com is a great tool for helping decide which package to use. JavaScript Learnings You’ll want to install underscore and use chain for data manipulation: _.chain(arr).map(...).uniq().value(). Lots of great tools you are missing from ruby or python. ES6 introduced computed property names so you can use a variable as an object key { [variableKey]: variableValue } I had trouble getting papaparse to read a local file without using a callback. I hate callbacks; here’s a promise wrapper that cleaned this up for me. Merge objects with _.extend. The dotenv package didn’t seem to parse .env with exports in the file. Got tripped up on this for a bit. require can be used to load a JSON file, not just a javascript file. Neat! There are nice iterators now! for(const i in list) There’s array destruction too const [a, b] = [1,2] Underscore JS has a nice memoize method. I hate the pattern of having a package-level variable for memoization. Just feels so ugly. There’s a in keyword that can be used with objects, but not arrays (at least in the way you’d expect). There’s a null-safe operator now. For instance, if you want to safely check a JSON blob for a field and set a default you can now do something like const accounts = json_blob?.accounts || [] You are iterate over the keys and values of an object using for (const [key, value] of Object.entries(object)) https://github.com/ccxt/ccxt is a neat project which transpiles JavaScript code into multiple languages. Hacking & Debugging The most disappointing part of the node ecosystem is the REPL experience. There are some tools that (very) slightly improve it, but there’s nothing like iPython or Pry. nbd is dead and hasn’t been updated in years node-help is dead as well and just made it slightly easier to view documentaiton. node-inspector is now included in node and basically enables you to use Chrome devtools local-repl looks neat, but also hasn’t been updated in ~year. The updated repl project wouldn’t load for me on v16. The debugging happy path seems to be using the GUI debugger. You can use toString() on a function to get the source code. Helpful alternative to show-source from ruby or ll from python. However, it has some gotchas: It’s specifically discouraged since it’s been removed from the standard Arguments and argument defaults are not specified It’s not obvious how to list local variables in the CLI debugger. There’s a seemingly undocumented exec .scope that you can run from the debugger context (but not from a repl!). You can change the target to ES6 to avoid some of the weird JS transpiling stuff, Run your script with node inspect and then before conting type breakOnUncaught to ensure that you can inspect any exceptions. I prefer terminal-based debugging, if you want to connect to a GUI (chrome or VS Code) use --inspect. There’s not a way I could find to add your own aliases to the debugging (i.e. c == continue == cont). It’s worth writing your own console.ts to generate a helpful repl environment to play with imports and some aliases defined. Unfortunately, this needs to be done on a per-project basis. You can’t redefine const variables in a repl, which makes it annoying to copy/paste code into a console. It looks like there are some hacks you can use to strip out the const and replace with a let before the copy/pasted code gets eval’d. This seems like a terrible hack and should just be a native flag added to node. In more recent versions of node (at least 16 or greater), you can use await within a repl session. If you are in a debugger session await does not work, unlike when you are a standard node repl. You cannot resolve promises and therefore cannot interact with async code. This is a known bug, will not be changed, and makes debugging async code interactively extremely hard. Very surprised this is still a limitation. console.dir is the easiest way to inspect all properties of an object within a REPL. This uses util.inspect under the hood, so you don’t need to import this package and remember the function arguments. There’s a set of functions only available in the web console. Most of these seem to model after jQuery functions. Open Questions How can I run commands without npx? Is there some shim I can add to my zsh config to conditionally load all npx-enabled bins when node_modules exists? Is there anything that can done to make the repl experience better? This is my biggest gripe with JavaScript development. https://github.com/11ways/janeway looks interesting but seems dead (no commits in over a year) This code looks like an interesting starting point to removing all const that are pasted into a repl. The number of configuration files you need to get started in a repo just to get started is insane (tsconfig.json, package*.json, .eslintc.json). Is there a better want to handle this? Some sort of single configuration file to rule them all?

Continue Reading

Building a SouthWest Price Monitor and Learning Server Side JavaScript

I originally wrote a draft of this post in early 2019. I’m spending some time learning TypeScript, so I wanted to finally get my JavaScript-related posts out of draft. Some notes and learnings here are out of date.

Both sides of our family live out of state. Over the last couple years, we’ve turned them on to credit card hacking to make visiting cheap (free). SouthWest has some awesome point bonuses on credit cards, but you can’t watch for price drops on Kayak and other flight aggregators.

After a bit of digging, I found a basic version of a tool to do just this. It’s a self-hosted bot to watch for flight cost drops so you can book (or rebook for free). I’ve been wanting to dig into server side JavaScript development, and this is the perfect excuse.

Here’s what I’d like to do:

Get the tool running somewhere simple: Heroku, Raspberry Pi, etc Convert the use of redis to mongodb. Redis isn’t a database, it’s a key-value store. But this project is using it for persistence. Why switch to MongoDB? I’ve been wanting to understand document databases a bit more. Postgres would have been easier for me, but this project is all about learning. Possibly add the option of searching for the best flight deal on a particular month

Below is a ‘learning log’ of what I discovered along the way. Let’s get started!

Learning JavaScript

As I mentioned in an earlier post, my JavaScript knowledge was very out of date (pre ES6). Some findings and musings below will be obvious to a seasoned JavaScript developer, but to someone more experienced in Ruby/Python/etc they’ll be some interesting tidbits.

Looks like express is the dominant HTTP router + server. It’s equivalent to the routing engine of Rails combined with rack and unicorn. It doesn’t seem like there are strong conventions to how you setup an express-based app. You bring your own ODM/ORM, testing library, etc. There is a consistent template/folder structure. However, express doesn’t make any assumptions about a database library, although it does support a couple of different templating languages and has a preferred default (pug). app.use adds additional middleware to the stack. Middleware is simply a function with three arguments. Very similar to rack in ruby-land or plugs in Elixir-land. There’s a part of me that loves the micro-modularity of the node/npm ecosystem, but the lack of declarative programming like DateTime.now + 1.day starts to feel really messy. The equivalent in node is (new Date()).setDate((new Date()).getDate() + 1);. Another example: there’s no built-in sortBy and sort mutates the original array. Some popular packages that solve this (moment, datefuncs, underscore, etc) and the popular choice is to just pull in these packages and use them heavily. I always find including many external decencies adds a lot of maintenance risk to your code. These packages can die, cause strange performance issues, cause weird compatibility issues with future iterations of the language, etc. The good news is the JavaScript ecosystem is so massive, the most popular packages have a very low risk of abandonment. Variable scoping is weird in debugger mode. If the variable isn’t referenced in the function, it’s not available to inspect in the debugger repl. Make sure you reference the variable to inspect/play with it in real time. Node, express, etc are not billed as full-stack web frameworks like rails. I find this super frustrating: not being able spin up a console (rails console) with your entire app’s environment loaded up is annoying. For this particular problem, it looks like the best alternative is to write your own console.js (here’s another guide) with the things you need and startup a repl. The annoying thing here is you need to manually connect to your DB and trigger the REPL after the DB connection is successful. Blitz and Redwood are solving these problems, although these didn’t exist when this post was written. It seems like node inspect + a debugger line doesn’t run the code ‘completely’. For instance, if the code runs past a mongodb.connection line it doesn’t connect. I wonder if this is because the .connection call runs async and doesn’t get a chance to execute before the debugger line is called? Is there a way to instruct the repl to execute anything in the async queue? I found that starting up a vanilla node console and requiring what you needed works better. There are some interesting utility libraries that convert all methods on an object to be promises (async). http://bluebirdjs.com/docs/api/promise.promisifyall.html Languages with declarative convenience methods are just so much nicer. args.priceHistory[args.priceHistory.length - 1] is just ugly compared to args.priceHistory.last. My time at a BigCo has helped me understand the value of typing. I still find the highest velocity developer experience is type-hinting (i.e. types are not required) combined with a linter. This lets you play with code without getting all the details hardened, but still enforces guardrails to avoid a class of production errors. I’m not seeing the value in the event-loop programming paradigm. I get how it allows you to handle more concurrent connections, but isn’t that something that should be handled by the language or in some lower level abstraction? It’s much easier to reason about code when it runs sequentially. For instance, not having object.save throw an exception right away is really annoying: I need to either use callbacks to act when the code has executed OR use async and await everywhere. I do not understand why this pattern has become so popular. https://repl.it is very cool. The idea of sending out links with a console running your code is very handy. This is used a lot in the JavaScript community. It’s fascinating to me how there’s always the 10x-er that becomes a hero of the community. https://github.com/substack has created a ridiculous number of npm packages. Think about let r = await promise as let r = null; promise.then(rr => r = rr) which is executed synchronously. Instead of hash.merge(h2) you write Object.assign({}, h2, hash). There are many unintuitive sharp edges to the language, as you learning, just googling "how to do X with JavaScript" is the best way to determine the JavaScript equivalent. http://jsnice.org is great at parsing obfuscated JS. It tries to rename variables based on the context. Very cool. ... is the splat operator used on objects It’s called the ‘rest’ operator. constructor is the magic method for class initialization Looks like function definitions within a class don’t need the function keyword Puppeteer, Proxies, and Scraping

Part of this project involved scraping information the web. Here’s some tidbits about scraping that I learned:

The node ecosystem is great for web scraping. Puppeteer is a well maintained chrome-controller package and there’s lot of sample code you can leverage to hack things together quickly. Websites have gotten very good at detecting scrapers. There are some workarounds to try to block bot detection, but if you are using a popular site, you will most likely be detected if you are using the default puppeteer installation. A common (and easy) detection method is IP address. If you are scraping from an AWS/cloud IP, you’ll be easily blocked. The way around this is a proxy to a residential IP address. Another option is to host your scraper locally on a Raspberry Pi or on your local computer. https://chrome.browserless.io cool way to test puppeteer scripts I learned a bit about web proxies. Firstly, there are a bunch of proxy protocols (SOCKS, HTTP with basic auth, etc). Different systems support different type of proxies. Package Management You can’t effectively use npm and yarn in the same project. Pick one or the other. Yarn is a more stable, more secure version of npm (but doesn’t have as many features / as much active development) module.exports lets a file expose constants to others which import the file, similar to python’s import system (but with default exports). I like this compared with ruby’s "everything is global" approach. It allows the other author to explicitly define what it wants other users to access. Npm will run pre & post scripts simply based on the name of the scripts. import Section, {SectionGroup} assigns Section to the default export of the file, and imports the SectionGroup explicitly. If you try to import something that isn’t defined in the module.exports of a file you will not get an error and will instead get an undefined value for that import. Testing tape is the test runner that this particular project used. It doesn’t look like it’s possible to run just a single test in a file without changing the test code to use test.only instead of test. The "Test Anything Protocol" is interesting http://testanything.org. Haven’t run into this before. I like consistent test output across languages. I do like how tape tests list out the status of each individual assertion. It becomes a bit verbose, but it’s helpful to see what assertions after the failing assertion succeeded or failed. VS Code + node debugging is very cool when you get it configured. You need to modify your VS Code launch.json in order to get it to work with test files. https://gist.github.com/dchowitz/83bdd807b5fa016775f98065b381ca4e#gistcomment-2204588 Debugging & Hacking

I’m a big fan of REPL driven development and I always put effort into understanding the repl environment in a language to increase development speed. Here are some tips & tricks I learned:

Tab twice (after inputting ob.) in a repl exposes everything that is available on the object under inspection. node inspect THE_FILE.js allows debugger statements to work. You can also debug remotely with chrome or with VS Code. Visual debugging is the happy path with node development, the CLI experience is poor. You don’t need to setup variables properly in the node repl. Nice! You can just a = 1 instead of let a = 1 I’ll often copy code into a live console to play around with it, but if it’s defined as const I need to restart the console and make sure I don’t copy the const part of the variable definition. That’s annoying. There’s a lot of sharp edges to the developer ergonomics. console.dir to output the entire javascript object Unlike pry you need to explicitly call repl after you hit a breakpoint when running node inspect. Also, debugger causes all promises not to resolve when testing puppeteer. https://github.com/berstend/puppeteer-extra/wiki/How-to-debug-puppeteer Cool! Navigating to about:inspect in Chrome allows you to inspect a node/puppeteer process. list is equivalent to whereami. You need to execute it explicitly with params list(5) _ exists like in ruby, but it doesn’t seem to work in a repl triggered by a debugger statement. _error is a neat feature which keeps the last exception that was thrown. .help while in a repl will output a list of "dot commands" you can use in the repl. I had a lot of trouble getting puppeteer to execute within a script executed with node inspect and paused with debugger. I’m not sure why, but I suspect it has something to do with how promises are resolved in inspect mode. You can enable await in your node console via --experimental-repl-await. This is really helpful to avoid having to write let r; promise.then(o => r) all of the time. Mongo & ODMs You’ll want to install mongo and the compass tool (brew install mongodb-compass) for GUI inspection. Running into startup problems? tail -f ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist If you had an old version of mongo install long ago, you may need to brew sevices stop mongodb-community && rm -rf /usr/local/var/mongodb && mkdir /usr/local/var/mongodb && brew services start mongodb-community -dv The connection string defaults to mongodb://localhost:27017 Mongoose looks like a well-liked JavaScript ODM for Mongo. You can think of each "row" (called a "document") as a JSON blob. You can nest things (arrays, objects, etc) in the blob. The blob is named using a UID, which is like a primary key but alphanumeric. You can do some fancy filtering that’s not possible with SQL and index specific keys on the blob. Looks like you define classes that map to tables ("schemas") but it doesn’t look like you can easily extend them. You can add individual methods to a class but you can’t extend a mongoose model class. It looks like a mongoose.connection call creates an event loop. Without closing the event loop, the process will hang. Use process.exit() to kill all event loops. Relatedly, all mongo DB calls are run async, so you’ll want to await them if you expect results synchronously. brew install mongodb-compass-community gives you a GUI to explore your mongo DB. Similar to Postico for Postgres. Open Questions How are event loops, like the one mongoose uses implemented? Is the node event loop built in Javascript or are there C-level hooks used for performance? There are lots of gaps in the default REPL experience. Is there an improved repl experience for hacking? Do Blitz/RedwoodJS/others materially improve the server side JS experience? What killer features does mongodb have? How does it compare to other document databases? Is there a real reason to use document databases now that most SQL databases have a jsonb column type with an array of json operators built in?

Continue Reading