Learning TypeScript by Migrating Mint Transactions

Years ago, I built a chrome extension to import transactions into Mint. Mint hasn’t been updated in nearly a decade at this point, and once it stopped connecting to my bank for over two months I decided to call it quits and switch to LunchMoney which is improved frequently and has a lot of neat developer-focused features.

However, I had years of historical data in Mint and I didn’t want to lose it when I transitioned. Luckily, Mint allows you to export all of your transaction data as a CSV and LunchMoney has an API.

I’ve spent some time brushing up on my JavaScript knowledge in the past, and have since used Flow (a TypeScript competitor) in my work, but I’ve heard great things about TypeScript and wanted to see how it compared. Building a simple importer tool like this in TypeScript seems like a great learning project, especially since the official bindings for the API is written in TypeScript.

Deno vs Node

Deno looks cool. It uses V8, Rust, supports TypeScript natively, and seems to have an improved REPL experience.

I started playing around with it, but it is not backwards compatible with Node/npm packages which is a non-starter for me. It still looks pretty early in its development and adoption. I hope Deno matures and is more backwards compatible in the future!

Learning TypeScript You can’t run TypeScript directly via node (this is one of the big benefits of Deno). There are some workarounds, although they all add another layer of indirection, which is the primary downfall of the JavaScript ecosystem in my opinion. ts-node looks like the easiest solution to run TypeScript without a compilation step. npm i ts-node will enable you to execute TypeScript directly using npx ts-node the_script.ts. However, if you use ESM you can’t use ts-node. This is a known issue, and although there’s a workaround it’s ugly and it feels easier just to have a watcher compile in the background and execute the raw JS. .d.ts within repos define types on top of raw JS. This reason this is done is to allow a single package to support both standard JavaScript and TypeScript: when you are using TypeScript the .js and .d.ts files are included in the TypeScript compilation process. Use npx tsc --init to setup an initial tsconfig.json. I turned off strict mode; it’s easier to learn a new typing system without hard mode enabled. Under the hood, typescript transpiles TypeScript into JavaScript. If you attempt to debug a TypeScript file with node inspect -r ts-node/register the code will look different and it’ll be challenging to copy/paste snippets to debug your application interactively. Same applies to debugging in a GUI like VS Code. You can enable sourcemaps, but the debugger is not smart enough to map variables dynamically for you when inputting strings into the console session. This is massive bummer for me: I’m a big fan of REPL-driven development. I can’t copy/paste snippets of code between my editor + REPL, it really slows me down. Similar to most other languages with gradual typing (python, ruby, etc), there are ‘community types’ for each package. TypeScript is very popular, so many/most packages includes types within the package itself. The typing packages need to be added to package.json. There’s a nice utility to do this automatically for you. If you want to be really fancy you can overload npm i and run typesync automatically. VS Code has great support for TypeScript: you can start a watcher process which emits errors directly into your VS Code status bar via cmd+shift+b. If you make any changes to tsconfig.json you’ll need to restart your watcher process. You can define a function signature that dynamically changes based on the input. For instance, if you have a configuration object, you can change the output of the function based on the structure of that object. Additionally, you can inline-assign an object to a type, which is a nice change from other languages (ruby, python). Example of inline type assignment: {download: true} as papaparse.ParseConfig<Object>. In this case, Object is an argument into the ParseConfig type and changes the type of the resulting return value. Very neat! I ran into Element implicitly has an 'any' type because expression of type 'string' can't be used to index type 'Object'. No index signature with a parameter of type 'string' was found on type 'Object. The solution was typing a map/object/hash with theVariable: { [key: string]: any } . I couldn’t change the any type of the value without causing additional typing errors since the returning function was typed as a simple Object return. There’s a great, free, extensive book on TypeScript development.

One of the most interesting pieces of TypeScript is how fast it’s improving. Just take a look at the changelog. Even though JavaScript isn’t the most well-designed language, "One by one, they are fixing the issues, and now it is an excellent product." A language that has wide adoption will iterate it’s way to greatness. There’s a polish that only high throughput can bring to a product and it’s clear that after a very long time JavaScript is finally getting a high level of polish.

Linting with ESLint, Code Formatting with Prettier ESLint looks like the most popular JavaScript linting tool. It has lots of plugins and huge community support. You can integrate prettier with eslint, which looks like the most popular code formatting tool. VS code couldn’t run ESLint after setting it up. Had trouble loading /node_modules/espree/dist/espree.cjs. Restarting VS Code fixed the problem.

Here’s the VS Code settings.json that auto-fixed ESLint issues on save:

{ "[typescript]": { "editor.formatOnSave": true, "editor.codeActionsOnSave": { "source.fixAll.eslint": true }, }, "eslint.validate": ["javascript"] }

And here’s the .eslintrc.json which allowed ESLint, prettier, and ESM to play well together:

{ "env": { "browser": false, "es2020": true }, "extends": [ "standard", "plugin:prettier/recommended" ], "parser": "@typescript-eslint/parser", "parserOptions": { "ecmaVersion": 2020, "sourceType": "module" }, "plugins": [ "@typescript-eslint" ], "rules": { } } Module Loading

As with most things in JavaScript-land, the module definition ecosystem has a bunch of different community implementation/conventions. It’s challenging to determine what the latest-and-best way to handle module definitions is. This was a great overview and I’ve summarized my learnings below.

require() == commonjs == CJS. You can spot modules in this format by module.exports in their package. This was originally designed for backend JavaScript code. AMD == Asynchronous Module Definition. You can spot packages in this style by define(['dep1', 'dep2'], function (dep1, dep2) { at the header of the index package. Designed for frontend components. UMD == Universal Module Definition. Designed to unify AMD + CJS so both backend and frontend code could import a package. The signature at the top of the UMD-packaged module is messy and basically checks for define, module.exports, etc. import == ESM == ES Modules. This is the latest-and-greatest module system officially baked into ES6. It has wide browser adoption at this point. This is most likely what you want to use. import requires module mode in TypeScript (or other compilers) not set to commonjs. If you use ESM, your transpiled JS code will look a lot less garbled, and you’ll still be able to use the VS Code debugger. The big win here is your import variable names will be consistent with your original source, which it makes it much easier to work with a REPL. There are certain compatibility issues between ESM and the rest of the older package types. I didn’t dig into this, but buyer beware. It looks like experimental support for loading modules from a URL exist. I hope this gets baked in to the runtime. There are downsides (major security risks), but it’s great for getting the initial version of something done. This was one of the features I thought was neat about Deno: you could write a script with a single JavaScript file without creating a mess of package*, tsconfig.json, etc files in a new folder. https://unpkg.com is a great tool for loading a JS file from any repo on GitHub. You’ll get Cannot use import statement inside the Node.js REPL, alternatively use dynamic import if you try to import inside of a repl. This is a known limitation.. The workaround (when in es2020 mode) is to use await import("./out/util.js"). When importing a commonjs formatted package, you’ll probably need to import specific exports via import {SpecificExport} from 'library'. However, if the thing you want to import is just the default export you’ll run into issues and probably need to modify the core library. Here’s an example commit which fixed the issue in the LunchMoney library When importing a local file, you need to specify the .js (not the ts) in the import statement import { readCSV, prettyPrintJSON } from "./util.js"; Package Management You can install a package directly from a GitHub reference npm i lunch-money/lunch-money-js You can’t put comments in package.json, which is terrible. Lots of situations where you want to document why you are importing a specific dependency, or a specific forked version of a dependency. npm install -g npm to update to the latest npm version. By default, npm update only updates packages to the latest minor semver. Use npx npm-check-updates -u && npm i to update all packages to the latest version. This is dangerous, and only makes sense if there are a small number of packages https://openbase.com is a great tool for helping decide which package to use. JavaScript Learnings You’ll want to install underscore and use chain for data manipulation: _.chain(arr).map(...).uniq().value(). Lots of great tools you are missing from ruby or python. ES6 introduced computed property names so you can use a variable as an object key { [variableKey]: variableValue } I had trouble getting papaparse to read a local file without using a callback. I hate callbacks; here’s a promise wrapper that cleaned this up for me. Merge objects with _.extend. The dotenv package didn’t seem to parse .env with exports in the file. Got tripped up on this for a bit. require can be used to load a JSON file, not just a javascript file. Neat! There are nice iterators now! for(const i in list) There’s array destruction too const [a, b] = [1,2] Underscore JS has a nice memoize method. I hate the pattern of having a package-level variable for memoization. Just feels so ugly. There’s a in keyword that can be used with objects, but not arrays (at least in the way you’d expect). There’s a null-safe operator now. For instance, if you want to safely check a JSON blob for a field and set a default you can now do something like const accounts = json_blob?.accounts || [] You are iterate over the keys and values of an object using for (const [key, value] of Object.entries(object)) https://github.com/ccxt/ccxt is a neat project which transpiles JavaScript code into multiple languages. Hacking & Debugging The most disappointing part of the node ecosystem is the REPL experience. There are some tools that (very) slightly improve it, but there’s nothing like iPython or Pry. nbd is dead and hasn’t been updated in years node-help is dead as well and just made it slightly easier to view documentaiton. node-inspector is now included in node and basically enables you to use Chrome devtools local-repl looks neat, but also hasn’t been updated in ~year. The updated repl project wouldn’t load for me on v16. The debugging happy path seems to be using the GUI debugger. You can use toString() on a function to get the source code. Helpful alternative to show-source from ruby or ll from python. However, it has some gotchas: It’s specifically discouraged since it’s been removed from the standard Arguments and argument defaults are not specified It’s not obvious how to list local variables in the CLI debugger. There’s a seemingly undocumented exec .scope that you can run from the debugger context (but not from a repl!). You can change the target to ES6 to avoid some of the weird JS transpiling stuff, Run your script with node inspect and then before conting type breakOnUncaught to ensure that you can inspect any exceptions. I prefer terminal-based debugging, if you want to connect to a GUI (chrome or VS Code) use --inspect. There’s not a way I could find to add your own aliases to the debugging (i.e. c == continue == cont). It’s worth writing your own console.ts to generate a helpful repl environment to play with imports and some aliases defined. Unfortunately, this needs to be done on a per-project basis. You can’t redefine const variables in a repl, which makes it annoying to copy/paste code into a console. It looks like there are some hacks you can use to strip out the const and replace with a let before the copy/pasted code gets eval’d. This seems like a terrible hack and should just be a native flag added to node. In more recent versions of node (at least 16 or greater), you can use await within a repl session. If you are in a debugger session await does not work, unlike when you are a standard node repl. You cannot resolve promises and therefore cannot interact with async code. This is a known bug, will not be changed, and makes debugging async code interactively extremely hard. Very surprised this is still a limitation. console.dir is the easiest way to inspect all properties of an object within a REPL. This uses util.inspect under the hood, so you don’t need to import this package and remember the function arguments. There’s a set of functions only available in the web console. Most of these seem to model after jQuery functions. Open Questions How can I run commands without npx? Is there some shim I can add to my zsh config to conditionally load all npx-enabled bins when node_modules exists? Is there anything that can done to make the repl experience better? This is my biggest gripe with JavaScript development. https://github.com/11ways/janeway looks interesting but seems dead (no commits in over a year) This code looks like an interesting starting point to removing all const that are pasted into a repl. The number of configuration files you need to get started in a repo just to get started is insane (tsconfig.json, package*.json, .eslintc.json). Is there a better want to handle this? Some sort of single configuration file to rule them all?

Continue Reading

Building a SouthWest Price Monitor and Learning Server Side JavaScript

I originally wrote a draft of this post in early 2019. I’m spending some time learning TypeScript, so I wanted to finally get my JavaScript-related posts out of draft. Some notes and learnings here are out of date.

Both sides of our family live out of state. Over the last couple years, we’ve turned them on to credit card hacking to make visiting cheap (free). SouthWest has some awesome point bonuses on credit cards, but you can’t watch for price drops on Kayak and other flight aggregators.

After a bit of digging, I found a basic version of a tool to do just this. It’s a self-hosted bot to watch for flight cost drops so you can book (or rebook for free). I’ve been wanting to dig into server side JavaScript development, and this is the perfect excuse.

Here’s what I’d like to do:

Get the tool running somewhere simple: Heroku, Raspberry Pi, etc Convert the use of redis to mongodb. Redis isn’t a database, it’s a key-value store. But this project is using it for persistence. Why switch to MongoDB? I’ve been wanting to understand document databases a bit more. Postgres would have been easier for me, but this project is all about learning. Possibly add the option of searching for the best flight deal on a particular month

Below is a ‘learning log’ of what I discovered along the way. Let’s get started!

Learning JavaScript

As I mentioned in an earlier post, my JavaScript knowledge was very out of date (pre ES6). Some findings and musings below will be obvious to a seasoned JavaScript developer, but to someone more experienced in Ruby/Python/etc they’ll be some interesting tidbits.

Looks like express is the dominant HTTP router + server. It’s equivalent to the routing engine of Rails combined with rack and unicorn. It doesn’t seem like there are strong conventions to how you setup an express-based app. You bring your own ODM/ORM, testing library, etc. There is a consistent template/folder structure. However, express doesn’t make any assumptions about a database library, although it does support a couple of different templating languages and has a preferred default (pug). app.use adds additional middleware to the stack. Middleware is simply a function with three arguments. Very similar to rack in ruby-land or plugs in Elixir-land. There’s a part of me that loves the micro-modularity of the node/npm ecosystem, but the lack of declarative programming like DateTime.now + 1.day starts to feel really messy. The equivalent in node is (new Date()).setDate((new Date()).getDate() + 1);. Another example: there’s no built-in sortBy and sort mutates the original array. Some popular packages that solve this (moment, datefuncs, underscore, etc) and the popular choice is to just pull in these packages and use them heavily. I always find including many external decencies adds a lot of maintenance risk to your code. These packages can die, cause strange performance issues, cause weird compatibility issues with future iterations of the language, etc. The good news is the JavaScript ecosystem is so massive, the most popular packages have a very low risk of abandonment. Variable scoping is weird in debugger mode. If the variable isn’t referenced in the function, it’s not available to inspect in the debugger repl. Make sure you reference the variable to inspect/play with it in real time. Node, express, etc are not billed as full-stack web frameworks like rails. I find this super frustrating: not being able spin up a console (rails console) with your entire app’s environment loaded up is annoying. For this particular problem, it looks like the best alternative is to write your own console.js (here’s another guide) with the things you need and startup a repl. The annoying thing here is you need to manually connect to your DB and trigger the REPL after the DB connection is successful. Blitz and Redwood are solving these problems, although these didn’t exist when this post was written. It seems like node inspect + a debugger line doesn’t run the code ‘completely’. For instance, if the code runs past a mongodb.connection line it doesn’t connect. I wonder if this is because the .connection call runs async and doesn’t get a chance to execute before the debugger line is called? Is there a way to instruct the repl to execute anything in the async queue? I found that starting up a vanilla node console and requiring what you needed works better. There are some interesting utility libraries that convert all methods on an object to be promises (async). http://bluebirdjs.com/docs/api/promise.promisifyall.html Languages with declarative convenience methods are just so much nicer. args.priceHistory[args.priceHistory.length - 1] is just ugly compared to args.priceHistory.last. My time at a BigCo has helped me understand the value of typing. I still find the highest velocity developer experience is type-hinting (i.e. types are not required) combined with a linter. This lets you play with code without getting all the details hardened, but still enforces guardrails to avoid a class of production errors. I’m not seeing the value in the event-loop programming paradigm. I get how it allows you to handle more concurrent connections, but isn’t that something that should be handled by the language or in some lower level abstraction? It’s much easier to reason about code when it runs sequentially. For instance, not having object.save throw an exception right away is really annoying: I need to either use callbacks to act when the code has executed OR use async and await everywhere. I do not understand why this pattern has become so popular. https://repl.it is very cool. The idea of sending out links with a console running your code is very handy. This is used a lot in the JavaScript community. It’s fascinating to me how there’s always the 10x-er that becomes a hero of the community. https://github.com/substack has created a ridiculous number of npm packages. Think about let r = await promise as let r = null; promise.then(rr => r = rr) which is executed synchronously. Instead of hash.merge(h2) you write Object.assign({}, h2, hash). There are many unintuitive sharp edges to the language, as you learning, just googling "how to do X with JavaScript" is the best way to determine the JavaScript equivalent. http://jsnice.org is great at parsing obfuscated JS. It tries to rename variables based on the context. Very cool. ... is the splat operator used on objects It’s called the ‘rest’ operator. constructor is the magic method for class initialization Looks like function definitions within a class don’t need the function keyword Puppeteer, Proxies, and Scraping

Part of this project involved scraping information the web. Here’s some tidbits about scraping that I learned:

The node ecosystem is great for web scraping. Puppeteer is a well maintained chrome-controller package and there’s lot of sample code you can leverage to hack things together quickly. Websites have gotten very good at detecting scrapers. There are some workarounds to try to block bot detection, but if you are using a popular site, you will most likely be detected if you are using the default puppeteer installation. A common (and easy) detection method is IP address. If you are scraping from an AWS/cloud IP, you’ll be easily blocked. The way around this is a proxy to a residential IP address. Another option is to host your scraper locally on a Raspberry Pi or on your local computer. https://chrome.browserless.io cool way to test puppeteer scripts I learned a bit about web proxies. Firstly, there are a bunch of proxy protocols (SOCKS, HTTP with basic auth, etc). Different systems support different type of proxies. Package Management You can’t effectively use npm and yarn in the same project. Pick one or the other. Yarn is a more stable, more secure version of npm (but doesn’t have as many features / as much active development) module.exports lets a file expose constants to others which import the file, similar to python’s import system (but with default exports). I like this compared with ruby’s "everything is global" approach. It allows the other author to explicitly define what it wants other users to access. Npm will run pre & post scripts simply based on the name of the scripts. import Section, {SectionGroup} assigns Section to the default export of the file, and imports the SectionGroup explicitly. If you try to import something that isn’t defined in the module.exports of a file you will not get an error and will instead get an undefined value for that import. Testing tape is the test runner that this particular project used. It doesn’t look like it’s possible to run just a single test in a file without changing the test code to use test.only instead of test. The "Test Anything Protocol" is interesting http://testanything.org. Haven’t run into this before. I like consistent test output across languages. I do like how tape tests list out the status of each individual assertion. It becomes a bit verbose, but it’s helpful to see what assertions after the failing assertion succeeded or failed. VS Code + node debugging is very cool when you get it configured. You need to modify your VS Code launch.json in order to get it to work with test files. https://gist.github.com/dchowitz/83bdd807b5fa016775f98065b381ca4e#gistcomment-2204588 Debugging & Hacking

I’m a big fan of REPL driven development and I always put effort into understanding the repl environment in a language to increase development speed. Here are some tips & tricks I learned:

Tab twice (after inputting ob.) in a repl exposes everything that is available on the object under inspection. node inspect THE_FILE.js allows debugger statements to work. You can also debug remotely with chrome or with VS Code. Visual debugging is the happy path with node development, the CLI experience is poor. You don’t need to setup variables properly in the node repl. Nice! You can just a = 1 instead of let a = 1 I’ll often copy code into a live console to play around with it, but if it’s defined as const I need to restart the console and make sure I don’t copy the const part of the variable definition. That’s annoying. There’s a lot of sharp edges to the developer ergonomics. console.dir to output the entire javascript object Unlike pry you need to explicitly call repl after you hit a breakpoint when running node inspect. Also, debugger causes all promises not to resolve when testing puppeteer. https://github.com/berstend/puppeteer-extra/wiki/How-to-debug-puppeteer Cool! Navigating to about:inspect in Chrome allows you to inspect a node/puppeteer process. list is equivalent to whereami. You need to execute it explicitly with params list(5) _ exists like in ruby, but it doesn’t seem to work in a repl triggered by a debugger statement. _error is a neat feature which keeps the last exception that was thrown. .help while in a repl will output a list of "dot commands" you can use in the repl. I had a lot of trouble getting puppeteer to execute within a script executed with node inspect and paused with debugger. I’m not sure why, but I suspect it has something to do with how promises are resolved in inspect mode. You can enable await in your node console via --experimental-repl-await. This is really helpful to avoid having to write let r; promise.then(o => r) all of the time. Mongo & ODMs You’ll want to install mongo and the compass tool (brew install mongodb-compass) for GUI inspection. Running into startup problems? tail -f ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist If you had an old version of mongo install long ago, you may need to brew sevices stop mongodb-community && rm -rf /usr/local/var/mongodb && mkdir /usr/local/var/mongodb && brew services start mongodb-community -dv The connection string defaults to mongodb://localhost:27017 Mongoose looks like a well-liked JavaScript ODM for Mongo. You can think of each "row" (called a "document") as a JSON blob. You can nest things (arrays, objects, etc) in the blob. The blob is named using a UID, which is like a primary key but alphanumeric. You can do some fancy filtering that’s not possible with SQL and index specific keys on the blob. Looks like you define classes that map to tables ("schemas") but it doesn’t look like you can easily extend them. You can add individual methods to a class but you can’t extend a mongoose model class. It looks like a mongoose.connection call creates an event loop. Without closing the event loop, the process will hang. Use process.exit() to kill all event loops. Relatedly, all mongo DB calls are run async, so you’ll want to await them if you expect results synchronously. brew install mongodb-compass-community gives you a GUI to explore your mongo DB. Similar to Postico for Postgres. Open Questions How are event loops, like the one mongoose uses implemented? Is the node event loop built in Javascript or are there C-level hooks used for performance? There are lots of gaps in the default REPL experience. Is there an improved repl experience for hacking? Do Blitz/RedwoodJS/others materially improve the server side JS experience? What killer features does mongodb have? How does it compare to other document databases? Is there a real reason to use document databases now that most SQL databases have a jsonb column type with an array of json operators built in?

Continue Reading

Building a Chrome Extension to Import Transactions into Mint

I originally wrote a draft of this post in early 2019. I’ve since stopped using Mint and switched to LunchMoney. However, I’m spending some time learning TypeScript so I wanted to finally get my JavaScript-related posts out of draft.

I use Mint (although it’s rotting on the vine after being acquired by Intuit), and want to import a list of transactions from a bank account that isn’t supported. However, there’s not a way to do this through the mint UI, but there is a hack someone documented.

I know old-school JavaScript but haven’t learned ES6, and I’ve never built a Chrome extension. Building a Chrome extension to use the private mint API to import transactions from a CSV is a perfect learning project.

As I built this extension, I ‘liveblogged’ my learnings which I’ve included below. Here’s the final source code of the project.

Reverse Engineering the Mint API

The first question I needed to answer is "Can I batch import transactions using a Mint API?". There is no public API, so I wasn’t sure; I had to attempt to reverse engineer how manual transactions were added.

The hacky blog post explaining how to batch import transactions is really old, so I wanted to validate the approach myself.

First, let’s ensure that the request hitting mint’s servers look about the same as the linked blog post. I pulled this curl command from the web console when adding a transaction manually in the Mint UI:

curl 'https://mint.intuit.com/updateTransaction.xevent' \ -H 'cookie: ...' \ -H 'origin: https://mint.intuit.com' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' \ -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \ -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=REDACTED' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' \ -H 'adrum: isAjax:true' \ --data ' cashTxnType=on& mtCashSplit=on& mtCheckNo=& tag806772=0& tag897426=0& tag1109697=0& tag975947=0& tag806773=0& tag806774=0& task=txnadd& txnId=%3A0& mtType=cash& mtAccount=3444067& symbol=& note=& isInvestment=false& catId=7& category=Food%20%26%20Dining& merchant=TEST& date=04%2F12%2F2019& amount=1& mtIsExpense=true& mtCashSplitPref=1& token=REDACTED' --compressed

It looks similar, but different enough from the old blog post I found. Some notes:

%3A0 url decoded is :0 mtAccount is not the same as the redacted accountId in the referrer I wonder if there is a query that can dump the catId list. Or if you can submit without a catId and category will auto match. Do we need a token? That would be a bummer.

I tried running this exact command again locally to see if it works with the tokens embedded in the command. I’d be surprised if it did, since some of those tokens look like a server-side generated CSRF token.

But, to my surprise, it worked! Here was the result.

{"task":"txnAdd","mtType":"CASH"}

I refreshed the Mint account and the transaction appears there as well. Great! At least we know it’s possible to push the data into mint.

Now, let’s see what parameters we can eliminate to make the request as simple as possible. the tag* and token params seem like the lowest hanging fruit…

<error><code>1</code><description>Session has expired.</description><name></name><type></type></error>

Hmm, let’s try adding in the token param. That’s probably tied to the session:

{"task":"txnAdd","mtType":"CASH"}

It worked! I’m guessing the token is embedded in the page source somewhere, or it could be pulled via another HTTP call (which would be a bummer). I poked around the

<input type="hidden" id="javascript-token" value="REDACTED_HASH"/>

I ran another request from the mint UI and the token used in the request matches up. We’ll have to parse the source for #javascript-token and extract that from the page. A pain, but doable.

Digging around the console a bit more it does look like there is a category API that is called!

curl 'https://mint.intuit.com/app/getJsonData.xevent?task=categories&rnd=1555119027945' \ -H 'cookie: REDACTED' \ -H 'accept-encoding: gzip, deflate, br' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' -H 'accept: */*' \ -H 'referer: https://mint.intuit.com/transaction.event?accountId=3871381' \ -H 'authority: mint.intuit.com' \ -H 'x-requested-with: XMLHttpRequest' --compressed

This returns a nice JSON blob:

{ "set": [ { "data": [ { "children": [ { "isStandard": true, "id": 1405, "value": "Auto Insurance", "isL1": false } ... ], "id": 14, "value": "Auto & Transport", "isL1": true } ] } ] }

Which we can use to set the catId in the previous API which added a transaction.

After discovering each category has an ID, I wanted to test if we could remove the text representation of the category in the previous API call (this might break future iterations of the mint internal API, so may not be worth doing). Additionally, this reminds me that we also will need to set the mtAccount ID dynamically. Let’s see if this ID exists in the page source.

It looks like when you initially load the transaction view the accountId query parameter is set. However, when you click on a different account the URL fragment (everything after the #) changes to include the account that was chosen:

https://mint.intuit.com/transaction.event?accountId=3444067#location:%7B%22accountId%22%3A3436098%2C%22offset%22%3A0%2C%22typeSort%22%3A8%7D

Searching through the dynamic page source (via the elements tab) I was able to find this reference to the ID:

<a id="transactionExport" href="https://mint.intuit.com/transactionDownload.event?accountId=3436098&offset=0&comparableType=8">Export all 1797 transactions</a>

Looks like we can either pull the accountId off of the URL of the #transactionExport element or parse the URL fragment. The fragment may be a bit more work given the weird format (URL encoded params look to be prefixed by location:). It looks like mint is using very old javascript technology, so we can’t make too many assumptions about the URL fragment structure.

Now we know we can build a prototype of a mint importing tool. Here’s what we need to do:

Pull the session cookie after the user logs in Pull the #javascript-token from the page source Pull the accountId either from the URL fragment or the transactionExport Hit the https://mint.intuit.com/updateTransaction.xevent endpoint to add transactions Optionally use the category API

Now to learn how google chrome extensions work!

Building a Chrome Extension

Firstly, how can we iterate on a Chrome extension while it’s in development? I google’d "best practices building a Chrome extension":

https://usersnap.com/blog/develop-chrome-extension

https://thoughtbot.com/blog/how-to-make-a-chrome-extension

https://github.com/kippt/kippt-chrome

https://github.com/yeoman/generator-chrome-extension

@@extension_id can be used to reference your extension ID in CSS. Most likely this works for JS and HTML as well.

To publish the extension, you need to register with Google and follow some guidelines.

"If you’re building a Chrome extension which needs to interact with web pages that are loaded by users, you definitely need a content script." This is our scenario.

"And luckily testing your new extension is pretty straightforward. Once you’ve activated the “developer mode”… You can simply add your unpacked extension to your Chrome browser to test it." I want to make sure I can test the Chrome extension quickly by editing some JS and reloading the page.

"When you change or add code in your extension, just come back to this page and reload the page." sounds like we need to reload the chrome://extensions extensions page?

Looks like we can load external libraries via a CDN using the manifest JSON. Great! This will make things easier for initial development.

You can limit which domains your extension activates on. We should do this and scope it down to Mint

Looks like all we need is a folder with a JS file and a manifest. Looks easy enough.

Yeoman scaffold looks cool. Uses babel which allows you to write ES6 (which I want to learn) and implements best practices. Looks kind of updated (last commit <1yr ago).

Ok, I think I have enough information. Let’s try that scaffold out:

npm install --global yo gulp-cli bower npm WARN deprecated bower@1.8.8: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/

Eek, not good. Scaffold looks to be too old. Let’s try to search for another newer scaffold…

https://github.com/edrpls/chrome-extension-template

This one looks newer. It uses webpack which I’ve heard good things about and been wanting to understand better.

git clone git@github.com:edrpls/chrome-extension-template.git mintporter cd mintporter/ npm install

The dependency graph that npm generated was massive, but it worked. Let’s try to run the start command.

yarn start -bash: yarn: command not found

Yarn is an npm alternative. I needed to install it via brew:

brew install yarn yarn start

With yarn running, I edited the manifest.json to use the "Mintporter" name, enable dev mode on chrome://extensions/, and loaded the extension via the dist/ folder. Now it’s loading on Chrome.

Now to understand how the template is structured:

Looks like webpack is similar to bower and other JS bundling tools. entry defines the output files. Interesting, looks like there is a library to specifically help with bundling chrome extensions. https://github.com/johnagan/crx-webpack-plugin Looks like mostly babel (transpiler) packages in package.json. I was confused by the rimraf in the template. It just adds the rm command in node.

Time to learn the latest JavaScript syntax!

Learning ES6

Googled "learn latest javascript":

https://javascript.info https://developer.mozilla.org/en-US/docs/Web/JavaScript/A_re-introduction_to_JavaScript https://derickbailey.com/2017/06/06/3-features-of-es7-and-beyond-that-you-should-be-using-now/ https://www.youtube.com/watch?v=cCOL7MC4Pl0 How the javascript event loop works. I’ve always found the javascript runtime a bit strange. https://www.smashingmagazine.com/2016/07/how-to-use-arguments-and-parameters-in-ecmascript-6/

Learnings:

let is the new var, but scoped to the block. You should use this instead of var const prevents the var from being reassigned. null/undefined/typeof seem to be unchanged We finally have default parameter values! Same syntax as ruby: function fn(arg="default") The => is just a shortcut for defining a new function. Left side is a list of function arguments, the right side is the code (normally a oner-liner) to be executed. data => port.postMessage(data) is the same as function(data) { return port.postMessage(data); } () => { console.log('in-content.js - disconnected from popup'); } is similar to the above syntax, but the {} does not default to returning the evaluated value. Brackets with arrow-defined functions allow multi-line functions but require an explicit return. => without brackets will return the last value in the one-liner function by default. Arrow-functions inherit the this value of the callee. async/await is the native way to write asynchronous code. Promises are now JavaScript-native and async/await use these under the hood. You can think of them as background jobs you can easily run in the browser. Template literals now exist! Use backticks: string text ${expression} string text. You can now define classes without prototypes! Yay. Use class as the keyword, constructor as the initializer. There’s no native Class object; it’s basically syntactic sugar on top of the old prototype model. Feels a lot like CoffeeScript. get and set language keywords exist inside a class to define custom getters and setters. Protected class variables are still convention-only. Still no private properties and methods, although this is in progress. Class-methods are defined using static Mixins are still messy. No language keywords to make this simple: you need to define objects and use low-level javascript calls to copy the methods of the object in. for...of loops are an easy way to iterate through all objects in an array. ...args is the "rest" syntax which allows you to represent multiple args passed to a function as an array. Similar to the splat argument in ruby *args. import is the newer version of require. I didn’t get a good sense of when import isn’t supported in various JavaScript versions. Destructuring is a mix between pattern matching and keyword arguments. function boom({ keyword }) => function boom(obj) { keyword = obj.keyword }. When looking at a function call: boom(a, b: c) is equivilent to boom(a: a, b: c)

Now, with enough new JavaScript knowledge under my belt, I can start hacking away at the extension!

Building the Prototype You need to manually click the reload button in chrome://extensions/ to pull in a new version of the javascript. Bummer. https://www.npmjs.com/package/webpack-chrome-extension-reloader looks like it may fix the issue. npm install webpack-chrome-extension-reloader --save-dev and NODE_ENV=development yarn start. https://stackoverflow.com/questions/2963260/how-do-i-auto-reload-a-chrome-extension-im-developing and https://github.com/arikw/chrome-extensions-reloader are also interesting. Use import $ from "jquery"; to pull jQuery in using the new import syntax. Babel will automatically backport this to be supported on an old JS version. Hmm, my extension is being automatically disabled. Weird. I’m getting an error on the extension page relating to inter-page communication. Cutting out that code from the in-content file. I’m going to start with two main classes: MintContext (to pull auth) and MintIntegrator to add transactions to mint. It would be nice to have an "Import Transactions" button. Looks like we can add it to #controls-top. I’ll need a way to watch for an element to appear on the page since transactions are loaded async. This page indicates I could just jQuery’s ajaxStop, but that isn’t working. DOMSubtreeModified event looks like the old-school way of watching for changes. #product-view-root looks like the best object to observe and MutationObserver looks like the API we want to use. Also, the chrome extension reloader isn’t working, I think I need to modify how webpack is interacting with yarn. To add that to that, my extension keeps getting disabled and I need to restart chrome to allow me to re-enable it again. https://www.ghacks.net/2017/07/04/hide-chromes-disable-developer-mode-extensions-warning/ let’s try using chrome canary and see if that fixes the chrome reloading issue. Looks like we don’t need the chrome reloader extension. The webpack plugin creates a websocket and listens for changes. Sidenote: I know some folks don’t like global namespaces, but I do find it annoying that you can mutate the name of any library into whatever global object you like via require or import. Maybe I’ll discover the benefits of this later on and change my mind. After more playing around, I got a better understanding of yarn. It’s just an improved npm and works off of the commands defined in package.json => scripts. It’s meant to be more stable than NPM. webpack-chrome-extension-reloader asks you to use --watch in webpack. However, the chrome extension template is setup to use nodemon with yarn build which I’m guessing monitors the filesystem and circumvents webpack‘s --watch. I wonder if there is a way to reload the extensions outside the --watch lifecycle. You can learn which process is using a port using lsof -i tcp:3000. Discovered this while trying to get the extension reload working. https://stackoverflow.com/questions/4075287/node-express-eaddrinuse-address-already-in-use-kill-server Hmm, I can’t get the reloader to work. Going to give up and manually reload. Ugh. #body-mint is the only element that exists on $(document).ready. Watching for changes in the DOM is going to kill performance (lots of events to comb through). Let’s just use a setTimeout based approach. https://gist.github.com/chrisjhoughton/7890303 Weird. My jQuery version is really old (1.x.x) but the npm package is the latest version. There is some sort of namespace conflict happening. Reverting back to the simple import $ from 'jquery'; to eliminate the issue. Working in the console uses the mint version of jQuery as opposed to the version you bundle with the app. This makes things tricky: you can’t run code live in the console. Hmm, the #javascript-token is empty. Mint.getToken() seems to work though. I’ll try using this instead. Looks like class vars are a new-ish feature. They are causing a compilation error in the version of babel I’m using. Looking into a bit deeper, they aren’t supported in babel yet. Bummer. Running into weird scoping issues when accessing the top-level Mint object. I’m guessing this has to do with my babel config. Man, the stack of javascript transforms on top of the raw browser is still such a pain. Actually, it’s not a babel config issue: chrome extensions can access the entire DOM of the page they are on, but they can’t access the javascript runtime of that page. All javascript runs in a separate sandboxed environment. However, you can access localStorage. This means Mint.getToken() won’t work. Luckily the CSFR token is stored in the session so we can just pull it from there. Now that we have a single transaction pushing into Mint when the "Import Transactions" button that we added to the page is pressed, we want to allow a CSV to be imported. From what I could tell, there’s not any great API to allow the user to select a file and read it into a string. You need to create a hidden file input, and "click" the input during a user-initiate click event, and then when the file input value has changed use the FileReader class to read the file into a string. Bummer. https://stackoverflow.com/questions/32490959/filereader-on-input-change-jquery and https://mariusschulz.com/blog/programmatically-opening-a-file-dialog-with-javascript. Ok, I have the file dialog opening within a click event triggered by a button (so the user doesn’t need to see the file input element). I wonder if I can wrap the entire "read a file" logic into a single function which returns a call back… Cool! We can insert the file input during the click call and the dialog still opens. This allows us to wrap the file input in a single method. Frustrating: you can’t execute code in the console referencing top-level constants defined via ES6. You need to use the webpack converted versions: __WEBPACK_IMPORTED_MODULE_1_papaparse___default. This makes it hard to fiddle around with code in the console. Ideally, we could run the import task (iterating over the CSV) async. We do want to run the mint request sync so we can easily aggregate errors and blow up if something breaks mid-way through. https://petetasker.com/using-async-await-jquerys-ajax/the I was curious what the difference is between TypeScript and the latest JS spec. Looks like TypeScript is just a typed version of the latest JS implementation. https://www.quora.com/What-is-the-difference-between-TypeScript-and-JavaScript Bah! You can’t add an account in mint that isn’t tied to a bank account login. Bummer. I can use an old credit card account as a hack, but this is a unfortunate limitation. I’m curious if someone else has reverse engineered mint’s API. https://webapps.stackexchange.com/questions/11398/does-mint-com-have-an-api-to-download-data-if-not-are-any-scraping-tools-avail and https://github.com/dhleong/pepper-mint. Interesting! Let’s see if there’s any interesting parameters that we can use. After digging a bit, it doesn’t look like there’s anything we didn’t find. Although, it does look like the accountId isn’t used in the API call. Transactions imported will always just hit the global transaction list. Bummer! Passing over a category name that doesn’t match a name in Mint doesn’t do anything. I could implement fuzzy matching on inputs from the CSV to the category API, but that wouldn’t help me learn anything new (I’ve worked with string distance algorithms before). Time to clean this up and call it done! Lessons Learned

This was fun! Great to way to learn Chrome Extension Development, refresh my JavaScript toolchain knowledge, and learn the new ES6 JavaScript standard.

When doing a "hack" project, spend more time fiddling with the system to make sure it can solve the use-cases you are trying to add functionality for. In this case, it looked like transactions could be added to a specific account but if you refreshed the page everything pushed to the "global" transaction list. Would have been helpful to know this ahead of time. The layers of indirection present in JavaScript development cause issues. It would have eliminated some frustration and wasted time to remove babel and develop directly against the browser’s JS engine as opposed to layering in babel all at once. JavaScript development is powerful (adding in a custom script to a webpage that inherits cookies and localStorage allows for some interesting and powerful hacks). There’s a lot of quirks (browser differences, nuances in what extensions can and can’t do, etc) but it’s the #1 language that allows you to manipulate a common interface that everyone has easy access to. This was my first time using VS Code. There was a lot of interesting contextual information VS code was able to provide while editing files. It seems very powerful and is a lot more snappy compared to Atom. However, I felt like an infant struggling with the keyboard shortcuts and lack of some nice Atom features I’m used to. I should spend some more time learning VS Code (especially now that MS owns GitHub, I can’t imagine they’ll invest in two similar editors for that much longer). It’s super useful to have a simple sample project to work with when attempting to learn new technologies. It’s worth spending some time to think about an interesting and genuinely useful project you can work on that involves new technologies you are interested in.

Continue Reading