Skip to content

Building a Chrome Extension to Import Transactions into Mint

Tags: javascript • Categories: Learning

Table of Contents

I originally wrote a draft of this post in early 2019. I’ve since stopped using Mint and switched to LunchMoney. However, I’m spending some time learning TypeScript so I wanted to finally get my JavaScript-related posts out of draft.

I use Mint (although it’s rotting on the vine after being acquired by Intuit), and want to import a list of transactions from a bank account that isn’t supported. However, there’s not a way to do this through the mint UI, but there is a hack someone documented.

I know old-school JavaScript but haven’t learned ES6, and I’ve never built a Chrome extension. Building a Chrome extension to use the private mint API to import transactions from a CSV is a perfect learning project.

As I built this extension, I ‘liveblogged’ my learnings which I’ve included below. Here’s the final source code of the project.

Reverse Engineering the Mint API

The first question I needed to answer is "Can I batch import transactions using a Mint API?". There is no public API, so I wasn’t sure; I had to attempt to reverse engineer how manual transactions were added.

The hacky blog post explaining how to batch import transactions is really old, so I wanted to validate the approach myself.

First, let’s ensure that the request hitting mint’s servers look about the same as the linked blog post. I pulled this curl command from the web console when adding a transaction manually in the Mint UI:

curl 'https://mint.intuit.com/updateTransaction.xevent' 
-H 'cookie: ...' 
-H 'origin: https://mint.intuit.com' 
-H 'accept-encoding: gzip, deflate, br' 
-H 'accept-language: en-US,en;q=0.9' 
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' 
-H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' 
-H 'accept: */*' 
-H 'referer: https://mint.intuit.com/transaction.event?accountId=REDACTED' 
-H 'authority: mint.intuit.com' 
-H 'x-requested-with: XMLHttpRequest' 
-H 'adrum: isAjax:true' 
--data '
cashTxnType=on&
mtCashSplit=on&
mtCheckNo=&
tag806772=0&
tag897426=0&
tag1109697=0&
tag975947=0&
tag806773=0&
tag806774=0&
task=txnadd&
txnId=%3A0&
mtType=cash&
mtAccount=3444067&
symbol=&
note=&
isInvestment=false&
catId=7&
category=Food%20%26%20Dining&
merchant=TEST&
date=04%2F12%2F2019&
amount=1&
mtIsExpense=true&
mtCashSplitPref=1&
token=REDACTED' --compressed

It looks similar, but different enough from the old blog post I found. Some notes:

  • %3A0 url decoded is :0
  • mtAccount is not the same as the redacted accountId in the referrer
  • I wonder if there is a query that can dump the catId list. Or if you can submit without a catId and category will auto match.
  • Do we need a token? That would be a bummer.

I tried running this exact command again locally to see if it works with the tokens embedded in the command. I’d be surprised if it did, since some of those tokens look like a server-side generated CSRF token.

But, to my surprise, it worked! Here was the result.

{"task":"txnAdd","mtType":"CASH"}

I refreshed the Mint account and the transaction appears there as well. Great! At least we know it’s possible to push the data into mint.

Now, let’s see what parameters we can eliminate to make the request as simple as possible. the tag* and token params seem like the lowest hanging fruit…

<error><code>1</code><description>Session has expired.</description><name></name><type></type></error>

Hmm, let’s try adding in the token param. That’s probably tied to the session:

{"task":"txnAdd","mtType":"CASH"}

It worked! I’m guessing the token is embedded in the page source somewhere, or it could be pulled via another HTTP call (which would be a bummer). I poked around the 

<input type="hidden" id="javascript-token" value="REDACTED_HASH"/>

I ran another request from the mint UI and the token used in the request matches up. We’ll have to parse the source for #javascript-token and extract that from the page. A pain, but doable.

Digging around the console a bit more it does look like there is a category API that is called!

curl 'https://mint.intuit.com/app/getJsonData.xevent?task=categories&rnd=1555119027945' 
-H 'cookie: REDACTED' 
-H 'accept-encoding: gzip, deflate, br' 
-H 'accept-language: en-US,en;q=0.9' 
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36' -H 'accept: */*' 
-H 'referer: https://mint.intuit.com/transaction.event?accountId=3871381' 
-H 'authority: mint.intuit.com' 
-H 'x-requested-with: XMLHttpRequest' --compressed

This returns a nice JSON blob:

{
  "set": [
    {
      "data": [
        {
          "children": [
            {
              "isStandard": true,
              "id": 1405,
              "value": "Auto Insurance",
              "isL1": false
            }
            ...
          ],
          "id": 14,
          "value": "Auto & Transport",
          "isL1": true
        }
      ]
    }
  ]
}

Which we can use to set the catId in the previous API which added a transaction.

After discovering each category has an ID, I wanted to test if we could remove the text representation of the category in the previous API call (this might break future iterations of the mint internal API, so may not be worth doing). Additionally, this reminds me that we also will need to set the mtAccount ID dynamically. Let’s see if this ID exists in the page source.

It looks like when you initially load the transaction view the accountId query parameter is set. However, when you click on a different account the URL fragment (everything after the #) changes to include the account that was chosen:

https://mint.intuit.com/transaction.event?accountId=3444067#location:%7B%22accountId%22%3A3436098%2C%22offset%22%3A0%2C%22typeSort%22%3A8%7D

Searching through the dynamic page source (via the elements tab) I was able to find this reference to the ID:

<a id="transactionExport" href="https://mint.intuit.com/transactionDownload.event?accountId=3436098&offset=0&comparableType=8">Export all 1797 transactions</a>

Looks like we can either pull the accountId off of the URL of the #transactionExport element or parse the URL fragment. The fragment may be a bit more work given the weird format (URL encoded params look to be prefixed by location:). It looks like mint is using very old javascript technology, so we can’t make too many assumptions about the URL fragment structure.

Now we know we can build a prototype of a mint importing tool. Here’s what we need to do:

  1. Pull the session cookie after the user logs in
  2. Pull the #javascript-token from the page source
  3. Pull the accountId either from the URL fragment or the transactionExport
  4. Hit the https://mint.intuit.com/updateTransaction.xevent endpoint to add transactions
  5. Optionally use the category API

Now to learn how google chrome extensions work!

Building a Chrome Extension

Firstly, how can we iterate on a Chrome extension while it’s in development? I google’d "best practices building a Chrome extension":

  • https://usersnap.com/blog/develop-chrome-extension

  • https://thoughtbot.com/blog/how-to-make-a-chrome-extension

  • https://github.com/kippt/kippt-chrome

  • https://github.com/yeoman/generator-chrome-extension

  • @@extension_id can be used to reference your extension ID in CSS. Most likely this works for JS and HTML as well.

  • To publish the extension, you need to register with Google and follow some guidelines.

  • "If you’re building a Chrome extension which needs to interact with web pages that are loaded by users, you definitely need a content script." This is our scenario.

  • "And luckily testing your new extension is pretty straightforward. Once you’ve activated the “developer mode”… You can simply add your unpacked extension to your Chrome browser to test it." I want to make sure I can test the Chrome extension quickly by editing some JS and reloading the page.

  • "When you change or add code in your extension, just come back to this page and reload the page." sounds like we need to reload the chrome://extensions extensions page?

  • Looks like we can load external libraries via a CDN using the manifest JSON. Great! This will make things easier for initial development.

  • You can limit which domains your extension activates on. We should do this and scope it down to Mint

  • Looks like all we need is a folder with a JS file and a manifest. Looks easy enough.

  • Yeoman scaffold looks cool. Uses babel which allows you to write ES6 (which I want to learn) and implements best practices. Looks kind of updated (last commit <1yr ago).

Ok, I think I have enough information. Let’s try that scaffold out:

npm install --global yo gulp-cli bower
npm WARN deprecated bower@1.8.8: We don't recommend using Bower for new projects. Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/

Eek, not good. Scaffold looks to be too old. Let’s try to search for another newer scaffold…

https://github.com/edrpls/chrome-extension-template

This one looks newer. It uses webpack which I’ve heard good things about and been wanting to understand better.

git clone git@github.com:edrpls/chrome-extension-template.git mintporter
cd mintporter/
npm install

The dependency graph that npm generated was massive, but it worked. Let’s try to run the start command.

yarn start
-bash: yarn: command not found

Yarn is an npm alternative. I needed to install it via brew:

brew install yarn
yarn start

With yarn running, I edited the manifest.json to use the "Mintporter" name, enable dev mode on chrome://extensions/, and loaded the extension via the dist/ folder. Now it’s loading on Chrome.

Now to understand how the template is structured:

  • Looks like webpack is similar to bower and other JS bundling tools. entry defines the output files.
  • Interesting, looks like there is a library to specifically help with bundling chrome extensions. https://github.com/johnagan/crx-webpack-plugin
  • Looks like mostly babel (transpiler) packages in package.json.
  • I was confused by the rimraf in the template. It just adds the rm command in node.

Time to learn the latest JavaScript syntax!

Learning ES6

Googled "learn latest javascript":

Learnings:

  • let is the new var, but scoped to the block. You should use this instead of var
  • const prevents the var from being reassigned.
  • null/undefined/typeof seem to be unchanged
  • We finally have default parameter values! Same syntax as ruby: function fn(arg="default")
  • The => is just a shortcut for defining a new function. Left side is a list of function arguments, the right side is the code (normally a oner-liner) to be executed. data => port.postMessage(data) is the same as function(data) { return port.postMessage(data); }
  • () => { console.log('in-content.js - disconnected from popup'); } is similar to the above syntax, but the {} does not default to returning the evaluated value.
  • Brackets with arrow-defined functions allow multi-line functions but require an explicit return. => without brackets will return the last value in the one-liner function by default.
  • Arrow-functions inherit the this value of the callee.
  • async/await is the native way to write asynchronous code. Promises are now JavaScript-native and async/await use these under the hood. You can think of them as background jobs you can easily run in the browser.
  • Template literals now exist! Use backticks: string text ${expression} string text.
  • You can now define classes without prototypes! Yay. Use class as the keyword, constructor as the initializer. There’s no native Class object; it’s basically syntactic sugar on top of the old prototype model. Feels a lot like CoffeeScript.
  • get and set language keywords exist inside a class to define custom getters and setters.
  • Protected class variables are still convention-only. Still no private properties and methods, although this is in progress.
  • Class-methods are defined using static
  • Mixins are still messy. No language keywords to make this simple: you need to define objects and use low-level javascript calls to copy the methods of the object in.
  • for...of loops are an easy way to iterate through all objects in an array.
  • ...args is the "rest" syntax which allows you to represent multiple args passed to a function as an array. Similar to the splat argument in ruby *args.
  • import is the newer version of require. I didn’t get a good sense of when import isn’t supported in various JavaScript versions.
  • Destructuring is a mix between pattern matching and keyword arguments. function boom({ keyword }) => function boom(obj) { keyword = obj.keyword }. When looking at a function call: boom(a, b: c) is equivilent to boom(a: a, b: c)

Now, with enough new JavaScript knowledge under my belt, I can start hacking away at the extension!

Building the Prototype

  • You need to manually click the reload button in chrome://extensions/ to pull in a new version of the javascript. Bummer.
  • https://www.npmjs.com/package/webpack-chrome-extension-reloader looks like it may fix the issue. npm install webpack-chrome-extension-reloader --save-dev and NODE_ENV=development yarn start. https://stackoverflow.com/questions/2963260/how-do-i-auto-reload-a-chrome-extension-im-developing and https://github.com/arikw/chrome-extensions-reloader are also interesting.
  • Use import $ from "jquery"; to pull jQuery in using the new import syntax. Babel will automatically backport this to be supported on an old JS version.
  • Hmm, my extension is being automatically disabled. Weird. I’m getting an error on the extension page relating to inter-page communication. Cutting out that code from the in-content file.
  • I’m going to start with two main classes: MintContext (to pull auth) and MintIntegrator to add transactions to mint.
  • It would be nice to have an "Import Transactions" button. Looks like we can add it to #controls-top. I’ll need a way to watch for an element to appear on the page since transactions are loaded async. This page indicates I could just jQuery’s ajaxStop, but that isn’t working. DOMSubtreeModified event looks like the old-school way of watching for changes. #product-view-root looks like the best object to observe and MutationObserver looks like the API we want to use. Also, the chrome extension reloader isn’t working, I think I need to modify how webpack is interacting with yarn. To add that to that, my extension keeps getting disabled and I need to restart chrome to allow me to re-enable it again.
  • https://www.ghacks.net/2017/07/04/hide-chromes-disable-developer-mode-extensions-warning/ let’s try using chrome canary and see if that fixes the chrome reloading issue.
  • Looks like we don’t need the chrome reloader extension. The webpack plugin creates a websocket and listens for changes.
  • Sidenote: I know some folks don’t like global namespaces, but I do find it annoying that you can mutate the name of any library into whatever global object you like via require or import. Maybe I’ll discover the benefits of this later on and change my mind.
  • After more playing around, I got a better understanding of yarn. It’s just an improved npm and works off of the commands defined in package.json => scripts. It’s meant to be more stable than NPM.
  • webpack-chrome-extension-reloader asks you to use --watch in webpack. However, the chrome extension template is setup to use nodemon with yarn build which I’m guessing monitors the filesystem and circumvents webpack‘s --watch. I wonder if there is a way to reload the extensions outside the --watch lifecycle.
  • You can learn which process is using a port using lsof -i tcp:3000. Discovered this while trying to get the extension reload working. https://stackoverflow.com/questions/4075287/node-express-eaddrinuse-address-already-in-use-kill-server
  • Hmm, I can’t get the reloader to work. Going to give up and manually reload. Ugh.
  • #body-mint is the only element that exists on $(document).ready. Watching for changes in the DOM is going to kill performance (lots of events to comb through). Let’s just use a setTimeout based approach. https://gist.github.com/chrisjhoughton/7890303
  • Weird. My jQuery version is really old (1.x.x) but the npm package is the latest version. There is some sort of namespace conflict happening. Reverting back to the simple import $ from 'jquery'; to eliminate the issue.
  • Working in the console uses the mint version of jQuery as opposed to the version you bundle with the app. This makes things tricky: you can’t run code live in the console.
  • Hmm, the #javascript-token is empty. Mint.getToken() seems to work though. I’ll try using this instead.
  • Looks like class vars are a new-ish feature. They are causing a compilation error in the version of babel I’m using. Looking into a bit deeper, they aren’t supported in babel yet. Bummer.
  • Running into weird scoping issues when accessing the top-level Mint object. I’m guessing this has to do with my babel config. Man, the stack of javascript transforms on top of the raw browser is still such a pain.
  • Actually, it’s not a babel config issue: chrome extensions can access the entire DOM of the page they are on, but they can’t access the javascript runtime of that page. All javascript runs in a separate sandboxed environment. However, you can access localStorage. This means Mint.getToken() won’t work. Luckily the CSFR token is stored in the session so we can just pull it from there.
  • Now that we have a single transaction pushing into Mint when the "Import Transactions" button that we added to the page is pressed, we want to allow a CSV to be imported. From what I could tell, there’s not any great API to allow the user to select a file and read it into a string. You need to create a hidden file input, and "click" the input during a user-initiate click event, and then when the file input value has changed use the FileReader class to read the file into a string. Bummer. https://stackoverflow.com/questions/32490959/filereader-on-input-change-jquery and https://mariusschulz.com/blog/programmatically-opening-a-file-dialog-with-javascript.
  • Ok, I have the file dialog opening within a click event triggered by a button (so the user doesn’t need to see the file input element). I wonder if I can wrap the entire "read a file" logic into a single function which returns a call back…
  • Cool! We can insert the file input during the click call and the dialog still opens. This allows us to wrap the file input in a single method.
  • Frustrating: you can’t execute code in the console referencing top-level constants defined via ES6. You need to use the webpack converted versions: __WEBPACK_IMPORTED_MODULE_1_papaparse___default. This makes it hard to fiddle around with code in the console.
  • Ideally, we could run the import task (iterating over the CSV) async. We do want to run the mint request sync so we can easily aggregate errors and blow up if something breaks mid-way through. https://petetasker.com/using-async-await-jquerys-ajax/the
  • I was curious what the difference is between TypeScript and the latest JS spec. Looks like TypeScript is just a typed version of the latest JS implementation. https://www.quora.com/What-is-the-difference-between-TypeScript-and-JavaScript
  • Bah! You can’t add an account in mint that isn’t tied to a bank account login. Bummer. I can use an old credit card account as a hack, but this is a unfortunate limitation.
  • I’m curious if someone else has reverse engineered mint’s API. https://webapps.stackexchange.com/questions/11398/does-mint-com-have-an-api-to-download-data-if-not-are-any-scraping-tools-avail and https://github.com/dhleong/pepper-mint. Interesting! Let’s see if there’s any interesting parameters that we can use. After digging a bit, it doesn’t look like there’s anything we didn’t find. Although, it does look like the accountId isn’t used in the API call. Transactions imported will always just hit the global transaction list. Bummer!
  • Passing over a category name that doesn’t match a name in Mint doesn’t do anything. I could implement fuzzy matching on inputs from the CSV to the category API, but that wouldn’t help me learn anything new (I’ve worked with string distance algorithms before). Time to clean this up and call it done!

Lessons Learned

This was fun! Great to way to learn Chrome Extension Development, refresh my JavaScript toolchain knowledge, and learn the new ES6 JavaScript standard.

  • When doing a "hack" project, spend more time fiddling with the system to make sure it can solve the use-cases you are trying to add functionality for. In this case, it looked like transactions could be added to a specific account but if you refreshed the page everything pushed to the "global" transaction list. Would have been helpful to know this ahead of time.
  • The layers of indirection present in JavaScript development cause issues. It would have eliminated some frustration and wasted time to remove babel and develop directly against the browser’s JS engine as opposed to layering in babel all at once.
  • JavaScript development is powerful (adding in a custom script to a webpage that inherits cookies and localStorage allows for some interesting and powerful hacks). There’s a lot of quirks (browser differences, nuances in what extensions can and can’t do, etc) but it’s the #1 language that allows you to manipulate a common interface that everyone has easy access to.
  • This was my first time using VS Code. There was a lot of interesting contextual information VS code was able to provide while editing files. It seems very powerful and is a lot more snappy compared to Atom. However, I felt like an infant struggling with the keyboard shortcuts and lack of some nice Atom features I’m used to. I should spend some more time learning VS Code (especially now that MS owns GitHub, I can’t imagine they’ll invest in two similar editors for that much longer).
  • It’s super useful to have a simple sample project to work with when attempting to learn new technologies. It’s worth spending some time to think about an interesting and genuinely useful project you can work on that involves new technologies you are interested in.

Keep in Touch

Subscribe to my email list to keep in touch. I’ll send you new blog posts and other thoughts.