Using the Multiprocess Library in Python 3

Python has a nifty multiprocessing library which comes with a lot of helpful abstractions. However, as with concurrent programming in most languages, there are lots of footguns.

Here some of the gotchas I ran into:

Logging does not work as you’d expect. Global state associated with your logger will be wiped out, although if you’ve already defined a logger variable it will continue to reference the same object from the parent process. It seems like the easiest solution for logging is to setup a new file-based logger in the child process. If you can’t do this, you’ll need to implement some sort of message queue logging which sounds terrible. Relatedly, be careful about using any database connections, file handles, etc in a forked process. This can cause strange, hard to debug errors. When you pass variables to a forked process, they are ‘pickled’. This serializes the python data structure and deserializes on the ‘other end’ (i.e. in the forked process). I was trying to decorate a function and pickle it, and ran into weird issues. Only top-level module functions can be pickled. If you are using the macos libraries via python, you cannot reference them both on a parent and child process. The solution here is to run all functions which hit the macos libraries in a subprocess. I was not able to get the decorator in this linked post working. Here’s a working example using a modified version of the source below.

I struggled to find full working examples of using the multiprocess library online (here’s the best I found). I’ve included an example of using multiprocessing to create a forked process to execute a function and result the results inline.

Send a signal from the parent process to the child process to start executing using multiprocessing.Condition. I want not able to get this working without first notify()ing the parent process. Kill the child process after 10m. This works around memory leaks I was running into with the applescript I was trying to execute. Configure logging in forked process. Return result synchronously to the caller using a shared queue implemented using multiprocessing.Queue import multiprocessing import time import logging forked_condition = None forked_result_queue = None forked_process = None forked_time = None logger = logging.getLogger(__name__) def _wrapped_function(condition, result_queue, function_reference): # this is run in a forked process, which wipes all logging configuration # you'll need to reconfigure your logging instance in the forked process logger.setLevel(logging.DEBUG) first_run = True while True: with condition: # notify parent process that we are ready to wait for notifications # an alternative here that I did not attempt is waiting for `is_alive()` if first_run: condition.notify() first_run = False condition.wait() try: logger.debug("running operation in fork") result_queue.put(function_reference()) except Exception as e: logger.exception("error running function in fork") result_queue.put(None) def _run_in_forked_process(function_reference): global forked_condition, forked_result_queue, forked_process, forked_time # terminate the process after 10m if forked_time and time.time() - forked_time > 60 * 10: assert forked_process logger.debug("killing forked process, 10 minutes have passed") forked_process.kill() forked_process = None if not forked_process: forked_condition = multiprocessing.Condition() forked_result_queue = multiprocessing.Queue() forked_process = multiprocessing.Process( target=_wrapped_function, args=(forked_condition, forked_result_queue, function_reference) ) forked_process.start() forked_time = time.time() # wait until fork is ready, if this isn't done the process seems to miss the # the parent process `notify()` call. My guess is `wait()` needs to be called before `notify()` with forked_condition: logger.debug("waiting for child process to indicate readiness") forked_condition.wait() # if forked_process is defined, forked_condition always should be as well assert forked_condition and forked_result_queue # signal to the process to run `getInfo` again and put the result on the queue with forked_condition: forked_condition.notify() logger.debug("waiting for result of child process") return forked_result_queue.get(block=True) def _exampleFunction(): # do something strange, like running applescript return "hello" def exampleFunction(): return _run_in_forked_process(_exampleFunction) # you can use the wrapped function like a normal python function print(exampleFunction()) # this doesn't make sense to use in a single-use script, but if you need to you'll need to terminate the forked process forked_process.kill()

Note that the target environment here was macos. This may not work perfectly on linux or windows, it seems as though there are additional footguns on windows in particular.

Continue Reading

Building a Crypto Index Bot and Learning Python

A long time ago, I was contracted to build a MacOS application using PyObjc. It was a neat little app that controlled the background music at high-end bars around London. That was the last time I used python (early 2.0 days if I remember properly). Since then, python has become the language of choice for ML/AI/data science and has grown to be the 2nd most popular language.

I’ve been wanting to brush up on my python knowledge and explore the language and community. Building a bot to buy a cryptocurrency index was the perfect learning project, especially since there was a bunch of existing code on GitHub doing similar things.

You can view the final crypto index bot project here. The notes from this learning project are below. These are mainly written for me to map my knowledge in other languages to python. Hopefully, it’s also helpful for others looking to get started quickly in the language!

Tooling & Package Management

I work primarily in ruby (and still enjoy the language after years of writing professionally in it). Some of the comparisons below are to the equivalent tooling in ruby-land.

pip == bundle Package versions are specified in a requirements.txt file if you are using pip. = There’s not really a rake equivalent that’s been adopted by the community. Poetry is an alternative to pip that seems to be the most popular choice for new projects. virtualenv = rbenv, but just for packages, not for the core python version, and is specific to each project. Poetry will autogen a virtualenv for you. There are dev and non-dev categories in poetry, but not a test category by default. Here’s how to add a dev dependency poetry add -D pytest If you are using the VS Code terminal, certain extensions will automatically source your virtualenv. I found this annoying and disabled this extension (can’t remember which extension was causing me issues). pyproject.toml alternative to requirements.txt, but also includes gemspec-like metadata about the package. It looks like poetry update consumes the .toml file and generates a poetry.lock. I’m guessing that other build tools also consume the .toml config and it’s not used just for poetry. The python community seems to be into toml configuration. This is used for poetry package specifications and project-specific variables. I don’t get it: it’s slightly nicer looking than JSON, but you can’t specify arrays or nested hash/dictionaries. Why not just use yaml instead? Or just keep it simple and use JSON? I ran into this issue where poetry was using the global ~/Library/Caches/pypoetry cache directory and I thought this was causing some package installation issues. I don’t think that ended up being the isweu poetry debug poetry config -vvv to see what configuration files are being loaded poetry config --list indicated that a global cache directory was being used. Tried upgrading pip, didn’t work: python3 -m pip install --upgrade pip I can’t remember how I fixed the issue, but these commands were helpful in understanding where poetry throws various code. If you want to hack on a package locally and use it in your project: vcrpy = { path = "/full/path/to/project", develop = true } in your toml file Note that you cannot use ~ in the path definition After adding this to your pyproject.toml run poetry lock && poetry install This will be easier in poetry 1.2 Want to make sure your project is pulling from your locally defined project? You can inspect the path that a module was pulled from via packagename.__file__ i.e. import vcr; print(vcr.__file__) I had trouble with a corrupted poetry env, I had to run poetry env use python to pick up my local package definition Working on a project not using poetry? Create a venv python -m venv venv && source ./venv/bin/activate If there’s a then run python install However, you can’t install ‘extra’ dependencies (like development/testing) via It looks like pip install -e '.[dev]' It sounds like and requirements.txt do not define dev dependencies. You’ll probably need to install these manually. Look at the CI definition in the project to determine what dev dependencies need to be installed. There’s a .spec file that seems to be used with pyinstaller, a python package, when packaging a python application for distribution. Pyinstaller is primarily aimed at distributing packages for execution locally on someone’s computer. This use-case is one of the areas where python shines: there’s decent tooling for building a multi-platform desktop application. You’ll see readme-like documents written in rst (restructure text format) instead of md. I have no idea why markdown just isn’t used. A ‘wheel’ is an architecture-specific package bundle that contained compiled binaries. This is helpful if a python package contains non-python code that needs to be compiled since it eliminates the compile step and reduces the change of any library compatibility issues (this is a major problem in PHP-land). black looks like the most popular python code formatter. Language Multiline strings (""") at the beginning of a class or function definition isn’t just a python idiom. They are ‘docstrings’ and get automatically pulled into the autogen’d python documentation. Similar to ruby, camelCase is used for class names, snake_case is used for function/variable names. Calling a function requires parens, unlike ruby or elixir. Like javascript, return values need to explicitly be defined by return val. Conditionals do not return values, which means you need to assign variables inside the block (unlike the ability to assign a variable to the return value of a block in ruby, a feature that I love). Each folder in a python project is transformed into a package that can you import. the __init__ file in the folder is automatically imported when you import the folder name. Imports have to be explicitly defined, like javascript, to use any functions outside the set of global/built-in functions. Speaking of built-in functions, python provides a pretty random group of global functions available to you without any imports. I found this confusing: round() is a built-in but ceil() is not. When you import with a . it looks at the local directory for matching imports first. Import everything in package with from math import *. This is not good practice, but helpful for debugging/hacking. Although you can import individual functions from a package, this is not good practice. Import modules or classes, not individual functions. You have to from package.path import ClassName to pull a classname from a module. You can’t import package.path.ClassName None is nil and capitalization matters True and False are the bool values; capitalization matters. Hashes are called dicts in python Arrays are called lists in python You can check the existence of an element in a list with element in list. Super handy! Triple-quoted strings are like heredocs in other languages. They can be used for long comments or multi-line strings. Substring extraction ranges are specified by the_string[0:-1]. If you omit a starting range, 0 is used: the_string[:-1]. The traditional boolean operators && and || aren’t used. Natural language and and or is what you use instead. Keyword arguments are grouped together using **kwargs in the method definition. You can splat a dict into keyword arguments using function_call(**dict) All arguments are keyword arguments in python. More info. You can lazy-evaluate a comprehension using () instead of [] When playing with comprehensions inside of a ipython session variable scoping will not act the same as if you weren’t executing within a breakpoint(). I don’t understand the reasons for this, but beware! In addition to list comprehensions, there are dictionary comprehensions. Use {...} for these. When logic gets complex for a list comprehension, you’ll need to use a for loop instead (even if you want to do basic log debugging within a comprehension). I miss ruby’s multi-line blocks and chained maps. List comprehensions are neat, but there doesn’t seem to be a way to do complex data transformations cleanly. I hate having to define an array, append to it, and then return it. The filter/map/etc functions can’t be easily chained like ruby or javascript. I wonder what I’m missing here? I’ve heard of pandas/numpy, maybe this is what those libraries solve? There are strange gaps in the stdlib, especially around manipulating data structures. For instance, there’s no dead-simple way to flatten an array-of-arrays. import operator; from functools import reduce; reduce(operator.concat, array_of_arrays) Similarly, there’s no easy way to get unique values from a list. Get all of the string values of an enum [choice.value for choice in MarketIndexStrategy] By subclassing str and enum.Enum (ex: class MarketIndexStrategy(str, enum.Enum):) you can use == to compare strings to enums. There’s no ? tertiary operator, instead you can do a one-liner if-else: assignment = result if condition else alternative To enable string interpolation that references variable names you need to use f"string {variable}". Otherwise you’ll need to run format on the string to get it interpolated: "string {}".format(variable) Python has built-in tuples (1, 2, 3). I’ve always found it annoying when languages just have arrays and don’t support tuples. Unlike ruby, not all code has a return value. You have to explicitly return from a function and you can’t assign the result of a code block to a variable. There’s some really neat python packages: natural language processing, pandas, numpy. Python has gained a lot of traction in the deep learning/AI space because of the high-quality packages available. is is NOT the same as ==. is tests if the variable references the same object, not if the objects are equal in value You can’t do an inline try/catch. Many bad patterns that ruby and other languages really shouldn’t let you do are blocked. In a lot of ways, python is a simpler language that forces you to be more explicit and write simpler code. I like this aspect of the language a lot. Sets are denoted with {}, which is also used for dictionaries/hashes. Here’s how decorators work: The @decorator on top of a method is like an elixir macro or ruby metaprogramming. It transforms the method beneath the decorator. The @ syntax ("pie" operator) calls the decorator function, passing the function below the decorator as an argument to the decorator function, and reassigning the passed function to the transformed function definition. The decorator function must return a function. There is no special syntax to designate a function as a ‘decorator function’. As long it accepts a function as an argument and returns a function, it can be used as a decorator. Referencing an unspecified key in a dict raises an exception. You need to specify a default: h.get(key, None) to safely grab a value from a dict. An empty array will evaluate to false. You don’t need to if len(l) == 0:. Instead you can if !l:. Same goes with empty dicts and sets. Lambdas can only be single-line. This is a bummer, and forces you to write code in a different style. := allows you to assign and test a value within a conditional. Interesting that there’s a completely separate syntax for ‘assign & test’. in a folder defines what happens when you import a folder reference. Here’s how classes work: class newClass(superClass): for defining a new class __init__ is the magic initialization method self.i_var within __init__ defines a new instance variable for a class. This is a good breakdown of instance and class variables. you can execute code within a class outside of a method definition for class-level variables and logic, new instances of a class are created via newClass(). Instance methods of a class are always passed self as the first argument Class variables are available on the instance as well, which is a bit strange. You can use class variables as default values for instance variables. This doesn’t seem like a great idea. newClass.__dict__ will give you a breakdown of everything on the class. Kind of like prototype in javascript. Python has multiple inheritance. class newClass(superClass1, superClass2). Inherited classes are searched left-to-right. There are not private variables built into the language, but the convention for indicating a variable is private is using a _ like self._private = value There’s a javascript-like async/await pattern (coroutines). I didn’t dig into it, but seems very similar to Javascript’s pattern. Debugging & Hacking

One of the important aspects of a language for me is the REPL and tinkering/hacking environment. If I can’t open up a REPL and interactively write/debug code, I’m a much slower developer. Thus far, ruby has the best interactive development environment that I’ve encountered:

binding.pry and binding.pry_remote when your console isn’t running your code directly to open a repl Automatic breakpoints on unhandled exceptions, in tests or when running the application locally Display code context in terminal when a breakpoint is hit Print and inspect local variables within a breakpoint Navigate up and down the callstack and inspect variables and state within each frame Overwrite/monkeypatch existing runtime code and rerun it with the new implementation within a repl Define new functions within the repl Inspect function implementation within the repl

I’d say that python is the first language that matches ruby’s debugging/hacking environment that I’ve used. It’s great, and better than ruby in many ways.

inspect is a very helpful stdlib package for poking at an object in a repl and figuring out the method, variables, etc available to it. traceback provides some great tools for inspecting the current stack. How you drop an interactive console at any point in your code? There are a couple ways: Uses the ipython enhanced repl in combination with the built in debugger import ipdb; ipdb.set_trace(). Requires you to install a separate package. There’s a breakpoint() builtin that launches the standard pdb debugger. You can configure breakpoint() to use ipdb via export PYTHONBREAKPOINT=ipdb.set_trace. All of the standard pdb functions work with ipdb import code; code.interact(local=dict(globals(), **locals())) can be used without any additional packages installed. bpython is a great improvement to the default python. You need to install this within your venv otherwise the packages within your projects venv won’t be available to it: pip install bpython && asdf reshim ipython is a bpython alternative that looks to be better maintained and integrates directly with ipdb. python -m ipdb to automatically open up ipython when an exception is raised when running Some misc ipython tips and tricks: If something is throwing an exception and you want to debug it: from ipdb import launch_ipdb_on_exception; with launch_ipdb_on_exception(): thing_causing_exception() who / whos in whereami %psource or source like show-source pp to pretty print an object ipython --pdb to break on unhandled exceptions Great grab bag of interesting tips %quickref for detailed help exit gets you out of the repl entirely All of the pypi information is pulled from a PKG-INFO file in the root of a package rich-powered tracebacks are neat, especially with locals=True The ruby-like metaprogramming/monkeypatching stuff happens via the __*__ functions which are mostly contained within the base object definitions. For instance, logging.__getattribute__('WARN') is equivalent to logging.WARN You can reload code in a REPL via from importlib import reload; reload(module_name). Super helpful for hacking on a module (definitely not as nice as Elixir’s recompile). Monkeypatching in python isn’t as clean as ruby, which in some ways is better since monkeypatching is really an antipattern and shouldn’t be used often. Making it harder and more ugly helps to dissuade folks from using it. To monkeypatch, you reassign the function/method to another method: ClassName.method_name = new_method. Here’s an example. Typing

I’ve become a huge fan of gradual types in dynamic languages. I never use them right away, but once the code hardens and I’m relatively sure I won’t need to iterate on the code design, I add some types in to improve self-documentation and make it safer to refactor in the future.

Python has a great gradual type system built-in. Way better than Ruby’s.

mypy . on the command line to test all python files within a folder. If your project fails to pass mypy, it won’t cause any runtime errors by default. There’s a VS Code extension. This extension is included in Pylance, which you should probably be using instead, but you need to set the typing mode to ‘basic’. Return value types are set with -> before the : at the end of the method definition. Otherwise, typing works very similar to other languages with gradular typing (TypeScript, Ruby, etc). A common pattern is importing types via import types as t t.Union[str, float] for union/any types, You can’t merge dictionaries if you are using a TypedDict (dict | dict_to_merge). Massive PITA when mutating API data. Verbose types can be assigned to a variable, and that variable can be used in type definintions. Handy way to make your code a bit cleaner. Enums defined with enum.Enum can be types. Testing Like Elixir, there are doctests that execute python within docstrings to ensure they work. Neat! There are built-in test libraries that look comparable to ruby’s testunit. pytest is similar to minitest: provides easy plugins, some better standard functionality, and builds on top of unittest. You probably want to use pytest for your testing framework. setup.cfg is parsed by pytest automatically and can change how tests work. is another magic file autoloaded by pytest which sets up hooks for various plugins. You can put this in the root of your project, or in test/ Test files must follow a naming convention test_*.py or * If you don’t follow this convention, they won’t be picked up by pytest by default. breakpoint()s won’t work by default, you need to pass the -s param to pytest Like ruby, there are some great plugins for recording and replaying HTTP requests. Checkout pytest-recording and vcrpy. To record HTTP request run pytest --record-mode=once If you want to be able to inspect & modify the API responses that are saved, use the VCR configuration option "decode_compressed_response": True There’s a mocking library in stdlib, which is comprehensive. I’m not sure why other languages don’t do this—everyone needs a mocking library. It looks like you set expectations on a mock after it runs, not before. Here’s how mocking works: The @patch decorator is a clean way to manage mocking if you don’t have too many methods or objects to mock in a single test. If you add multiple patch decorators to a method, the mocks for those methods are passed in as additional arguments. The last patch applied is the first argument. mock.call_count, mock.mock_calls, mock.mock_calls[0].kwargs are the main methods you’ll want for assertions asset without parens is used in tests. This confused me, until I looked it up in the stdlib docs and realized assert is a language construct not a method. tox is much more complex that pytest. It’s not a replacement for pytest, but seems to run on top of it, adding a bunch of functionality like running against multiple environments and installing additional packages. It feels confusing—almost like GitHub actions running locally. If you want to just run a single test file, you need to specify an environment identifier and test file tox -epy38-requests -- -x tests/unit/ My thoughts on Python

Overall, I’m impressed with how python is improved over the years. Here are some things I enjoyed:

Gradual typing included in the core language Comprehensions are natural to write Syntax simplicity: there are not too many ways to do things, which makes code more straightforward to read. Mature, well-designed libraries Virtual environments out of the box Robust, well-maintained developer tooling (ibpd, ipython, etc) with a advanced REPL Great built-in testing libraries Lots of example code to grep through for usage examples Explicit imports and local-by-default logic (unlike ruby, where it’s much easier to modify global state) Easy to understand runtime environment (in comparison to JavaScript & Elixir/BEAM)

The big question is if Django is a good alternative to Rails. I love Rails: it’s expansive, well-maintained, thoughtfully designed and constantly improving. It provides a massive increase in development velocity and I haven’t found a framework that’s as complete as Rails. If Django is close to rails, I don’t see a strong argument for not using anything python over ruby for a web product.

Open Questions

Some questions I didn’t have time to answer. If I end up working on this project further, this is a list of questions I’d love to answer:

How good is django? Does it compare to Rails, or is it less batteries-included and more similar to phoenix/JS in that sense. Does numpy/pandas solve the data manipulation issue? My biggest gripe with python is the lack of chained data manipulation operators like ruby. How does the ML/AI/data science stuff work? This was one of my primary motivations for brushing up on my python skills and I’d love to deeply explore this. How does async/await work in python? Learning Resources

General guides:

Monkeypatching: Open Source Example Code

There are some great, large open source python projects to learn from: – opens source APM

Download these in a folder on your local to easily grep through.

Continue Reading

How to install an old HomeBrew package

I accidentally ran brew upgrade on a set of packages which caused brew to bump most library packages that were installed on my computer. This caused issues compiling PHP via asdf.

I thought it would be easy to install an older version of the offending package, icu4c, but I was very wrong. In recent (2021?) versions of brew. I ended up learning a bit about the homebrew internals and solving

Installing an old version of icu4c to fix PHP compilation on macOS

I discovered the offending package was icu4c, but the PHP compile script did not indicate that I was using an unsupported package version. make emitted an error with an ‘international’ (intl) identifier in the offending file, which after some googling indicated that icu4c was the offending package:

3 errors generated. make: *** [ext/intl/dateformat/dateformat_attr.lo] Error 1

Brew used to support installing packages via a formula URL. Unfortunately, this method of installation was removed in a recent version of brew. Instead of something like brew install you need to execute (found this snippet through this blog post):

brew tap-new $USER/local-tap brew extract --version=68.2 icu4c $USER/local-tap brew install icu4c@68.2

You can determine the last supported version in homebrew using git history via the URL exposed when running brew info icu4c.

However, extracting a specific version through this method does not use the same package name. The package name is icu4c@68.2 with version 68.2 which means any scripts which check for a package via brew list --version icu4c will fail. If it wasn’t for this nuance, this StackOverflow answer would have had all of the information I needed.

Additionally, even if the package name was the same, brew removed the switch command. Now, the preferred way to switch versions is to relink a specific version:

brew link --overwrite --force icu4c@68.2

However, this doesn’t work:

Warning: Refusing to link macOS provided/shadowed software: icu4c@68.2 If you need to have icu4c first in your PATH, run: echo 'export PATH="/usr/local/opt/icu4c@68.2/bin:$PATH"' >> ~/.zshrc echo 'export PATH="/usr/local/opt/icu4c@68.2/sbin:$PATH"' >> ~/.zshrc For compilers to find icu4c you may need to set: export LDFLAGS="-L/usr/local/opt/icu4c@68.2/lib" export CPPFLAGS="-I/usr/local/opt/icu4c@68.2/include" For pkg-config to find icu4c you may need to set: export PKG_CONFIG_PATH="/usr/local/opt/icu4c@68.2/lib/pkgconfig"

From what I can tell, the only real workaround for this error is to manually specify the environment variables indicated in the installation post-install message. The downside with this approach is if you aren’t installing the package directly you’d have to modify the underlying script which could be a major pain.

If you don’t want to go that route, here’s the easiest alternative I found:

# uninstall the newer version of the package that you accidentally installed brew uninstall --ignore-dependencies icu4c # `extract` the version you'd like to install into a custom tap brew tap-new $USER/local-tap brew extract --version=68.2 icu4c $USER/local-tap # jump into the new tap you created cd $(brew --repository $USER/local-tap)/Formula # rename the formula mv icu4c@68.2.rb icu4c.rb # change the name of the formula by removing "AT682" from the `class` definition nano icu4c.rb # then, install this specific formula directly brew install $(brew --repository $USER/local-tap)/Formula/icu4c.rb

PHP 7.4.x & 8.x works with icu4c 68.2. PHP 7.2 did not compile properly with this version. To my surprise the PHP compilation script will not notify you that you are using an unsupported version of a package. php-build is a great project, which is probably worth using instead of asdf, and is a look place to look for PHP compilation tips & tricks.

Worth noting that the only reason I couldn’t use the simple extract and install route is I needed brew list --version $package_name to split out a valid package reference. If you don’t need that, you should be able to get away with the extract route and setting export PKG_CONFIG_PATH directly in the shell.

I’m very surprised installing an older version of a package was this messy. I can see why using a docker container is a better way to go for PHP development. The build story for PHP isn’t much better than what it was 10 years ago.

Installing the PHP ImageMagick Pecl extension

I also ran into issues installing the imagemagick extension with PHP8 (pecl install imagick) . Here’s what I found as far as version compatibility goes:

You need to brew install imagemagick to install the pecl extension Imagick pecl extension works with the latest ImageMagick from homebrew 7 Imagick doesn’t work with PHP8 yet You need to enable the extension "" > $(asdf where php)/conf.d/php.ini Upgrading an old brew-powered MySQL installation

I had MySQL data from an older installation on my machine that was causing issues. MySQL would not start properly and brew services was not listing the correct status.

Here’s what worked for me to fix it, although you might need to do more:

# this will wipe all mysql data # if you want to keep data from your old mysql installation, do not do this sudo rm -rf /usr/local/var/mysql/ brew remove mysql brew cleanup brew install mysql

I was then able to mysql.server start without an issue. If you want MySQL to start automatically on boot, run brew services start mysql. Quick pro-tip: take a look Sequel ace, a great GUI MySQL client.

Continue Reading

Migrating from bash to zsh

I love productivity tools. Anyone who works with me knows I love my keyboard shortcuts and tiny productivity hacks. Small, incremental productivity improvements add up over time: feeling fast makes you fast. Plus, I just enjoy tinkering and making things more productive.

One of the rabbit holes I love to go down is optimizing my development environment. I spend a lot of time in a terminal, so it’s a fun place to optimize my setup. Ever since hearing of Oh My ZSH I wanted to try out zsh, so I set aside some time to update my dotfiles to use zsh as the default shell.

Below are some notes & learnings from the transition.

What’s new in zsh? There are lots of small packages out there for neat things like autocomplete, async prompts, etc. This is the best part about zsh and the main reason I put the effort into switching. There’s a bunch of configuration managers out there. Oh My ZSH, zplug, antigen, antibody, zinit, etc. These managers pull various bundles of zsh scripts together and source them for you. Antibody was the best manager that I could find (when I originally wrote this post in 2020). Allows you to pull directly from GitHub repositories, and load shell scripts that aren’t packaged as a "plugin". However, in less than a year it died out and is unmaintained. Here’s my plugin list with antibody Zinit looks like the best package manager nowadays (2021). Here’s how I moved from antibody to zinit and the change that enabled turbo mode. The syntax is strange. ice is a command that modifies the next command (why not just add a modifier to the command itself? Who knows.) for allows you to execute a command as a loop (like you’d expect) without having to separate ice from the actual command. Helpful if you don’t need separate ice modifiers for each command lucid eliminates the loading messages. Not sure why this isn’t enabled by default. I found this example setup to be the most helpful in decoding the zinit syntax. zi update to update all plugins Packaging something as a plugin is super simple. Create a name.plugin.zsh file in your repo. This file is autoloaded by plugin managers. I’ve always struggled to understand where I can map key pressed to the strange double-bracket definitions I see (e.g. ^[[A] is equivalent to the up arrow key). Run /bin/cat -v and when you press a key it’ll output the key definition you can use in key bindings. There are many options for up/down history matching. I like the substring search package, but there are great builtins for this as well There are many little changes to the shell which make life easier. For instance, mv something.{js,ts} will rename a file. zsh variables have different types. Run type var_name to inspect types of various variables. zsh line editor is zle. zle -N widget-name adds the widget to the line editor so you can bindkey the widget. bindkey lists out all of your keyboard shortcuts zle -la lists out all ‘widgets’ (zsh commands, not sure why they are called widgets). You can bind keyboard sequences to these widgets. The edit-command-line widget ‘parks’ the current command until the next command you type is done executing. Here’s how to bind to ctrl-e (the default ctrl-q binding wasn’t working for me). Function path is fpath, the list of paths to search for the definition of a function definition. This is distinct from $PATH in zsh. A big improvement with zsh is the ability to async run commands. For instance, you can display your prompt and then run various git commands and update your prompt status. This is critical for large repos (where git commands can take seconds to run) and is the main reason I switched to zsh. <<< is a here string. Useful for passing a string to stdin (echo 'hi' | cat is equal to cat <<< 'hi'). zsh also has here docs with the standard <<EOL syntax. Nifty command to list out all autocompletions. zinit also has a similar (cleaner) command zi clist. Snippet to list aliases, functions, and variables. Globs support regex-like syntax. It’s worth spending some time reading about this and getting familiar with it. There’s a neat trend of folks rewritten common utilities (cd, cat, find, etc) in rust. Here’s a great writeup of improved utilities you can use. You can find my set of tools here. Plugins

Some notes on my plugin configuration:

Here’s my list of zsh plugins. It took some extra bindkey config to get substring history search working zsh-autosuggestions caused weird formatting issues when deleting and pasting text (the autocomplete text wouldn’t use a different color and I couldn’t tell what was actually deleted). Modifying ZSH_AUTOSUGGEST_IGNORE_WIDGETS fixed the issue for me. I tried to get larkery/zsh-histdb working (really neat project) but it doesn’t play well with the fzf reverse-i search, which I really love. Hoping to give this another go in a year or so to see if the integration with fzf and other standard tooling is improved. Being able to filter out failed commands from your zsh history search would be neat. zsh-autosuggest and bracketed paste don’t play well together. This snippet fixed it for me. fasd is a really neat tool, but I wanted to customize the j shortcut to automatically pick the first result. Here’s how I did it. Resources

Some helpful posts and guides I ran into:

Really awesome guide to fancy zsh features & syntax…

Continue Reading

Dumping a AWS RDS Database to Your Local Machine

I’m a big Heroku fan. I used it’s hosted Redis and Postgres services for my startup and it scaled incredibly well and saved me a ton of time not having to ever worry about devops.

One of things I loved about Heroku was it’s CLI API. You could very easily manage infrastructure through a very thoughtful CLI.

For instance, a common process I would run was:

Dump production database to my local Import the dump into my local Postgres Clean & sanitize the data

This process looked something like:

curl -o latest.dump `heroku pg:backups public-url` pg_restore --verbose --clean --no-acl --no-owner -h localhost -U postgres -d app latest.dump bundle exec rake app:sanitize

That first line was all you needed to download a production copy of your DB.

I’ve been playing around with AWS for a side project and replicating a similar process was surprisingly challenging. AWS RDS (Amazon hosted relational databases) has a concept of ‘snapshots’ which sounds like exactly what you’d want, but all of the instructions I found looked complicated and there wasn’t a simple GUI or CDK interface to create one. Very frustrating!

The easiest solution I was able to find is to tunnel a port from your local to the RDS instance through the EC2 instance (or a bastion host, if you have one) connecting to the RDS DB.

Here’s what this looks like:

# don't bind to 5432 on your local, you probably have pg running on that port already local_host=localhost:5433 # pull the remote host from your db connection string attached to your app # you shouldn't be abler to access RDS # proxy connection to RDS from your local via your EC2 box ssh -N -L $local_host:$remote_host # assumes that `postgres` is your local username # on your local, in another terminal, you'll run this command to dump the entire remote database # you'll need your pg password on hand in order to run this command pg_dump -c -p 5433 -h localhost -U postgres -f ./latest.dump postgres # after that's completed, you can pull the database into your local psql app_dev < ./latest.dump Resources:

Continue Reading

Learning Clojure by Automating an RSS Reader

I’ve been working on revamping how I consume information. Most of my information consumption has been moved to RSS feeds, but I can’t keep up with the number of articles in my feeds. When I take a look at my reader I tend to get overwhelmed and spend more time than I’d like to trying to "catch up" on information I generally was consuming out of curiosity.

Not good.

I want articles to be automatically marked as read after they are a month old to eliminate the feeling of being "behind". This is a perfect little project to learn a programming language that’s looked interesting for a while!

Building a small project in a new language or technology is the best way to learn. While I was building this tool, I documented what questions I was asking, answers to these questions, and what articles and resources I found helpful.

Posts like this have been interesting to me, hopefully this is a fun read for others!

What do I want to build?

I want to build a Clojure script for FeedBin that will:

Inspect all unread articles If the publish date is more than two weeks in the past, mark the article as unread Automatically run every day

Let’s get started!


Here are some helpful blogs & tutorials I used while learning:

Also, I always try to grab a couple of large open-source repos to look at when I’m learning a new language. Here are some places I searched:

Some repos I found interesting: This is probably the largest full-blown open-source Clojure application out there. Most other projects I found were libraries, not applications. Syntax & Structure

Now that I have some browser tabs open with documentation, let’s start learning!

How do I install this thing? => brew install clojure/tools/clojure Going through the "Learn X in Y" guide, some interesting takeaways: Clojure is built on the JVM and uses Java classes for things like arrays. Code in Clojure is essentially a list-of-lists. A list is how you execute code: the first element is the method name, and then arguments separated by spaces. This feels very weird at first, but it’s a really powerful concept. Simple made Easy explains the philosophy behind this a bit. "Quoting" (prefacing a list with a single quote) prevents the list from executing. This is helpful for defining a list, passing code as a data structure that can be mutated later on. Sequences (Arrays/Lists) seem to have some important different properties from vectors. I need to understand this a bit more. When you define a function it doesn’t get a name. You need to assign it (def) to a variable to give it a name. The [] in a function definition is the list of arguments. There are lots of ways to create functions: fn, defn, def ... #() multi-variadic function is a new word for me! It’s a function with a variable number of arguments. Looks like you can define different execution paths depending on the arguments, kind-of like Elixir’s pattern matching. [& args] is equivalent to (*args) in ruby The beginner (me!) can treat ArrayMap and HashMap as the same. Keywords == ruby symbols The language looks to execute from the inside out, and the composition of functions is done via spaces not commas, parens, etc. Looks like everything is immutable in Clojure. Everything is a function. So much so, that even basic control flow is managed the same way as a standard function. Looks like "STM" is an escape hatch if you need to store state. Similar to Elixir’s process state. The Clojure community is big on "repl driven development", but what exactly do they mean? How is that different from binding.pry in a ruby process to play around with code? Looks like it’s not that different. Some nice editor integrations make things a bit more clean, but more or less the same as opening up rails console with pry enabled. I’ve always disliked the ability to alias a module or function to a custom name. It makes it much harder for newcomers to the codebase to navigate what is going on. Looks like this is a pretty common pattern in Clojure, the require at the top of a file can setup custom aliases for all functions. "forms" have been mentioned a couple of times, but I still don’t get it. What is a form? I’ve heard that Clojure is a Lisp. What is a "lisp"? There was an original LISP programming language, but "a lisp" is a language patterned after the original LISP Seems like the unique property of a lisp-style language is code is essentially is a linked list data structure. Since all code is a data structure, you can define really interesting macros to modify your source code. Another property is the parentheses-based syntax. It’s interesting to look at the different lisp styles available. I feel like the only language that is popular today is Clojure. Sounds like immutability is unique to Clojure and isn’t a core structure other lisps.

I think I know just enough to start coding.

Coding in Clojure

Here’s the learning process which generated the final source code:

Let’s define the namespace and get a "Hello World" to make sure I have the runtime executing locally without an issue. 184408626bb41b87d53f9b0bb5485a8e9201d8d5 Ok, now let’s outline the logic we’ll need to implement. 7e018b05ff8ad925ef2bfe9c56c4a702dce4c3d0 Now, let’s pick a HTTP library and figure out how to add it as a dependency. looks like the most popular package repository. It doesn’t seem like there’s any download/popularity indicator that you can sort by. Bummer. Hard to figure out what sort of HTTP library I should use. Looks like project.clj is a gemspec type definition file. Metabase’s http library is clj-http. Let’s use that. We’ll also need to figure out how to setup this dependency file. is linked in the project.clj files I’ve seen. It’s listed as a dependency manager on the clj-http library: Let’s install it via brew install leiningen. lein new feedbin and mv ./feedbin./ ./ to setup the project structure. Looks like lein will help us with dependencies and deployment. b0b4022618abac840af6679f900584d04de510c1 There’s this skip-aot thing in the main: definition which I don’t understand. In any case, if I stuff a defn -main in the file for the namespace defined in main lein run works! 764d7a1e2a537d61b036df4229a2c96671725dd8 It looks like this ^: syntax is used often. What is it? Ok, let’s copy our logic outline from the other file we were working on over to the src/feedbin/core.clj and try to add our HTTP dependency. Added [clj-http "3.10.0"] to the dependency list in project.clj, lein run seemed to pull down a bunch of files and run successfully. Now, let’s pull the FeedBin variable from the ENV and store it to a var. Looks like you have to wrap let in parens, and include commands that rely on the var within the scope of the parens. I could see how this would force you to keep methods relatively short. 6f1f8099ffd0ed5f997be93685d18d1c574efb6b Let’s hit the API and get all unread entries and store them in a var. Looks like cheshire is a popular JSON decoder, let’s use that. It looks like let is only when you want temporary bindings within a specific scope. Otherwise, you should use def to setup a variable. 5b63cd289052d9fcebec2cb2965d598927b0616a Convention is - for word separation, not _ or camel case. Let’s refactor the getenv to use def. Much better! a6a95a1e4703c07e76ecce32b56b6b0f1903acca Time to select entries that are two months old. A debugger is going to be helpful here to poke at the API responses. Looks like debugger is the pry equivalent. I had trouble getting this to work and deep-dived on this a bit: (pst) displays the stacktrace associated with the last exception. This is not dependent on clj-debugger Looking closer at clj-debugger it has ~no documentation and hasn’t been updated in nearly two years. Is there a better option? Doesn’t look like it (require 'feedbin.core :reload-all) seems like the best way to hot reload the code in a repl. Then you can rerun (feedbin.core/-main) Ah, figured it out! (break) on it’s own won’t do anything. It needs an input to debug. (break true) works. You need to run this in lien repl for it to work. As a side note, I’ve found the REPL/debugging aspect of learning a new programming language to be really important. Languages that don’t have great tooling and accessible documentation around this make it much harder for newcomers to come up to speed. The REPL feedback loop is just so much faster and in developer tooling speed matters. I was able to extract the published date, now I just need to do some date comparison to figure out which entries are over a month old. ca16f54f66a39753933168c3f8deac636144ca47 Now to mark the entries as "read" (in feedbin this is deleting the entries). Should be able to just iterate through the ID list and POST to the delete endpoint. I started running into rate limiting errors as I was testing this. # turns a string into a regex, but appears to do much more. Looks like it’s a shorthand for creating lambda. macroexpansion is an interesting command to help with debugging. With the rate limit errors gone, I can finally get this working for good. I tried passing in the article IDs as a comma-separated list as a query string and it didn’t work. I need to send this data in as a JSON blob. 166ea49439ed690ff08c8fd987530b170b9bb80e Got the delete call working. You can pass a hash directly to clj-http and it’ll convert it into JSON. Nice. 63ac8bf1d4fd969326fffa9ad7b50ad1f0a4b56d

Great! We have the script working. Now, let’s deploy it.

Clojure Deployment Using AWS Serverless

I have a friend who is constantly talking about how awesome serverless is (i.e. AWS Lambda). I also remember hearing that you can setup cron-like jobs in AWS that hit a lambda. Let’s see if that’s the case and if we can get this script working on lambda.

Some things we’ll need to figure out:

How/where do I specify that an endpoint should be hit every X hours? How do I specify where the entrypoint is for the lambda function? How do we specify environment variables?


I jumped into AWS lambda dashboard and created a function named "Mark-Feedbin-Entries-As-Read" with Java 11. It looks like the crazy AWS permission structure is generated for me. I added the com.amazonaws/aws-lambda-java-core package and it looks like I need to run gen-class to expose my handler. What is gen-class? It generates a .class file when compiling, which I vaguely remember is a file which is bundled into the .jar executable. Looks like aot compilation needs to be enabled as well. Still need to understand what aot is. I ran lein uberjar and specified feedbin.core::handler as my handler. Created a test event with "testing" as the input. Used the -standalone jar version that was generated. Looks like environment variables can be setup directly in the Lambda GUI. "Cron jobs" are setup via CloudWatch events. What is CloudWatch? It’s AWS’s monitoring stack. Strange that this is the recommended way to setup cron jobs. I would have thought there was a dedicated service for recurring job schedules. "Serverless" (looks like a CDK-like YML configuration syntax for AWS serverless) makes it look easy to deploy a lambda which executes on a schedule, but doesn’t indicate how it’s actually managed in AWS in the blog post. Aside: It’s interesting the more you dig into AWS, the more it feels like a programming language. Each of the services is a library and the interface to configure them in yaml. It looks like "Amazon EventBridge" is the new "CloudWatch Events". Looks like we can setup a rule which triggers a lambda function at a particular rate. Neat, you can setup a rule directly with the AWS Lambda GUI. Use a EventBridge trigger with rate(1 day) to trigger the function every day. Really easy! I checked on it the next day and it’s failing. How can we inspect the request? It’s probably failing due to the input data being some sort of JSON object vs a simple string that I tested with. Here’s what I found: you can inspect the logs, use CloudTrail to view an event, enable X-Ray tracing, and send failed events to a dead letter queue. I enabled all of this stuff: my end goal to inspect the event JSON passed the lambda to determine how to fix it. Ah! After a bit more digging, if you find the event in CloudTrail there’s a "View event" button that will give you the JSON output. I can then copy the JSON into the test event in the configuration for the lambda and run it there to get helpful debugging information. Feels a bit primitive, but it works. I wonder how you would run the function and locally and write integration tests using example AWS JSON? Looks like the function signature for my handler is incorrect. When handling events, the handler accepts two arguments [Object]. This fixed the issue! 8520e8a319bd5d41a67a01f9517ce4cf559ab381

Resources: Open Questions

Here’s a list of questions that I wasn’t able to answer during my learning process:

How can you parallelize operations in Clojure? How easy is deployment? How does interop with Java work? Is there a rails-like web stack? Is there a style guide?

Continue Reading

Archiving a QuickBooks Online Account to QuickBooks Desktop

If you run a small business, you probably use QuickBooks. I’ve been impressed with the product: the rate of improvement continues to stay constant over the last couple of years (one of the most important criteria in picking a software platform for your business!) and it’s surprisingly pleasant to use.

If you close a company, you’ll want to archive all of your QuickBooks data for at least a couple of years in case you get audited. However, QuickBooks Online does not have a low-cost “audit backup” option to access a read-only version of your data. If you cancel your subscription, you only have 90 days to reactivate your subscription and restore your data. There are some QuickBooks Online apps that claim to backup your data, but there are significant caveats with most of these cloud apps.

The best (and cheapest) option is to download a QuickBooks Desktop version of your data which can then be later used to create a new QuickBooks Online account if you ever need to. However, this is more complicated than it should be (most likely, by design).

Hopefully, this guide makes it a bit easier!

1. Get Access to Windows 7 with Internet Explorer 11

In order to actually export the QuickBooks Desktop file from QuickBooks Online, you need Windows + IE. User-agent spoofing won’t work, there’s some wacky ActiveX plugin you need to install in order to complete the export process.

Other versions of Windows and IE may work, but this is the ‘official’ version stated to work when you login to QuickBooks Online > Exports > “Moving to QuickBooks Desktop?” > “Download company data”.

“How can I possibly get those exact versions, especially when I don’t have a Microsoft computer anywhere in sight?!” you may ask. Great question.

Luckily, Microsoft has virtual machines with a bunch of different Microsoft and IE versions (designed to be used for website testing) you can download and use for 90 days for free. Download the virtual machine here.

2. Download VirtualBox

You’ll need VirtualBox to run the virtual machine you downloaded from Microsoft. Download it here. You’ll also want to set up a couple of configuration options by navigating to Machine > Settings once your virtual machine is running:

General > Advanced > Shared Clipboard > Bidirectional. Helpful for sharing passwords for QB. Shared Folder > Add New. Set up a folder and make sure to auto-mount it. General > Displays. Make sure video memory is at least 128mb. For some reason the default was 4mb. 3. Download QuickBooks Desktop Trial Version

In order to download the QuickBooks Desktop file, you’ll need to have QuickBooks Desktop installed on your virtual machine. You can download a trial here.

4. Generate QuickBooks Desktop Download File

Follow these instructions to download your QuickBooks Online as a QuickBooks Desktop file. Here’s another guide that’s more simple (but won’t be kept up to date).

5. Verify & Backup

After you generate the QuickBooks Download file you’ll get a strange email with the subject “QuickBooks Online Simple Start: Company Data Ready for Download” asking you to “2. Click the task or to do item called ‘Download the company file created on xx/yy/zzz'”. This must be old email copy, because there is no such thing as a “task or to do item” in QuickBooks.

Here’s what you need to do:

Navigate to QuickBooks Online > Exports > “Moving to QuickBooks Desktop?” > “Download company data”. You’ll be walked through a process to download the file and import it into QuickBooks Desktop. If things don’t work, restart your computer and open up QuickBooks Desktop before opening up IE. After you’ve opened the file in QuickBooks desktop, run the P&L and Balance Sheet reports to ensure the numbers match up with QuickBooks Online. Create a backup from QuickBooks Desktop and store it in your shared folder Create a copy of the company for QuickBooks online. File > Utilities. Copy the backup to DropBox/Google Drive/whatever Recreate a new company in QuickBooks Desktop using your backup to ensure you can recreate it if you need it.

Continue Reading

How to Create a Custom Field to a NetSuite Customer or Transaction

NetSuite custom fields are incredibly powerful and flexible, but they are not easy to work with! Below is a quick guide that walks through how to create customer and transaction custom fields and some tips and tricks for working with custom fields.

How to create a custom customer (i.e. entity) field in NetSuite: Navigate to: Customization > Lists, Records & Fields > Entity Fields > New Label: use a name you’d like to see displayed on your customer. Type: “Free-form Text”. There are cases where you won’t want to use free-form text, but in most cases this is the best option. ID: “_a_internal_identifier”. Note that you should start your identifier with a _ since custentity will be prepended to whatever you provide here. You’ll want to change “_a_internal_identifier” to an ID that is descriptive of this field in your setup. Store value: checked Applies to: Customer Save How to create a transaction (invoice, cash sale, order, payment, etc) field in NetSuite: Navigate to: Customization > Transaction Body Fields > New Label: use a name you’d like to see displayed on your transaction. ID: _a_internal_identifier. Note that you should start your identifier with a _ since custbody will be prepended to whatever you provide here. Type: free form text Store Value: checked Applies to: Sale, Payment, etc. Where you want this field displayed is up to your use-case. Global Search: checked. This enables you to search for the value in the NetSuite global search Save Other Tips & Tricks for Working With NetSuite Custom Fields Customer (entity) custom fields will always start with custentity. Transaction (invoice, sales order, payment, etc) custom fields will always start with custbody. You can create custom fields on most, but not all records. For instance, you can’t create a custom field on the currency record in NetSuite but you can create one on a subsidiary. It’s possible to access “hidden” fields that are not available through SuiteTalk or SuiteScript using field defaults or formula field defaults on custom fields. You can source values from a related record. For instance, if you want to source a value from a CashSale connected to a CashRefund you could use a formula like {createdfrom.externalid} as the default value for a field that doesn’t store a value. A field which does not store value (i.e. a field that just contains a default value) is not available via a formula on another field. A field not found error will be reported. In other words, you can’t nest formulas. This is a huge bummer. If “formula” is not checked when creating the custom field you can still substitute field values using the {field} syntax You’ll get ERROR: Invalid Expression for lots of different types of errors. For instance, if your formula doesn’t output the correct field type you’ll get this error. This makes it very challenging to debug why a formula isn’t working. Join fields (i.e. {entity.datecreated}) strips the value of its type. Everything comes over as a string. You can use SQL functions in a formula field default

Continue Reading

Notes on Working with Jobs (Projects) using the SuiteTalk NetSuite API

Jobs are a confusing part of the NetSuite system. Here’s some notes and resources to help make handling jobs easier:

Jobs are named Projects in the GUI There are two versions of jobs: “standard” and “Advanced Jobs”. They both use the same core job record but change how the data is structured on the customer. Jobs have a kcustomer attribute that is not accessible via SuiteTalk which represents the customer that the job is associated with. This field is accessible via SuiteScript. You could most likely expose this field by creating a formula field on the job record with {kcustomer} as the formula. There is no customer field on the job schema However, there is a customer field on the job search columns. You can use an advanced search to return the customer associated with a job. If “Project Management” is enabled (aka ADVANCEDJOBS) the entity reference on the customer is a customer during the edit view, but will switch to be a job reference when the record is saved. When ADVANCEDJOBS is disabled the job reference could be either a customer or a job and the job field is not displayed. The parent field on a job can reference a customer. I believe you can have an heirarchy of jobs so it can be possible for the parent to be a job reference. Since the kcustomer field is not exposed via SuiteTalk the only way to determine the correct customer reference is to walk up the hierarchy.


Determine the customer associated with a job via SuiteTalk SuiteScript to extract a valid customer reference from an entity ID Determining if an entity ID is a customer or a project

Continue Reading

Notes on Dates & TimeZones with NetSuite’s SuiteTalk API

Setting date and datetime fields in NetSuite using the SuiteTalk API is tricky business. Here’s some notes on how to ensure that the date you push over to NetSuite persists the way you’d expect regardless of the server, local server, user, and company timezone.

The API is sensitive to the timezone of the current user. If you push the same datetime value to the XML via SuiteTalk, and change the timezone settings on the user preferences page, the resulting date set on the record will most likely change. The timezone is not available on the employee record. Also, the timezone is not available on the company/subsidiary record. It is impossible to determine the timezone set on the user or the company from the SuiteTalk API. The get_server_time method does not return the time in the current user’s timezone preference. There is no SuiteScript-specific method to getting a timezone. Although it looks like you can infer it from the stdlib JS responses. It could be possible to create a RESTlet bundle that could retrieve the timezone of the current user as a JSON response. If the timezone of the user (employee) differs from the timezone of the request NetSuite will convert the datetime in the request to the timezone of the current user. If you push a date of 2017-04-17T00:00:00-07:00 and the user’s timezone is GMT -04:00 the resulting date field in NetSuite returns 2017-04-16T21:00:00-07:00. This appears properly as 04/17 in the GUI for the current user as well as for other users with different timezone configurations. If you push a date in the timezone of the NetSuite servers (PDT) NetSuite seems to ignore the timezone of the user used to make the request. Note that accounting for DST offset and zeroing out the hours, minutes, seconds is essential to making this work. Here’s an example of what your date should look like 2017-04-17T00:00:00-07:00. If you push a date one second after midnight NetSuite will take into account the user’s timezone. Best practice: convert your date to UTC0 and use NetSuite::Utilities.normalize_time_to_netsuite_date(time_stamp). This utility will handle DST, zeroing out hr/minute/seconds, and convert the datetime to the right format for SuiteTalk. The date returned by the API is adjusted based on the user’s timezone. For instance, if you push 2017-04-17T00:00:00-07:00 to NetSuite with a user whose timezone is set to EST the date returned by the API will be 2017-04-16T17:00:00-07:00 although it will properly display as 2017-04-17 in the GUI.


NetSuite ruby client conversion utility Converting between time and datetime in ruby Modifying time without railscn

Continue Reading