Building a Crypto Index Bot and Learning Python
A long time ago, I was contracted to build a MacOS application using PyObjc. It was a neat little app that controlled the background music at high-end bars around London. That was the last time I used python (early 2.0 days if I remember properly). Since then, python has become the language of choice for ML/AI/data science and has grown to be the 2nd most popular language.
I've been wanting to brush up on my python knowledge and explore the language and community. Building a bot to buy a cryptocurrency index was the perfect learning project, especially since there was a bunch of existing code on GitHub doing similar things.
You can view the final crypto index bot project here. The notes from this learning project are below. These are mainly written for me to map my knowledge in other languages to python. Hopefully, it's also helpful for others looking to get started quickly in the language!Tooling & Package Management
One of the important aspects of a language for me is the REPL and tinkering/hacking environment. If I can't open up a REPL and interactively write/debug code, I'm a much slower developer. Thus far, ruby has the best interactive development environment that I've encountered:binding.pry and binding.pry_remote when your console isn't running your code directly to open a repl Automatic breakpoints on unhandled exceptions, in tests or when running the application locally Display code context in terminal when a breakpoint is hit Print and inspect local variables within a breakpoint Navigate up and down the callstack and inspect variables and state within each frame Overwrite/monkeypatch existing runtime code and rerun it with the new implementation within a repl Define new functions within the repl Inspect function implementation within the repl
I'd say that python is the first language that matches ruby's debugging/hacking environment that I've used. It's great, and better than ruby in many ways.inspect is a very helpful stdlib package for poking at an object in a repl and figuring out the method, variables, etc available to it. traceback provides some great tools for inspecting the current stack. How you drop an interactive console at any point in your code? There are a couple ways: Uses the ipython enhanced repl in combination with the built in debugger import ipdb; ipdb.set_trace(). Requires you to install a separate package. There's a breakpoint() builtin that launches the standard pdb debugger. You can configure breakpoint() to use ipdb via export PYTHONBREAKPOINT=ipdb.set_trace. All of the standard pdb functions work with ipdb import code; code.interact(local=dict(globals(), **locals())) can be used without any additional packages installed. bpython is a great improvement to the default python. You need to install this within your venv otherwise the packages within your projects venv won't be available to it: pip install bpython && asdf reshim ipython is a bpython alternative that looks to be better maintained and integrates directly with ipdb. python -m ipdb script.py to automatically open up ipython when an exception is raised when running script.py Some misc ipython tips and tricks: If something is throwing an exception and you want to debug it: from ipdb import launch_ipdb_on_exception; with launch_ipdb_on_exception(): thing_causing_exception() who / whos in whereami %psource or source like show-source pp to pretty print an object ipython --pdb script.py to break on unhandled exceptions Great grab bag of interesting tips %quickref for detailed help exit gets you out of the repl entirely All of the pypi information is pulled from a PKG-INFO file in the root of a package rich-powered tracebacks are neat, especially with locals=True The ruby-like metaprogramming/monkeypatching stuff happens via the __*__ functions which are mostly contained within the base object definitions. For instance, logging.__getattribute__('WARN') is equivalent to logging.WARN You can reload code in a REPL via from importlib import reload; reload(module_name). Super helpful for hacking on a module (definitely not as nice as Elixir's recompile). Monkeypatching in python isn't as clean as ruby, which in some ways is better since monkeypatching is really an antipattern and shouldn't be used often. Making it harder and more ugly helps to dissuade folks from using it. To monkeypatch, you reassign the function/method to another method: ClassName.method_name = new_method. Here's an example. Typing
I've become a huge fan of gradual types in dynamic languages. I never use them right away, but once the code hardens and I'm relatively sure I won't need to iterate on the code design, I add some types in to improve self-documentation and make it safer to refactor in the future.
Python has a great gradual type system built-in. Way better than Ruby's.mypy . on the command line to test all python files within a folder. If your project fails to pass mypy, it won't cause any runtime errors by default. There's a VS Code extension. This extension is included in Pylance, which you should probably be using instead, but you need to set the typing mode to 'basic'. Return value types are set with -> before the : at the end of the method definition. Otherwise, typing works very similar to other languages with gradular typing (TypeScript, Ruby, etc). A common pattern is importing types via import types as t t.Union[str, float] for union/any types, You can't merge dictionaries if you are using a TypedDict (dict | dict_to_merge). Massive PITA when mutating API data. Verbose types can be assigned to a variable, and that variable can be used in type definintions. Handy way to make your code a bit cleaner. Enums defined with enum.Enum can be types. Testing Like Elixir, there are doctests that execute python within docstrings to ensure they work. Neat! There are built-in test libraries that look comparable to ruby's testunit. pytest is similar to minitest: provides easy plugins, some better standard functionality, and builds on top of unittest. You probably want to use pytest for your testing framework. setup.cfg is parsed by pytest automatically and can change how tests work. conftest.py is another magic file autoloaded by pytest which sets up hooks for various plugins. You can put this in the root of your project, or in test/ Test files must follow a naming convention test_*.py or *_test.py. If you don't follow this convention, they won't be picked up by pytest by default. breakpoint()s won't work by default, you need to pass the -s param to pytest Like ruby, there are some great plugins for recording and replaying HTTP requests. Checkout pytest-recording and vcrpy. To record HTTP request run pytest --record-mode=once If you want to be able to inspect & modify the API responses that are saved, use the VCR configuration option "decode_compressed_response": True There's a mocking library in stdlib, which is comprehensive. I'm not sure why other languages don't do this—everyone needs a mocking library. It looks like you set expectations on a mock after it runs, not before. Here's how mocking works: The @patch decorator is a clean way to manage mocking if you don't have too many methods or objects to mock in a single test. If you add multiple patch decorators to a method, the mocks for those methods are passed in as additional arguments. The last patch applied is the first argument. mock.call_count, mock.mock_calls, mock.mock_calls.kwargs are the main methods you'll want for assertions asset without parens is used in tests. This confused me, until I looked it up in the stdlib docs and realized assert is a language construct not a method. tox is much more complex that pytest. It's not a replacement for pytest, but seems to run on top of it, adding a bunch of functionality like running against multiple environments and installing additional packages. It feels confusing—almost like GitHub actions running locally. If you want to just run a single test file, you need to specify an environment identifier and test file tox -epy38-requests -- -x tests/unit/test_persist.py My thoughts on Python
The big question is if Django is a good alternative to Rails. I love Rails: it's expansive, well-maintained, thoughtfully designed and constantly improving. It provides a massive increase in development velocity and I haven't found a framework that's as complete as Rails. If Django is close to rails, I don't see a strong argument for not using anything python over ruby for a web product.Open Questions
Some questions I didn't have time to answer. If I end up working on this project further, this is a list of questions I'd love to answer:How good is django? Does it compare to Rails, or is it less batteries-included and more similar to phoenix/JS in that sense. Does numpy/pandas solve the data manipulation issue? My biggest gripe with python is the lack of chained data manipulation operators like ruby. How does the ML/AI/data science stuff work? This was one of my primary motivations for brushing up on my python skills and I'd love to deeply explore this. How does async/await work in python? Learning Resources
General guides:https://python-patterns.guide/python/module-globals/ https://book.pythontips.com/en/latest/ternary_operators.html https://realpython.com/python-lambda/#anonymous-functions https://google.github.io/styleguide/pyguide.html
Monkeypatching:https://sharmapacific.in/monkey-patching-in-python/ https://github.com/ytdl-org/youtube-dl/commit/00fcc17aeeab11ce694699bf183d33a3af75aab6 https://filippo.io/instance-monkey-patching-in-python/ https://tryolabs.com/blog/2013/07/05/run-time-method-patching-python/ Open Source Example Code
There are some great, large open source python projects to learn from:https://github.com/getsentry/sentry https://github.com/arachnys/cabot - opens source APM https://github.com/vitorfs/bootcamp https://github.com/rafalp/Misago
Download these in a folder on your local to easily grep through.