Using GitHub Actions With Python, Django, Pytest, and More

GitHub actions is a powerful tool. When GitHub was first released, it felt magical. Clean, simple, extensible, and adds so much value that it felt like you should be paying for it. GitHub actions feel similarly powerful and positively affected the package ecosystem of many languages.

I finally had a chance to play around with it as part of building a crypto index fund bot. I wanted to setup a robust CI run which included linting, type checking, etc.

Here’s what I learned:

• It’s not possible to test changes to GitHub actions locally. You can use the GH CLI locally to run them, but GH will use the latest version of the workflow that exists in your repo. The best workflow I found is working on a branch and then squashing the changes.
• You can use GitHub actions to run arbitrary scripts on a schedule. This may sound obvious, but it can be used in really interesting ways, like updating a repo everyday with the results of a script.
• You can setup dependabot to submit automatic package update PRs using a .github/dependabot.yml file.
• The action/package ecosystem seems relatively weak. The GitHub-owned actions are great and work well, but even very popular flows outside of the default action set do not seem widely used and seem to have quirks.
• There are some nice linting tools available with VS Code so you don’t need to remember the exact key structure of the GitHub actions yaml.
• Unlike docker’s depends_on, containers running in the services key, are not linked to the CI jobs in a similar way to docker compose yaml files. By ‘linked’ I’m referring to exposing ports, host IP, etc to the other images that are running your jobs. You need to explicitly define ports to expose on these service images, and they are all bound to localhost.
• on: workflow_dispatch does not allow you to manually trigger a workflow to run with locally modified yaml. This will only run a job in your yaml already pushed to GitHub.
• Matrix builds are easy to setup to run parallelized builds across different runtime/dependency versions. Here’s an example.
• Some details about the postgres service:
  • Doesn’t seem like you can create new databases using the default postgres/postgres username + password pair. You must use the default database, postgres.
  • Unlike docker, the image does not resolve the domain postgres to an IP. Use 127.0.0.1 instead.
  • You must expose the ports using ports: otherwise redis is inaccessible.
  • You must set the password on the image, which felt very strange to me. You’ll run into errors if you don’t do this.

Here’s an example .github/workflows/ci.yml file with the following features:

• Redis & postgres services for Django ORM, Django cache, and Celery queue store support.
• Django test configuration specification using DJANGO_SETTINGS_MODULE. This pattern is not standard to django, here’s more information about how this works and why you probably want to use it.
• Database migrations against postgres using Django
• Package installation via Poetry
• Caching package installation based on VM type and SHA of the poetry/package lock file
• Code formatting checks using black and isort
• Type checking using pyright
• Linting using pylint
• Test runs using pytest

name: Django CI

on:
  workflow_dispatch:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    # each step can define `env` vars, but it's easiest to define them on the build level
    # if you'll add additional jobs testing the same application later (which you probably will
    env:
      DJANGO_SECRET_KEY: django-insecure-@o-)qrym-cn6_*mx8dnmy#m4*$j%8wyy+l=)va&pe)9e7@o4i)
      DJANGO_SETTINGS_MODULE: botweb.settings.test

      REDIS_URL: redis://localhost:6379
      TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres

    # port mapping for each of these services is required otherwise it's inaccessible to the rest of the jobs
    services:
      redis:
        image: redis
        # these options are recommended by GitHub to ensure the container is fully operational before moving 
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379
      postgres:
        image: postgres
        ports:
          - 5432:5432
        env:
          POSTGRES_PASSWORD: postgres

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.9.6

    # install packages via poetry and cache result so future CI runs are fast
    # the result is only cached if the build is successful
    # https://stackoverflow.com/questions/62977821/how-to-cache-poetry-install-for-github-actions
    - name: Install poetry
      uses: snok/install-poetry@v1.2.0
      with:
        version: 1.1.8
        virtualenvs-create: true
        virtualenvs-in-project: true
    - name: Load cached venv
      id: cached-poetry-dependencies
      uses: actions/cache@v2
      with:
        path: .venv
        key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
    - name: Install dependencies
      run: poetry install
      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'

    - name: Linting
      run: |
        source .venv/bin/activate
        pylint **/*.py

    - name: Code Formatting
      run: |
        # it's unclear to me if `set` is required to ensure errors propagate, or if that's by default in some way
        # the examples I found did not consistently set these options or indicate that it wasn't required
        set -eax
        source .venv/bin/activate
        black --version
        black --check .
        isort **/*.py -c -v

    - name: Setup node.js (for pyright)
      uses: actions/setup-node@v2.4.0
      with:
        node-version: "12"
    - name: Run type checking
      run: |
        npm install -g pyright
        source .venv/bin/activate
        pyright .

    - name: Run DB migrations
      run: |
        source .venv/bin/activate
        python manage.py migrate
    - name: Run Tests
      run: |
        source .venv/bin/activate
        pytest