Jump to content

Python and Typescript in a monorepo

From JOHNWICK

Many companies already run their SaaS with one or a few microservices — often a backend and some background workers — and some use a monorepo to support this. That’s the setup we have at Taskworld. With AI now everywhere and most of the tooling already in Python, rewriting everything in TypeScript is hard, costly, or simply unrealistic. You can either create a second monorepo for Python or adopt tools like Bazel. This post shares our experience navigating those choices.

Why a Monorepo? Our monorepo originally supported only TypeScript, as the whole company stack was built around it. It helped us avoid the overhead of spinning up new repos, syncing PR policies, and switching between codebases. It hosts our services, internal packages, and a toolbox of scripts for infrastructure. Design choices

We avoided tools like turbo, which aim to optimize multi-workspace CI pipelines. The overhead of adoption — ownership, team ramp-up, and inevitable workarounds — was too much for a first attempt. Instead, we focused on simplicity and atomic workflows. We leaned on GitHub Actions and community tooling (e.g., @actions/cache, Renovate) and kept things familiar with pnpm. Our setup resembles what Lerna used to offer: pnpm workspaces and a CI workflow that detects changed workspaces and runs jobs in parallel, and that’s it..

Core principles Initially, we parsed the output of pnpm --filter="[origin/main]" list to build a matrix and distribute it to another job. Each matrix job would run in isolation. Test jobs would pnpm lint, pnpm build, and pnpm test. Build jobs would pnpm build and either push services (pnpm push) or publish packages (pnpm publish).

Since CI jobs are created dynamically, we had a brief issue where auto-merge would trigger before all jobs were done. To fix that, we added a passed job to enforce full matrix completion before marking the CI as green: 

passed:
    if: ${{ always() }}
    name: passed
    needs: [resolve, test]
    runs-on: ubuntu-latest

    steps:
      - run: |
          result="${{ needs.test.result }}"
          if [[ $result == "success" || $result == "skipped" ]]; then
            exit 0
          else
            exit 1
          fi

Publishing changes Our initial strategy mimicked changesets: files committed in a .versions folder described what to publish (workspace + version), and merges to main would trigger publish workflows based on that. This handled package publishing and staging releases. Production deployments are managed via Kubernetes manifests in a separate repo. Updating dependencies With Taskworld being SoC2 compliant, we also had to find a way to address dependency drifting and allow self-updates to happen. We picked renovate for its better handling of monorepo (at the time).

Optimizations Thanks to its very modular approach, we fined-tuned our usage: 1. we added a rule to only bump the version of a workspace when production dependencies were bumped (that way, we wouldn’t redeploy a service when eslint got an update). 

{
  "bumpVersion": null,
  "packageRules": [
  {
     // only enable semantic commits if production dependencies have changed
     "matchDepTypes": ["dependencies"],
     "bumpVersion": "minor",
  ]
},
2. we used it for node-version management too:
{
  "major": {
    "enabled": false
  },
  "rangeStrategy": "bump",
  "updatePinnedDependencies": false,
  "packageRules": [
    {
      // nodenv are always pinned, so we want to keep updating them like this
      "matchManagers": ["nodenv"],
      "matchNewValue": "/^[12][02468]\\./",
      "major": {
        "enabled": true
      },
      "updatePinnedDependencies": true,
    },
  ]
}

Tradeoffs On the bad side of it, since we used a “changeset” approach, it came with the need of a post-processing step to commit the dotversion file. In order to fit into our CI process, we had to create a dedicated workflow for it for which we had to play with Github identities to make the new commits run the testing workflows. 

on:
  pull_request:
    types:
      - opened
      - reopened
jobs:
  renovate:
    if: startsWith(github.head_ref, 'renovate/')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          token: ${{ secrets.ADMIN_TOKEN }}
          fetch-depth: 0

      - run: |
          git config user.name 'github-actions'
          git config user.email '41898282+github-actions[bot]@users.noreply.github.com'

      - name: create a dotversion lockfile
        run: pnpm run create-dotversion

      - name: commit version files
        run: |
          git add .versions
          git commit -m "renovate dotversion"

      - name: push to branch
        continue-on-error: true
        run: |
          git push origin ${{ github.event.pull_request.head.ref }}

(As mentioned in the design principles, the create-dotversion script was running pnpm --filter=[origin/main] list and would reformat it into a Record<name, version>-like data structure.)

Adding rust to the mix Rust was the second runtime we added to the monorepo. We introduced it for Taskworld’s GraphQL API, which we built on Apollo Router using a GraphQL federation.

To integrate Rust seamlessly, we treated it like another nodejs workspace: we added a package.json to define its CI tasks and wire rust tooling in it (rustfmt for lint, rustc for build, etc.). We also leveraged the package.jsonname and version fields rather than the Cargo.tomland only kept the Cargo files for dependency management. This worked ok for the most parts, but came with tradeoffs. First, toolchain was a consideration. How to install Rust only where it’s needed? Thankfully, Cargo manages its own toolchain so there wasn’t much to worry. We simply added a parameter in our composite setup action to call rustup: 

name: Setup action
description: Install pnpm and node runtime

inputs:
  workspace:
    required: false
    description: name of the workspace to set up, optional

  with-rust:
    required: false
    description: update rust toolchain to latest

runs:
  using: composite
  steps:
    - if: ${{ inputs.with-rust }}
      run: |
        rustup update stable

        rustc --version
        cargo --version
      shell: bash

    - uses: pnpm/action-setup@v3
      with:
        version: 9.7.0

    - if: ${{ !inputs.workspace }}
      uses: actions/setup-node@v4
      with:
        node-version: 20
        cache: 'pnpm'
        cache-dependency-path: ./pnpm-lock.yaml
        registry-url: https://npm.pkg.github.com/
        scope: '@taskworld'

    - if: ${{ inputs.workspace }}
      run: |
        abs=$(pnpm --silent --filter "${{ inputs.workspace }}" exec -- pwd)
        rel=$(echo ${abs} | sed "s|$(pwd)/||g")

        echo "abs=${abs}" >> $GITHUB_OUTPUT
        echo "rel=${rel}" >> $GITHUB_OUTPUT

      id: paths
      shell: bash

    - if: ${{ inputs.workspace }}
      uses: actions/setup-node@v4
      with:
        node-version-file: ${{ format('{0}/.node-version', steps.paths.outputs.rel) }}
        cache: 'pnpm'
        cache-dependency-path: ${{ format('{0}/pnpm-lock.yaml', steps.paths.outputs.rel) }}
        registry-url: https://npm.pkg.github.com/
        scope: '@taskworld'

In our test workflow it was set to workspace.name == 'graphql-gateway' ; in the renovate action, it was set to true which wasn’t ideal but we thought that was ok enough.

Second, using package.json for versioning broke compatibility with Renovate’s native bumpVersion. Renovate didn’t account for our CI workaround and attempted to update Cargo.toml version field rather than the one we used. As a result, our pnpm --filter=[origin/main] list wouldn’t catch it. We moved to handle version bumps manually during a CI post-processing step instead. Since we limited bumps to production dependency updates only, that meant re-implementing a versioning strategy from scratch and introducing exceptions for Rust (since its dependencies still lived in Cargo.toml). 

- id: workspaces
  name: bump package.json versions
  run: |
    # this is doing the job of searching if a given workspace had its prod dependencies
    # updated or not. we take the dependencies node of the package.json on main, and the one
    # from this branch and perform a strict == comparison. if !=, it's been changed, so
    # it requires a bump.

    workspaces=$(pnpm --silent --filter="[origin/main]" exec -- jq -r .name package.json)
    workspaces=($(echo "${workspaces}"))
    length="${#workspaces[@]}"

    echo "length=${length}"
    echo "length=${length}" >> $GITHUB_OUTPUT

    for workspace in "${workspaces[@]}"; do
      service_path=$(pnpm --silent --filter "${workspace}" exec -- pwd | sed "s|$(pwd)/||g")

      # special case for rust, good enough for now.
      if [ -f "${service_path}/Cargo.toml" ]; then
        old=$(git show "origin/main:${service_path}/Cargo.toml" | toml2json | jq -r -S -c ".dependencies")
        new=$(cat "./${service_path}/Cargo.toml" | toml2json | jq -r -S -c ".dependencies")
      # fallback is to use package.json
      else
        old=$(git show "origin/main:${service_path}/package.json" | jq -r -S ".dependencies")
        new=$(cat "./${service_path}/package.json" | jq -r -S ".dependencies")
      fi

      if [[ "${old}" != "${new}" ]]; then
        echo "Updating ${workspace}..."
        pnpm --silent --filter="${workspace}" exec -- pnpm version minor --no-git-tag
      else
        echo "${workspace} skipped!"
      fi
    done

That worked for a while, until it didn’t. Adding python…

Adding python meant that our hardcoded workaround wouldn’t work anymore. Python toolchain and dependencies are handled differently so we couldn’t expand on the existing strategy. Before adding python, we had to stop pushing forward and rearrange things and smoothen the bumps we created with rust. The first thing we did was use mise-en-place for toolchain management: Instead of hoping that rustc and cargo would always be natively available in our CI runner, we wanted to make sure things would be smooth no matter what.

Mise is a great tool in the sense of that it supports a lot of older existing solutions already. It is not invasive. It can read the .node-version files we already had, the package.json engine field, as well as the rust-toolchain.toml file we had for our gateway. Looking at python, we knew it could support pyproject.toml files as well. Our setup action evolved to use mise install in the monorepo root and a second mise install in the workspace directory. It removed the need for that with-rust that felt hacky — it worked like a charm from the first push.

We could have stopped here and add another if [ -f “${service_path}/pyproject.toml” ]; then in the renovate CI flow. The thing is that this separation of version fields was starting to be confusing, and also the conflicts to be resolved created by the commits added necessarily by the manual management of the renovate flow was a mess. Could we eventually go back to the days where renovate was self-sufficient before adding python?

Monorepo, 2nd iteration Looking deeper at thepost-processing step in the renovate flow made us realize two things that we did which were wrong:

  • the commit for dotversion files
  • the manual handling of version diffing

About changesets Spoiler: it didn’t work for us. At Taskworld, engineers rarely work across multiple services in the same sprint. Most PRs target a single workspace, and changes are typically intended to go to staging immediately after merge. Since production releases are handled elsewhere, we don’t need synchronized staging releases.

After a year of using the monorepo, we noticed that nearly every PR bumped a service version — and just as often, engineers forgot to include the .versions file. The extra step didn’t feel natural, and many didn’t understand why merging alone wasn’t enough.

Changesets are also useful for avoiding version downgrades caused by concurrent PRs (when each are bumping the same workspace’s version independently). But we realized we could solve that differently.

What if we didn’t use changeset and rely on git diff for version diffing? This would work but would be prone to a few errors, so we enforced two simple rules:

  • No one can push — or force-push — to main (a SoC2 requirement anyway)
  • Linear history is mandatory (you must rebase before merging)

That way, we’re sure version can’t never go downward. Then, it was safe to try ourgit diff based approach which would eliminate the need for dotversion files (and their commit).

The first version of this script was: 

// utility
const fs = require('fs')
const path = require('path')
const util = require('util')

// async shell
const $ = util.promisify(require('child_process').exec)

// semver comparison
function gt(vA, vB) {
  const [majA, minA, patchA] = vA.split('.', 3).map(n => parseInt(n, 10))
  const [majB, minB, patchB] = vB.split('.', 3).map(n => parseInt(n, 10))

  return majA > majB || minA > minB || patchA > patchB 
}

// params
const base = '${{ inputs.base }}';
const head = '${{ inputs.head }}';

// script
const diff = await $(`git diff --name-only ${base} ${head}`)
  .then(({ stdout }) => stdout.split('\n').filter(Boolean));

const prefixes = ['jobs/', 'packages/', 'services/']

const workspaces = diff
  // only known workspaces
  .filter(path => prefixes.some(prefix => path.startsWith(prefix)))
  // workspace root folder
  .map(path => path.split('/').slice(0, 2).join('/'))
  // remove duplicates
  .filter((e, i, a) => a.indexOf(e) === i)
  // discard removed
  .filter(path => fs.existsSync(`./${path}`))

let outdated = [] // workspace has changes, version did bump but is behind base
let untouched = [] // workspace has changes but version didn't move
let updated = [] // workspace has changes and version did bump properly

for (const workspacePath of workspaces) {
  const packagePath = `${workspacePath}/package.json`
  const touched = diff.includes(packagePath)

  const { name, version } = JSON.parse(
    fs.readFileSync(packagePath, 'utf-8')
  )

  const workspace = {
    name,
    version,
    relativePath: workspacePath,
  }

  if (!touched) {
    untouched.push(workspace)
  } else {
    try {
      const { stdout: file } = await $(`git show ${base}:${packagePath}`)
      const { version: published } = JSON.parse(file)

      if (workspace.version === published) {
        untouched.push(workspace)
      } else if (gt(workspace.version, published)) {
        updated.push(workspace)
      } else {
        outdated.push(workspace)
      }
    } catch (err) {
      core.error(err)
    }
  }
}

const reports = {
  outdated,
  untouched,
  updated,
  workspaces: [...outdated, ...untouched, ...updated],
}

for (const [category, workspaces] of Object.entries(reports)) {
  core.setOutput(`${category}`, workspaces)
  core.setOutput(`${category}-count`, workspaces.length)
  core.setOutput(
    `${category}-list`,
     workspaces
       .map(({ name, version }) => `- ${name} (v${version})`)
       .join('\n')
  )
}

console.log(reports)

It was simple enough to let the engineers work freely without having to worry about dotversion files. It was well received and made the workflow feel much lighter. That said, it still relied on having a package.json in every workspace and was not rust or python friendly.

Introducing wr, a runtime agnostic Workspace Resolver On the side, I had already been managing a personal monorepo and, inspired by how mise handles multiple runtimes, I realized I could apply a similar approach of its backend management to our monorepo.

The initial scope was simple: reference different backends with their respective manifest files, read their name and version, and use that git diff approach we had earlier to build a map of the monorepo’s workspaces. That led to y-nk/wr, a small workspace resolver tool that provides 4GitHub Actions for diffing, resolving, and listing workspaces across Node.js, Rust, Python, and Ruby projects. We eventually swapped our old diffing script (above) for y-nk/wr/diff@main which produced the same output—but it supported reading Cargo.toml for Rust. Reading version from Cargo.toml meant we could use Renovate's native bumpVersion again, and therefore that the whole rewrite of renovate’s packageRule we initially had could go too.

In short, we no longer needed the custom post-processing workflow we had built just for Renovate.

Consequences of a “package.json free” monorepo Besides being cleaner, dropping the need for package.json everywhere had real benefits. Renovate now works better, independently, and without needing workarounds. We couldn’t rely on our previous convention of having package.json file in every workspace to act as a task runner ; but it wasn’t a problem since we had a better one: mise.

The switch was painless: pnpm install changed to mise run install for dependency management, and pnpm run build to mise run build for CI tasks.

mise also has a very handy lookup feature which we instantly adopted: one .mise.toml at the root of the monorepo contains the “most common nodejs” tasks and if a workspace needs customization, they can create their own locally. Our “generic nodejs workspace” mise config file looks like this: 

[tools]
"node" = "22"
"npm:pnpm" = "10.7.1"

# these tasks are global since we mostly use nodejs/pnpm
# so they are defined at root level to avoid repetition
# since the file is not defined in the workspace, we must
# add {{cwd}} to them otherwise the commands will run where
# the mise file is located rather than cwd.

[tasks.build]
dir = "{{cwd}}"
run = "pnpm build"

[tasks.install]
dir = "{{cwd}}"
run = "pnpm install --frozen-lockfile"

[tasks.lint]
dir = "{{cwd}}"
run = """
#!/bin/bash
pnpm run lint;

MONOREPO_ROOT="$(pnpm -w exec -- pwd)"
WORKSPACE_DIR=$(pwd)

pnpm -w exec prettier --ignore-path ${MONOREPO_ROOT}/.gitignore --ignore-path ${MONOREPO_ROOT}/.prettierignore --ignore-path $WORKSPACE_DIR/.gitignore --ignore-path $WORKSPACE_DIR/.prettierignore --check $WORKSPACE_DIR;
"""

[tasks.publish]
dir = "{{cwd}}"
run = "pnpm publish --no-git-checks"

[tasks.test]
dir = "{{cwd}}"
run = "pnpm test"

With this in place, CI was now agnostic of package.json file to 100% of its needs and engineers could still decide to use pnpm instead of opting-in to mise. (if you noticed, we have a pnpm -w exec prettier which means we installed nodejs linting at monorepo root to keep it in sync accross the entire monorepo) For our rust graphql-gateway, the .mise.toml file looks like this: 

[tasks.build]
run = "exit 0"

[tasks.install]
run = "exit 0"

[tasks.lint]
run = "rustfmt --check ./**/*.rs --edition 2021"

[tasks.push]
run = "./scripts/push.sh"

[tasks.test]
run = [ 'mise run test-unit', 'mise run test-spec' ]

[tasks.test-unit]
hide = true
run = """
docker build . -f ./Dockerfile.build -t router-build;
docker run router-build cargo test --release;
"""

[tasks.test-spec]
hide = true
run = """
cd tests;
pnpm install --ignore-workspace;
pnpm test;
"""

[_]
"docker-cache" = ["Cargo.lock", "Dockerfile.build"]

A couple of things to note here: There are no traditional install or build steps because the build happens within Docker. We took this approach because the gateway takes around 20 minutes to build due to the massive number of dependencies, and we wanted to cache the build for unit and integration tests. The test-unit task builds the Docker container and caches steps up to a certain point, while the test-spec task reuses those cached steps to continue building further.

Another neat feature of mise we leveraged is its free-form section [_] in which you can store anything you need. We used this to let each workspace define its own Docker build cache keys, and implemented Docker layer caching using the scribmd/docker-cache action. 

- id: cache-key
  working-directory: ${{ steps.workspace.outputs.path }}
  run: |
    keys=$(mise config get _.docker-cache || echo '[]')
    echo "result=${keys}" >> $GITHUB_OUTPUT

- if: steps.cache-key.outputs.result != '[]'
　uses: ScribeMD/[email protected]
  with:
  # we can't do better since hashFiles does not support passing an array so, we'll accept that 4 globs are a limitation. it should be enough to target: dockerfile, pnpm lock, cargo lock, and an extra glob.
  key: |
  　0:${{ hashFiles(fromJson(steps.cache-key.outputs.result)[0]) }};1:${{ hashFiles(fromJson(steps.cache-key.outputs.result)[1]) }};2:${{ hashFiles(fromJson(steps.cache-key.outputs.result)[2]) }};3:${{ hashFiles(fromJson(steps.cache-key.outputs.result)[3]) }};

Once this was done… Adding python! It was a breeze. We added the workspace, moved from pip’s requirement.txt to uv and its pyproject.toml. The .mise.toml was extremely simple to write: 

[tools]
uv = "latest"

[settings]
python.uv_venv_auto = true

[tasks.build]
run = "exit 0"

[tasks.install]
run = "uv sync --frozen --only-dev"

[tasks.lint]
run = "uvx ruff check"

[tasks.test]
run = "docker compose up --detach --wait; uv run --no-project pytest tests/*"

The only small tweaks we had to make were adding --only-dev and --no-project. Since we wanted integration tests to run within Docker, we moved the build step into the Dockerfile and used Docker Compose to build and run the server. However, we ran into an issue: production dependencies were being downloaded twice — once during the Docker build and again when running uv run pytest — which eventually led to disk space errors in the CI runner. By limiting installs to --only-dev, we ensured only pytest was downloaded. The --no-project flag further prevented any project discovery from triggering additional dependency downloads when running uv run.

Further tricks and optimizations Now that Renovate can update workspaces in large batches regardless of the tech stack, we’re much more comfortable letting it manage a high number of workspaces in a single PR. However, that came with a drawback: if just one workspace required manual intervention after an update (to fix a failing test, for instance), it would block the entire PR. Any push would re-trigger all workspace workflows, leading to unnecessary CI usage and costs. Caching individual workspace

To fix this, we introduced a lightweight caching strategy:

  • In the first step of the workflow, we use @actions/cache to cache a single .passed file. The hash key is based on the workspace folder content.
  • Every test/build step for that workspace is guarded with if: steps.cache.outputs.cache-hit != 'true' to skip execution if nothing changed.
  • The .passed file is only generated and committed to the CI runner at the very end of the workflow, and only if success().

This way, if our rust workspace’s workflow passed, its own .passed file would be created and the post action would create the cache entry for the corresponding hash key. Later when our python workspace would be modified, the rust’s workflow would still run—but since its source didn’t change, it’s cache key would remain the same and it would instantly hit the cache and complete in under a minute, skipping all the costly steps.

Spacing renovate triggers smartly Renovate supports human expressions to setup advanced schedules, which we leveraged to achieve bi-weekly crons. Our package rules currently look like this: 

  "packageRules": [
    {
      // only enable semantic commits if production dependencies have changed
      "matchDepTypes": ["dependencies"],
      "bumpVersion": "minor",
    },

    // monthly updates
    {
      "managers": ["cargo"],
      "groupName": "graphql-gateway",
      "schedule": ["on the 1st day instance on monday before 9am"],
      "matchFileNames": ["services/graphql-gateway/**"],
    },
    {
      "managers": ["dockerfile"],
      "groupName": "elasticsearch",
      "schedule": ["on the 1st day instance on monday before 9am"], 
      "matchFileNames": ["services/elasticsearch/**"],
    },
    {
      "managers": ["pep621"],
      "groupName": "embeddings-server",
      "schedule": ["on the 1st day instance on monday before 9am"],
      "matchFileNames": ["services/assistant-embeddings-server/**"],
      "bumpVersion": "minor", // forced
    },
    // nodejs packages, bi-weekly (1, 3)
    {
      "managers": ["npm"],
      "groupName": "packages",
      "schedule": ["on the 1st and 3rd day instance on monday before 9am"],
      "matchFileNames": ["packages/**"],
      "postUpdateOptions": ["pnpmDedupe"],
    },
    // the rest of node (services, jobs), bi-weekly (2, 4)
    {
      "managers": ["npm"],
      "groupName": "all",
      "schedule": ["on the 2nd and 4th day instance on monday before 9am"],
      "matchFileNames": [
        "package.json",
        "jobs/**",
        "services/!(elasticsearch|graphql-gateway)/**"
      ],
      "postUpdateOptions": ["pnpmDedupe"],
    },
    // node runtime
    {
      "matchManagers": ["nodenv", "npm"],
      "matchDepNames": ["@types/node"],
      "matchNewValue": "/^[12][02468]\\./",
      "updatePinnedDependencies": true,
      "major": {
        "enabled": true
      },
    },
    // nodejs major dependencies management: one per workspace
    // (every 1st thursday of the month, rate-limited at 5 concurrents)
    {
      "managers": ["npm"],
      "groupName": "{{packageFileDir}}",
      "schedule": ["on the 1st day instance on thursday before 9am"], 
      "major": {
        "enabled": true
      },
      "updateTypes": ["major"],
    },
  ],

Providing a default docker path We have a lot of files to maintain — and a good portion of them are just copypasta. With over 70% of our monorepo made up of standard TypeScript services, Dockerfiles quickly became redundant across workspaces. To reduce duplication, we moved the common Dockerfile to .github/Dockerfile, and made it the fallback when a workspace didn’t provide its own.

We applied the same logic to task definitions: if a workspace didn’t define a push task in its .mise.toml, we’d fall back to a default using the docker/build-push-action. This setup gave us a convention-over-configuration approach without forcing any rigid constraints—services could always opt out by providing their own files or script, but the default path was simple, predictable, and consistent. 

runs:
  using: composite
  steps:
    - id: resolve
      uses: taskworld/wr/resolve@main
      with:
        name: ${{ inputs.name }}
        backends: |
          rust
          python
          nodejs

    - id: vars
      run: |
        hasDockerfile=$(test -f "${{ steps.resolve.outputs.path }}/Dockerfile" && echo 'true' || echo 'false')
        echo "has-dockerfile=${hasDockerfile}" >> $GITHUB_OUTPUT

        hasPush=$(mise --cd "$(pwd)/${{ steps.resolve.outputs.path }}" config get tasks.push > /dev/null && echo 'true' || echo 'false')
        echo "has-push=${hasPush}" >> $GITHUB_OUTPUT

        runtimeVersion=$(${{ steps.resolve.outputs.eval }})
        echo "runtime-version=${runtimeVersion}" >> $GITHUB_OUTPUT
      shell: bash

    - uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ inputs.aws-access-key-id }}
        aws-secret-access-key: ${{ inputs.aws-secret-access-key }}
        aws-region: ap-southeast-1

    - id: login-ecr
      uses: aws-actions/amazon-ecr-login@v2

    - uses: docker/setup-buildx-action@v3

    - name: use generic dockerfile
      if: ${{ steps.vars.outputs.has-dockerfile == 'false' }}
      run: cp ./.github/Dockerfile "${{ steps.resolve.outputs.path }}"
      shell: bash

    - if: ${{ steps.vars.outputs.has-push == 'true' }}
      working-directory: ${{ steps.resolve.outputs.path }}
      run: mise run push
      env:
        DRY_RUN: ${{ inputs.push == 'true' && 0 || 1 }}
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
      shell: bash

    - uses: docker/build-push-action@v6
      if: ${{ steps.vars.outputs.has-push != 'true' }}
      with:
        context: ${{ steps.resolve.outputs.path }}
        platforms: |
            linux/amd64
            linux/arm64
        build-args: |
            NODE_VERSION=${{ steps.vars.outputs.runtime-version }}
        tags: |
            ${{ format('{0}/{1}:v{2}', steps.login-ecr.outputs.registry, inputs.name, steps.resolve.outputs.version) }}
        push: ${{ inputs.push == 'true' }}

Additional action for branching workflows We have two main types of workspaces in the monorepo: packages and services. To manage them efficiently, we created a dedicated GitHub Action that categorizes each workspace into separate arrays. This allowed us to split the CI logic into two distinct workflows: package.yml and service.yml.

Since workspace names are already resolved by wr, we just needed a consistent convention: packages are scoped (e.g., @taskworld/utils), while services are not. From there, a small bash script was enough to discriminate between the two types and route them accordingly. 

name: workspace splitter
description: switch/case splitting packages from services

inputs:
  workspaces:
    required: true
    description: a JSON array of workspaces with { name, version }

outputs:
  packages:
    description: an array of { name, version } of updated packages
    value: ${{ steps.split.outputs.packages }}

  packages-count:
    description: the number of packages found
    value: ${{ steps.split.outputs.packages-count }}

  services:
    description: an array of { name, version } of updated services
    value: ${{ steps.split.outputs.services }}

  services-count:
    description: the number of services found
    value: ${{ steps.split.outputs.services-count }}

runs:
  using: composite
  steps:
    - run: |
        json=${{ toJSON(inputs.workspaces) }}

        packages=$(echo "$json" | jq -c '[.[] | select(.name | startswith("@taskworld/"))]')
        packages_count=$(echo "$packages" | jq 'length')

        services=$(echo "$json" | jq -c '[.[] | select(.name | startswith("@taskworld/") | not)]')
        services_count=$(echo "$services" | jq 'length')

        echo "packages=${packages}" >> "$GITHUB_OUTPUT"
        echo "packages-count=${packages_count}" >> "$GITHUB_OUTPUT"
        echo "services=${services}" >> "$GITHUB_OUTPUT"
        echo "services-count=${services_count}" >> "$GITHUB_OUTPUT"

      id: split
      shell: bash

Leveraging composite workflows We designed the CI to avoid blockers too, so the two mainsworkflows (and also the test workflow) were designed to support manual trigger with workflow_dispatch. We did that by sticking to a common interface between workflow_call and workflow_dispatch and using the y-nk/wr/resolve action at almost every entry-point. 

name: PR checks

concurrency:
  group: pr-check-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  checks: write
  issues: write
  pull-requests: write

on:
  pull_request:
    types:
      - opened
      - reopened
      - ready_for_review
      - synchronize

jobs:
  resolve:
    if: github.event.pull_request.draft == false
    name: filter workspaces
    runs-on: ubuntu-latest

    outputs:
      workspaces: ${{ steps.diff.outputs.all }}
      workspaces-count: ${{ steps.diff.outputs.all-count }}

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: ./.github/actions/setup

      - id: diff
        uses: taskworld/wr/diff@main
        with:
          base: ${{ github.event.pull_request.base.sha }}
          head: ${{ github.event.pull_request.head.sha }}
          backends: |
            rust
            python
            nodejs

      - if: steps.diff.outputs.conflicting-count > 0
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: diff
          message: |
            ### Those workspaces are conflicting and must be rebased before continuing
            ${{ steps.diff.outputs.conflicting-md }}

      - if: steps.diff.outputs.conflicting-count > 0
        run: exit 1

      - if: steps.diff.outputs.bumped-count > 0 || steps.diff.outputs.modified-count > 0
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: diff
          message: |
            ### Scheduled for publication
            ${{ steps.diff.outputs.bumped-md || 'ø' }}

            ### Silent updates
            ${{ steps.diff.outputs.modified-md || 'ø' }}

  tests:
    if: needs.resolve.outputs.workspaces-count > 0
    name: lint & tests (${{ matrix.workspace.name }})
    needs: resolve

    strategy:
      fail-fast: ${{ !contains(github.event.pull_request.labels.*.name, 'no-fail-fast') }}
      matrix:
        workspace: ${{ fromJSON(needs.resolve.outputs.workspaces) }}

    secrets: inherit
    uses: ./.github/workflows/test.yml
    with:
      name: ${{ matrix.workspace.name }}
      path: ${{ matrix.workspace.path }}

and our test.yml : 

name: Test
run-name: test ${{ inputs.name }} workspace

on:
  workflow_dispatch:
    inputs:
      name:
        type: string
        required: true
        description: name of workspace to build

  workflow_call:
    inputs:
      name:
        type: string
        required: true
        description: name of workspace to build

      path:
        type: string
        required: false
        description: relative path of workspace to build (used for caching)

jobs:
  pr-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1

      # caching tests results to avoid re-running if already passed and unchanged
      - id: cache
        uses: actions/cache@v4
        with:
          path: .passed
          key: ${{ inputs.name }}-${{ hashFiles(inputs.path) }}${{ contains(github.event.pull_request.labels.*.name, 'no-cache') && format('-{0}', github.run_id) || '' }}

      - if: steps.cache.outputs.cache-hit != 'true'
        id: setup
        uses: ./.github/actions/setup
        with:
          workspace: ${{ inputs.name }}

      - if: steps.cache.outputs.cache-hit != 'true'
        working-directory: ${{ steps.setup.outputs.path }}
        run: mise run lint

      - if: steps.cache.outputs.cache-hit != 'true'
        working-directory: ${{ steps.setup.outputs.path }}
        run: mise run test

      - if: steps.cache.outputs.cache-hit != 'true'
        working-directory: ${{ steps.setup.outputs.path }}
        run: mise run build

      - if: steps.cache.outputs.cache-hit != 'true' && success()
        run: touch .passed

      - if: failure() && contains(github.event.pull_request.labels.*.name, 'enable-ssh-debugging')
        uses: mxschmitt/action-tmate@v3
        timeout-minutes: 5

Conclusion We could certainly have gone with Bazel or Please.build. That route would likely have been a stronger investment in personal skills — but it would also have introduced a steep learning curve. It would have meant teaching, supporting, and enforcing the use of a dedicated tool across the company, shifting engineering habits and, ultimately, slowing teams down.

Instead, we went with mise — a highly versatile tool that aligned better with our existing workflows. The only thing it lacked was workspace support, which they seem to avoid. Still, we were able to fill that gap with a simple but effective implementation through wr and i’m sure that, by leveraging again the [_] config node and some CI, we could manage to add it.

Today, what we’re still missing is proper support for workspace dependencies (e.g., building B before A and reusing assets), but it’s not a constraint we actually feel in practice. On the contrary, engineers are free to continue using the tool they’re used to (pnpm) across most of the monorepo, running pnpm build just like they always have.

And when they cd into another codebase, they can still rely on tools like rustup, uv, or ruff without caring for mise if they didn’t want to. This opens a broader range of support from the “native” communities if there’s ever a need rather than digging into a specific tool which has a smaller community.

If they do choose to install mise they gain an optional but powerful layer: automatic toolchain management and a consistent task runner. It’s their choice to move forward or stay comfy and performant in their habits—and that flexibility is arguably more valuable in the long run.

Read the full article here: https://medium.com/@julien.barbay/python-and-typescript-in-a-monorepo-c862a3bacddb

Retrieved from "https://johnwick.cc/index.php?title=Python_and_Typescript_in_a_monorepo&oldid=3151"