Tests in CI

There are two lanes. The managed lane is the one to set up for a team: your repo keeps zero secrets, and results land everywhere your team looks. The manual lane is plain exit codes for any CI system.

#The managed lane: tests on every pull request

If your repo uses the SootSim GitHub integration for PR previews, tests are one input away. The workflow step that runs the preview also runs your Maestro flows:

- uses: sootbean/soot-preview-action@v1
  with:
    start: npm run dev
    port: 8081
    maestro: auto

maestro: auto discovers every flow in your repo’s .maestro/ directory. You can also pass explicit paths, comma-separated, and control which events run tests with maestro-on (default pull_request; add push:main style entries for branch builds):

    maestro: .maestro/login.yaml,.maestro/checkout.yaml
    maestro-on: pull_request,push:main

Each flow runs against the same app the preview booted, records itself, and uploads its own replay. On every PR you get:

• A comment with a Tests table: each flow, its result, and a replay link.
• A “Soot Tests” check on the commit whose conclusion is the real aggregate, so a failing flow blocks merges under branch protection.
• Dashboard rows: every flow indexed in your org’s Runs view with branch, commit, PR, and a per-step trace.

The repo adds no API keys and no secrets; the workflow authenticates with its own GitHub token. A failing flow marks the check red and the run failed without aborting the preview itself. This lane requires the org’s plan to include previews; see Plans.

Setup from zero: install the GitHub App and accept the bootstrap PR, per GitHub PR Previews.

#The manual lane: any CI, exit codes

Test runs are free and account-less: the runner exits nonzero when any flow or spec fails, so any CI gates on the exit code.

name: sootsim tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
      - run: bun install
      - run: npx playwright install chromium

      - run: bun dev &
      - run: |
          for i in $(seq 1 60); do
            curl -sf http://localhost:8081 > /dev/null && break
            sleep 2
          done

      - run: sootsim maestro test .maestro/

The same shape works on any CI system: Node, Playwright’s Chromium, your dev server, and the sootsim CLI are the whole dependency list.

To ALSO get hosted replays and dashboard rows from the manual lane, add a SOOTSIM_API_KEY secret (created in the SootSim account UI, on the account that should own the runs) and pass --preview:

      - run: sootsim maestro test .maestro/ --preview
        env:
          SOOTSIM_API_KEY: ${{ secrets.SOOTSIM_API_KEY }}

Runs registered from CI carry their git provenance automatically (branch, commit, PR number, workflow run) so the dashboard can group and link them.

#Next

• Read the results
• GitHub PR Previews setup