Which test frameworks are supported?

Anything that writes JUnit-style XML: Maven Surefire, Gradle, TestNG, pytest, Jest, Vitest, Playwright, Cypress, Mocha, PHPUnit, RSpec, gotestsum, k6, CTest and other xUnit emitters.

Your test failed. Did you break it, or is CI just noisy?

FlakeTriage reads your CI history and your diff, then tells you which failures to ignore and which one to fix — with evidence you can verify, not a guess from a language model.

Run it on your repo See how it decides

- uses: AleksandrJelohhin/flaketriage@v1

Free GitHub Action · JUnit XML from any framework · works offline with --no-llm

flaketriagebot commented 2 minutes ago

FlakeTriage — 3 failures, 1 needs you

🔴 Needs you — likely broken by this PR

checkout.spec.ts › applies discount code

Passed on a1b2c3d, fails here. Frame 2 points at src/pricing/discount.ts:88, which is inside a hunk you changed.

→ Start at src/pricing/discount.ts:88

⚪ Safe to ignore — 2 flakes

auth.spec.ts › refreshes token — passed on attempt 1, failed on attempt 2 of this same commit. Confirmed flake.

search.spec.ts › debounces input — flipped 7 of 30 recent runs, test file unchanged in all of them. Flip rate 23%.

2 verdicts from history · 1 from model · $0.04

Why FlakeTriage

Flaky test detection that shows its work.

Evidence, not opinion

“Passed on attempt 1, failed on attempt 2 of the same commit.” That’s proof, not a probability score. Every verdict cites history you can check yourself.

flaketriage explain checkout.spec.ts

Works without an API key

Most verdicts come from deterministic history analysis — no model, no network, no cost. The model handles only genuinely ambiguous cases, with your own key.

flaketriage run --no-llm

Your logs never leave your CI

History lives in SQLite next to your runner. No stack traces uploaded, no vendor to put through security review.

flaketriage run --print-payload

How it decides

The model is the last step, not the first.

The model is the last step, not the first. That’s why the verdicts hold up.

Verdicts — first match wins
Verdict	Meaning	What you do
`flake_confirmed`	Failed, then passed on retry of the same commit	Ignore
`infra_failure`	Matches a known infrastructure signature — DNS, connection refused, runner timeout	Ignore
`always_failing`	Already failing before this change	Look when you can
`flake_likely`	Flips pass/fail on unchanged code	Ignore
`real_regression`	Green on the parent commit and blamed on the diff	Needs you
`ambiguous`	Not enough signal — goes to the model, if enabled	Look when you can

Install

Add it after your test step. Keep your tests as they are.

GitHub Action

It doesn’t run your tests — it triages the JUnit reports they leave behind, keeps one sticky comment per PR, and only fails the build on real regressions.

# .github/workflows/flaketriage.yml
on: pull_request
permissions:
  contents: read
  pull-requests: write
  actions: write
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: npm test || true
      - uses: AleksandrJelohhin/flaketriage@v1
        with:
          reports: "**/junit*.xml"
          fail-on: regression

CLI, in any CI

Needs only JUnit XML and a git checkout. Jenkins, GitLab, Buildkite, your laptop.

git clone https://github.com/AleksandrJelohhin/flaketriage
cd flaketriage && npm install && npm run build

node dist/cli.js run --repo ../my-app \
  --reports 'build/test-results/**/*.xml'
node dist/cli.js explain 'applies discount code'
node dist/cli.js stats --min-runs 5
node dist/cli.js backfill --repo owner/name --days 90

Supported: anything that writes JUnit-style XML. The parser is tested against 303 real failing reports from 212 public repositories.

Maven Surefire
Gradle
TestNG
pytest
Jest
Vitest
Playwright
Cypress
Mocha
PHPUnit
RSpec
gotestsum
k6
CTest

FAQ

Questions engineers ask first.

How is this different from retrying flaky tests?

Retries hide flakes. FlakeTriage records them and tells flaky tests apart from real regressions, so a retry that turns green is evidence rather than silence.

Do I need an LLM or an API key?

No. History, fingerprinting, blame and infra signatures handle most failures on their own. The model only sees what those can’t decide — Claude, Gemini, OpenAI, or a local model, with your own key.

What leaves my CI?

With --no-llm, nothing. Otherwise only ambiguous failures, at most 15 per run, with truncated stacks and secrets redacted. --print-payload prints the exact request and exits without opening a socket.

Does it work outside GitHub Actions?

The CLI is CI-agnostic: it only needs JUnit XML and a git checkout, so it runs in any CI system. A native integration ships for GitHub Actions.

Where is the test history stored?

In a local SQLite file. In the GitHub Action it is persisted with actions/cache. backfill imports past GitHub Actions runs, so a first install already knows which tests are flaky.