Introducing Flakiness.io

After 2.5 years of development, Flakiness.io — a test analytics service for GitHub — is generally available.

Why would you need it?

Consider WordPress/Gutenberg — a mature open source project that takes testing seriously. They run tens of thousands of tests across dozens of configurations on every commit and every pull request.

Here’s one of their draft pull requests:

“Some checks were not successful.” Does this pull request break something, or is it just some flaky tests misbehaving? Or maybe the main branch is broken?

The usual investigation steps consist of opening the failing workflow(s), scrolling terminal logs to find failing tests, and checking the most recent commits from the main branch to get a baseline.

None of this is needed if every test result is annotated with the commit and cross-referenced with the repository history.

We can then reliably detect new test failures and report them back to GitHub via a check:

And if you click that Flakiness.io check, you’ll get a detailed test report that shows the actual failures, along with debugging artifacts like screenshots, videos, and Playwright Traces:

Notice how the test is marked as “regression” since it has a perfect track record on the target branch.

Interested in the status of your release branch? Flakiness.io gives you a single place for all your tests: you can sort them by duration, duration trends, fail and flip rates, and so on:

When you click into a test’s history, you can explore its performance over months, days, and commits:

Just as GitHub stores your code, Flakiness.io stores your test results.

Why develop another service?

Because most services are designed to work with hundreds of tests, maybe a thousand, but fall apart if you give them 100,000+ test results.

Let’s take pricing, for example. Lots of test analytics services charge per uploaded test result. A typical price is around $40 per 10,000 uploaded test results. Given how similar this pricing is across different test analytics providers, we don’t think it’s unreasonable: holding and processing tons of testing data can get expensive.

However, back in 2023, the Microsoft Playwright team was generating about 280,000 test results for each commit (~4,000 tests across dozens of configurations). Test analytics would cost $1,120 for every single commit.

This is not only prohibitively expensive, but it also penalizes testing: adding one more testing environment to run tests might result in a sizable price bump.

Can we do better?

Turns out we can!

Real-world testing data has a nice set of properties that allow it to be packed and processed in bulk. We built a specialized test analytics engine that is inspired by interval unions and is happily chewing through millions and millions of test results on modest hardware.

Thanks to this engine, we have a dead-simple pricing model where our clients pay just for the data they upload. Playwright’s 280,000 test results (without artifacts) would take only about 7 MB of space. At that density, our free 1 GB plan could hold roughly 40 million test results, or about $160,000 worth of testing data at typical per-result pricing.

For a real-world example, check out Gutenberg’s usage page.

How does it work?

To use Flakiness.io, you’ll need to create a Flakiness.io project and link it to your GitHub repository. The project is where all your testing data will be stored and processed.

Next, you’ll need to install a test runner integration — we either support your test runner already, or you can use our JUnit XML converter. And that’s it: as your CI runs your tests, the integration will automatically upload the results to Flakiness.io.

What else can it do?

• First-class GitHub support. We fully respect your GitHub repository permissions: someone with access to your repository will automatically have access to your Flakiness.io dashboard. We also support GitHub OIDC for secretless report uploads.
• We ship Flakiness CLI with agent skills to help drive test investigations.
• We have a small but mighty Flakiness Query Language to help you (and your agents) filter tests. Check it out, it’s fun!

There’s more, but we’ll stop here to keep it short.

How is Flakiness.io funded?

Flakiness.io is a bootstrapped project with a small team. Our goal is to provide a well-maintained, useful service at a reasonable price.

And if you ever grow dissatisfied with Flakiness.io, you can always download all your data with the Flakiness CLI: the data is stored as Flakiness JSON reports — an open source MIT-licensed format that is very easy to work with.

Flakiness.io is generally available

We soft-launched in fall 2025 with a select group of customers. Now, having ironed out the early issues, we’re finally ready for the official introduction:

We thrive on feedback, and we’d love to know what you think! Here are a few ways to reach us:

• Email us at support@flakiness.io
• File issues at flakiness/flakiness-feedback
• Reach out directly on X at @flakinessio

Happy coding!

Flakiness.io Team