Managed GitHub runners are fast. Just how fast?

Last month we introduced managed GitHub runners. With a one-line change to the workflow definitions, your GitHub actions will run on Namespace infrastructure. And we will run them fast!

Surely I don’t need to sell you the benefits of fast CI. We all remember times we were impatiently watching the GitHub Actions page. Waiting to see if your pull request can finally be merged or if it needs one more cycle of fixing the tests and waiting for the CI again.

We noticed that our colleagues at Sentry feel the pain of slow CI too (great write-up, btw!). Our takeaway is that having access to parallelism in CI while keeping the costs manageable is the key. So we thought, can Namespace be of help?

Sentry Case Study

At Namespace we are fans of Sentry. We use it for both our backend and frontend. And there were countless times it saved us when an unforeseen code path misbehaved in production.

Sentry tests turned out to be a perfect way to put our managed runners to the test (pun intended). It’s fairly representative of a typical project with a Python backend and a Node.js frontend. Its CI runs 14k backend tests and starts six backends in the ephemeral test environment. The frontend test suite includes 5.5k browser-based tests that run in parallel making full use of the RAM available on the runner.

So we did a trial run of Sentry CI on Namespace managed runners. We are not affiliated with Sentry, so we did this on our own based on the public source code available on GitHub. We set up a shadow clone of the sentry repo and modified the workflow files in our fork.

Results

I will let the numbers speak for themselves:

Frontend: 5m 31s vs 11m 27s.

Backend: 15m 3s vs 21m 30s.

Hence, for the Frontend workflow our Runners are 55% faster, while for the Backend workflow they are 28% faster.

Still Affordable

How do we achieve this performance? It’s no magic. Our default runner has a 4x16 shape (4 CPU cores and 16G RAM) while GitHub default runners are only 2x7 (2 cores, 7G RAM).

Indeed you can get beefier runners from GitHub too. However, that’s when it gets expensive. Larger runners are not eligible for the free tier GitHub has for public repositories. So you get charged per minute.

Did you know that that 21.5 minute backend run will actually cost 165 runner minutes? What many don’t realize is that GitHub charges for each parallel job inside the workflow individually. That backend workflow consists of 19 jobs, each running for up to 21 minutes. They will add up for the billing.

Let’s do a back-of-the-envelope cost calculation using real-world data from the Sentry repo. For simplicity, let’s only consider CI runs on the master branch (additional runs on pull requests and other branches would scale proportionally).

Last week (i.e. April 30—May 6 2023) there were 574 runs of frontend and backend workflows, totaling 31k runner minutes. That would translate to a monthly cost of

Namespace pricing works differently: you pay a flat monthly price for concurrent compute capacity. There are no overruns. When you run out of concurrent capacity, CI jobs may be queued for some time but will make progress eventually.

For a fair comparison, let’s calculate Namespace pricing so that all jobs can run in parallel (thus actually achieving the fast execution times shown above). The workflows we are testing would use up 21 workers (4x16) at the same time. That translates to 10 Namespace Pro seats:

That’s a 79% saving for 28-55% speed up!

What’s next?

Setting up our Managed Runners takes 10 minutes. Just get started.