How DuckDB runs their CI/CD pipelines 10x faster with Namespace
“Even though Namespace saved us money, that wasn’t the point. What matters is that it’s reliable and fast. In the end, it’s more about the cost of time than the cost of money.”
Sander van VeenDeveloper Experience Engineer, DuckLabsDuckDB is an MIT-licensed open source analytical database that has been under development for eight years. It began as an in-process system you could embed directly in your Python, Node.js, or Rust process, aimed at data scientists who wanted something faster and more ergonomic than dataframe libraries but with a familiar SQL interface. Over time it grew into a full-fledged database. They are trusted by Nvidia, Fivetran, Hugging Face, Okta, and so many more.
DuckLabs is the company behind DuckDB. Its engineers are exceptional at building a database, but as Sander, who joined the team in February to focus on developer experience, puts it, being great at building a database does not automatically mean the development environment around it keeps pace. Sander had used Namespace at a previous company and brought that experience with him when DuckLabs’ CI pipeline started to slow the team down.
Problem: An open source project outgrowing GitHub
DuckLabs runs its CI on GitHub Actions. Because the project is open source, GitHub granted them generous compute, but it still was not enough. DuckDB is built for all major operating systems and architectures so it is easy to use by most users. DuckDB draws a large volume of external contributors on top of a growing team of employees, and that combined load, together with the need to build a core extension repository plus a community extension repository of 100+ extensions, regularly overwhelmed the system.
The result was long build queues and slow feedback. On a public repository, GitHub runners come with just four CPU cores and roughly 20 GB of disk, and moving to larger instances would have meant a different, much more expensive, plan. DuckDB has a fairly complicated build process, so jobs would sometimes simply run out of disk space, a failure mode that is not only painful to troubleshoot but also forces engineers to pull tricks to free up space mid-build. Making it worse, GitHub offers log output but not system resource metrics that show what a runner is actually doing, so diagnosing why a job ran out of space or slowed down often came down to guesswork. Major DuckDB releases were the worst case: cutting a new version of DuckDB meant rebuilding every extension for many platforms and architectures, which could take anywhere from several hours to half a day.
To cope, engineers learned to work around the problem. Many ran their pull requests in personal forks to reduce load on the main organization’s runners, and the team adopted informal policies about when it was acceptable to open a PR at all.
“During a big release we’d tell people internally: now is not the time to open fix-typo pull requests. Please just work on your own fork, and you’re welcome to open it tomorrow. It’s not the end of the world, but it does add friction to the development process.”
At its worst, getting full confirmation on a pull request took around six hours. That is a long time to wait just to learn whether a change is worth iterating on.
Solution: Selecting Namespace over GitHub and self-hosting on AWS
Sander already knew what great looked like. At a previous company he had used Namespace and been happy with both the reliability of the platform and the observability it gave him into CI. When CI became a bottleneck at DuckLabs, Namespace was the natural starting point.
Rather than commit blindly, Sander built a proposal for the team. He worked through the questions management would ask: how many concurrent runners the project would need, whether the platform supported their full matrix of Linux, macOS, Windows, and other architectures, how responsive support would be, and what the fallback plan was if the platform ever went down. He invited two engineers into Namespace’s Discord channel to see the support experience firsthand, and he weighed the alternatives, including a larger GitHub plan and self-hosted runners on EC2, an option he knew well from a previous employer.
“Self-hosted works, but it costs a lot to maintain and keep up. If something breaks, how do you even know it broke? There’s a lot of setup there that I simply don’t have time for.”
Because Namespace runners are a drop-in replacement for GitHub Actions, trying it out was as easy as pointing the workflows at Namespace with no rearchitecting required. And because those runners are built on the Namespace platform, which is optimized for performance and offers a range of machine shapes across Linux, macOS, and Windows, Sander got the full build matrix the project needed out of the box. For Sander, the bigger draw was the observability he knew he would finally get: instead of guessing at why a job ran out of space or slowed down the way he had to on GitHub, he could drill down into a job to see exactly how it was behaving and where its time, CPU, RAM, and other resources were going.
Results: 10x faster CI and observability GitHub couldn’t offer
Today, pull request validation that used to take about six hours completes in 15 to 30 minutes. Because Namespace instances are larger, individual jobs got dramatically faster on their own: a Linux debug build that took around 30 minutes on a fork runner now finishes in about 3 minutes, a 10x speedup. Where GitHub gave the project four CPU cores, Namespace runs these jobs on 16 cores with about 160 GB of disk. Since disk scales up proportionally with the larger instances, the out-of-space failures that used to plague complex builds are largely a thing of the past, even as the team runs more compute and more testing.
The speed compounds across the pipeline. Fast early jobs, like verifying that an engineer’s changes are formatted correctly, now surface a failure within the first 30 seconds, so contributors get an immediate signal about whether a PR is worth waiting on. Namespace’s cache volumes trim overhead like git checkouts to keep that feedback even tighter.
Just as important is what the team can now see. Namespace’s per-job observability lets Sander check whether each job is sized correctly for its hardware and right-size instances up or down to balance latency against cost, while the Insights view shows queue times and trends over time so the team can measure the impact of changes to the CI pipeline.
“If I made some changes and CI jobs got slower, being able to drill down and find that is really useful. On GitHub, that's very difficult to do.”
If you’re running an open source project with heavy contributor load and your CI can’t keep up, or you just want observability into where your pipeline is spending time, reach out to join our OSS sponsorship program.
