Run Buildkite Jobs on Namespace

Namespace provides ephemeral, self-hosted Buildkite agents without requiring you to operate agent infrastructure. Prefer to manage the integration through Buildkite? Buildkite Hosted Agents use Namespace infrastructure and are configured directly in Buildkite. Learn about Buildkite Hosted Agents →

How it works

Connecting a Buildkite organization creates a Namespace-managed queue and an agent token in either a new Namespace Cluster or an existing Buildkite cluster. A Buildkite webhook notifies Namespace when a job is scheduled on that cluster. Namespace then provisions an isolated instance, starts an agent for the queue, runs the job, and tears the instance down when the job completes. You can select the operating system, architecture, instance shape, base image, and persistent cache for each step using agent tags.

Reach out to get enrolled into Namespace-managed agents.Contact support →

Getting Started

Create a Buildkite API access token

Create an API access token for the Buildkite organization you want to connect. Namespace uses this token for initial setup, ongoing build reconciliation, and build annotations, so its expiry must be set to Never. The token can be revoked at any time through the Buildkite UI. Enable these REST API scopes:

read_pipelines
read_builds and write_builds
read_clusters and write_clusters
read_organizations

Connect your Buildkite organization

Open Buildkite → Get started in the Namespace dashboard. If another organization is already connected, select Connect organization from the Organizations page instead.Enter the API token and click Next. Namespace uses the token to identify your Buildkite organization and list its clusters.

Select a cluster and queue

Choose where Namespace should create its managed queue:

Select Create a new cluster to create the Namespace Cluster with a default queue.
Select an existing cluster to add a queue to it. Enter a queue name, or keep the default name, namespace.

Configure any optional connection settings, then click Associate. Namespace creates the cluster if needed, creates the queue and its agent token, and displays the webhook setup instructions. Copy the generated webhook URL for the next step.

Configure the Buildkite webhook

On the next page, click on Create new Webhook and copy and paste the provided webhook URL and token.Select the following events in the Buildkite webhook interface:

ping
job.scheduled
job.started
job.finished

Make sure the webhook is enabled for All Pipelines.Click Add Webhook Notification, then return to Namespace. The connection becomes active after Namespace receives the webhook ping.

Add a pipeline to the selected cluster

In your Buildkite pipeline’s settings, select the cluster you chose during setup under General > Cluster. If you added a queue to an existing cluster, set the pipeline or step’s queue agent tag to the queue name you chose:

agents:
  queue: "namespace"

Manage queues

When you connect a Buildkite organization, Namespace creates and registers the first managed queue during setup. A new Namespace Cluster uses the default queue; an existing cluster uses the queue name selected during setup.

Register another queue

In Buildkite, navigate to Agents and select the cluster where you want to run Namespace agents.
Create a new self-hosted queue.

Open Buildkite → Organizations in the Namespace dashboard. Click on the organization you want to manage, and then click Queues. This page lists all registered and unregistered queues, grouped by Buildkite cluster.
Find the queue and click Register.
Confirm the registration. Namespace creates an agent token and begins acquiring jobs from the queue.

Make sure that your webhook is configured to send events from the pipelines using this queue.Webhook Settings →

Select a queue for a job

Set the queue agent tag on a pipeline or step to send jobs to the registered queue:

agents:
  queue: "my-queue"

Unregister a queue

In the Namespace Queues page, open the menu next to a registered queue, select Unregister, and confirm. A queue cannot be unregistered while it has scheduled or running jobs in Namespace. Unregistering a queue stops Namespace from provisioning agents for it and revokes the agent token created during registration. It does not delete the queue or its cluster from Buildkite.

Checking out code

Cloning with SSH

Cloning repositories over HTTPS works by default and requires no additional configuration. You can configure cloning over SSH by uploading your private key either to Namespace Vault or as a secret in Buildkite. Currently, signed pipelines with SSH checkout are supported only with secrets stored in Namespace. To configure cloning over SSH in Namespace:

Upload your private key as a secret, using either the Vault dashboard or CLI command

nsc vault add --from_file /path/to/private/key --description "SSH private key"

In the Namespace dashboard, click on Edit Secrets and then Add another secret.
Select SSH Key from the dropdown, select your key, and then click Save Variables.

If you prefer to store your SSH key in Buildkite, follow Buildkite’s code access setup for self-hosted agents. Store your private key as a Buildkite secret, then set checkout.ssh_secret on each of your pipeline’s steps that require SSH access:

steps:
  - label: "Build"
    command: "make build"
    checkout:
      ssh_secret: "MY_SSH_KEY"

Git snapshots

Git snapshots are in early access. Reach out to enable them for your workspace.Contact support →

Git snapshots speed up checkouts by replacing Buildkite’s default git clone with a pre-fetched snapshot of the repository. To set up git snapshots for your Buildkite pipelines, you must first enable snapshots for your Buildkite installation:

Navigate to the Vault dashboard.
Scroll down to GitHub checkout associations and click Connect Organization to connect your GitHub organization to Namespace.
Open Buildkite → Organizations in the Namespace dashboard and select your Buildkite organization.
Click the toggle to enable git snapshots.

You can also select Enable Git snapshots when connecting a new Buildkite organization. Enabling this setting grants these workload permissions to Namespace instances run for jobs with git snapshots enabled:

ensure on code/git/snapshots/*, which allows the instance to start or reuse the workspace’s git snapshot service.
list and issue_token on github-association/*, which allow the snapshot service to find the associated GitHub organization and request a short-lived repository access token.

Use a snapshot in a job

After enabling snapshots for the installation, set the nsc-git-checkout agent tag to snapshot on a pipeline or individual step:

steps:
  - label: "Build"
    command: "make build"
    agents:
      queue: "default"
      nsc-git-checkout: "snapshot"

Only jobs with this tag use snapshots. Other jobs continue to use Buildkite’s default checkout behavior. To explicitly select the default Git clone, set nsc-git-checkout to clone or default.

Next Steps

Configure Buildkite Agents. Choose compute resources, base images, and persistent caches with agent tags.

Products

Platform

Administration

How it works

Getting Started

Create a Buildkite API access token

Connect your Buildkite organization

Select a cluster and queue

Configure the Buildkite webhook

Add a pipeline to the selected cluster

Manage queues

Register another queue

Select a queue for a job

Unregister a queue

Checking out code

Cloning with SSH

Git snapshots

Use a snapshot in a job

Next Steps

​How it works

​Getting Started

Create a Buildkite API access token

Connect your Buildkite organization

Select a cluster and queue

Configure the Buildkite webhook

Add a pipeline to the selected cluster

​Manage queues

​Register another queue

​Select a queue for a job

​Unregister a queue

​Checking out code

​Cloning with SSH

​Git snapshots

​Use a snapshot in a job

​Next Steps

How it works

Getting Started

Manage queues

Register another queue

Select a queue for a job

Unregister a queue

Checking out code

Cloning with SSH

Git snapshots

Use a snapshot in a job

Next Steps