Run Buildkite Jobs on Namespace

Namespace provides ephemeral, self-hosted Buildkite agents without requiring you to operate agent infrastructure.

How it works

Connecting a Buildkite organization creates a Namespace-managed queue and an agent token in either a new Namespace Cluster or an existing Buildkite cluster. A Buildkite webhook notifies Namespace when a job is scheduled on that cluster. Namespace then provisions an isolated instance, starts an agent for the queue, runs the job, and tears the instance down when the job completes.

You can select the operating system, architecture, instance shape, base image, and persistent cache for each step using agent tags.

Getting Started

Create a Buildkite API access token

Create an API access token for the Buildkite organization you want to connect. Namespace uses this token for initial setup, ongoing build reconciliation, and build annotations, so its expiry must be set to Never. The token can be revoked at any time through the Buildkite UI. Enable these REST API scopes:

• read_pipelines
• read_builds and write_builds
• read_clusters and write_clusters
• read_organizations

Connect your Buildkite organization

Open Buildkite → Get started in the Namespace dashboard. If another organization is already connected, select Connect a new organization from the Organizations page instead.

Enter the API token and click Next. Namespace uses the token to identify your Buildkite organization and list its clusters.

Select a cluster and queue

Choose where Namespace should create its managed queue:

• Select Create a new cluster to create the Namespace Cluster with a default queue.
• Select an existing cluster to add a queue to it. Enter a queue name, or keep the default name, namespace.

Configure any optional connection settings, then click Associate. Namespace creates the cluster if needed, creates the queue and its agent token, and displays the webhook setup instructions. Copy the generated webhook URL for the next step.

Configure the Buildkite webhook

On the next page, click on Create new Webhook and copy and paste the provided webhook URL and token.

Select the following events in the Buildkite webhook interface:

• ping
• job.scheduled
• job.started
• job.finished

Make sure the webhook is enabled for All Pipelines.

Click Add Webhook Notification, then return to Namespace. The connection becomes active after Namespace receives the webhook ping.

Add a pipeline to the selected cluster

In your Buildkite pipeline's settings, select the cluster you chose during setup under General > Cluster. If you added a queue to an existing cluster, set the pipeline or step's queue agent tag to the queue name you chose:

agents:
  queue: "namespace"

Manage queues

When you connect a Buildkite organization, Namespace creates and registers the first managed queue during setup. A new Namespace Cluster uses the default queue; an existing cluster uses the queue name selected during setup.

Register another queue

1. In Buildkite, navigate to Agents and select the cluster where you want to run Namespace agents.
2. Create a new self-hosted queue.

3. Open Buildkite → Overview in the Namespace dashboard. The overview lists each connected organization's registered and unregistered queues, grouped by Buildkite cluster.
4. Find the queue and click Register.
5. Confirm the registration. Namespace creates an agent token and begins acquiring jobs from the queue.

Select a queue for a job

Set the queue agent tag on a pipeline or step to send jobs to the registered queue:

agents:
  queue: "my-queue"

Unregister a queue

In the Namespace Buildkite overview, open the menu next to a registered queue, select Unregister, and confirm. A queue cannot be unregistered while it has scheduled or running jobs in Namespace.

Unregistering a queue stops Namespace from provisioning agents for it and revokes the agent token created during registration. It does not delete the queue or its cluster from Buildkite.

Checking out code

Git snapshots

Git snapshots speed up checkouts by replacing Buildkite's default git clone with a pre-fetched snapshot of the repository. Git snapshots are currently supported on Linux only.

To set up git snapshots for your Buildkite pipelines, you must first enable snapshots for your Buildkite installation:

1. Connect your GitHub organization to Namespace.
2. Open Buildkite → Organizations in the Namespace dashboard.
3. Open the menu for your Buildkite organization and select Edit.
4. In Edit Connection Settings, select Enable Git snapshots, then save the settings.

You can also select Enable Git snapshots while connecting a new Buildkite organization.

Enabling this setting grants these workload permissions to Namespace instances run for jobs with git snapshots enabled:

• ensure on code/git/snapshots/*, which allows the instance to start or reuse the workspace's git snapshot service.
• list and issue_token on github-association/*, which allow the snapshot service to find the associated GitHub organization and request a short-lived repository access token.

Use a snapshot in a job

After enabling snapshots for the installation, set the nsc-git-checkout agent tag to snapshot on a pipeline or individual step:

steps:
  - label: "Build"
    command: "make build"
    agents:
      queue: "default"
      nsc-git-checkout: "snapshot"

Only jobs with this tag use snapshots. Other jobs continue to use Buildkite's default checkout behavior. To explicitly select the default Git clone, set nsc-git-checkout to clone or default.

Cloning with SSH

Cloning repositories over HTTPS works by default and requires no additional configuration.

You can configure cloning over SSH by uploading your private key either to Namespace Vault or as a secret in Buildkite. Currently, signed pipelines with SSH checkout are supported only with secrets stored in Namespace.

To configure cloning over SSH in Namespace:

1. Upload your private key as a secret, using either the Vault dashboard or CLI command

nsc vault add --from_file /path/to/private/key --description "SSH private key"

2. In the Namespace dashboard, click on Edit Secrets and then Add another secret.
3. Select SSH Key from the dropdown, select your key, and then click Save Variables.

If you prefer to store your SSH key in Buildkite, follow Buildkite's code access setup for self-hosted agents. Store your private key as a Buildkite secret, then set checkout.ssh_secret on each of your pipeline's steps that require SSH access:

steps:
  - label: "Build"
    command: "make build"
    checkout:
      ssh_secret: "MY_SSH_KEY"

Next Steps

Configure Buildkite Agents. Choose compute resources, base images, and persistent caches with agent tags.