Bazel Remote Execution

How it works

A Bazel Remote Execution cluster has three components:

• Scheduler: accepts Bazel Remote Execution API requests and assigns actions to workers.
• Storage: stores content-addressable storage (CAS) entries durably across the cluster.
• Workers: execute Bazel actions.

Each worker advertises a fixed number of execution slots. A slot represents one unit of concurrent work, so a worker with four slots can run up to four actions at the same time. Namespace starts workers on demand as the build requires more capacity.

Bazel's remote execution protocol supports multi-platform builds. For example, you can run Bazel from a Linux host while selected actions execute on macOS workers. This is useful for repositories that need to produce or test artifacts on multiple operating systems.

Getting started

First, make sure Remote Execution is enabled for your workspace. Then use the CLI to create a Bazel configuration file for the execution cluster:

$
nsc bazel setup --bazelrc=~/.namespace.bazelrc

$
bazel --bazelrc=~/.namespace.bazelrc build //...

Example: Linux host to Linux workers

This example runs Bazel from a Linux host and executes actions on Linux workers. It uses a public Bazel repository so you can try the flow without changing your own project first.

$ git clone https://github.com/bazelbuild/examples.git
$ cd examples/cpp-tutorial/stage3
 
$ nsc bazel setup --bazelrc=~/.namespace.bazelrc --key=examples-linux
$ bazel --bazelrc=~/.namespace.bazelrc build //...

Actions without explicit platform properties are routed to Linux workers.

Example: Linux host to macOS workers

Bazel can also route selected actions to macOS workers from a Linux client. The linux-to-mac-rbe-demo repository demonstrates that setup.

This example requires macos/arm64 workers to be enabled for your workspace. If you see an error such as no on-demand worker capacity configured for platform "macos/arm64", reach out to the Namespace team to enable macOS worker capacity.

$ git clone https://github.com/namespacelabs/linux-to-mac-rbe-demo.git
$ cd linux-to-mac-rbe-demo
 
$ nsc bazel setup --bazelrc=~/.namespace.bazelrc --key=linux-to-mac-demo
$ bazelisk --bazelrc=~/.namespace.bazelrc build //DemoApp:DemoApp

For your own targets, route an action to macOS by setting Bazel execution properties on that target or platform, for example:

exec_properties = {
    "OSFamily": "macos",
    "Arch": "arm64",
}

Actions without macOS execution properties continue to run on the default Linux workers, allowing one Bazel invocation to use both platforms.

Additional configuration options

Bazel execution properties can reserve worker CPU capacity and select the isolation mode on Linux workers:

exec_properties = {
    "cpu": "2",
    "namespace_action_isolation": "sandboxed",
    "namespace_requires_network": "true",
}

Use --remote_default_exec_properties to apply a property to actions that do not override it:

$ bazel --bazelrc=~/.namespace.bazelrc build //... \
  --remote_default_exec_properties=cpu=2 \
  --remote_default_exec_properties=namespace_action_isolation=sandboxed \

Reserve CPU capacity

The lowercase cpu property tells the scheduler how much CPU capacity to reserve while an action runs. Values can be whole or fractional numbers, such as "2" or "0.5". The scheduler considers both execution slots and CPU reservations when assigning concurrent actions to a worker.

The reservation is a scheduling input, not a CPU limit applied to the action. An omitted or zero value does not reserve CPU capacity beyond the action's execution slot.

resources:cpu is accepted as an alternative property name.

Select action isolation

The namespace_action_isolation property controls how a Linux worker runs an action:

The workspace default is none unless configured otherwise. Sandboxed isolation is supported only on Linux workers at the moment. Contact support@namespace.so to change the setting.

Sandboxed actions receive an isolated, loopback-only network namespace by default. Set namespace_requires_network to true on an action that needs network access. The action keeps its filesystem, process, mount, UTS, and IPC isolation, but shares the worker's network namespace. Omit the property for actions that do not need network access.

Custom worker pools

Named worker pools let actions select a worker machine type, execution slot count, and custom base image. Namespace creates a pool when its first action is queued, starts workers for it on demand, and routes only actions that name that pool to those workers. Actions without a pool use the workspace's default workers.

To use custom worker pools, define each pool as a Bazel execution platform. The platform's exec_properties describe the properties of the pool. A constraint value then lets targets select that execution platform with exec_compatible_with:

constraint_setting(name = "worker_pool")
 
constraint_value(
    name = "compile_pool_constraint",
    constraint_setting = ":worker_pool",
)
 
constraint_value(
    name = "high_memory_pool_constraint",
    constraint_setting = ":worker_pool",
)
 
platform(
    name = "compile_pool",
    constraint_values = [":compile_pool_constraint"],
    exec_properties = {
        "namespace_pool": "compile",
        "namespace_pool_machine_type": "linux/amd64:16x32",
        "namespace_pool_slots": "4",
    },
)
 
platform(
    name = "high_memory_pool",
    constraint_values = [":high_memory_pool_constraint"],
    exec_properties = {
        "namespace_pool": "high-memory",
        "namespace_pool_machine_type": "linux/amd64:16x64",
        "namespace_pool_slots": "2",
    },
)
 
genrule(
    name = "compile_probe",
    outs = ["compile_probe.txt"],
    cmd = "printf compile > $@",
 
    exec_compatible_with = [":compile_pool_constraint"],
)
 
genrule(
    name = "high_memory_probe",
    outs = ["high_memory_probe.txt"],
    cmd = "printf high-memory > $@",
 
    exec_compatible_with = [":high_memory_pool_constraint"],
)

Register the pool platforms when invoking Bazel:

$ bazel --bazelrc=~/.namespace.bazelrc build //... \
  --extra_execution_platforms=//:compile_pool,//:high_memory_pool

You can add --extra_execution_platforms to your project bazelrc instead of passing it on every invocation.

The machine type uses the format <os>/<arch>:<cpu>x<memory>. For example, linux/amd64:16x64 selects a Linux AMD64 worker with 16 vCPUs and 64 GB of memory. See Machine Shapes for supported CPU and memory configurations.

The following execution properties define a pool:

Every execution platform that uses the same pool name must provide the same pool definition. The scheduler rejects an action if its machine type, slot count, or worker image conflicts with the pool's existing definition.

Custom worker base images

By default, actions run using one of Namespace's managed worker images. If your actions need extra tools, libraries, or a specific base environment, you can run them inside your own container image instead.

Custom images are selected per action through the standard Bazel container-image execution property.

A few requirements apply:

• The image reference must be pinned to an immutable digest (repo@sha256:...). Tags are rejected.
• The image must be optimized before it can be used. Namespace converts the image into a fast-booting disk variant.
• The worker image must be hosted in the Namespace container registry nscr.io.
• Custom images are supported on Linux workers only.

Build, push, and optimize a custom image

$ nsc docker login
$ nsc build . -t nscr.io/<tenant>/my-worker:latest --platform linux/amd64 --push

$ nsc base-image upload <source-image> my-worker:latest

$ docker inspect --format '{{index .RepoDigests 0}}' nscr.io/<tenant>/my-worker:latest
nscr.io/<tenant>/my-worker@sha256:<digest>

$
nsc base-image optimize --image_ref nscr.io/<tenant>/my-worker@sha256:<digest>

jobs:
  rbe:
    runs-on:
      - namespace-profile-foobar;permissions.additional_grant=baseimage:*:*

jobs:
  rbe:
    runs-on:
      - nscloud-ubuntu-22.04-amd64-4x8-with-features
      - namespace-features:permissions.additional_grant=baseimage:*:*

Use the custom image

Set the container-image execution property on a specific target or platform so only those actions run with the custom worker image:

exec_properties = {
    "container-image": "docker://<repo>@sha256:<digest>",
}

The docker:// scheme prefix is optional. To apply an image through a custom worker pool instead, set namespace_pool_worker_image as described in Custom worker pools.

Run preparation scripts at worker startup

A custom worker image can ship preparation scripts that run once before the worker starts handling any actions. These hooks handle setup that must run on every worker boot, such as mounting an external /nix store or seeding caches.

Place executable files in /etc/namespace/prepare.d inside your image. On startup, the worker runs every executable file in that directory in lexical order (e.g. 10-mount.sh before 20-warmup.sh). Non-executable files and subdirectories are skipped.

Observability

To list recent invocations and then generate a report for an invocation, use:

$ nsc bazel invocation list
$ nsc bazel invocation report <invocation-id>

The report is JSON intended for machine consumption, including use by agents. It includes most events observed during an invocation, such as action execution, cache state, worker assignment and timing, queue times, and hydration times.

Future reports will also include operation joins, which indicate whether an invocation joins action invocations from other invocations, and hot or warm cache information.

Using revocable tokens

By default, nsc bazel setup uses your interactive login and writes a bazelrc that authenticates to the cluster with a short-lived mTLS client certificate. For CI/CD pipelines, automation, or any environment without an interactive login, you can instead authenticate with a revocable token.

Revocable tokens are long-lived but can be revoked at any time from cloud.namespace.so/user/sessions or with nsc token revoke.

$ nsc bazel create-token \
  --token rbe-token.json \
  --expires_in 10d

$ nsc bazel setup --token rbe-token.json --bazelrc=namespace.bazelrc

$ bazel --bazelrc=namespace.bazelrc build //...

Current caveats

Remote Execution is in early access, with the following current limitations:

• Action isolation within macOS workers is not available yet.
• An observability web UI is currently missing.

If any of these caveats block your use case, contact support@namespace.so so we can help find an appropriate solution.