Running GitHub Self-Hosted Runners Reliably

Are you considering running GitHub self-hosted runners? We’ve been managing ours for quite some time. Here’s everything you need to know.
Get started with Namespace — it takes less than a minute.

GitHub Actions are a fantastic CI/CD solution loved by many developers. They are intuitive to use, and their flexibility covers many use cases.

At Namespace Labs, we are huge fans of GitHub Actions, and we’ve been managing self-hosted runners for us and our customers for quite some time now. We decided to share our experience and tips to make the self-hosted runners’ infrastructure more reliable.

Why even bother with Self-Hosted Runners?

Since you are here, you probably already know the pros and cons of self-hosting GitHub runners and are interested in running your own. If not, we’ve discussed self-hosted runners in more detail in a previous blog post; go and check it out!

All in all, self-hosted runners allow for finer OS, hardware, and environment customization. This high flexibility comes at a maintenance cost. You are responsible for managing and securing the compute infrastructure that powers your self-hosted runners. You need to upgrade the OS when required and install all packages and tools your workflow jobs use.

On the bright side, self-hosted runners are free for public and private repositories.

Running Persistent Self-Host Runners

Persistent runners are the opposite of ephemeral runners. The latter process one workflow job and then terminate themselves. Persistent runners still process one job at a time but remain active after processing the job finishes.

The most straightforward deployment for self-hosted runners is to run an always-on persistent runner in one of your machines.

Download the runner package

The runner package is released for architecture amd64 and arm64, and MacOS, Linux, and Windows OS. You can identify the correct package on the release page.

Download the package archive after selecting the desired runner OS, arch, and version variables. For example, with the below, I am downloading the runner image for MacOS Apple silicon.

$ mkdir actions-runner && cd actions-runner
$ RUNNER_OS="osx" RUNNER_ARCH="arm64" RUNNER_VERSION="2.307.1"
$ curl -fLo runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-${RUNNER_OS}-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz
$ tar xzf ./runner.tar.gz

Generate a registration token

The self-hosted runner needs a token to authenticate itself to GitHub. You can generate a token to register the runner to a specific repository or organization. We will register our runner to a single repository with the related registration token API. But you can follow the same steps using the organization token API.

Note the following steps require the GitHub gh CLI.

Install it now

$ OWNER="gmichelo" REPOSITORY="commerce"
$ gh api \
  --method POST \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  "token": "AAAXXX",
  "expires_at": "2023-08-03T15:55:21.860+02:00"

Configure and start the runner

You are ready to configure and start the runner. The config.sh script has many options to configure the runner environment. Most importantly, you want to set the runners labels, which are the only properties GitHub uses to schedule workflow jobs onto runners. When scheduling a job, GitHub selects a runner that matches all workflow labels. See official GitHub docs for more details.

In the following example, I've decided to label my runner my-permanent-runner.

$ ./config.sh --url https://github.com/${OWNER}/${REPOSITORY} \
	--token AAAXXX \
	--name my-permanent-runner \
	--labels my-permanent-runner \
# Authentication
 Connected to GitHub
# Runner Registration
 Runner successfully added
 Runner connection is good
# Runner settings
 Settings Saved.

Finally, run it!

$ ./run.sh
 Connected to GitHub
Current runner version: '2.307.1'
2023-08-03 13:05:34Z: Listening for Jobs

Confirm runner is registered on GitHub

The runner is active and waiting for jobs from GitHub. You can confirm that it was successfully registered to GitHub from the UI or API.

$ gh api \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  "total_count": 1,
  "runners": [
      "id": 22,
      "name": "my-permanent-runner",
      "os": "macOS",
      "status": "online",
      "busy": false,
      "labels": [
          "id": 1,
          "name": "self-hosted",
          "type": "read-only"
          "id": 2,
          "name": "macOS",
          "type": "read-only"
          "id": 3,
          "name": "ARM64",
          "type": "read-only"
          "id": 4,
          "name": "my-permanent-runner",
          "type": "custom"

The API shows that the runner status is online and that it is not processing any job (i.e. busy equals false).

Use the self-hosted runner in a workflow

Modify your workflow file to match the runs-on field to the runner’s label. GitHub will schedule jobs to the runner having every label you specify in that field.

name: Build with self-hosted runner
- runs-on: ubuntu-latest
+ runs-on: my-permanent-runner

Once the job is submitted, GitHub will schedule it on our self-hosted runner. When our runner receives the job, it prints the job information to standard output.

 Connected to GitHub
Current runner version: '2.307.1'
2023-08-03 13:05:34Z: Listening for Jobs
2023-08-03 13:11:21Z: Running job: Build with self-hosted runner
2023-08-03 13:13:27Z: Job Build with self-hosted runner completed with result: Succeeded

And the runner status transitions to active.

active runner

So, we made it. We've just run GitHub Actions on our self-hosted runner! Registering runners by hand is a good approach for initial experimentation and testing the environment configuration. But it won't scale. Manually managing many runners will soon become cumbersome, and starting a single runner application per machine leads to poor resource utilization.

The recommended deployment uses auto-scaling, ephemeral runners, and scale-to-0.

Auto-scaling Runners with Kubernetes

If you use Kubernetes, then GitHub got you covered. They developed a Kubernetes operator to dynamically scale up and down runners based on the number of queued jobs.

Note the following steps require kubectl, helm 3, and, most importantly, a Kubernetes cluster.

Installing ARC

$ NAMESPACE="arc-runners"
$ helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \

You can confirm ARC is installed by checking CRDs and pods.

$ kubectl get crd
autoscalinglisteners.actions.github.com    2023-08-03T15:10:05Z
autoscalingrunnersets.actions.github.com   2023-08-03T15:10:05Z
ephemeralrunners.actions.github.com        2023-08-03T15:10:05Z
ephemeralrunnersets.actions.github.com     2023-08-03T15:10:05Z
$ kubectl get pods -n arc-runners
NAME                                                   READY   STATUS    RESTARTS   AGE
arc-gha-runner-scale-set-controller-755f574df6-z6c48   1/1     Running   0          118s

Generate a Personal Access Token

ARC authenticates against GitHub using personal access tokens. If you want to create self-hosted runners for specific repositories, you need to create a PAT with repo scope. If you want to register organization runners instead, you need a PAT with admin:org scope.

Follow the official GitHub docs to create a PAT. Remember to save the token value somewhere safe and then come back here.

Create a Runner Scale Set

Now, we create a AutoscalingRunnerSets resource with the name arc-runner-set. This name will also be the runner’s label in your workflow file’s runs-on field.

Set the personal access token you created at the step before in variable GITHUB_PAT.

$ INSTALLATION_NAME="arc-runner-set"
$ GITHUB_CONFIG_URL="https://github.com/${OWNER}/${REPOSITORY}"
$ helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \

With helm list, we can confirm that the CRD resource is deployed with Helm.

$ helm list -A
NAME          	NAMESPACE  	REVISION	UPDATED                              	STATUS  	CHART                                	APP VERSION
arc           	arc-runners	1       	2023-08-03 17:10:06.376484 +0200 CEST	deployed	gha-runner-scale-set-controller-0.4.0	0.4.0
arc-runner-set	arc-runners	1       	2023-08-03 17:26:06.035833 +0200 CEST	deployed	gha-runner-scale-set-0.4.0           	0.4.0

Use the ARC runners in a workflow

As before, we need to change the workflow file and set the INSTALLATION_NAME value we used above (e.g. arc-runner-set).

name: Build with ARC runner
- runs-on: ubuntu-latest
+ runs-on: arc-runner-set

ARC can create ephemeral runners on the fly, as new job requests come from GitHub. In my example, I started three jobs, and ARC auto-scaled three pods to serve them in parallel. You can check the pods with kubectl.

$ kubectl get pods -n arc-runners
NAME                                                   READY   STATUS    RESTARTS   AGE
arc-gha-runner-scale-set-controller-755f574df6-z6c48   1/1     Running   0          26m
arc-runner-set-754b578d-listener                       1/1     Running   0          10m
arc-runner-set-sn5hq-runner-bxzh4                      1/1     Running   0          5s
arc-runner-set-sn5hq-runner-j75br                      1/1     Running   0          5s
arc-runner-set-sn5hq-runner-nhwfw                      1/1     Running   0          5s

Once the jobs finish, the ephemeral runners terminate, and the pods are deleted. This automatic release of unused resources is a significant improvement from before when we had a single always-on runner application installed in our machine, capable of serving only one job at a time.

Namespace Runners

Does the above look like a lot of work? Yes, it is a lot of work. We went through similar steps to deploy our self-hosted runners, and it took a lot of engineering to get it right.

In an upcoming blog post, we will share the design of our self-hosted runner offering and how we made it possible to scale to thousands of ephemeral runner VMs in a matter of seconds. We'll discuss the challenges of managing ephemeral capacity and designing a robust scheduler resilient to outages, and how to react when GitHub misbehaves.

Are you looking for the advantages of self-hosted runners but want to avoid handling the associated complexity (managing a Kubernetes cluster and writing your own GitHub runners scheduler)? Then Namespace Runners are the perfect fit.

After three simple installation steps, you can access powerful AMD64 or ARM64 runners with up to 32 CPU and 64 GB of RAM. Our runners have access to an integrated environment with cache volumes, optimized builder machines for faster Docker builds, a private Docker registry for your team, and preview environments to run QA tests.

Get started now.

Share this story onTwitteror join the team on Discord