Computer Controls

Use display controls to capture screenshots of a macOS Devbox and click anywhere on its screen. Together they let an agent see what an app shows and act on it, for example to check that a page renders or to click through a flow that has no command-line equivalent.

Display controls are available on macOS Devboxes only. On a Linux Devbox, they reject with DevboxDisplayUnavailableError. See Create a Devbox.

Take screenshots

Capture the full screen as a PNG. The screenshot is 2560 by 1600 pixels. A display call starts a stopped Devbox and keeps its connection open for later screenshots and clicks. See display.screenshot() for its options and return value.

TS SDK

import { writeFile } from "node:fs/promises";
import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Open a page, and wait for it to load
await devbox.exec(["open", "-a", "Safari", "https://namespace.so"]);
await new Promise((resolve) => setTimeout(resolve, 2_000));

// Capture the screen and save it as a PNG
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("screen.png", screenshot.png);

client.close();

The result holds the PNG bytes in png, the screen size in width and height, and the desktopName reported by the display server. Pass the PNG to your model as an image.

A screenshot of a macOS Devbox desktop with namespace.so open in Safari

Each screenshot takes about a second. open returns before an app has finished loading, so wait briefly, or take screenshots until the screen stops changing, before acting on what you see.

Click the mouse

Click at a point on the screen. Coordinates are in screenshot pixels, with (0, 0) at the top-left corner, so a point read from a screenshot can be passed as is. See display.click() for its options, including the mouse button.

TS SDK

import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Left click on the Contact Us button
await devbox.display.click(2200, 280);

// Wait for the page to load and capture a screenshot
await new Promise((resolve) => setTimeout(resolve, 2_000));
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("contact-us.png", screenshot.png);

client.close();

click() returns as soon as the click is sent, before the app reacts. Wait briefly, then take a new screenshot to confirm the result before choosing the next action.

The screenshot after the click, with the namespace.so contact page open in a new Safari tab

Agent loop

Screenshots and clicks combine into a loop that lets an agent work on an app until a task is done:

Start the app with devbox.exec(), for example open -a Safari https://namespace.so.
Take a screenshot to capture what the screen shows.
Send the screenshot and the task to a vision model, and ask it for the next action.
Run the action the model returns, such as a click, then wait for the app to react.
Repeat from step 2 until the model reports that the task is done.

The loop below follows that pattern. decideNextAction() stands for your model call, for example to the Claude or OpenAI computer use tools. It receives the screenshot and returns the next action: a click, an app or URL to open, or done.

TS SDK

import { createDevboxClient } from "@namespacelabs/sdk/devbox";

type Action =
  | { type: "click"; x: number; y: number; button?: "left" | "middle" | "right" }
  | { type: "open"; target: string }
  | { type: "done" };

// Your model call: sends the task and screenshot, returns the next action
declare function decideNextAction(task: string, screenshot: Uint8Array): Promise<Action>;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

const task = "List Devbox docs pages from the sidenav. Expand collapsed sections where needed.";

// 1. Start the app
await devbox.exec(["open", "-a", "Safari", "https://namespace.so/docs"]);
await sleep(2_000);

while (true) {
  // 2. Capture the current screen
  const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });

  // 3. Ask the model for the next action
  const action = await decideNextAction(task, screenshot.png);

  // 4. Stop the loop when done
  if (action.type === "done") break;

  // 5. Run the action, then give the app time to react
  switch (action.type) {
    case "click":
      await devbox.display.click(action.x, action.y, { button: action.button });
      break;
    case "open":
      await devbox.exec(["open", action.target]);
      break;
  }
  await sleep(2_000);
}

client.close();

Tell the model the screen size, 2560 by 1600 pixels, so that the coordinates it returns match the screenshot. The open action shows how to mix commands into the loop. For anything other than clicks, such as launching apps or opening URLs, run a command with devbox.exec(). Commands are usually faster and more reliable than working through the screen.

Products

Platform

Administration

Take screenshots

Click the mouse

Agent loop

​Take screenshots

​Click the mouse

​Agent loop

Take screenshots

Click the mouse

Agent loop