Computer Controls

Use display controls to capture screenshots of a Devbox, click anywhere on its screen, and type into the focused app. Together they let an agent see what an app shows and act on it, for example to check that a page renders or to click through a flow that has no command-line equivalent.

Display controls are available on macOS Devboxes only. See Create a Devbox.

Take screenshots

Capture the full screen as a PNG. The screenshot is 2560 by 1600 pixels.

CLI
TS SDK

# Open a page, and wait for it to load
devbox exec my-mac-devbox -- open -a Safari https://namespace.so
sleep 2

# Capture the screen and save it as a PNG
devbox desktop screenshot my-mac-devbox --file screen.png

The CLI prints the screenshot size, in the same pixels that devbox desktop click uses. Pass --file - to write the PNG to stdout instead.

import { writeFile } from "node:fs/promises";
import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Open a page, and wait for it to load
await devbox.exec(["open", "-a", "Safari", "https://namespace.so"]);
await new Promise((resolve) => setTimeout(resolve, 2_000));

// Capture the screen and save it as a PNG
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("screen.png", screenshot.png);

client.close();

A display call starts a stopped Devbox and keeps its connection open for later screenshots and clicks. See display.screenshot() for its options and return value.

The result is a full-screen capture, ready to pass to your model:

A screenshot of a macOS Devbox desktop with namespace.so open in Safari

Click the mouse

Click at a point on the screen. Coordinates are in screenshot pixels, with (0, 0) at the top-left corner, so a point read from a screenshot can be passed as is.

CLI
TS SDK

# Left click on the Contact Us button
devbox desktop click my-mac-devbox --x 2200 --y 280

# Wait for the page to load and capture a screenshot
sleep 2
devbox desktop screenshot my-mac-devbox --file contact-us.png

Pass --button right or --button middle to click another mouse button.

import { createDevboxClient } from "@namespacelabs/sdk/devbox";

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

// Left click on the Contact Us button
await devbox.display.click(2200, 280);

// Wait for the page to load and capture a screenshot
await new Promise((resolve) => setTimeout(resolve, 2_000));
const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });
await writeFile("contact-us.png", screenshot.png);

client.close();

See display.click() for its options, including the mouse button.

A click returns as soon as it is sent, before the app reacts. Wait briefly, then take a new screenshot to confirm the result before choosing the next action.

The screenshot after the click, with the namespace.so contact page open in a new Safari tab

Use the keyboard

Type text into the app that has focus, and press keys and shortcuts. Click a text field first, or use a shortcut, to choose where the text goes.

# Focus the Safari address bar
devbox desktop press my-mac-devbox --key cmd+l

# Type a URL and press Enter
devbox desktop type my-mac-devbox --text "namespace.so/docs"
devbox desktop press my-mac-devbox --key enter

# Press several keys in order
devbox desktop press my-devbox --key down --key down --key enter

devbox desktop type types text into the focused app.
devbox desktop press presses keys and shortcuts, such as enter or cmd+l.

Agent loop

Screenshots and clicks combine into a loop that lets an agent work on an app until a task is done:

Start the app with a command, for example open -a Safari https://namespace.so.
Take a screenshot to capture what the screen shows.
Send the screenshot and the task to a vision model, and ask it for the next action.
Run the action the model returns, such as a click, typing, or a key press, then wait for the app to react.
Repeat from step 2 until the model reports that the task is done.

The loops below follow that pattern. decide_next_action in the CLI script and decideNextAction() in the TS SDK stand for your model call, for example to the Claude or OpenAI computer use tools. It receives the screenshot and returns the next action: a click, text to type or a key to press (CLI only), a URL to open, or done with the task result.

CLI
TS SDK

#!/usr/bin/env bash
set -euo pipefail

devbox=my-mac-devbox
task="List Devbox docs pages from the sidenav. Expand collapsed sections where needed."

# Your model call: reads the task and a screenshot, prints the next action as JSON,
# such as {"type":"click","x":120,"y":340}, {"type":"type","text":"docs"},
# {"type":"press","key":"enter"}, or {"type":"done","result":"..."}
decide_next_action() {
  ./decide-next-action "$1" "$2"
}

# 1. Start the app
devbox exec "$devbox" -- open -a Safari https://namespace.so/docs
sleep 2

while true; do
  # 2. Capture the current screen
  devbox desktop screenshot "$devbox" --file screen.png

  # 3. Ask the model for the next action
  action=$(decide_next_action "$task" screen.png)
  type=$(jq -r .type <<<"$action")

  # 4. Print the result and stop the loop when done
  if [ "$type" = done ]; then
    jq -r .result <<<"$action"
    break
  fi

  # 5. Run the action, then give the app time to react
  case "$type" in
    click) devbox desktop click "$devbox" --x "$(jq -r .x <<<"$action")" --y "$(jq -r .y <<<"$action")" ;;
    type) devbox desktop type "$devbox" --text "$(jq -r .text <<<"$action")" ;;
    press) devbox desktop press "$devbox" --key "$(jq -r .key <<<"$action")" ;;
    open) devbox exec "$devbox" -- open "$(jq -r .target <<<"$action")" ;;
  esac
  sleep 2
done

import { createDevboxClient } from "@namespacelabs/sdk/devbox";

type Action =
  | { type: "click"; x: number; y: number; button?: "left" | "middle" | "right" }
  | { type: "open"; target: string }
  | { type: "done"; result: string };

// Your model call: sends the task and screenshot, returns the next action
declare function decideNextAction(task: string, screenshot: Uint8Array): Promise<Action>;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

const client = createDevboxClient();
const devbox = await client.devboxes.get("my-mac-devbox");

const task = "List Devbox docs pages from the sidenav. Expand collapsed sections where needed.";

// 1. Start the app
await devbox.exec(["open", "-a", "Safari", "https://namespace.so/docs"]);
await sleep(2_000);

while (true) {
  // 2. Capture the current screen
  const screenshot = await devbox.display.screenshot({ timeoutMs: 60_000 });

  // 3. Ask the model for the next action
  const action = await decideNextAction(task, screenshot.png);

  // 4. Print the result and stop the loop when done
  if (action.type === "done") {
    console.log(action.result);
    break;
  }

  // 5. Run the action, then give the app time to react
  switch (action.type) {
    case "click":
      await devbox.display.click(action.x, action.y, { button: action.button });
      break;
    case "open":
      await devbox.exec(["open", action.target]);
      break;
  }
  await sleep(2_000);
}

client.close();

Tell the model the screen size, 2560 by 1600 pixels, so that the coordinates it returns match the screenshot. The open action shows how to mix commands into the loop. For anything that does not need the screen, such as launching apps or opening URLs, run a command with devbox exec or devbox.exec(). Commands are usually faster and more reliable than working through the screen.

Watch the desktop

Open the desktop in a VNC client on your machine to watch an agent work, unblock it when it is stuck on a dialog, or take over with your own mouse and keyboard. See Remote Desktop.

devbox desktop connect my-mac-devbox

You and an agent share one pointer and keyboard, so input from both at the same time collides. Ask the agent to pause before you take over, and have it take a new screenshot before it acts again.

Products

Platform

Administration

Take screenshots

Click the mouse

Use the keyboard

Agent loop

Watch the desktop

​Take screenshots

​Click the mouse

​Use the keyboard

​Agent loop

​Watch the desktop

Take screenshots

Click the mouse

Use the keyboard

Agent loop

Watch the desktop