Build AI computer use for a browser, a shell, a container, or a desktop.

Start

Install

Automify requires Node.js 20.12.2 or newer. Playwright is a dependency because browser automation is a built-in surface, and the package postinstall step installs Chromium automatically.

AUTOMIFY_SKIP_BROWSER_INSTALL=1 npm install automify

Use the environment variable above when your CI image or deployment environment manages browsers separately.

Zod support is optional. Install Zod only when you want to define structured output with Zod schemas:

npm install zod

The main `automify` package works without Zod. The Zod helper lives behind the separate `automify/zod` import and uses Zod as an optional peer dependency.

Optional Docker setup

Docker is required only for `automify.dockerCli()` and `automify.dockerComputer()`. On macOS and Windows, install Docker Desktop from the official Docker website: Docker Desktop for Mac or Docker Desktop for Windows.

On Ubuntu, install Docker from the Ubuntu repositories:

sudo apt-get update
sudo apt-get install -y docker.io

Use `docker.io`, not the `docker` package. In Ubuntu packages, `docker.io` provides the Docker Engine/runtime and CLI.

Start Docker on Ubuntu and enable it after reboot:

sudo systemctl enable --now docker
sudo docker run hello-world

To run Docker commands without `sudo`, add your user to the `docker` group, then log out and back in:

sudo usermod -aG docker $USER

Optional QEMU setup

QEMU is required only for `automify.virtualCli()`, `automify.virtualComputer()`, and `createVirtualDesktopComputer()`. When no image is configured, Automify downloads the official Debian genericcloud qcow2 image into a local cache, prepares a reusable Automify-ready Debian qcow2 with the `automify` SSH user already provisioned. Runtime VMs boot from short-lived overlays backed by that prepared image. Pass `image` or `vm.image` only when you want to use your own bootable Linux disk image with SSH access.

# Ubuntu
sudo apt-get install -y qemu-system qemu-utils

# macOS
brew install qemu

# Windows
# Install QEMU from https://www.qemu.org/download/

On Windows, the QEMU installer commonly places binaries in C:\Program Files\qemu without making them available in every terminal. Add that directory to your user PATH from PowerShell, then open a new terminal and verify both commands work:

setx PATH "$env:PATH;C:\Program Files\qemu"
qemu-system-x86_64 --version
qemu-img --version

Pre-warm or refresh QEMU image caches:

# Pre-warm the minimal QEMU CLI cache.
npx automify-qemu-image

# Pre-warm a QEMU CLI cache for examples that need Node.js in the VM.
npx automify-qemu-image --package coreutils --package nodejs

# Use a larger timeout on slow hardware or slow package mirrors.
npx automify-qemu-image --package nodejs --timeout-ms 1200000

# Pre-warm the QEMU desktop cache with Xvfb/openbox/xterm/xdotool/scrot.
npx automify-qemu-image --desktop

# Re-download the Debian base image and rebuild the prepared cache.
npx automify-qemu-image --force-download

# Pre-warm an alternate qcow2 URL into its own cache.
npx automify-qemu-image \
  --image-url https://example.com/path/linux.qcow2 \
  --cache-dir ~/.cache/automify/qemu-custom

Use `--image-url` for an alternate apt-based Linux cloud qcow2 that Automify should download, boot, and prepare. Use the same URL and cache directory at runtime with `vm.imageUrl` or `qemuImageUrl` plus `defaultImageCache`. If you pass a local disk with `image` or `vm.image`, Automify uses that qcow2 directly instead of preparing it; the image must already boot, accept SSH, and include the desktop packages required for virtual desktop runs.

Core idea

Initialize once, then create the surface you need.

import { initAutomify } from "automify";

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const browser = await automify.browser({
  startUrl: "https://example.com"
});

try {
  const result = await browser.do("Summarize what you see on the page.");
  console.log(result.text);
} finally {
  await browser.close();
}

`initAutomify()` creates shared provider configuration and returns factory methods for browser, CLI, desktop, and custom computer runs. Every surface exposes `.do(instruction, run?)`, where run data lives under `data` and output/options live beside it in the same object.

Arguments

Use grouped options for new code.

`.do()` accepts a single run object. Prefer grouped options so input, evaluation files, screenshots, recording, safety, hooks, limits, and CLI command policy stay easy to scan.

await automify.cli({
  command: {
    cwd: process.cwd(),
    allow: ["npm test", "npm run build"],
    block: [/^rm\b/]
  }
}).do("Run tests and summarize failures", {
  data: { packageName: "automify" },
  evaluate: [{ path: "/tmp/screenshot.png", detail: "high" }],
  output: jsonOutput("test_report", { passed: "boolean", summary: "string" }),
  limits: { steps: 20 },
  screenshots: {
    final: "/tmp/automify-final.png",
    actions: "/tmp/automify-actions"
  },
  recording: {
    path: "/tmp/automify-run.mp4",
    fps: 6
  },
  screenshot: {
    detail: "high",
    maxWidth: 1600
  },
  safety: {
    domains: ["example.com"],
    onCheck: async ({ checks }) => checks.every((check) => check.code !== "external_form_submit")
  },
  hooks: {
    step: (event) => console.log(event.phase),
    complete: ({ result }) => console.log(result.status)
  }
});

The grouped names map to the flat names: `evaluate` maps to `filesToEvaluate`, `limits.steps` to `maxSteps`, `screenshots.final` to `finalScreenshot`, `recording` to `screenRecording`, `command.allow` to `allowedCommands`, and `safety.domains` to `allowedDomains`.

Build longer tasks as steps

Use the task builder when a workflow has distinct phases and should read like an ordered checklist. The builder creates one `.do()` instruction, so data, output parsing, screenshots, hooks, safety options, and limits behave the same as a normal run.

const run = await browser
  .addStep("Open the contacts page.")
  .addWait("the contacts table is visible")
  .addStep("Create the lead from data.")
  .addExtract("Return the saved record JSON.", {
    key: "lead",
    shape: { id: "string", firstName: "string", lastName: "string" }
  })
  .addData({ firstName: "Ada", lastName: "Lovelace" })
  .run();

console.log(run.parsed.lead.id);

Start with `browser.task()` when you want to keep a reusable builder. Alongside `addStep()`, Automify provides `addAct()`, `addWait()`, `addPause()`, `addObserve()`, `addExtract()`, and `addAssert()` plus short aliases without `add`. `addExtract()` can accept `{ key, shape }`; multiple extracts are merged into one structured result under `run.parsed[key]`.

Pass `task({ mode: "sequential" })` when each builder step should be its own model run. Sequential mode keeps browser or desktop state between steps, returns a `taskSteps` audit trail, and aggregates underlying action steps into `run.steps`. `addPause(ms)` is deterministic and does not call the model; `addAssert()` performs a structured pass/fail check and fails the task when the assertion is not true.

const run = await browser
  .task({ mode: "sequential" })
  .addStep("Fill the first name field from data.")
  .addStep("Fill the last name field from data.")
  .addStep("Submit the form.")
  .addExtract("Return the saved record JSON.", {
    key: "record",
    shape: { id: "string", firstName: "string", lastName: "string" }
  })
  .addData({ firstName: "Dorothy", lastName: "Vaughan" })
  .run({ recording: "/tmp/automify-sequential.mp4" });

console.log(run.taskSteps.length);
console.log(run.parsed.record.id);
console.log(run.recording.path);

In sequential mode, a run-level `output` applies only to the final non-pause step. Extract outputs still belong on `addExtract()` steps; if you define multiple extracts, give each one a `key`. Browser and desktop recordings cover the whole sequential task. CLI tasks can run sequentially, but CLI adapters do not record the screen.

Record visual runs

Browser and computer-use adapters can record a run by polling screenshots and encoding them with `ffmpeg`. Pass `recording` or `screenRecording` as a path, `true` for a temp MP4, or an object when you need `fps`, `framesDir`, `keepFrames`, `ffmpegCommand`, or a custom `execFile`.

const run = await browser.do("Run the checkout smoke test.", {
  screenRecording: {
    path: "/tmp/automify-checkout.mp4",
    fps: 6
  }
});

console.log(run.recording.path);

CLI adapters do not produce screen recordings. The host process needs `ffmpeg` on PATH unless you configure the encoder command.

Adapter factories use the same grouped shape. Browser uses `startUrl`, `viewport`, `launch`, `context`, and `navigation`; CLI uses `command`; Docker-backed adapters use `container`, `desktop`, `shared`, and `sharedFiles`; QEMU-backed adapters use `vm`, `ssh`, `desktop`, `shared`, and `sharedFiles`; local desktop uses `viewport`, `mouse`, `keyboard`, `calibration`, and `virtualDisplay`. Linux local desktop requires X11/Xorg or Xvfb; Wayland is not supported. Automify validates adapter option names at runtime and suggests likely fixes. Docker-backed and QEMU-backed APIs are named explicitly. The generated Markdown reference is `docs/argument-reference.md`.

Presets are optional shortcuts: `repo` for CLI, Docker CLI, and virtual CLI; `browser-review` for browser inspection; and `desktop-review` for review-friendly Docker desktop and virtual desktop viewport and settle timing.

Status

Handle automation errors with the Promise.

A resolved run means the requested model tool loop finished without provider, adapter, safety, policy, or max-step errors. If one of those fails, `.do()` rejects so regular `try`/`catch` is the primary error boundary.

try {
  const run = await browser.do("Submit the lead form");

  console.log("Actions executed:", run.steps.length);
} catch (error) {
  console.error("Automation failed:", error);
}

Treat returned structured data as the business result you asked the model to produce. If you need to prove a page or system actually changed, read back the saved data and compare it with your expected values.

Environment

Environment describes the surface the model is operating.

Browser runs default to `environment: "browser"`. Desktop adapters usually provide values such as `mac`, `windows`, or `linux`. The environment is passed through the computer tool payload so the model knows what kind of system it is controlling.

Local desktop computer use is optional and needs native desktop dependencies that are not installed by default. When you use the local desktop adapter, your OS may ask for permission to control the desktop.

Before running the desktop installer, install the native build tools for your operating system:

# Windows
winget install --id Microsoft.VisualStudio.2022.BuildTools --exact --override "--passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"
winget install --id Kitware.CMake --exact --source winget

# macOS
xcode-select --install
brew install cmake

# Ubuntu
sudo apt-get install -y git build-essential cmake pkg-config libx11-dev libxtst-dev libpng++-dev

On macOS and Windows, `cmake --version` must work in the same terminal where you run the installer. The VS Code CMake Tools extension is not enough by itself, and Visual Studio 2026 is not currently recognized by the native build chain used by nut.js.

npx automify-install-desktop

Linux local desktop support is documented for Ubuntu. Capture is X11-based: use Xorg/X11 or Xvfb, not Wayland. On headless Ubuntu hosts, also install Xvfb unless you manage `DISPLAY` yourself. Install the full package list before running the desktop installer; it checks for command-line build tools but does not verify every native library package.

const browser = await automify.browser({
  startUrl: "https://example.com",
  environment: "browser"
});

// Local desktop support requires the desktop installer once for this project.
const desktop = await automify.localComputer();

You can use any string if your adapter targets a different surface, but the built-in browser and desktop flows are designed around browser, macOS, Windows, and Linux X11/Xorg-style environments. Linux Wayland is not supported by the built-in local desktop adapter.

Defaults

Runtime defaults are optimized for practical automation.

`limits.steps` controls how many model-action turns a run may execute before Automify stops it with `MaxStepsExceededError`. The default is `100` for browser, desktop, custom computer, CLI, Docker CLI, and QEMU virtual CLI surfaces. Set a higher value for long workflows, or a lower value when a task should fail fast.

// Set the default for every run created by this adapter.
const browser = await automify.browser({
  startUrl: "https://example.com",
  limits: { steps: 250 }
});

await browser.do("Audit this long report workflow");

// Override the limit for one run only.
await browser.do("Do a quick smoke test", {
  limits: { steps: 25 }
});

The flat `maxSteps` option is equivalent to `limits.steps` and is kept for existing code. If both are provided, `maxSteps` wins. Per-run options passed to `.do()` override the adapter default for that call only.

// Equivalent to limits: { steps: 250 }.
const cli = automify.cli({ maxSteps: 250 });

// Per-run maxSteps also works, and overrides the adapter default.
await cli.do("Refactor and run tests", { maxSteps: 40 });

Production checklist

Before you let an automation touch real accounts.

Providers

Choose a provider and keep the same surface API.

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const automify = initAutomify({
  provider: {
    type: "anthropic",
    apiKey: process.env.ANTHROPIC_API_KEY,
    model: "claude-sonnet-4-20250514"
  }
});

Use `provider.type: "custom"` for another vendor, a model gateway, internal routing, mocks, or a provider release that Automify does not support directly yet.

`provider` is required in `initAutomify()`. Keep `apiKey` and `model` inside the provider object so there is no ambiguity about which provider owns the model string. Use direct factories such as `createAutomify()` only when you need the lower-level client API.

Choosing a GPT-5.6 model

The examples use the explicit `gpt-5.6-sol` model ID so logs, usage, and evaluations identify the selected tier unambiguously. The `gpt-5.6` alias currently routes to Sol. Choose `gpt-5.6-terra` when cost and latency need a stronger balance, or `gpt-5.6-luna` for high-volume, cost-sensitive work.

GPT-5.6 defaults to `medium` reasoning. Use it as the initial baseline, then compare `low` on representative tasks if latency or token use matters. Pass `reasoning: { effort: "low", summary: "concise" }` to `initAutomify()` or an individual run to override it. Review the official GPT-5.6 model guidance before changing model tier or reasoning effort in production.

Structured output

Use JSON for application data.

`.do()` returns run metadata plus the raw provider response. That is useful for tracing, steps, and callbacks, but your actual business data should usually come from structured model output.

Ask for the exact shape you want back. `jsonOutput()` expands the compact field map into a strict JSON schema-style output configuration.

const run = await browser.do(
  "Extract the support email.",
  {
    output: jsonOutput("support_contact", { email: "string" })
  }
);

console.log(run.parsed.email);

Compact shape

The compact form is designed for the common case: a flat object where every listed field is required.

jsonOutput("created_lead", {
  recordId: "string",
  portalUrl: "string"
})

That expands to a strict object schema with `recordId` and `portalUrl` in `required`, and `additionalProperties: false`.

Optional fields

Compact fields are required by default. For optional fields and richer validation, pass full JSON Schema.

const run = await browser.do(
  "Create the lead.",
  {
    output: jsonOutput("created_lead", {
      type: "object",
      properties: {
        recordId: { type: "string" },
        note: { type: "string" }
      },
      required: ["recordId"],
      additionalProperties: false
    })
  }
);

console.log(run.parsed.recordId, run.parsed.note);

Zod schemas

Projects using Zod 4 can use the optional adapter from `automify/zod`. This is separate from the main `automify` import so applications that only use `jsonOutput()` do not need to install Zod.

npm install zod

import { z } from "zod";
import { zodOutput } from "automify/zod";

const Lead = z.object({
  id: z.string(),
  firstName: z.string(),
  lastName: z.string()
});

const run = await browser.do("Create the lead.", {
  output: zodOutput("lead", Lead)
});

console.log(run.parsed.id);

`zodOutput()` converts the schema to JSON Schema with Zod 4's `z.toJSONSchema()` and sends that schema to the model. After the model returns JSON, Automify parses the response and validates it with the original Zod schema's `schema.parse()` before setting `run.parsed`.

Pass `{ parse: false }` to request the Zod-derived JSON Schema while skipping automatic JSON parsing and Zod validation of the final response.

Richer schemas

Full JSON Schema lets you use enums, arrays, nested objects, number constraints, string formats, and provider-specific schema features.

For an array of objects, prefer returning an object with a named array property. That keeps `jsonOutput()` in its object-shaped path and gives your application a stable place to read from `run.parsed`.

const run = await browser.do(
  "Analyze the order.",
  {
    output: jsonOutput("order_summary", {
      type: "object",
      properties: {
        priority: {
          type: "string",
          enum: ["low", "normal", "high"]
        },
        items: {
          type: "array",
          items: {
            type: "object",
            properties: {
              sku: { type: "string" },
              quantity: { type: "integer", minimum: 1 }
            },
            required: ["sku", "quantity"],
            additionalProperties: false
          }
        }
      },
      required: ["priority", "items"],
      additionalProperties: false
    })
  }
);

console.log(run.parsed.items);

Top-level arrays

If your application needs `run.parsed` itself to be an array, pass the `json_schema` output format directly. `jsonOutput()` intentionally accepts object-shaped schemas, while `output` can receive the lower-level structured output configuration.

const run = await browser.do(
  "Extract the products.",
  {
    output: {
      type: "json_schema",
      name: "products",
      strict: true,
      schema: {
        type: "array",
        items: {
          type: "object",
          properties: {
            sku: { type: "string" },
            title: { type: "string" },
            price: { type: "number" }
          },
          required: ["sku", "title", "price"],
          additionalProperties: false
        }
      }
    }
  }
);

console.log(run.parsed[0].sku);

Zod arrays

Zod schemas can describe arrays too. Use an object with a named array field when you want the same application-friendly shape as `jsonOutput()`.

import { z } from "zod";
import { zodOutput } from "automify/zod";

const ProductList = z.object({
  products: z.array(
    z.object({
      sku: z.string(),
      title: z.string(),
      price: z.number()
    })
  )
});

const run = await browser.do("Extract the products.", {
  output: zodOutput("product_list", ProductList)
});

console.log(run.parsed.products);

Provider compatibility

The practical rule: with OpenAI, structured output is provider-enforced. With Anthropic, Automify makes the request explicit and parses the JSON response, but final enforcement depends on the model following the instruction.

If you want plain text instead, use `run.text` or `getOutputText(run.response)`.

import { parseOutputJson } from "automify";

// Helper-based fallback
const run = await browser.do(
  "Extract the support contact as JSON."
);

const support = parseOutputJson(run.response);
console.log(support.email);

Anthropic details

Anthropic computer-use settings stay explicit and overridable.

With the built-in Anthropic provider, a model such as `claude-sonnet-4-20250514` uses Automify's current Anthropic computer-use defaults unless you override them.

const automify = initAutomify({
  provider: {
    type: "anthropic",
    apiKey: process.env.ANTHROPIC_API_KEY,
    model: "claude-sonnet-4-20250514",
    betas: ["computer-use-2025-01-24"],
    computerToolType: "computer_20250124"
  }
});

If Anthropic updates the beta header or computer tool schema for the model you want, set both fields together so the request stays internally consistent.

Browser

Use Playwright-backed browser automation.

const browser = await automify.browser({
  startUrl: "https://example.com",
  viewport: { width: 1280, height: 800 },
  headless: true,
  safety: {
    domains: ["example.com"]
  }
});

try {
  const result = await browser.do(
    "Summarize what you see on the page.",
    { data: { audience: "new visitor" } }
  );
} finally {
  await browser.close();
}

Browser runs take screenshots, feed them back to the model, and execute computer actions such as click, double click, move, drag, scroll, keypress, type, wait, and screenshot.

CLI

Run model-requested commands with explicit policies.

const cli = automify.cli({
  logFile: "/tmp/automify-cli.jsonl",
  command: {
    cwd: process.cwd(),
    allow: ["npm test", "npm run build", "ls", "pwd"]
  }
});

await cli.do("Run tests and summarize any failures");

CLI approval is `never` by default. Add `approval: "always"` or `confirmCommand` when you want human approval or policy logic before a command executes.

const cli = automify.cli({
  command: {
    approval: "always",
    confirm: async ({ command }) => {
      return command.command.startsWith("npm test");
    }
  }
});

Use `automify.dockerCli()` when commands should run in an isolated Docker container instead of the host shell. Docker must be installed and running before you create the adapter. It supports the same command policies and can mount a shared folder into the container. See Optional Docker setup if Docker is not installed yet.

import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { initAutomify, jsonOutput } from "automify";

const sharedDir = await mkdtemp(join(tmpdir(), "automify-docker-cli-"));
const dataDir = join(sharedDir, "data");
const reportPath = join(dataDir, "report.csv");
const summaryPath = join(dataDir, "summary.json");

await mkdir(dataDir, { recursive: true });
await writeFile(
  reportPath,
  "region,customer,revenue\n" +
    "North,Ada Corp,1250\n" +
    "South,Byron Ltd,980\n" +
    "North,Lovelace Labs,2230\n"
);
await writeFile(summaryPath, "{}\n");

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const cli = automify.dockerCli({
  logFile: "/tmp/automify-docker-cli.jsonl",
  additionalAptPackages: ["coreutils", "nodejs"],
  shared: { hostPath: sharedDir, containerPath: "/workspace" }
});

try {
  const run = await cli.do("Read data/report.csv, use a Node.js script to calculate revenue by region, update data/summary.json with the result, and return the top region.", {
    output: jsonOutput("report_summary", {
      topRegion: "string",
      totalRevenue: "number",
      outputFile: "string",
      summary: "string"
    })
  });

  const summaryFile = JSON.parse(await readFile(summaryPath, "utf8"));
  console.log(run.parsed, summaryFile);
  console.log("Shared output file:", summaryPath);
} finally {
  await cli.close();
}

Use `automify.virtualCli()` when commands should run inside a real QEMU VM instead of the host shell or a Docker container. QEMU must be installed. Without an image override, Automify boots short-lived overlays from the cached prepared Debian image. It supports the same command policies as local CLI and Docker CLI, plus `shared` / `sharedFiles` for workspace access.

const cli = automify.virtualCli({
  vm: { memory: "2g", cpus: 2 },
  additionalAptPackages: ["coreutils", "nodejs"],
  shared: { hostPath: process.cwd(), containerPath: "/workspace" },
  command: {
    allow: ["node --version", "pwd"]
  }
});

try {
  const run = await cli.do("Run 'node --version' and 'pwd', then summarize the VM environment.");
  console.log(run.text);
} finally {
  await cli.close();
}

To reuse an alternate qcow2 URL that you pre-warmed with `automify-qemu-image --image-url`, pass the same URL and cache directory:

const cli = automify.virtualCli({
  vm: {
    imageUrl: "https://example.com/path/linux.qcow2"
  },
  defaultImageCache: {
    dir: "/var/cache/automify-qemu-custom"
  }
});

Desktop

Desktop automation supports local and sandboxed Linux sessions.

There are three desktop modes. `localComputer()` controls the native desktop on the machine running your Node.js process, so it targets macOS, Windows, or Linux depending on the host. On Linux, this means X11/Xorg or Xvfb; Wayland is not supported for local desktop control. `dockerComputer()` requires Docker to be installed and running, and can be started from any host that can run Docker Linux containers, but the desktop it controls is always a Linux desktop inside the container. `virtualComputer()` requires QEMU and uses a default Debian cloud image unless you provide a custom disk; it controls a Linux Xvfb desktop inside a real VM. See Optional Docker setup and Optional QEMU setup before using those adapters.

import { initAutomify, jsonOutput } from "automify";

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

// Reminder: local desktop support requires `npx automify-install-desktop` once for this project.
const desktop = await automify.localComputer();

try {
  const run = await desktop.do("Open the Calendar app installed on this computer, find the next event, and return its title and start time. Do not create or edit events.", {
    screenshots: {
      initial: "/tmp/automify-local-desktop-initial.png",
      final: "/tmp/automify-local-desktop-final.png"
    },
    output: jsonOutput("next_event", {
      title: "string",
      startsAt: "string"
    })
  });

  console.log(run.parsed);
} finally {
  await desktop.close();
}

The desktop adapter is installed manually with `npx automify-install-desktop`; it is not installed by postinstall. The command builds nut.js, libnut-core, and macOS permission helpers from source, so it can take some time. If the command is run again and the cached runtime still matches the current platform, CPU architecture, Node ABI, and pinned native refs, Automify prints a skip message and exits without rebuilding. Use `npx automify-install-desktop --force` to rebuild a compatible cache anyway. Local desktop control may require OS permissions. The adapter provides `execute(action, context)` and `screenshot(context)` for native desktop control.

Native desktop prerequisites

Install the native build tools for your OS before running `npx automify-install-desktop`.

winget install --id Microsoft.VisualStudio.2022.BuildTools --exact --override `
  "--passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"
winget install --id Kitware.CMake --exact --source winget

xcode-select --install
brew install cmake

sudo apt-get install -y git build-essential cmake pkg-config libx11-dev libxtst-dev libpng++-dev

Linux local desktop capture is X11-based: use Xorg/X11 or Xvfb, not Wayland.

`createLocalDesktopComputer()` is the local desktop mode. On Linux servers without `DISPLAY`, it starts an in-memory Xvfb display automatically before loading nut.js. Install `xvfb` on headless hosts, or pass `virtualDisplay: false` if you manage an X11 `DISPLAY` outside Automify. Wayland `DISPLAY`/XWayland sessions are not supported for local desktop capture and can fail with native X11 errors such as `BadMatch` / `X_GetImage`. On macOS and Windows, it controls the host desktop directly through the native nut.js backend.

`createLocalDesktopComputer()` takes an exclusive cross-process lock while the adapter is alive. This prevents two scripts, including scripts launched from separate terminals, from controlling the same real mouse, keyboard, and screen at the same time. Always call `computer.close()` in a `finally` block so Automify can release the lock and stop any virtual display it started. If a process crashes, Automify stores lock ownership under the system temp directory at `automify-locks`, checks whether the recorded PID is still alive, and removes stale locks automatically when the owning process has exited. If the owner still appears to be running, a second adapter fails with an error that includes the lock path.

import { initAutomify, jsonOutput } from "automify";

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const desktop = await automify.dockerComputer({
  desktop: { startupCommand: "xterm" }
});

try {
  const run = await desktop.do("Use the open terminal to run 'uname -a', then return the kernel name and machine architecture shown on screen.", {
    screenshots: {
      initial: "/tmp/automify-docker-desktop-initial.png",
      final: "/tmp/automify-docker-desktop-final.png"
    },
    output: jsonOutput("system_info", {
      kernelName: "string",
      machine: "string",
      summary: "string"
    })
  });

  console.log(run.parsed);
} finally {
  await desktop.close();
}

`createDockerDesktopComputer()` creates a Docker-backed Linux desktop. By default it creates a fresh container from the default public Linux image, keeps Docker networking enabled, installs a minimal desktop package set with apt on startup, and runs Xvfb, a lightweight window manager, `xdotool` input, and PNG screenshots. There is no `automify-docker-desktop:latest` image to pull by default: Automify prepares the desktop inside the container unless you pass a custom `image`. Docker desktop does not control the host operating system desktop and does not use `automify-install-desktop`; it needs a running Docker daemon and a startup command instead.

Docker desktop requires an initial app command. Pass `startupCommand` or `desktop.startupCommand`, for example `desktop.startupCommand: "xterm"`.

The default Docker desktop setup installs `xvfb`, `openbox`, `xterm`, `x11-utils`, `xdotool`, `imagemagick`, `scrot`, and `ca-certificates`. Browser packages are opt-in; pass `desktop.additionalAptPackages: ["chromium"]` when you want Chromium installed during container startup. Pass `container.image` to use another apt-based distro or a prebuilt image. When `desktop.installDependencies: false`, the image must already include the required desktop tools, plus Chromium only if you plan to launch it.

Pass `container.network: false` or `container.network: "none"` when you want to disable Docker networking. Use `desktop.packages` to replace the base package list, or `desktop.additionalAptPackages` to append packages to the default list. Use `desktop.startupCommand: "xterm"` to start with a terminal, or pass `desktop.additionalAptPackages: ["chromium"]` with `desktop.startupCommand: "chromium --no-sandbox"` if you want Chromium open at startup.

Docker desktop locks are scoped to the container name. Default Docker desktop adapters use a random container name, so separate calls create separate containers and can run in parallel. If you pass `containerName` or `container.name`, Automify takes a cross-process lock for that exact name until `desktop.close()` or `computer.close()` releases it. Two adapters with different names can run at the same time; two adapters with the same name fail fast instead of racing Docker or driving the same display. The same rule applies when using `existingContainer`: only one adapter should control a given existing container at a time.

Docker desktop, Docker CLI, QEMU virtual desktop, and QEMU virtual CLI sessions all accept `shared` and `sharedFiles`. Pass `shared` to expose a host directory, or pass `sharedFiles` to copy specific host files into a temporary shared directory before the run starts.

const desktop = await automify.virtualComputer({
  vm: { memory: "2g", cpus: 2 },
  desktop: { startupCommand: "xterm" }
});

try {
  const run = await desktop.do("Use the open terminal to run 'uname -a', then return the kernel name and machine architecture shown on screen.", {
    screenshots: {
      initial: "/tmp/automify-qemu-desktop-initial.png",
      final: "/tmp/automify-qemu-desktop-final.png"
    },
    output: jsonOutput("system_info", {
      kernelName: "string",
      machine: "string",
      summary: "string"
    })
  });

  console.log(run.parsed);
} finally {
  await desktop.close();
}

`createVirtualDesktopComputer()` creates a QEMU-backed Linux desktop. It starts QEMU with user-mode networking and SSH port forwarding, waits for SSH readiness, then starts Xvfb, a lightweight window manager, `xdotool` input, and PNG screenshots inside the guest. Without an image override, Automify prepares a default Debian cloud image with SSH access. Pass `vm.image` or `image` only for a custom disk, `ssh.user` and `ssh.keyPath` for custom login, and `desktop.startupCommand` or `startupCommand` for the initial app.

QEMU virtual desktop locks are scoped to the VM name. Default virtual desktop adapters use a random VM name, so separate calls can run in parallel. If you pass `vmName` or `vm.name`, Automify takes a cross-process lock for that exact name until `desktop.close()` or `computer.close()` releases it.

QEMU virtual desktop and virtual CLI sessions both accept `shared` and `sharedFiles`. The default shared folder path is exposed through QEMU `virtfs` / 9p with the mount tag `automify_shared`; the guest must have 9p support for live host folder mounting.

Pass `screenshots.initial` and/or `screenshots.final` on a `do()` call to save screenshots at the beginning and end of a run. Pass `screenshots.actions: "/tmp/automify-actions"` to save a before and after PNG for every browser, desktop, or custom computer action. Action screenshots are named by step and action index, and each step includes the saved paths under `step.actionScreenshots`.

node examples/desktop-docker.js

On macOS, `createLocalDesktopComputer()` calibrates screenshot pixels to native mouse coordinates. For unusual monitor scaling, pass `pixelScale`, `mouseScaleX`, `mouseScaleY`, `mouseOffsetX`, or `mouseOffsetY`.

Pass `debug: true` to log useful progress to stderr: model request phases, step/action execution, full CLI command results, browser input, desktop screenshots, and Docker container setup. Pass `debug: (message, details) => { ... }` to collect the same events yourself. On `automify.cli()` and `automify.dockerCli()`, pass `logFile: "/tmp/automify-cli.jsonl"` to append debug events as JSON Lines without enabling stderr logs.

For desktop speed, the nut.js adapter disables nut.js mouse and keyboard auto-delays by default. You can override this with `mouseAutoDelayMs` or `keyboardAutoDelayMs` if a specific application needs slower input. Screenshot uploads default to `detail: "auto"` and are resized to fit within 1440x1440 before upload when the optional image pipeline is available; Automify maps model coordinates back to the original screenshot space before executing actions. Pass `screenshotDetail`, `screenshotMaxWidth`, or `screenshotMaxHeight` to `initAutomify()`, `createAutomify()`, or a single `do()` call when you want to tune precision, image cost, and latency. Pass `trace: true` to include response/action/screenshot timings in the run result.

Desktop source builds are pinned to known working nut.js, libnut-core, and node-mac-permissions commits. Override with `AUTOMIFY_DESKTOP_NUT_REF`, `AUTOMIFY_DESKTOP_LIBNUT_CORE_REF`, or `AUTOMIFY_DESKTOP_MAC_PERMISSIONS_REF` when testing a newer upstream revision. Changing those refs changes the desktop runtime compatibility key, so the next `npx automify-install-desktop` build will not be skipped.

Files

Pass files as data or mount them into virtual environments.

Use `fileToData()` or `filesToData()` when you want file metadata, text, base64, or data URLs inside `data`. Use `evaluate` on `.do()` when the model should evaluate files directly; images are sent as image input and text-like files are sent as text input. Use `shared` or `sharedFiles` when Docker CLI, Docker desktop, QEMU virtual CLI, or QEMU virtual desktop also needs filesystem access to those files.

Factory options define runner defaults. Per-run `.do()` options override those defaults for one run, so a CLI can have a default command policy and still narrow or change it for a specific task.

await automify.cli().do("Evaluate this screenshot and list visible issues.", {
  evaluate: [
    { path: "/tmp/screenshot.png", detail: "high" }
  ]
});

import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { filesToData, initAutomify } from "automify";

const sharedDir = await mkdtemp(join(tmpdir(), "automify-docker-cli-"));
const dataDir = join(sharedDir, "data");
const reportPath = join(dataDir, "report.csv");
const summaryPath = join(dataDir, "summary.json");

await mkdir(dataDir, { recursive: true });
await writeFile(
  reportPath,
  "region,customer,revenue\n" +
    "North,Ada Corp,1250\n" +
    "South,Byron Ltd,980\n" +
    "North,Lovelace Labs,2230\n"
);
await writeFile(summaryPath, "{}\n");

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const cli = automify.dockerCli({
  additionalAptPackages: ["coreutils", "nodejs"],
  shared: { hostPath: sharedDir, containerPath: "/workspace" }
});

try {
  await cli.do("Read data/report.csv, use a Node.js script to calculate revenue by region, update data/summary.json with the result, and summarize the result.", {
    data: {
      files: await filesToData(reportPath, { format: "metadata" })
    }
  });
  console.log(JSON.parse(await readFile(summaryPath, "utf8")));
  console.log("Shared output file:", summaryPath);
} finally {
  await cli.close();
}

In Docker adapters, `shared` mounts a host folder into the container, so files the command creates or edits under `/workspace` are visible on the host after the run. In QEMU adapters, `shared` is exposed through QEMU `virtfs` / 9p by default, so the guest image must support 9p mounts. `sharedFiles` is still useful when you only need to copy specific inputs into a temporary shared directory before the first request.

import { createDockerDesktopComputer } from "automify";

const computer = await createDockerDesktopComputer({
  desktop: { startupCommand: "xterm" },
  shared: {
    hostPath: "/tmp/automify-work",
    containerPath: "/workspace"
  }
});

console.log(computer.sharedFolder.containerPath);

import { createVirtualDesktopComputer } from "automify";

const computer = await createVirtualDesktopComputer({
  vm: { memory: "2g", cpus: 2 },
  desktop: { startupCommand: "xterm" },
  shared: {
    hostPath: "/tmp/automify-work",
    containerPath: "/workspace"
  }
});

console.log(computer.sharedFolder.containerPath);

Input

Pass task input the same way on every adapter.

The first `.do()` argument is the instruction. Structured values go under `data`, and adapter setup goes in the factory options. The model receives both the instruction and the JSON data.

import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { initAutomify } from "automify";

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  }
});

const sharedDir = await mkdtemp(join(tmpdir(), "automify-docker-cli-"));
const dataDir = join(sharedDir, "data");
const reportPath = join(dataDir, "report.csv");
const summaryPath = join(dataDir, "summary.json");

await mkdir(dataDir, { recursive: true });
await writeFile(
  reportPath,
  "region,customer,revenue\n" +
    "North,Ada Corp,1250\n" +
    "South,Byron Ltd,980\n" +
    "North,Lovelace Labs,2230\n"
);
await writeFile(summaryPath, "{}\n");

// Browser
const browser = await automify.browser({ startUrl: "https://aldovincenti.github.io/automify/demo.html" });
await browser.do("Create the lead from data.", {
  data: { firstName: "Ada", lastName: "Lovelace", email: "ada@example.com" }
});

// CLI
const cli = automify.cli({ command: { cwd: process.cwd(), allow: ["npm test"] } });
await cli.do("Run the requested script and summarize the result.", {
  data: { script: "npm test" }
});

// Docker CLI
const dockerCli = automify.dockerCli({
  additionalAptPackages: ["coreutils", "nodejs"],
  shared: { hostPath: sharedDir, containerPath: "/workspace" }
});
try {
  await dockerCli.do("Read data/report.csv, use a Node.js script to calculate revenue by region, update data/summary.json with the result, and report the top region.", {
    data: { report: "data/report.csv" }
  });
  console.log(JSON.parse(await readFile(summaryPath, "utf8")));
  console.log("Shared output file:", summaryPath);
} finally {
  await dockerCli.close();
}

// Virtual CLI
const virtualCli = automify.virtualCli({
  vm: { memory: "2g", cpus: 2 },
  additionalAptPackages: ["coreutils"],
  shared: { hostPath: sharedDir, containerPath: "/workspace" }
});
try {
  await virtualCli.do("Read data/report.csv, calculate revenue by region, and report the top region.", {
    data: { report: "data/report.csv" }
  });
} finally {
  await virtualCli.close();
}


// Local desktop
// Reminder: local desktop support requires `npx automify-install-desktop` once for this project.
const localDesktop = await automify.localComputer();
try {
  await localDesktop.do("Open the Calendar app installed on this computer, find the next event, and summarize it. Do not create or edit events.", {
    screenshots: { final: "/tmp/local-final.png" }
  });
} finally {
  await localDesktop.close();
}

// Docker desktop
const dockerDesktop = await automify.dockerComputer({
  desktop: {
    startupCommand: "xterm"
  }
});
try {
  await dockerDesktop.do("Use the open terminal to run 'uname -a' and summarize the system information shown on screen.", {
    screenshots: { final: "/tmp/docker-final.png" }
  });
} finally {
  await dockerDesktop.close();
}

// Virtual desktop
const virtualDesktop = await automify.virtualComputer({
  vm: { memory: "2g", cpus: 2 },
  desktop: {
    startupCommand: "xterm"
  }
});
try {
  await virtualDesktop.do("Use the open terminal to run 'uname -a' and summarize the VM system information shown on screen.", {
    screenshots: { final: "/tmp/qemu-final.png" }
  });
} finally {
  await virtualDesktop.close();
}

Completion

Use the Promise result, `onComplete`, or both.

Every `.do()` returns a Promise that resolves when the requested tool loop succeeds and rejects when the provider, adapter, safety, policy, or step limit fails. Use the resolved result for `run.status`, `run.steps`, `run.text`, and optional `run.parsed`. Register `onComplete` globally, per surface, or per run when your app should react to completion together with the original input data.

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  },
  onComplete: async ({ surface, data, result, response }) => {
    await auditLog.write({
      surface,
      customerId: data.customer?.id,
      status: result.status,
      responseId: response.id
    });
  }
});

const result = await automify.browser({ startUrl: "https://example.com" }).do(
  "Summarize the visible page for this customer record.",
  { data: { customer: { id: "cus_123", email: "ada@example.com" } } }
);

Adapters

Custom adapters keep provider integration small and ergonomic.

import {
  initAutomify,
  response,
  message,
  computerCall,
  runCommandCall,
  getInputText,
  getComputerTool,
  testModelAdapter
} from "automify";

const automify = initAutomify({
  provider: {
    type: "custom",
    model: "my-model",
    adapter: {
      async respond(payload, context) {
        const text = getInputText(payload);

        if (context.surface === "cli") {
          return response({
            id: "cli_step_1",
            output: [runCommandCall("npm test")]
          });
        }

        if (getComputerTool(payload)) {
          return response({
            id: "computer_step_1",
            output: [computerCall({ type: "click", x: 120, y: 180 })]
          });
        }

        return response({
          id: "done",
          output: [message(`Handled: ${text}`)]
        });
      }
    }
  },
  requestOptions: {
    temperature: 0.2,
    metadata: { app: "ops-agent" }
  }
});

await testModelAdapter(automify.client);

Adapter context includes `surface`, `phase`, `step`, and `requestOptions`. `requestOptions` is the place for provider-specific request parameters like temperature, tracing IDs, routing hints, or gateway metadata.

Safety

Use explicit guardrails in production.

const browser = await automify.browser({
  startUrl: "https://example.com",
  safety: {
    domains: ["example.com"]
  },
  screenshot: {
    redact: async (screenshot) => screenshot
  }
});

await browser.do("Summarize the page without leaving the allowed domain.", {
  safety: {
    onCheck: async ({ checks, action }) => true
  }
});

Recommended production controls include domain allowlists, command allowlists, screenshot redaction, bounded `limits.steps`, isolated accounts, and human approval for sensitive operations.

Treat desktop and CLI runs as privileged automation surfaces. Keep credentials scoped, prefer short-lived sessions, and save screenshots only where your data policy allows it.

Observability

Inspect requests, responses, steps, and completion.

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.6-sol"
  },
  onRequest: (payload, meta) => logRequest(payload, meta),
  onResponse: (response, meta) => logResponse(response, meta),
  onStep: (event) => logStep(event),
  onComplete: (event) => logCompletion(event),
  debug: true
});

Use hooks for traces, audit logs, policy decisions, action screenshots, screen recordings, full CLI logs, and provider debugging.

const run = await browser.do("Review the checkout flow.", {
  screenshots: {
    actions: "/tmp/automify-action-screenshots"
  },
  recording: "/tmp/automify-checkout.mp4",
  trace: true
});

console.log(run.recording.path);

Development

Running tests

npm test
npm run test:e2e
OPENAI_API_KEY=... npm run test:live

`npm run test:live:qemu` runs only the real QEMU Debian boot smoke test, without OpenAI. `npm run test:live:qemu:desktop` runs the real QEMU desktop smoke test with the default Debian image. Set `AUTOMIFY_QEMU_IMAGE=/path/to/linux.qcow2` only when you want the desktop smoke test or the QEMU live tests to use a custom image. The equivalent direct flags are `RUN_QEMU_DEBIAN_E2E=1 npm run test:e2e` and `RUN_QEMU_DESKTOP_E2E=1 npm run test:e2e`.

`npm run test:live` runs `test/e2e/live-openai.e2e.test.js` with `RUN_OPENAI_E2E=1`. By default, it runs the live OpenAI CLI and Docker CLI checks and skips the browser, Docker desktop, and QEMU checks.

Run every live test through the npm script:

OPENAI_API_KEY=... \
RUN_OPENAI_BROWSER_E2E=1 \
RUN_OPENAI_DOCKER_DESKTOP_E2E=1 \
RUN_OPENAI_QEMU_CLI_E2E=1 \
RUN_OPENAI_QEMU_DESKTOP_E2E=1 \
npm run test:live

The equivalent direct command is:

OPENAI_API_KEY=... \
RUN_OPENAI_E2E=1 \
RUN_OPENAI_BROWSER_E2E=1 \
RUN_OPENAI_DOCKER_DESKTOP_E2E=1 \
RUN_OPENAI_QEMU_CLI_E2E=1 \
RUN_OPENAI_QEMU_DESKTOP_E2E=1 \
node --test test/e2e/live-openai.e2e.test.js

Set `AUTOMIFY_QEMU_IMAGE=/path/to/linux.qcow2` only when you want QEMU live tests to use a custom image instead of the default Debian image.

The default Debian base image and prepared Automify-ready images are cached on the user's computer and reused across VM runs. CLI cache variants bake requested `packages` and `additionalAptPackages`; the desktop cache is a separate variant that bakes Xvfb/openbox/xterm/xdotool/scrot so warm desktop boots do not reinstall apt packages. Configure image caching with `defaultImageCache`, for example `defaultImageCache: { dir: "/var/cache/automify-qemu", forcePrepare: true }`. Run `npx automify-qemu-image` to pre-warm the minimal QEMU CLI cache, or add flags such as `--package coreutils --package nodejs` to pre-warm a package-specific CLI cache. The default preparation timeout is generous, but `--timeout-ms 1200000` can help on slow hardware or package mirrors. Run `npx automify-qemu-image --desktop` to pre-warm the QEMU desktop cache. Run `npx automify-qemu-image --image-url https://example.com/path/linux.qcow2 --cache-dir /var/cache/automify-qemu-custom` to pre-warm an alternate cloud qcow2 cache, then use the same URL and cache directory at runtime with `vm.imageUrl` or `qemuImageUrl` plus `defaultImageCache`. Run `npx automify-qemu-image --force-download` to replace the cached base image and rebuild the prepared image. Local disks passed with `image` or `vm.image` are not prepared by the cache command; they must already be bootable with SSH access.

API map

Primary exports

Disclaimer

Distributed as is.

Automify is provided "as is", without warranty of any kind. Automation can control browsers, shells, desktops, files, and external services; you are responsible for how you configure and run it, and for any events associated with that use. To the maximum extent permitted by law, the author is not liable for losses, damages, data loss, service disruption, or other consequences arising from use of the software.