This is page 21 of 29. Use http://codebase.md/trycua/cua?lines=true&page={x} to view the full context.
# Directory Structure
```
├── .cursorignore
├── .dockerignore
├── .editorconfig
├── .gitattributes
├── .github
│ ├── FUNDING.yml
│ ├── scripts
│ │ ├── get_pyproject_version.py
│ │ └── tests
│ │ ├── __init__.py
│ │ ├── README.md
│ │ └── test_get_pyproject_version.py
│ └── workflows
│ ├── bump-version.yml
│ ├── ci-lume.yml
│ ├── docker-publish-cua-linux.yml
│ ├── docker-publish-cua-windows.yml
│ ├── docker-publish-kasm.yml
│ ├── docker-publish-xfce.yml
│ ├── docker-reusable-publish.yml
│ ├── link-check.yml
│ ├── lint.yml
│ ├── npm-publish-cli.yml
│ ├── npm-publish-computer.yml
│ ├── npm-publish-core.yml
│ ├── publish-lume.yml
│ ├── pypi-publish-agent.yml
│ ├── pypi-publish-computer-server.yml
│ ├── pypi-publish-computer.yml
│ ├── pypi-publish-core.yml
│ ├── pypi-publish-mcp-server.yml
│ ├── pypi-publish-som.yml
│ ├── pypi-reusable-publish.yml
│ ├── python-tests.yml
│ ├── test-cua-models.yml
│ └── test-validation-script.yml
├── .gitignore
├── .pre-commit-config.yaml
├── .prettierignore
├── .prettierrc.yaml
├── .vscode
│ ├── docs.code-workspace
│ ├── extensions.json
│ ├── launch.json
│ ├── libs-ts.code-workspace
│ ├── lume.code-workspace
│ ├── lumier.code-workspace
│ ├── py.code-workspace
│ └── settings.json
├── blog
│ ├── app-use.md
│ ├── assets
│ │ ├── composite-agents.png
│ │ ├── docker-ubuntu-support.png
│ │ ├── hack-booth.png
│ │ ├── hack-closing-ceremony.jpg
│ │ ├── hack-cua-ollama-hud.jpeg
│ │ ├── hack-leaderboard.png
│ │ ├── hack-the-north.png
│ │ ├── hack-winners.jpeg
│ │ ├── hack-workshop.jpeg
│ │ ├── hud-agent-evals.png
│ │ └── trajectory-viewer.jpeg
│ ├── bringing-computer-use-to-the-web.md
│ ├── build-your-own-operator-on-macos-1.md
│ ├── build-your-own-operator-on-macos-2.md
│ ├── cloud-windows-ga-macos-preview.md
│ ├── composite-agents.md
│ ├── computer-use-agents-for-growth-hacking.md
│ ├── cua-hackathon.md
│ ├── cua-playground-preview.md
│ ├── cua-vlm-router.md
│ ├── hack-the-north.md
│ ├── hud-agent-evals.md
│ ├── human-in-the-loop.md
│ ├── introducing-cua-cli.md
│ ├── introducing-cua-cloud-containers.md
│ ├── lume-to-containerization.md
│ ├── neurips-2025-cua-papers.md
│ ├── sandboxed-python-execution.md
│ ├── training-computer-use-models-trajectories-1.md
│ ├── trajectory-viewer.md
│ ├── ubuntu-docker-support.md
│ └── windows-sandbox.md
├── CONTRIBUTING.md
├── Development.md
├── Dockerfile
├── docs
│ ├── .env.example
│ ├── .gitignore
│ ├── content
│ │ └── docs
│ │ ├── agent-sdk
│ │ │ ├── agent-loops.mdx
│ │ │ ├── benchmarks
│ │ │ │ ├── index.mdx
│ │ │ │ ├── interactive.mdx
│ │ │ │ ├── introduction.mdx
│ │ │ │ ├── meta.json
│ │ │ │ ├── osworld-verified.mdx
│ │ │ │ ├── screenspot-pro.mdx
│ │ │ │ └── screenspot-v2.mdx
│ │ │ ├── callbacks
│ │ │ │ ├── agent-lifecycle.mdx
│ │ │ │ ├── cost-saving.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ ├── logging.mdx
│ │ │ │ ├── meta.json
│ │ │ │ ├── pii-anonymization.mdx
│ │ │ │ └── trajectories.mdx
│ │ │ ├── chat-history.mdx
│ │ │ ├── custom-tools.mdx
│ │ │ ├── customizing-computeragent.mdx
│ │ │ ├── integrations
│ │ │ │ ├── hud.mdx
│ │ │ │ ├── meta.json
│ │ │ │ └── observability.mdx
│ │ │ ├── mcp-server
│ │ │ │ ├── client-integrations.mdx
│ │ │ │ ├── configuration.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ ├── installation.mdx
│ │ │ │ ├── llm-integrations.mdx
│ │ │ │ ├── meta.json
│ │ │ │ ├── tools.mdx
│ │ │ │ └── usage.mdx
│ │ │ ├── message-format.mdx
│ │ │ ├── meta.json
│ │ │ ├── migration-guide.mdx
│ │ │ ├── prompt-caching.mdx
│ │ │ ├── supported-agents
│ │ │ │ ├── composed-agents.mdx
│ │ │ │ ├── computer-use-agents.mdx
│ │ │ │ ├── grounding-models.mdx
│ │ │ │ ├── human-in-the-loop.mdx
│ │ │ │ └── meta.json
│ │ │ ├── supported-model-providers
│ │ │ │ ├── cua-vlm-router.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ └── local-models.mdx
│ │ │ ├── telemetry.mdx
│ │ │ └── usage-tracking.mdx
│ │ ├── cli-playbook
│ │ │ ├── commands.mdx
│ │ │ ├── index.mdx
│ │ │ └── meta.json
│ │ ├── computer-sdk
│ │ │ ├── cloud-vm-management.mdx
│ │ │ ├── commands.mdx
│ │ │ ├── computer-server
│ │ │ │ ├── Commands.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ ├── meta.json
│ │ │ │ ├── REST-API.mdx
│ │ │ │ └── WebSocket-API.mdx
│ │ │ ├── computer-ui.mdx
│ │ │ ├── computers.mdx
│ │ │ ├── custom-computer-handlers.mdx
│ │ │ ├── meta.json
│ │ │ ├── sandboxed-python.mdx
│ │ │ └── tracing-api.mdx
│ │ ├── example-usecases
│ │ │ ├── form-filling.mdx
│ │ │ ├── gemini-complex-ui-navigation.mdx
│ │ │ ├── meta.json
│ │ │ ├── post-event-contact-export.mdx
│ │ │ └── windows-app-behind-vpn.mdx
│ │ ├── get-started
│ │ │ ├── meta.json
│ │ │ └── quickstart.mdx
│ │ ├── index.mdx
│ │ ├── macos-vm-cli-playbook
│ │ │ ├── lume
│ │ │ │ ├── cli-reference.mdx
│ │ │ │ ├── faq.md
│ │ │ │ ├── http-api.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ ├── installation.mdx
│ │ │ │ ├── meta.json
│ │ │ │ └── prebuilt-images.mdx
│ │ │ ├── lumier
│ │ │ │ ├── building-lumier.mdx
│ │ │ │ ├── docker-compose.mdx
│ │ │ │ ├── docker.mdx
│ │ │ │ ├── index.mdx
│ │ │ │ ├── installation.mdx
│ │ │ │ └── meta.json
│ │ │ └── meta.json
│ │ └── meta.json
│ ├── next.config.mjs
│ ├── package-lock.json
│ ├── package.json
│ ├── pnpm-lock.yaml
│ ├── postcss.config.mjs
│ ├── public
│ │ └── img
│ │ ├── agent_gradio_ui.png
│ │ ├── agent.png
│ │ ├── bg-dark.jpg
│ │ ├── bg-light.jpg
│ │ ├── cli.png
│ │ ├── computer.png
│ │ ├── grounding-with-gemini3.gif
│ │ ├── hero.png
│ │ ├── laminar_trace_example.png
│ │ ├── som_box_threshold.png
│ │ └── som_iou_threshold.png
│ ├── README.md
│ ├── source.config.ts
│ ├── src
│ │ ├── app
│ │ │ ├── (home)
│ │ │ │ ├── [[...slug]]
│ │ │ │ │ └── page.tsx
│ │ │ │ └── layout.tsx
│ │ │ ├── api
│ │ │ │ ├── posthog
│ │ │ │ │ └── [...path]
│ │ │ │ │ └── route.ts
│ │ │ │ └── search
│ │ │ │ └── route.ts
│ │ │ ├── favicon.ico
│ │ │ ├── global.css
│ │ │ ├── layout.config.tsx
│ │ │ ├── layout.tsx
│ │ │ ├── llms.mdx
│ │ │ │ └── [[...slug]]
│ │ │ │ └── route.ts
│ │ │ ├── llms.txt
│ │ │ │ └── route.ts
│ │ │ ├── robots.ts
│ │ │ └── sitemap.ts
│ │ ├── assets
│ │ │ ├── discord-black.svg
│ │ │ ├── discord-white.svg
│ │ │ ├── logo-black.svg
│ │ │ └── logo-white.svg
│ │ ├── components
│ │ │ ├── analytics-tracker.tsx
│ │ │ ├── cookie-consent.tsx
│ │ │ ├── doc-actions-menu.tsx
│ │ │ ├── editable-code-block.tsx
│ │ │ ├── footer.tsx
│ │ │ ├── hero.tsx
│ │ │ ├── iou.tsx
│ │ │ ├── mermaid.tsx
│ │ │ └── page-feedback.tsx
│ │ ├── lib
│ │ │ ├── llms.ts
│ │ │ └── source.ts
│ │ ├── mdx-components.tsx
│ │ └── providers
│ │ └── posthog-provider.tsx
│ └── tsconfig.json
├── examples
│ ├── agent_examples.py
│ ├── agent_ui_examples.py
│ ├── browser_agent_example.py
│ ├── browser_tool_example.py
│ ├── cloud_api_examples.py
│ ├── computer_examples_windows.py
│ ├── computer_examples.py
│ ├── computer_ui_examples.py
│ ├── computer-example-ts
│ │ ├── .env.example
│ │ ├── .gitignore
│ │ ├── package-lock.json
│ │ ├── package.json
│ │ ├── pnpm-lock.yaml
│ │ ├── README.md
│ │ ├── src
│ │ │ ├── helpers.ts
│ │ │ └── index.ts
│ │ └── tsconfig.json
│ ├── docker_examples.py
│ ├── evals
│ │ ├── hud_eval_examples.py
│ │ └── wikipedia_most_linked.txt
│ ├── pylume_examples.py
│ ├── sandboxed_functions_examples.py
│ ├── som_examples.py
│ ├── tracing_examples.py
│ ├── utils.py
│ └── winsandbox_example.py
├── img
│ ├── agent_gradio_ui.png
│ ├── agent.png
│ ├── cli.png
│ ├── computer.png
│ ├── logo_black.png
│ └── logo_white.png
├── libs
│ ├── kasm
│ │ ├── Dockerfile
│ │ ├── LICENSE
│ │ ├── README.md
│ │ └── src
│ │ └── ubuntu
│ │ └── install
│ │ └── firefox
│ │ ├── custom_startup.sh
│ │ ├── firefox.desktop
│ │ └── install_firefox.sh
│ ├── lume
│ │ ├── .cursorignore
│ │ ├── CONTRIBUTING.md
│ │ ├── Development.md
│ │ ├── img
│ │ │ └── cli.png
│ │ ├── Package.resolved
│ │ ├── Package.swift
│ │ ├── README.md
│ │ ├── resources
│ │ │ └── lume.entitlements
│ │ ├── scripts
│ │ │ ├── build
│ │ │ │ ├── build-debug.sh
│ │ │ │ ├── build-release-notarized.sh
│ │ │ │ └── build-release.sh
│ │ │ └── install.sh
│ │ ├── src
│ │ │ ├── Commands
│ │ │ │ ├── Clone.swift
│ │ │ │ ├── Config.swift
│ │ │ │ ├── Create.swift
│ │ │ │ ├── Delete.swift
│ │ │ │ ├── Get.swift
│ │ │ │ ├── Images.swift
│ │ │ │ ├── IPSW.swift
│ │ │ │ ├── List.swift
│ │ │ │ ├── Logs.swift
│ │ │ │ ├── Options
│ │ │ │ │ └── FormatOption.swift
│ │ │ │ ├── Prune.swift
│ │ │ │ ├── Pull.swift
│ │ │ │ ├── Push.swift
│ │ │ │ ├── Run.swift
│ │ │ │ ├── Serve.swift
│ │ │ │ ├── Set.swift
│ │ │ │ └── Stop.swift
│ │ │ ├── ContainerRegistry
│ │ │ │ ├── ImageContainerRegistry.swift
│ │ │ │ ├── ImageList.swift
│ │ │ │ └── ImagesPrinter.swift
│ │ │ ├── Errors
│ │ │ │ └── Errors.swift
│ │ │ ├── FileSystem
│ │ │ │ ├── Home.swift
│ │ │ │ ├── Settings.swift
│ │ │ │ ├── VMConfig.swift
│ │ │ │ ├── VMDirectory.swift
│ │ │ │ └── VMLocation.swift
│ │ │ ├── LumeController.swift
│ │ │ ├── Main.swift
│ │ │ ├── Server
│ │ │ │ ├── Handlers.swift
│ │ │ │ ├── HTTP.swift
│ │ │ │ ├── Requests.swift
│ │ │ │ ├── Responses.swift
│ │ │ │ └── Server.swift
│ │ │ ├── Utils
│ │ │ │ ├── CommandRegistry.swift
│ │ │ │ ├── CommandUtils.swift
│ │ │ │ ├── Logger.swift
│ │ │ │ ├── NetworkUtils.swift
│ │ │ │ ├── Path.swift
│ │ │ │ ├── ProcessRunner.swift
│ │ │ │ ├── ProgressLogger.swift
│ │ │ │ ├── String.swift
│ │ │ │ └── Utils.swift
│ │ │ ├── Virtualization
│ │ │ │ ├── DarwinImageLoader.swift
│ │ │ │ ├── DHCPLeaseParser.swift
│ │ │ │ ├── ImageLoaderFactory.swift
│ │ │ │ └── VMVirtualizationService.swift
│ │ │ ├── VM
│ │ │ │ ├── DarwinVM.swift
│ │ │ │ ├── LinuxVM.swift
│ │ │ │ ├── VM.swift
│ │ │ │ ├── VMDetails.swift
│ │ │ │ ├── VMDetailsPrinter.swift
│ │ │ │ ├── VMDisplayResolution.swift
│ │ │ │ └── VMFactory.swift
│ │ │ └── VNC
│ │ │ ├── PassphraseGenerator.swift
│ │ │ └── VNCService.swift
│ │ └── tests
│ │ ├── Mocks
│ │ │ ├── MockVM.swift
│ │ │ ├── MockVMVirtualizationService.swift
│ │ │ └── MockVNCService.swift
│ │ ├── VM
│ │ │ └── VMDetailsPrinterTests.swift
│ │ ├── VMTests.swift
│ │ ├── VMVirtualizationServiceTests.swift
│ │ └── VNCServiceTests.swift
│ ├── lumier
│ │ ├── .dockerignore
│ │ ├── Dockerfile
│ │ ├── README.md
│ │ └── src
│ │ ├── bin
│ │ │ └── entry.sh
│ │ ├── config
│ │ │ └── constants.sh
│ │ ├── hooks
│ │ │ └── on-logon.sh
│ │ └── lib
│ │ ├── utils.sh
│ │ └── vm.sh
│ ├── python
│ │ ├── agent
│ │ │ ├── .bumpversion.cfg
│ │ │ ├── agent
│ │ │ │ ├── __init__.py
│ │ │ │ ├── __main__.py
│ │ │ │ ├── adapters
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── azure_ml_adapter.py
│ │ │ │ │ ├── cua_adapter.py
│ │ │ │ │ ├── huggingfacelocal_adapter.py
│ │ │ │ │ ├── human_adapter.py
│ │ │ │ │ ├── mlxvlm_adapter.py
│ │ │ │ │ └── models
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── generic.py
│ │ │ │ │ ├── internvl.py
│ │ │ │ │ ├── opencua.py
│ │ │ │ │ └── qwen2_5_vl.py
│ │ │ │ ├── agent.py
│ │ │ │ ├── callbacks
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── budget_manager.py
│ │ │ │ │ ├── image_retention.py
│ │ │ │ │ ├── logging.py
│ │ │ │ │ ├── operator_validator.py
│ │ │ │ │ ├── pii_anonymization.py
│ │ │ │ │ ├── prompt_instructions.py
│ │ │ │ │ ├── telemetry.py
│ │ │ │ │ └── trajectory_saver.py
│ │ │ │ ├── cli.py
│ │ │ │ ├── computers
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── cua.py
│ │ │ │ │ └── custom.py
│ │ │ │ ├── decorators.py
│ │ │ │ ├── human_tool
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── __main__.py
│ │ │ │ │ ├── server.py
│ │ │ │ │ └── ui.py
│ │ │ │ ├── integrations
│ │ │ │ │ └── hud
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── agent.py
│ │ │ │ │ └── proxy.py
│ │ │ │ ├── loops
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── anthropic.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── composed_grounded.py
│ │ │ │ │ ├── fara.py
│ │ │ │ │ ├── gelato.py
│ │ │ │ │ ├── gemini.py
│ │ │ │ │ ├── generic_vlm.py
│ │ │ │ │ ├── glm45v.py
│ │ │ │ │ ├── gta1.py
│ │ │ │ │ ├── holo.py
│ │ │ │ │ ├── internvl.py
│ │ │ │ │ ├── model_types.csv
│ │ │ │ │ ├── moondream3.py
│ │ │ │ │ ├── omniparser.py
│ │ │ │ │ ├── openai.py
│ │ │ │ │ ├── opencua.py
│ │ │ │ │ ├── uiins.py
│ │ │ │ │ ├── uitars.py
│ │ │ │ │ └── uitars2.py
│ │ │ │ ├── playground
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ └── server.py
│ │ │ │ ├── proxy
│ │ │ │ │ ├── examples.py
│ │ │ │ │ └── handlers.py
│ │ │ │ ├── responses.py
│ │ │ │ ├── tools
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ └── browser_tool.py
│ │ │ │ ├── types.py
│ │ │ │ └── ui
│ │ │ │ ├── __init__.py
│ │ │ │ ├── __main__.py
│ │ │ │ └── gradio
│ │ │ │ ├── __init__.py
│ │ │ │ ├── app.py
│ │ │ │ └── ui_components.py
│ │ │ ├── benchmarks
│ │ │ │ ├── .gitignore
│ │ │ │ ├── contrib.md
│ │ │ │ ├── interactive.py
│ │ │ │ ├── models
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ └── gta1.py
│ │ │ │ ├── README.md
│ │ │ │ ├── ss-pro.py
│ │ │ │ ├── ss-v2.py
│ │ │ │ └── utils.py
│ │ │ ├── example.py
│ │ │ ├── pyproject.toml
│ │ │ ├── README.md
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ └── test_computer_agent.py
│ │ ├── bench-ui
│ │ │ ├── bench_ui
│ │ │ │ ├── __init__.py
│ │ │ │ ├── api.py
│ │ │ │ └── child.py
│ │ │ ├── examples
│ │ │ │ ├── folder_example.py
│ │ │ │ ├── gui
│ │ │ │ │ ├── index.html
│ │ │ │ │ ├── logo.svg
│ │ │ │ │ └── styles.css
│ │ │ │ ├── output_overlay.png
│ │ │ │ └── simple_example.py
│ │ │ ├── pyproject.toml
│ │ │ ├── README.md
│ │ │ └── tests
│ │ │ └── test_port_detection.py
│ │ ├── computer
│ │ │ ├── .bumpversion.cfg
│ │ │ ├── computer
│ │ │ │ ├── __init__.py
│ │ │ │ ├── computer.py
│ │ │ │ ├── diorama_computer.py
│ │ │ │ ├── helpers.py
│ │ │ │ ├── interface
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── factory.py
│ │ │ │ │ ├── generic.py
│ │ │ │ │ ├── linux.py
│ │ │ │ │ ├── macos.py
│ │ │ │ │ ├── models.py
│ │ │ │ │ └── windows.py
│ │ │ │ ├── logger.py
│ │ │ │ ├── models.py
│ │ │ │ ├── providers
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── cloud
│ │ │ │ │ │ ├── __init__.py
│ │ │ │ │ │ └── provider.py
│ │ │ │ │ ├── docker
│ │ │ │ │ │ ├── __init__.py
│ │ │ │ │ │ └── provider.py
│ │ │ │ │ ├── factory.py
│ │ │ │ │ ├── lume
│ │ │ │ │ │ ├── __init__.py
│ │ │ │ │ │ └── provider.py
│ │ │ │ │ ├── lume_api.py
│ │ │ │ │ ├── lumier
│ │ │ │ │ │ ├── __init__.py
│ │ │ │ │ │ └── provider.py
│ │ │ │ │ ├── types.py
│ │ │ │ │ └── winsandbox
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── provider.py
│ │ │ │ │ └── setup_script.ps1
│ │ │ │ ├── tracing_wrapper.py
│ │ │ │ ├── tracing.py
│ │ │ │ ├── ui
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── __main__.py
│ │ │ │ │ └── gradio
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ └── app.py
│ │ │ │ └── utils.py
│ │ │ ├── poetry.toml
│ │ │ ├── pyproject.toml
│ │ │ ├── README.md
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ ├── test_computer.py
│ │ │ └── test_helpers.py
│ │ ├── computer-server
│ │ │ ├── .bumpversion.cfg
│ │ │ ├── computer_server
│ │ │ │ ├── __init__.py
│ │ │ │ ├── __main__.py
│ │ │ │ ├── browser.py
│ │ │ │ ├── cli.py
│ │ │ │ ├── diorama
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── diorama_computer.py
│ │ │ │ │ ├── diorama.py
│ │ │ │ │ ├── draw.py
│ │ │ │ │ ├── macos.py
│ │ │ │ │ └── safezone.py
│ │ │ │ ├── handlers
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── factory.py
│ │ │ │ │ ├── generic.py
│ │ │ │ │ ├── linux.py
│ │ │ │ │ ├── macos.py
│ │ │ │ │ └── windows.py
│ │ │ │ ├── main.py
│ │ │ │ ├── server.py
│ │ │ │ ├── utils
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ └── wallpaper.py
│ │ │ │ └── watchdog.py
│ │ │ ├── examples
│ │ │ │ ├── __init__.py
│ │ │ │ └── usage_example.py
│ │ │ ├── pyproject.toml
│ │ │ ├── README.md
│ │ │ ├── run_server.py
│ │ │ ├── test_connection.py
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ └── test_server.py
│ │ ├── core
│ │ │ ├── .bumpversion.cfg
│ │ │ ├── core
│ │ │ │ ├── __init__.py
│ │ │ │ └── telemetry
│ │ │ │ ├── __init__.py
│ │ │ │ └── posthog.py
│ │ │ ├── poetry.toml
│ │ │ ├── pyproject.toml
│ │ │ ├── README.md
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ └── test_telemetry.py
│ │ ├── mcp-server
│ │ │ ├── .bumpversion.cfg
│ │ │ ├── build-extension.py
│ │ │ ├── CONCURRENT_SESSIONS.md
│ │ │ ├── desktop-extension
│ │ │ │ ├── cua-extension.mcpb
│ │ │ │ ├── desktop_extension.png
│ │ │ │ ├── manifest.json
│ │ │ │ ├── README.md
│ │ │ │ ├── requirements.txt
│ │ │ │ ├── run_server.sh
│ │ │ │ └── setup.py
│ │ │ ├── mcp_server
│ │ │ │ ├── __init__.py
│ │ │ │ ├── __main__.py
│ │ │ │ ├── server.py
│ │ │ │ └── session_manager.py
│ │ │ ├── pdm.lock
│ │ │ ├── pyproject.toml
│ │ │ ├── QUICK_TEST_COMMANDS.sh
│ │ │ ├── quick_test_local_option.py
│ │ │ ├── README.md
│ │ │ ├── scripts
│ │ │ │ ├── install_mcp_server.sh
│ │ │ │ └── start_mcp_server.sh
│ │ │ ├── test_mcp_server_local_option.py
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ └── test_mcp_server.py
│ │ ├── pylume
│ │ │ └── tests
│ │ │ ├── conftest.py
│ │ │ └── test_pylume.py
│ │ └── som
│ │ ├── .bumpversion.cfg
│ │ ├── LICENSE
│ │ ├── poetry.toml
│ │ ├── pyproject.toml
│ │ ├── README.md
│ │ ├── som
│ │ │ ├── __init__.py
│ │ │ ├── detect.py
│ │ │ ├── detection.py
│ │ │ ├── models.py
│ │ │ ├── ocr.py
│ │ │ ├── util
│ │ │ │ └── utils.py
│ │ │ └── visualization.py
│ │ └── tests
│ │ ├── conftest.py
│ │ └── test_omniparser.py
│ ├── qemu-docker
│ │ ├── linux
│ │ │ ├── Dockerfile
│ │ │ ├── README.md
│ │ │ └── src
│ │ │ ├── entry.sh
│ │ │ └── vm
│ │ │ ├── image
│ │ │ │ └── README.md
│ │ │ └── setup
│ │ │ ├── install.sh
│ │ │ ├── setup-cua-server.sh
│ │ │ └── setup.sh
│ │ ├── README.md
│ │ └── windows
│ │ ├── Dockerfile
│ │ ├── README.md
│ │ └── src
│ │ ├── entry.sh
│ │ └── vm
│ │ ├── image
│ │ │ └── README.md
│ │ └── setup
│ │ ├── install.bat
│ │ ├── on-logon.ps1
│ │ ├── setup-cua-server.ps1
│ │ ├── setup-utils.psm1
│ │ └── setup.ps1
│ ├── typescript
│ │ ├── .gitignore
│ │ ├── .nvmrc
│ │ ├── agent
│ │ │ ├── examples
│ │ │ │ ├── playground-example.html
│ │ │ │ └── README.md
│ │ │ ├── package.json
│ │ │ ├── README.md
│ │ │ ├── src
│ │ │ │ ├── client.ts
│ │ │ │ ├── index.ts
│ │ │ │ └── types.ts
│ │ │ ├── tests
│ │ │ │ └── client.test.ts
│ │ │ ├── tsconfig.json
│ │ │ ├── tsdown.config.ts
│ │ │ └── vitest.config.ts
│ │ ├── computer
│ │ │ ├── .editorconfig
│ │ │ ├── .gitattributes
│ │ │ ├── .gitignore
│ │ │ ├── LICENSE
│ │ │ ├── package.json
│ │ │ ├── README.md
│ │ │ ├── src
│ │ │ │ ├── computer
│ │ │ │ │ ├── index.ts
│ │ │ │ │ ├── providers
│ │ │ │ │ │ ├── base.ts
│ │ │ │ │ │ ├── cloud.ts
│ │ │ │ │ │ └── index.ts
│ │ │ │ │ └── types.ts
│ │ │ │ ├── index.ts
│ │ │ │ ├── interface
│ │ │ │ │ ├── base.ts
│ │ │ │ │ ├── factory.ts
│ │ │ │ │ ├── index.ts
│ │ │ │ │ ├── linux.ts
│ │ │ │ │ ├── macos.ts
│ │ │ │ │ └── windows.ts
│ │ │ │ └── types.ts
│ │ │ ├── tests
│ │ │ │ ├── computer
│ │ │ │ │ └── cloud.test.ts
│ │ │ │ ├── interface
│ │ │ │ │ ├── factory.test.ts
│ │ │ │ │ ├── index.test.ts
│ │ │ │ │ ├── linux.test.ts
│ │ │ │ │ ├── macos.test.ts
│ │ │ │ │ └── windows.test.ts
│ │ │ │ └── setup.ts
│ │ │ ├── tsconfig.json
│ │ │ ├── tsdown.config.ts
│ │ │ └── vitest.config.ts
│ │ ├── core
│ │ │ ├── .editorconfig
│ │ │ ├── .gitattributes
│ │ │ ├── .gitignore
│ │ │ ├── LICENSE
│ │ │ ├── package.json
│ │ │ ├── README.md
│ │ │ ├── src
│ │ │ │ ├── index.ts
│ │ │ │ └── telemetry
│ │ │ │ ├── clients
│ │ │ │ │ ├── index.ts
│ │ │ │ │ └── posthog.ts
│ │ │ │ └── index.ts
│ │ │ ├── tests
│ │ │ │ └── telemetry.test.ts
│ │ │ ├── tsconfig.json
│ │ │ ├── tsdown.config.ts
│ │ │ └── vitest.config.ts
│ │ ├── cua-cli
│ │ │ ├── .gitignore
│ │ │ ├── .prettierrc
│ │ │ ├── bun.lock
│ │ │ ├── CLAUDE.md
│ │ │ ├── index.ts
│ │ │ ├── package.json
│ │ │ ├── README.md
│ │ │ ├── src
│ │ │ │ ├── auth.ts
│ │ │ │ ├── cli.ts
│ │ │ │ ├── commands
│ │ │ │ │ ├── auth.ts
│ │ │ │ │ └── sandbox.ts
│ │ │ │ ├── config.ts
│ │ │ │ ├── http.ts
│ │ │ │ ├── storage.ts
│ │ │ │ └── util.ts
│ │ │ └── tsconfig.json
│ │ ├── package.json
│ │ ├── pnpm-lock.yaml
│ │ ├── pnpm-workspace.yaml
│ │ └── README.md
│ └── xfce
│ ├── .dockerignore
│ ├── .gitignore
│ ├── Development.md
│ ├── Dockerfile
│ ├── Dockerfile.dev
│ ├── README.md
│ └── src
│ ├── scripts
│ │ ├── resize-display.sh
│ │ ├── start-computer-server.sh
│ │ ├── start-novnc.sh
│ │ ├── start-vnc.sh
│ │ └── xstartup.sh
│ ├── supervisor
│ │ └── supervisord.conf
│ └── xfce-config
│ ├── helpers.rc
│ ├── xfce4-power-manager.xml
│ └── xfce4-session.xml
├── LICENSE.md
├── Makefile
├── notebooks
│ ├── agent_nb.ipynb
│ ├── blog
│ │ ├── build-your-own-operator-on-macos-1.ipynb
│ │ └── build-your-own-operator-on-macos-2.ipynb
│ ├── composite_agents_docker_nb.ipynb
│ ├── computer_nb.ipynb
│ ├── computer_server_nb.ipynb
│ ├── customizing_computeragent.ipynb
│ ├── eval_osworld.ipynb
│ ├── ollama_nb.ipynb
│ ├── README.md
│ ├── sota_hackathon_cloud.ipynb
│ └── sota_hackathon.ipynb
├── package-lock.json
├── package.json
├── pnpm-lock.yaml
├── pyproject.toml
├── pyrightconfig.json
├── README.md
├── scripts
│ ├── install-cli.ps1
│ ├── install-cli.sh
│ ├── playground-docker.sh
│ ├── playground.sh
│ ├── run-docker-dev.sh
│ └── typescript-typecheck.js
├── TESTING.md
├── tests
│ ├── agent_loop_testing
│ │ ├── agent_test.py
│ │ └── README.md
│ ├── pytest.ini
│ ├── shell_cmd.py
│ ├── test_files.py
│ ├── test_mcp_server_session_management.py
│ ├── test_mcp_server_streaming.py
│ ├── test_shell_bash.py
│ ├── test_telemetry.py
│ ├── test_tracing.py
│ ├── test_venv.py
│ └── test_watchdog.py
└── uv.lock
```
# Files
--------------------------------------------------------------------------------
/libs/typescript/computer/tests/interface/macos.test.ts:
--------------------------------------------------------------------------------
```typescript
1 | import { afterEach, beforeEach, describe, expect, it } from 'vitest';
2 | import { WebSocket, WebSocketServer } from 'ws';
3 | import { MacOSComputerInterface } from '../../src/interface/macos.ts';
4 |
5 | describe('MacOSComputerInterface', () => {
6 | // Define test parameters
7 | const testParams = {
8 | ipAddress: 'localhost',
9 | username: 'testuser',
10 | password: 'testpass',
11 | // apiKey: "test-api-key", No API Key for local testing
12 | vmName: 'test-vm',
13 | };
14 |
15 | // WebSocket server mock
16 | let wss: WebSocketServer;
17 | let serverPort: number;
18 | let connectedClients: WebSocket[] = [];
19 |
20 | // Track received messages for verification
21 | interface ReceivedMessage {
22 | action: string;
23 | [key: string]: unknown;
24 | }
25 | let receivedMessages: ReceivedMessage[] = [];
26 |
27 | // Set up WebSocket server before all tests
28 | beforeEach(async () => {
29 | receivedMessages = [];
30 | connectedClients = [];
31 |
32 | // Create WebSocket server on a random available port
33 | wss = new WebSocketServer({ port: 0 });
34 | serverPort = (wss.address() as { port: number }).port;
35 |
36 | // Update test params with the actual server address
37 | testParams.ipAddress = `localhost:${serverPort}`;
38 |
39 | // Handle WebSocket connections
40 | wss.on('connection', (ws) => {
41 | connectedClients.push(ws);
42 |
43 | // Handle incoming messages
44 | ws.on('message', (data) => {
45 | try {
46 | const message = JSON.parse(data.toString());
47 | receivedMessages.push(message);
48 |
49 | // Send appropriate responses based on action
50 | switch (message.command) {
51 | case 'screenshot':
52 | ws.send(
53 | JSON.stringify({
54 | image_data: Buffer.from('fake-screenshot-data').toString('base64'),
55 | success: true,
56 | })
57 | );
58 | break;
59 | case 'get_screen_size':
60 | ws.send(
61 | JSON.stringify({
62 | size: { width: 1920, height: 1080 },
63 | success: true,
64 | })
65 | );
66 | break;
67 | case 'get_cursor_position':
68 | ws.send(
69 | JSON.stringify({
70 | position: { x: 100, y: 200 },
71 | success: true,
72 | })
73 | );
74 | break;
75 | case 'copy_to_clipboard':
76 | ws.send(
77 | JSON.stringify({
78 | content: 'clipboard content',
79 | success: true,
80 | })
81 | );
82 | break;
83 | case 'file_exists':
84 | ws.send(
85 | JSON.stringify({
86 | exists: true,
87 | success: true,
88 | })
89 | );
90 | break;
91 | case 'directory_exists':
92 | ws.send(
93 | JSON.stringify({
94 | exists: true,
95 | success: true,
96 | })
97 | );
98 | break;
99 | case 'list_dir':
100 | ws.send(
101 | JSON.stringify({
102 | files: ['file1.txt', 'file2.txt'],
103 | success: true,
104 | })
105 | );
106 | break;
107 | case 'read_text':
108 | ws.send(
109 | JSON.stringify({
110 | content: 'file content',
111 | success: true,
112 | })
113 | );
114 | break;
115 | case 'read_bytes':
116 | ws.send(
117 | JSON.stringify({
118 | content_b64: Buffer.from('binary content').toString('base64'),
119 | success: true,
120 | })
121 | );
122 | break;
123 | case 'run_command':
124 | ws.send(
125 | JSON.stringify({
126 | stdout: 'command output',
127 | stderr: '',
128 | success: true,
129 | })
130 | );
131 | break;
132 | case 'get_accessibility_tree':
133 | ws.send(
134 | JSON.stringify({
135 | role: 'window',
136 | title: 'Test Window',
137 | bounds: { x: 0, y: 0, width: 1920, height: 1080 },
138 | children: [],
139 | success: true,
140 | })
141 | );
142 | break;
143 | case 'to_screen_coordinates':
144 | case 'to_screenshot_coordinates':
145 | ws.send(
146 | JSON.stringify({
147 | coordinates: [message.params?.x || 0, message.params?.y || 0],
148 | success: true,
149 | })
150 | );
151 | break;
152 | default:
153 | // For all other actions, just send success
154 | ws.send(JSON.stringify({ success: true }));
155 | break;
156 | }
157 | } catch (error) {
158 | ws.send(JSON.stringify({ error: (error as Error).message }));
159 | }
160 | });
161 |
162 | ws.on('error', (error) => {
163 | console.error('WebSocket error:', error);
164 | });
165 | });
166 | });
167 |
168 | // Clean up WebSocket server after each test
169 | afterEach(async () => {
170 | // Close all connected clients
171 | for (const client of connectedClients) {
172 | if (client.readyState === WebSocket.OPEN) {
173 | client.close();
174 | }
175 | }
176 |
177 | // Close the server
178 | await new Promise<void>((resolve) => {
179 | wss.close(() => resolve());
180 | });
181 | });
182 |
183 | describe('Connection Management', () => {
184 | it('should connect with proper authentication headers', async () => {
185 | const macosInterface = new MacOSComputerInterface(
186 | testParams.ipAddress,
187 | testParams.username,
188 | testParams.password,
189 | undefined,
190 | testParams.vmName
191 | );
192 |
193 | await macosInterface.connect();
194 |
195 | // Verify the interface is connected
196 | expect(macosInterface.isConnected()).toBe(true);
197 | expect(connectedClients.length).toBe(1);
198 |
199 | await macosInterface.disconnect();
200 | });
201 |
202 | it('should handle connection without API key', async () => {
203 | // Create a separate server that doesn't check auth
204 | const noAuthWss = new WebSocketServer({ port: 0 });
205 | const noAuthPort = (noAuthWss.address() as { port: number }).port;
206 |
207 | noAuthWss.on('connection', (ws) => {
208 | ws.on('message', () => {
209 | ws.send(JSON.stringify({ success: true }));
210 | });
211 | });
212 |
213 | const macosInterface = new MacOSComputerInterface(
214 | `localhost:${noAuthPort}`,
215 | testParams.username,
216 | testParams.password,
217 | undefined,
218 | undefined
219 | );
220 |
221 | await macosInterface.connect();
222 | expect(macosInterface.isConnected()).toBe(true);
223 |
224 | await macosInterface.disconnect();
225 | await new Promise<void>((resolve) => {
226 | noAuthWss.close(() => resolve());
227 | });
228 | });
229 | });
230 |
231 | describe('Mouse Actions', () => {
232 | let macosInterface: MacOSComputerInterface;
233 |
234 | beforeEach(async () => {
235 | macosInterface = new MacOSComputerInterface(
236 | testParams.ipAddress,
237 | testParams.username,
238 | testParams.password,
239 | undefined,
240 | testParams.vmName
241 | );
242 | await macosInterface.connect();
243 | });
244 |
245 | afterEach(async () => {
246 | if (macosInterface) {
247 | await macosInterface.disconnect();
248 | }
249 | });
250 |
251 | it('should send mouse_down command', async () => {
252 | await macosInterface.mouseDown(100, 200, 'left');
253 |
254 | const lastMessage = receivedMessages[receivedMessages.length - 1];
255 | expect(lastMessage).toEqual({
256 | command: 'mouse_down',
257 | params: {
258 | x: 100,
259 | y: 200,
260 | button: 'left',
261 | },
262 | });
263 | });
264 |
265 | it('should send mouse_up command', async () => {
266 | await macosInterface.mouseUp(100, 200, 'right');
267 |
268 | const lastMessage = receivedMessages[receivedMessages.length - 1];
269 | expect(lastMessage).toEqual({
270 | command: 'mouse_up',
271 | params: {
272 | x: 100,
273 | y: 200,
274 | button: 'right',
275 | },
276 | });
277 | });
278 |
279 | it('should send left_click command', async () => {
280 | await macosInterface.leftClick(150, 250);
281 |
282 | const lastMessage = receivedMessages[receivedMessages.length - 1];
283 | expect(lastMessage).toEqual({
284 | command: 'left_click',
285 | params: {
286 | x: 150,
287 | y: 250,
288 | },
289 | });
290 | });
291 |
292 | it('should send right_click command', async () => {
293 | await macosInterface.rightClick(200, 300);
294 |
295 | const lastMessage = receivedMessages[receivedMessages.length - 1];
296 | expect(lastMessage).toEqual({
297 | command: 'right_click',
298 | params: {
299 | x: 200,
300 | y: 300,
301 | },
302 | });
303 | });
304 |
305 | it('should send double_click command', async () => {
306 | await macosInterface.doubleClick(250, 350);
307 |
308 | const lastMessage = receivedMessages[receivedMessages.length - 1];
309 | expect(lastMessage).toEqual({
310 | command: 'double_click',
311 | params: {
312 | x: 250,
313 | y: 350,
314 | },
315 | });
316 | });
317 |
318 | it('should send move_cursor command', async () => {
319 | await macosInterface.moveCursor(300, 400);
320 |
321 | const lastMessage = receivedMessages[receivedMessages.length - 1];
322 | expect(lastMessage).toEqual({
323 | command: 'move_cursor',
324 | params: {
325 | x: 300,
326 | y: 400,
327 | },
328 | });
329 | });
330 |
331 | it('should send drag_to command', async () => {
332 | await macosInterface.dragTo(400, 500, 'left', 1.5);
333 |
334 | const lastMessage = receivedMessages[receivedMessages.length - 1];
335 | expect(lastMessage).toEqual({
336 | command: 'drag_to',
337 | params: {
338 | x: 400,
339 | y: 500,
340 | button: 'left',
341 | duration: 1.5,
342 | },
343 | });
344 | });
345 |
346 | it('should send drag command with path', async () => {
347 | const path: Array<[number, number]> = [
348 | [100, 100],
349 | [200, 200],
350 | [300, 300],
351 | ];
352 | await macosInterface.drag(path, 'middle', 2.0);
353 |
354 | const lastMessage = receivedMessages[receivedMessages.length - 1];
355 | expect(lastMessage).toEqual({
356 | command: 'drag',
357 | params: {
358 | path: path,
359 | button: 'middle',
360 | duration: 2.0,
361 | },
362 | });
363 | });
364 | });
365 |
366 | describe('Keyboard Actions', () => {
367 | let macosInterface: MacOSComputerInterface;
368 |
369 | beforeEach(async () => {
370 | macosInterface = new MacOSComputerInterface(
371 | testParams.ipAddress,
372 | testParams.username,
373 | testParams.password,
374 | undefined,
375 | testParams.vmName
376 | );
377 | await macosInterface.connect();
378 | });
379 |
380 | afterEach(async () => {
381 | if (macosInterface) {
382 | await macosInterface.disconnect();
383 | }
384 | });
385 |
386 | it('should send key_down command', async () => {
387 | await macosInterface.keyDown('a');
388 |
389 | const lastMessage = receivedMessages[receivedMessages.length - 1];
390 | expect(lastMessage).toEqual({
391 | command: 'key_down',
392 | params: {
393 | key: 'a',
394 | },
395 | });
396 | });
397 |
398 | it('should send key_up command', async () => {
399 | await macosInterface.keyUp('b');
400 |
401 | const lastMessage = receivedMessages[receivedMessages.length - 1];
402 | expect(lastMessage).toEqual({
403 | command: 'key_up',
404 | params: {
405 | key: 'b',
406 | },
407 | });
408 | });
409 |
410 | it('should send type_text command', async () => {
411 | await macosInterface.typeText('Hello, World!');
412 |
413 | const lastMessage = receivedMessages[receivedMessages.length - 1];
414 | expect(lastMessage).toEqual({
415 | command: 'type_text',
416 | params: {
417 | text: 'Hello, World!',
418 | },
419 | });
420 | });
421 |
422 | it('should send press_key command', async () => {
423 | await macosInterface.pressKey('enter');
424 |
425 | const lastMessage = receivedMessages[receivedMessages.length - 1];
426 | expect(lastMessage).toEqual({
427 | command: 'press_key',
428 | params: {
429 | key: 'enter',
430 | },
431 | });
432 | });
433 |
434 | it('should send hotkey command', async () => {
435 | await macosInterface.hotkey('cmd', 'c');
436 |
437 | const lastMessage = receivedMessages[receivedMessages.length - 1];
438 | expect(lastMessage).toEqual({
439 | command: 'hotkey',
440 | params: {
441 | keys: ['cmd', 'c'],
442 | },
443 | });
444 | });
445 | });
446 |
447 | describe('Scrolling Actions', () => {
448 | let macosInterface: MacOSComputerInterface;
449 |
450 | beforeEach(async () => {
451 | macosInterface = new MacOSComputerInterface(
452 | testParams.ipAddress,
453 | testParams.username,
454 | testParams.password,
455 | undefined,
456 | testParams.vmName
457 | );
458 | await macosInterface.connect();
459 | });
460 |
461 | afterEach(async () => {
462 | if (macosInterface) {
463 | await macosInterface.disconnect();
464 | }
465 | });
466 |
467 | it('should send scroll command', async () => {
468 | await macosInterface.scroll(10, -5);
469 |
470 | const lastMessage = receivedMessages[receivedMessages.length - 1];
471 | expect(lastMessage).toEqual({
472 | command: 'scroll',
473 | params: {
474 | x: 10,
475 | y: -5,
476 | },
477 | });
478 | });
479 |
480 | it('should send scroll_down command', async () => {
481 | await macosInterface.scrollDown(3);
482 |
483 | const lastMessage = receivedMessages[receivedMessages.length - 1];
484 | expect(lastMessage).toEqual({
485 | command: 'scroll_down',
486 | params: {
487 | clicks: 3,
488 | },
489 | });
490 | });
491 |
492 | it('should send scroll_up command', async () => {
493 | await macosInterface.scrollUp(2);
494 |
495 | const lastMessage = receivedMessages[receivedMessages.length - 1];
496 | expect(lastMessage).toEqual({
497 | command: 'scroll_up',
498 | params: {
499 | clicks: 2,
500 | },
501 | });
502 | });
503 | });
504 |
505 | describe('Screen Actions', () => {
506 | let macosInterface: MacOSComputerInterface;
507 |
508 | beforeEach(async () => {
509 | macosInterface = new MacOSComputerInterface(
510 | testParams.ipAddress,
511 | testParams.username,
512 | testParams.password,
513 | undefined,
514 | testParams.vmName
515 | );
516 | await macosInterface.connect();
517 | });
518 |
519 | afterEach(async () => {
520 | if (macosInterface) {
521 | await macosInterface.disconnect();
522 | }
523 | });
524 |
525 | it('should get screenshot', async () => {
526 | const screenshot = await macosInterface.screenshot();
527 |
528 | expect(screenshot).toBeInstanceOf(Buffer);
529 | expect(screenshot.toString()).toBe('fake-screenshot-data');
530 |
531 | const lastMessage = receivedMessages[receivedMessages.length - 1];
532 | expect(lastMessage).toEqual({
533 | command: 'screenshot',
534 | params: {},
535 | });
536 | });
537 |
538 | it('should get screen size', async () => {
539 | const size = await macosInterface.getScreenSize();
540 |
541 | expect(size).toEqual({ width: 1920, height: 1080 });
542 |
543 | const lastMessage = receivedMessages[receivedMessages.length - 1];
544 | expect(lastMessage).toEqual({
545 | command: 'get_screen_size',
546 | params: {},
547 | });
548 | });
549 |
550 | it('should get cursor position', async () => {
551 | const position = await macosInterface.getCursorPosition();
552 |
553 | expect(position).toEqual({ x: 100, y: 200 });
554 |
555 | const lastMessage = receivedMessages[receivedMessages.length - 1];
556 | expect(lastMessage).toEqual({
557 | command: 'get_cursor_position',
558 | params: {},
559 | });
560 | });
561 | });
562 |
563 | describe('Clipboard Actions', () => {
564 | let macosInterface: MacOSComputerInterface;
565 |
566 | beforeEach(async () => {
567 | macosInterface = new MacOSComputerInterface(
568 | testParams.ipAddress,
569 | testParams.username,
570 | testParams.password,
571 | undefined,
572 | testParams.vmName
573 | );
574 | await macosInterface.connect();
575 | });
576 |
577 | afterEach(async () => {
578 | if (macosInterface) {
579 | await macosInterface.disconnect();
580 | }
581 | });
582 |
583 | it('should copy to clipboard', async () => {
584 | const text = await macosInterface.copyToClipboard();
585 |
586 | expect(text).toBe('clipboard content');
587 |
588 | const lastMessage = receivedMessages[receivedMessages.length - 1];
589 | expect(lastMessage).toEqual({
590 | command: 'copy_to_clipboard',
591 | params: {},
592 | });
593 | });
594 |
595 | it('should set clipboard', async () => {
596 | await macosInterface.setClipboard('new clipboard text');
597 |
598 | const lastMessage = receivedMessages[receivedMessages.length - 1];
599 | expect(lastMessage).toEqual({
600 | command: 'set_clipboard',
601 | params: {
602 | text: 'new clipboard text',
603 | },
604 | });
605 | });
606 | });
607 |
608 | describe('File System Actions', () => {
609 | let macosInterface: MacOSComputerInterface;
610 |
611 | beforeEach(async () => {
612 | macosInterface = new MacOSComputerInterface(
613 | testParams.ipAddress,
614 | testParams.username,
615 | testParams.password,
616 | undefined,
617 | testParams.vmName
618 | );
619 | await macosInterface.connect();
620 | });
621 |
622 | afterEach(async () => {
623 | if (macosInterface) {
624 | await macosInterface.disconnect();
625 | }
626 | });
627 |
628 | it('should check file exists', async () => {
629 | const exists = await macosInterface.fileExists('/path/to/file');
630 |
631 | expect(exists).toBe(true);
632 |
633 | const lastMessage = receivedMessages[receivedMessages.length - 1];
634 | expect(lastMessage).toEqual({
635 | command: 'file_exists',
636 | params: {
637 | path: '/path/to/file',
638 | },
639 | });
640 | });
641 |
642 | it('should check directory exists', async () => {
643 | const exists = await macosInterface.directoryExists('/path/to/dir');
644 |
645 | expect(exists).toBe(true);
646 |
647 | const lastMessage = receivedMessages[receivedMessages.length - 1];
648 | expect(lastMessage).toEqual({
649 | command: 'directory_exists',
650 | params: {
651 | path: '/path/to/dir',
652 | },
653 | });
654 | });
655 |
656 | it('should list directory', async () => {
657 | const files = await macosInterface.listDir('/path/to/dir');
658 |
659 | expect(files).toEqual(['file1.txt', 'file2.txt']);
660 |
661 | const lastMessage = receivedMessages[receivedMessages.length - 1];
662 | expect(lastMessage).toEqual({
663 | command: 'list_dir',
664 | params: {
665 | path: '/path/to/dir',
666 | },
667 | });
668 | });
669 |
670 | it('should read text file', async () => {
671 | const content = await macosInterface.readText('/path/to/file.txt');
672 |
673 | expect(content).toBe('file content');
674 |
675 | const lastMessage = receivedMessages[receivedMessages.length - 1];
676 | expect(lastMessage).toEqual({
677 | command: 'read_text',
678 | params: {
679 | path: '/path/to/file.txt',
680 | },
681 | });
682 | });
683 |
684 | it('should write text file', async () => {
685 | await macosInterface.writeText('/path/to/file.txt', 'new content');
686 |
687 | const lastMessage = receivedMessages[receivedMessages.length - 1];
688 | expect(lastMessage).toEqual({
689 | command: 'write_text',
690 | params: {
691 | path: '/path/to/file.txt',
692 | content: 'new content',
693 | },
694 | });
695 | });
696 |
697 | it('should read binary file', async () => {
698 | const content = await macosInterface.readBytes('/path/to/file.bin');
699 |
700 | expect(content).toBeInstanceOf(Buffer);
701 | expect(content.toString()).toBe('binary content');
702 |
703 | const lastMessage = receivedMessages[receivedMessages.length - 1];
704 | expect(lastMessage).toEqual({
705 | command: 'read_bytes',
706 | params: {
707 | path: '/path/to/file.bin',
708 | },
709 | });
710 | });
711 |
712 | it('should write binary file', async () => {
713 | const buffer = Buffer.from('binary data');
714 | await macosInterface.writeBytes('/path/to/file.bin', buffer);
715 |
716 | const lastMessage = receivedMessages[receivedMessages.length - 1];
717 | expect(lastMessage).toEqual({
718 | command: 'write_bytes',
719 | params: {
720 | path: '/path/to/file.bin',
721 | content_b64: buffer.toString('base64'),
722 | },
723 | });
724 | });
725 |
726 | it('should delete file', async () => {
727 | await macosInterface.deleteFile('/path/to/file');
728 |
729 | const lastMessage = receivedMessages[receivedMessages.length - 1];
730 | expect(lastMessage).toEqual({
731 | command: 'delete_file',
732 | params: {
733 | path: '/path/to/file',
734 | },
735 | });
736 | });
737 |
738 | it('should create directory', async () => {
739 | await macosInterface.createDir('/path/to/new/dir');
740 |
741 | const lastMessage = receivedMessages[receivedMessages.length - 1];
742 | expect(lastMessage).toEqual({
743 | command: 'create_dir',
744 | params: {
745 | path: '/path/to/new/dir',
746 | },
747 | });
748 | });
749 |
750 | it('should delete directory', async () => {
751 | await macosInterface.deleteDir('/path/to/dir');
752 |
753 | const lastMessage = receivedMessages[receivedMessages.length - 1];
754 | expect(lastMessage).toEqual({
755 | command: 'delete_dir',
756 | params: {
757 | path: '/path/to/dir',
758 | },
759 | });
760 | });
761 |
762 | it('should run command', async () => {
763 | const [stdout, stderr] = await macosInterface.runCommand('ls -la');
764 |
765 | expect(stdout).toBe('command output');
766 | expect(stderr).toBe('');
767 |
768 | const lastMessage = receivedMessages[receivedMessages.length - 1];
769 | expect(lastMessage).toEqual({
770 | command: 'run_command',
771 | params: {
772 | command: 'ls -la',
773 | },
774 | });
775 | });
776 | });
777 |
778 | describe('Accessibility Actions', () => {
779 | let macosInterface: MacOSComputerInterface;
780 |
781 | beforeEach(async () => {
782 | macosInterface = new MacOSComputerInterface(
783 | testParams.ipAddress,
784 | testParams.username,
785 | testParams.password,
786 | undefined,
787 | testParams.vmName
788 | );
789 | await macosInterface.connect();
790 | });
791 |
792 | afterEach(async () => {
793 | if (macosInterface) {
794 | await macosInterface.disconnect();
795 | }
796 | });
797 |
798 | it('should get accessibility tree', async () => {
799 | const tree = await macosInterface.getAccessibilityTree();
800 |
801 | expect(tree).toEqual({
802 | role: 'window',
803 | title: 'Test Window',
804 | bounds: { x: 0, y: 0, width: 1920, height: 1080 },
805 | children: [],
806 | success: true,
807 | });
808 |
809 | const lastMessage = receivedMessages[receivedMessages.length - 1];
810 | expect(lastMessage).toEqual({
811 | command: 'get_accessibility_tree',
812 | params: {},
813 | });
814 | });
815 |
816 | it('should convert to screen coordinates', async () => {
817 | const [x, y] = await macosInterface.toScreenCoordinates(100, 200);
818 |
819 | expect(x).toBe(100);
820 | expect(y).toBe(200);
821 |
822 | const lastMessage = receivedMessages[receivedMessages.length - 1];
823 | expect(lastMessage).toEqual({
824 | command: 'to_screen_coordinates',
825 | params: {
826 | x: 100,
827 | y: 200,
828 | },
829 | });
830 | });
831 |
832 | it('should convert to screenshot coordinates', async () => {
833 | const [x, y] = await macosInterface.toScreenshotCoordinates(300, 400);
834 |
835 | expect(x).toBe(300);
836 | expect(y).toBe(400);
837 |
838 | const lastMessage = receivedMessages[receivedMessages.length - 1];
839 | expect(lastMessage).toEqual({
840 | command: 'to_screenshot_coordinates',
841 | params: {
842 | x: 300,
843 | y: 400,
844 | },
845 | });
846 | });
847 | });
848 |
849 | describe('Error Handling', () => {
850 | it('should handle WebSocket connection errors', async () => {
851 | // Use a valid but unreachable IP to avoid DNS errors
852 | const macosInterface = new MacOSComputerInterface(
853 | 'localhost:9999',
854 | testParams.username,
855 | testParams.password,
856 | undefined,
857 | testParams.vmName
858 | );
859 |
860 | // Connection should fail
861 | await expect(macosInterface.connect()).rejects.toThrow();
862 | });
863 |
864 | it('should handle command errors', async () => {
865 | // Create a server that returns errors
866 | const errorWss = new WebSocketServer({ port: 0 });
867 | const errorPort = (errorWss.address() as { port: number }).port;
868 |
869 | errorWss.on('connection', (ws) => {
870 | ws.on('message', () => {
871 | ws.send(JSON.stringify({ error: 'Command failed', success: false }));
872 | });
873 | });
874 |
875 | const macosInterface = new MacOSComputerInterface(
876 | `localhost:${errorPort}`,
877 | testParams.username,
878 | testParams.password,
879 | undefined,
880 | testParams.vmName
881 | );
882 |
883 | await macosInterface.connect();
884 |
885 | // Command should throw error
886 | await expect(macosInterface.leftClick(100, 100)).rejects.toThrow('Command failed');
887 |
888 | await macosInterface.disconnect();
889 | await new Promise<void>((resolve) => {
890 | errorWss.close(() => resolve());
891 | });
892 | });
893 |
894 | it('should handle disconnection gracefully', async () => {
895 | const macosInterface = new MacOSComputerInterface(
896 | testParams.ipAddress,
897 | testParams.username,
898 | testParams.password,
899 | undefined,
900 | testParams.vmName
901 | );
902 |
903 | await macosInterface.connect();
904 | expect(macosInterface.isConnected()).toBe(true);
905 |
906 | // Disconnect
907 | macosInterface.disconnect();
908 | expect(macosInterface.isConnected()).toBe(false);
909 |
910 | // Should reconnect automatically on next command
911 | await macosInterface.leftClick(100, 100);
912 | expect(macosInterface.isConnected()).toBe(true);
913 |
914 | await macosInterface.disconnect();
915 | });
916 |
917 | it('should handle force close', async () => {
918 | const macosInterface = new MacOSComputerInterface(
919 | testParams.ipAddress,
920 | testParams.username,
921 | testParams.password,
922 | undefined,
923 | testParams.vmName
924 | );
925 |
926 | await macosInterface.connect();
927 | expect(macosInterface.isConnected()).toBe(true);
928 |
929 | // Force close
930 | macosInterface.forceClose();
931 | expect(macosInterface.isConnected()).toBe(false);
932 | });
933 | });
934 | });
935 |
```
--------------------------------------------------------------------------------
/docs/content/docs/macos-vm-cli-playbook/lume/http-api.mdx:
--------------------------------------------------------------------------------
```markdown
1 | ---
2 | title: HTTP Server API
3 | description: Lume exposes a local HTTP API server that listens at localhost for programmatic management of VMs.
4 | ---
5 |
6 | import { Tabs, Tab } from 'fumadocs-ui/components/tabs';
7 | import { Callout } from 'fumadocs-ui/components/callout';
8 |
9 | ## Default URL
10 |
11 | ```
12 | http://localhost:7777
13 | ```
14 |
15 | <Callout type="info">
16 | The HTTP API service runs on port `7777` by default. If you'd like to use a different port, pass
17 | the `--port` option during installation or when running `lume serve`.
18 | </Callout>
19 |
20 | ## Endpoints
21 |
22 | ---
23 |
24 | ### Create VM
25 |
26 | Create a new virtual machine.
27 |
28 | `POST: /lume/vms`
29 |
30 | #### Parameters
31 |
32 | | Name | Type | Required | Description |
33 | | -------- | ------- | -------- | ------------------------------------ |
34 | | name | string | Yes | Name of the VM |
35 | | os | string | Yes | Guest OS (`macOS`, `linux`, etc.) |
36 | | cpu | integer | Yes | Number of CPU cores |
37 | | memory | string | Yes | Memory size (e.g. `4GB`) |
38 | | diskSize | string | Yes | Disk size (e.g. `64GB`) |
39 | | display | string | No | Display resolution (e.g. `1024x768`) |
40 | | ipsw | string | No | IPSW version (e.g. `latest`) |
41 | | storage | string | No | Storage type (`ssd`, etc.) |
42 |
43 | #### Example Request
44 |
45 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
46 | <Tab value="Curl">
47 |
48 | ```bash
49 | curl --connect-timeout 6000 \
50 | --max-time 5000 \
51 | -X POST \
52 | -H "Content-Type: application/json" \
53 | -d '{
54 | "name": "lume_vm",
55 | "os": "macOS",
56 | "cpu": 2,
57 | "memory": "4GB",
58 | "diskSize": "64GB",
59 | "display": "1024x768",
60 | "ipsw": "latest",
61 | "storage": "ssd"
62 | }' \
63 | http://localhost:7777/lume/vms
64 | ```
65 |
66 | </Tab>
67 | <Tab value="Python">
68 |
69 | ```python
70 | import requests
71 |
72 | payload = {
73 | "name": "lume_vm",
74 | "os": "macOS",
75 | "cpu": 2,
76 | "memory": "4GB",
77 | "diskSize": "64GB",
78 | "display": "1024x768",
79 | "ipsw": "latest",
80 | "storage": "ssd"
81 | }
82 | r = requests.post("http://localhost:7777/lume/vms", json=payload, timeout=50)
83 | print(r.json())
84 | ```
85 |
86 | </Tab>
87 | <Tab value="TypeScript">
88 |
89 | ```typescript
90 | const payload = {
91 | name: 'lume_vm',
92 | os: 'macOS',
93 | cpu: 2,
94 | memory: '4GB',
95 | diskSize: '64GB',
96 | display: '1024x768',
97 | ipsw: 'latest',
98 | storage: 'ssd',
99 | };
100 |
101 | const res = await fetch('http://localhost:7777/lume/vms', {
102 | method: 'POST',
103 | headers: { 'Content-Type': 'application/json' },
104 | body: JSON.stringify(payload),
105 | });
106 | console.log(await res.json());
107 | ```
108 |
109 | </Tab>
110 | </Tabs>
111 |
112 | ---
113 |
114 | ### Run VM
115 |
116 | Run a virtual machine instance.
117 |
118 | `POST: /lume/vms/:name/run`
119 |
120 | #### Parameters
121 |
122 | | Name | Type | Required | Description |
123 | | ----------------- | --------------- | -------- | --------------------------------------------------- |
124 | | noDisplay | boolean | No | If true, do not start VNC client |
125 | | sharedDirectories | array of object | No | List of shared directories (`hostPath`, `readOnly`) |
126 | | recoveryMode | boolean | No | Start in recovery mode |
127 | | storage | string | No | Storage type (`ssd`, etc.) |
128 |
129 | #### Example Request
130 |
131 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
132 | <Tab value="Curl">
133 |
134 | ```bash
135 | # Basic run
136 | curl --connect-timeout 6000 \
137 | --max-time 5000 \
138 | -X POST \
139 | http://localhost:7777/lume/vms/my-vm-name/run
140 |
141 | # Run with VNC client started and shared directory
142 | curl --connect-timeout 6000 \
143 | --max-time 5000 \
144 | -X POST \
145 | -H "Content-Type: application/json" \
146 | -d '{
147 | "noDisplay": false,
148 | "sharedDirectories": [
149 | {
150 | "hostPath": "~/Projects",
151 | "readOnly": false
152 | }
153 | ],
154 | "recoveryMode": false,
155 | "storage": "ssd"
156 | }' \
157 | http://localhost:7777/lume/vms/lume_vm/run
158 | ```
159 |
160 | </Tab>
161 | <Tab value="Python">
162 |
163 | ```python
164 | import requests
165 |
166 | # Basic run
167 | r = requests.post("http://localhost:7777/lume/vms/my-vm-name/run", timeout=50)
168 | print(r.json())
169 |
170 | # With VNC and shared directory
171 | payload = {
172 | "noDisplay": False,
173 | "sharedDirectories": [
174 | {"hostPath": "~/Projects", "readOnly": False}
175 | ],
176 | "recoveryMode": False,
177 | "storage": "ssd"
178 | }
179 | r = requests.post("http://localhost:7777/lume/vms/lume_vm/run", json=payload, timeout=50)
180 | print(r.json())
181 | ```
182 |
183 | </Tab>
184 | <Tab value="TypeScript">
185 |
186 | ```typescript
187 | // Basic run
188 | let res = await fetch('http://localhost:7777/lume/vms/my-vm-name/run', {
189 | method: 'POST',
190 | });
191 | console.log(await res.json());
192 |
193 | // With VNC and shared directory
194 | const payload = {
195 | noDisplay: false,
196 | sharedDirectories: [{ hostPath: '~/Projects', readOnly: false }],
197 | recoveryMode: false,
198 | storage: 'ssd',
199 | };
200 | res = await fetch('http://localhost:7777/lume/vms/lume_vm/run', {
201 | method: 'POST',
202 | headers: { 'Content-Type': 'application/json' },
203 | body: JSON.stringify(payload),
204 | });
205 | console.log(await res.json());
206 | ```
207 |
208 | </Tab>
209 | </Tabs>
210 |
211 | ---
212 |
213 | ### List VMs
214 |
215 | List all virtual machines.
216 |
217 | `GET: /lume/vms`
218 |
219 | #### Example Request
220 |
221 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
222 | <Tab value="Curl">
223 |
224 | ```bash
225 | curl --connect-timeout 6000 \
226 | --max-time 5000 \
227 | http://localhost:7777/lume/vms
228 | ```
229 |
230 | </Tab>
231 | <Tab value="Python">
232 |
233 | ```python
234 | import requests
235 |
236 | r = requests.get("http://localhost:7777/lume/vms", timeout=50)
237 | print(r.json())
238 | ```
239 |
240 | </Tab>
241 | <Tab value="TypeScript">
242 |
243 | ```typescript
244 | const res = await fetch('http://localhost:7777/lume/vms');
245 | console.log(await res.json());
246 | ```
247 |
248 | </Tab>
249 | </Tabs>
250 |
251 | ```json
252 | [
253 | {
254 | "name": "my-vm",
255 | "state": "stopped",
256 | "os": "macOS",
257 | "cpu": 2,
258 | "memory": "4GB",
259 | "diskSize": "64GB"
260 | },
261 | {
262 | "name": "my-vm-2",
263 | "state": "stopped",
264 | "os": "linux",
265 | "cpu": 2,
266 | "memory": "4GB",
267 | "diskSize": "64GB"
268 | }
269 | ]
270 | ```
271 |
272 | ---
273 |
274 | ### Get VM Details
275 |
276 | Get details for a specific virtual machine.
277 |
278 | `GET: /lume/vms/:name`
279 |
280 | #### Parameters
281 |
282 | | Name | Type | Required | Description |
283 | | ------- | ------ | -------- | -------------------------- |
284 | | storage | string | No | Storage type (`ssd`, etc.) |
285 |
286 | #### Example Request
287 |
288 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
289 | <Tab value="Curl">
290 |
291 | ```bash
292 | # Basic get
293 | curl --connect-timeout 6000 \
294 | --max-time 5000 \
295 | http://localhost:7777/lume/vms/lume_vm
296 |
297 | # Get with specific storage
298 | curl --connect-timeout 6000 \
299 | --max-time 5000 \
300 | http://localhost:7777/lume/vms/lume_vm?storage=ssd
301 | ```
302 |
303 | </Tab>
304 | <Tab value="Python">
305 |
306 | ```python
307 | import requests
308 |
309 | # Basic get
310 | details = requests.get("http://localhost:7777/lume/vms/lume_vm", timeout=50)
311 | print(details.json())
312 |
313 | # Get with specific storage
314 | details = requests.get("http://localhost:7777/lume/vms/lume_vm", params={"storage": "ssd"}, timeout=50)
315 | print(details.json())
316 | ```
317 |
318 | </Tab>
319 | <Tab value="TypeScript">
320 |
321 | ```typescript
322 | // Basic get
323 | let res = await fetch('http://localhost:7777/lume/vms/lume_vm');
324 | console.log(await res.json());
325 |
326 | // Get with specific storage
327 | res = await fetch('http://localhost:7777/lume/vms/lume_vm?storage=ssd');
328 | console.log(await res.json());
329 | ```
330 |
331 | </Tab>
332 | </Tabs>
333 |
334 | ```json
335 | {
336 | "name": "lume_vm",
337 | "state": "stopped",
338 | "os": "macOS",
339 | "cpu": 2,
340 | "memory": "4GB",
341 | "diskSize": "64GB",
342 | "display": "1024x768",
343 | "ipAddress": "192.168.65.2",
344 | "vncPort": 5900,
345 | "sharedDirectories": [
346 | {
347 | "hostPath": "~/Projects",
348 | "readOnly": false,
349 | "tag": "com.apple.virtio-fs.automount"
350 | }
351 | ]
352 | }
353 | ```
354 |
355 | ---
356 |
357 | ### Update VM Configuration
358 |
359 | Update the configuration of a virtual machine.
360 |
361 | `PATCH: /lume/vms/:name`
362 |
363 | #### Parameters
364 |
365 | | Name | Type | Required | Description |
366 | | -------- | ------- | -------- | ------------------------------------- |
367 | | cpu | integer | No | Number of CPU cores |
368 | | memory | string | No | Memory size (e.g. `8GB`) |
369 | | diskSize | string | No | Disk size (e.g. `100GB`) |
370 | | display | string | No | Display resolution (e.g. `1920x1080`) |
371 | | storage | string | No | Storage type (`ssd`, etc.) |
372 |
373 | #### Example Request
374 |
375 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
376 | <Tab value="Curl">
377 |
378 | ```bash
379 | curl --connect-timeout 6000 \
380 | --max-time 5000 \
381 | -X PATCH \
382 | -H "Content-Type: application/json" \
383 | -d '{
384 | "cpu": 4,
385 | "memory": "8GB",
386 | "diskSize": "100GB",
387 | "display": "1920x1080",
388 | "storage": "ssd"
389 | }' \
390 | http://localhost:7777/lume/vms/lume_vm
391 | ```
392 |
393 | </Tab>
394 | <Tab value="Python">
395 |
396 | ```python
397 | import requests
398 |
399 | payload = {
400 | "cpu": 4,
401 | "memory": "8GB",
402 | "diskSize": "100GB",
403 | "display": "1920x1080",
404 | "storage": "ssd"
405 | }
406 | r = requests.patch("http://localhost:7777/lume/vms/lume_vm", json=payload, timeout=50)
407 | print(r.json())
408 | ```
409 |
410 | </Tab>
411 | <Tab value="TypeScript">
412 |
413 | ```typescript
414 | const payload = {
415 | cpu: 4,
416 | memory: '8GB',
417 | diskSize: '100GB',
418 | display: '1920x1080',
419 | storage: 'ssd',
420 | };
421 | const res = await fetch('http://localhost:7777/lume/vms/lume_vm', {
422 | method: 'PATCH',
423 | headers: { 'Content-Type': 'application/json' },
424 | body: JSON.stringify(payload),
425 | });
426 | console.log(await res.json());
427 | ```
428 |
429 | </Tab>
430 | </Tabs>
431 |
432 | ---
433 |
434 | ### Stop VM
435 |
436 | Stop a running virtual machine.
437 |
438 | `POST: /lume/vms/:name/stop`
439 |
440 | #### Parameters
441 |
442 | | Name | Type | Required | Description |
443 | | ------- | ------ | -------- | -------------------------- |
444 | | storage | string | No | Storage type (`ssd`, etc.) |
445 |
446 | #### Example Request
447 |
448 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
449 | <Tab value="Curl">
450 |
451 | ```bash
452 | # Basic stop
453 | curl --connect-timeout 6000 \
454 | --max-time 5000 \
455 | -X POST \
456 | http://localhost:7777/lume/vms/lume_vm/stop
457 |
458 | # Stop with storage location specified
459 | curl --connect-timeout 6000 \
460 | --max-time 5000 \
461 | -X POST \
462 | http://localhost:7777/lume/vms/lume_vm/stop?storage=ssd
463 | ```
464 |
465 | </Tab>
466 | <Tab value="Python">
467 |
468 | ```python
469 | import requests
470 |
471 | # Basic stop
472 | r = requests.post("http://localhost:7777/lume/vms/lume_vm/stop", timeout=50)
473 | print(r.json())
474 |
475 | # Stop with storage location specified
476 | r = requests.post("http://localhost:7777/lume/vms/lume_vm/stop", params={"storage": "ssd"}, timeout=50)
477 | print(r.json())
478 | ```
479 |
480 | </Tab>
481 | <Tab value="TypeScript">
482 |
483 | ```typescript
484 | // Basic stop
485 | let res = await fetch('http://localhost:7777/lume/vms/lume_vm/stop', {
486 | method: 'POST',
487 | });
488 | console.log(await res.json());
489 |
490 | // Stop with storage location specified
491 | res = await fetch('http://localhost:7777/lume/vms/lume_vm/stop?storage=ssd', {
492 | method: 'POST',
493 | });
494 | console.log(await res.json());
495 | ```
496 |
497 | </Tab>
498 | </Tabs>
499 |
500 | ---
501 |
502 | ### Delete VM
503 |
504 | Delete a virtual machine instance.
505 |
506 | `DELETE: /lume/vms/:name`
507 |
508 | #### Parameters
509 |
510 | | Name | Type | Required | Description |
511 | | ------- | ------ | -------- | -------------------------- |
512 | | storage | string | No | Storage type (`ssd`, etc.) |
513 |
514 | #### Example Request
515 |
516 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
517 | <Tab value="Curl">
518 |
519 | ```bash
520 | # Basic delete
521 | curl --connect-timeout 6000 \
522 | --max-time 5000 \
523 | -X DELETE \
524 | http://localhost:7777/lume/vms/lume_vm
525 |
526 | # Delete with specific storage
527 | curl --connect-timeout 6000 \
528 | --max-time 5000 \
529 | -X DELETE \
530 | http://localhost:7777/lume/vms/lume_vm?storage=ssd
531 | ```
532 |
533 | </Tab>
534 | <Tab value="Python">
535 |
536 | ```python
537 | import requests
538 |
539 | # Basic delete
540 | r = requests.delete("http://localhost:7777/lume/vms/lume_vm", timeout=50)
541 | print(r.status_code)
542 |
543 | # Delete with specific storage
544 | r = requests.delete("http://localhost:7777/lume/vms/lume_vm", params={"storage": "ssd"}, timeout=50)
545 | print(r.status_code)
546 | ```
547 |
548 | </Tab>
549 | <Tab value="TypeScript">
550 |
551 | ```typescript
552 | // Basic delete
553 | let res = await fetch('http://localhost:7777/lume/vms/lume_vm', {
554 | method: 'DELETE',
555 | });
556 | console.log(res.status);
557 |
558 | // Delete with specific storage
559 | res = await fetch('http://localhost:7777/lume/vms/lume_vm?storage=ssd', {
560 | method: 'DELETE',
561 | });
562 | console.log(res.status);
563 | ```
564 |
565 | </Tab>
566 | </Tabs>
567 |
568 | ---
569 |
570 | ### Clone VM
571 |
572 | Clone an existing virtual machine.
573 |
574 | `POST: /lume/vms/clone`
575 |
576 | #### Parameters
577 |
578 | | Name | Type | Required | Description |
579 | | -------------- | ------ | -------- | ----------------------------------- |
580 | | name | string | Yes | Source VM name |
581 | | newName | string | Yes | New VM name |
582 | | sourceLocation | string | No | Source storage location (`default`) |
583 | | destLocation | string | No | Destination storage location |
584 |
585 | #### Example Request
586 |
587 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
588 | <Tab value="Curl">
589 |
590 | ```bash
591 | curl --connect-timeout 6000 \
592 | --max-time 5000 \
593 | -X POST \
594 | -H "Content-Type: application/json" \
595 | -d '{
596 | "name": "source-vm",
597 | "newName": "cloned-vm",
598 | "sourceLocation": "default",
599 | "destLocation": "ssd"
600 | }' \
601 | http://localhost:7777/lume/vms/clone
602 | ```
603 |
604 | </Tab>
605 | <Tab value="Python">
606 |
607 | ```python
608 | import requests
609 |
610 | payload = {
611 | "name": "source-vm",
612 | "newName": "cloned-vm",
613 | "sourceLocation": "default",
614 | "destLocation": "ssd"
615 | }
616 | r = requests.post("http://localhost:7777/lume/vms/clone", json=payload, timeout=50)
617 | print(r.json())
618 | ```
619 |
620 | </Tab>
621 | <Tab value="TypeScript">
622 |
623 | ```typescript
624 | const payload = {
625 | name: 'source-vm',
626 | newName: 'cloned-vm',
627 | sourceLocation: 'default',
628 | destLocation: 'ssd',
629 | };
630 | const res = await fetch('http://localhost:7777/lume/vms/clone', {
631 | method: 'POST',
632 | headers: { 'Content-Type': 'application/json' },
633 | body: JSON.stringify(payload),
634 | });
635 | console.log(await res.json());
636 | ```
637 |
638 | </Tab>
639 | </Tabs>
640 |
641 | ---
642 |
643 | ### Pull VM Image
644 |
645 | Pull a VM image from a registry.
646 |
647 | `POST: /lume/pull`
648 |
649 | #### Parameters
650 |
651 | | Name | Type | Required | Description |
652 | | ------------ | ------ | -------- | ------------------------------------- |
653 | | image | string | Yes | Image name (e.g. `macos-sequoia-...`) |
654 | | name | string | No | VM name for the pulled image |
655 | | registry | string | No | Registry host (e.g. `ghcr.io`) |
656 | | organization | string | No | Organization name |
657 | | storage | string | No | Storage type (`ssd`, etc.) |
658 |
659 | #### Example Request
660 |
661 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
662 | <Tab value="Curl">
663 |
664 | ```bash
665 | curl --connect-timeout 6000 \
666 | --max-time 5000 \
667 | -X POST \
668 | -H "Content-Type: application/json" \
669 | -d '{
670 | "image": "macos-sequoia-vanilla:latest",
671 | "name": "my-vm-name",
672 | "registry": "ghcr.io",
673 | "organization": "trycua",
674 | "storage": "ssd"
675 | }' \
676 | http://localhost:7777/lume/pull
677 | ```
678 |
679 | </Tab>
680 | <Tab value="Python">
681 |
682 | ```python
683 | import requests
684 |
685 | payload = {
686 | "image": "macos-sequoia-vanilla:latest",
687 | "name": "my-vm-name",
688 | "registry": "ghcr.io",
689 | "organization": "trycua",
690 | "storage": "ssd"
691 | }
692 | r = requests.post("http://localhost:7777/lume/pull", json=payload, timeout=50)
693 | print(r.json())
694 | ```
695 |
696 | </Tab>
697 | <Tab value="TypeScript">
698 |
699 | ```typescript
700 | const payload = {
701 | image: 'macos-sequoia-vanilla:latest',
702 | name: 'my-vm-name',
703 | registry: 'ghcr.io',
704 | organization: 'trycua',
705 | storage: 'ssd',
706 | };
707 | const res = await fetch('http://localhost:7777/lume/pull', {
708 | method: 'POST',
709 | headers: { 'Content-Type': 'application/json' },
710 | body: JSON.stringify(payload),
711 | });
712 | console.log(await res.json());
713 | ```
714 |
715 | </Tab>
716 | </Tabs>
717 |
718 | ---
719 |
720 | ### Push VM Image
721 |
722 | Push a VM to a registry as an image (asynchronous operation).
723 |
724 | `POST: /lume/vms/push`
725 |
726 | #### Parameters
727 |
728 | | Name | Type | Required | Description |
729 | | ------------ | ----------- | -------- | ------------------------------------ |
730 | | name | string | Yes | Local VM name to push |
731 | | imageName | string | Yes | Image name in registry |
732 | | tags | array | Yes | Image tags (e.g. `["latest", "v1"]`) |
733 | | organization | string | Yes | Organization name |
734 | | registry | string | No | Registry host (e.g. `ghcr.io`) |
735 | | chunkSizeMb | integer | No | Chunk size in MB for upload |
736 | | storage | string/null | No | Storage type (`ssd`, etc.) |
737 |
738 | #### Example Request
739 |
740 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
741 | <Tab value="Curl">
742 |
743 | ```bash
744 | curl --connect-timeout 6000 \
745 | --max-time 5000 \
746 | -X POST \
747 | -H "Content-Type: application/json" \
748 | -d '{
749 | "name": "my-local-vm",
750 | "imageName": "my-image",
751 | "tags": ["latest", "v1"],
752 | "organization": "my-org",
753 | "registry": "ghcr.io",
754 | "chunkSizeMb": 512,
755 | "storage": null
756 | }' \
757 | http://localhost:7777/lume/vms/push
758 | ```
759 |
760 | </Tab>
761 | <Tab value="Python">
762 |
763 | ```python
764 | import requests
765 |
766 | payload = {
767 | "name": "my-local-vm",
768 | "imageName": "my-image",
769 | "tags": ["latest", "v1"],
770 | "organization": "my-org",
771 | "registry": "ghcr.io",
772 | "chunkSizeMb": 512,
773 | "storage": None
774 | }
775 | r = requests.post("http://localhost:7777/lume/vms/push", json=payload, timeout=50)
776 | print(r.json())
777 | ```
778 |
779 | </Tab>
780 | <Tab value="TypeScript">
781 |
782 | ```typescript
783 | const payload = {
784 | name: 'my-local-vm',
785 | imageName: 'my-image',
786 | tags: ['latest', 'v1'],
787 | organization: 'my-org',
788 | registry: 'ghcr.io',
789 | chunkSizeMb: 512,
790 | storage: null,
791 | };
792 | const res = await fetch('http://localhost:7777/lume/vms/push', {
793 | method: 'POST',
794 | headers: { 'Content-Type': 'application/json' },
795 | body: JSON.stringify(payload),
796 | });
797 | console.log(await res.json());
798 | ```
799 |
800 | </Tab>
801 | </Tabs>
802 |
803 | **Response (202 Accepted):**
804 |
805 | ```json
806 | {
807 | "message": "Push initiated in background",
808 | "name": "my-local-vm",
809 | "imageName": "my-image",
810 | "tags": ["latest", "v1"]
811 | }
812 | ```
813 |
814 | ---
815 |
816 | ### List Images
817 |
818 | List available VM images.
819 |
820 | `GET: /lume/images`
821 |
822 | #### Example Request
823 |
824 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
825 | <Tab value="Curl">
826 |
827 | ```bash
828 | curl --connect-timeout 6000 \
829 | --max-time 5000 \
830 | http://localhost:7777/lume/images
831 | ```
832 |
833 | </Tab>
834 | <Tab value="Python">
835 |
836 | ```python
837 | import requests
838 |
839 | r = requests.get("http://localhost:7777/lume/images", timeout=50)
840 | print(r.json())
841 | ```
842 |
843 | </Tab>
844 | <Tab value="TypeScript">
845 |
846 | ```typescript
847 | const res = await fetch('http://localhost:7777/lume/images');
848 | console.log(await res.json());
849 | ```
850 |
851 | </Tab>
852 | </Tabs>
853 |
854 | ```json
855 | {
856 | "local": ["macos-sequoia-xcode:latest", "macos-sequoia-vanilla:latest"]
857 | }
858 | ```
859 |
860 | ---
861 |
862 | ### Prune Images
863 |
864 | Remove unused VM images to free up disk space.
865 |
866 | `POST: /lume/prune`
867 |
868 | #### Example Request
869 |
870 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
871 | <Tab value="Curl">
872 |
873 | ```bash
874 | curl --connect-timeout 6000 \
875 | --max-time 5000 \
876 | -X POST \
877 | http://localhost:7777/lume/prune
878 | ```
879 |
880 | </Tab>
881 | <Tab value="Python">
882 |
883 | ```python
884 | import requests
885 |
886 | r = requests.post("http://localhost:7777/lume/prune", timeout=50)
887 | print(r.json())
888 | ```
889 |
890 | </Tab>
891 | <Tab value="TypeScript">
892 |
893 | ```typescript
894 | const res = await fetch('http://localhost:7777/lume/prune', {
895 | method: 'POST',
896 | });
897 | console.log(await res.json());
898 | ```
899 |
900 | </Tab>
901 | </Tabs>
902 |
903 | ---
904 |
905 | ### Get Latest IPSW URL
906 |
907 | Get the URL for the latest macOS IPSW file.
908 |
909 | `GET: /lume/ipsw`
910 |
911 | #### Example Request
912 |
913 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
914 | <Tab value="Curl">
915 |
916 | ```bash
917 | curl --connect-timeout 6000 \
918 | --max-time 5000 \
919 | http://localhost:7777/lume/ipsw
920 | ```
921 |
922 | </Tab>
923 | <Tab value="Python">
924 |
925 | ```python
926 | import requests
927 |
928 | r = requests.get("http://localhost:7777/lume/ipsw", timeout=50)
929 | print(r.json())
930 | ```
931 |
932 | </Tab>
933 | <Tab value="TypeScript">
934 |
935 | ```typescript
936 | const res = await fetch('http://localhost:7777/lume/ipsw');
937 | console.log(await res.json());
938 | ```
939 |
940 | </Tab>
941 | </Tabs>
942 |
943 | ---
944 |
945 | ## Configuration Management
946 |
947 | ### Get Configuration
948 |
949 | Get current Lume configuration settings.
950 |
951 | `GET: /lume/config`
952 |
953 | #### Example Request
954 |
955 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
956 | <Tab value="Curl">
957 |
958 | ```bash
959 | curl --connect-timeout 6000 \
960 | --max-time 5000 \
961 | http://localhost:7777/lume/config
962 | ```
963 |
964 | </Tab>
965 | <Tab value="Python">
966 |
967 | ```python
968 | import requests
969 |
970 | r = requests.get("http://localhost:7777/lume/config", timeout=50)
971 | print(r.json())
972 | ```
973 |
974 | </Tab>
975 | <Tab value="TypeScript">
976 |
977 | ```typescript
978 | const res = await fetch('http://localhost:7777/lume/config');
979 | console.log(await res.json());
980 | ```
981 |
982 | </Tab>
983 | </Tabs>
984 |
985 | ```json
986 | {
987 | "homeDirectory": "~/.lume",
988 | "cacheDirectory": "~/.lume/cache",
989 | "cachingEnabled": true
990 | }
991 | ```
992 |
993 | ### Update Configuration
994 |
995 | Update Lume configuration settings.
996 |
997 | `POST: /lume/config`
998 |
999 | #### Parameters
1000 |
1001 | | Name | Type | Required | Description |
1002 | | -------------- | ------- | -------- | ------------------------- |
1003 | | homeDirectory | string | No | Lume home directory path |
1004 | | cacheDirectory | string | No | Cache directory path |
1005 | | cachingEnabled | boolean | No | Enable or disable caching |
1006 |
1007 | #### Example Request
1008 |
1009 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
1010 | <Tab value="Curl">
1011 |
1012 | ```bash
1013 | curl --connect-timeout 6000 \
1014 | --max-time 5000 \
1015 | -X POST \
1016 | -H "Content-Type: application/json" \
1017 | -d '{
1018 | "homeDirectory": "~/custom/lume",
1019 | "cacheDirectory": "~/custom/lume/cache",
1020 | "cachingEnabled": true
1021 | }' \
1022 | http://localhost:7777/lume/config
1023 | ```
1024 |
1025 | </Tab>
1026 | <Tab value="Python">
1027 |
1028 | ```python
1029 | import requests
1030 |
1031 | payload = {
1032 | "homeDirectory": "~/custom/lume",
1033 | "cacheDirectory": "~/custom/lume/cache",
1034 | "cachingEnabled": True
1035 | }
1036 | r = requests.post("http://localhost:7777/lume/config", json=payload, timeout=50)
1037 | print(r.json())
1038 | ```
1039 |
1040 | </Tab>
1041 | <Tab value="TypeScript">
1042 |
1043 | ```typescript
1044 | const payload = {
1045 | homeDirectory: '~/custom/lume',
1046 | cacheDirectory: '~/custom/lume/cache',
1047 | cachingEnabled: true,
1048 | };
1049 | const res = await fetch('http://localhost:7777/lume/config', {
1050 | method: 'POST',
1051 | headers: { 'Content-Type': 'application/json' },
1052 | body: JSON.stringify(payload),
1053 | });
1054 | console.log(await res.json());
1055 | ```
1056 |
1057 | </Tab>
1058 | </Tabs>
1059 |
1060 | ---
1061 |
1062 | ## Storage Location Management
1063 |
1064 | ### Get VM Storage Locations
1065 |
1066 | List all configured VM storage locations.
1067 |
1068 | `GET: /lume/config/locations`
1069 |
1070 | #### Example Request
1071 |
1072 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
1073 | <Tab value="Curl">
1074 |
1075 | ```bash
1076 | curl --connect-timeout 6000 \
1077 | --max-time 5000 \
1078 | http://localhost:7777/lume/config/locations
1079 | ```
1080 |
1081 | </Tab>
1082 | <Tab value="Python">
1083 |
1084 | ```python
1085 | import requests
1086 |
1087 | r = requests.get("http://localhost:7777/lume/config/locations", timeout=50)
1088 | print(r.json())
1089 | ```
1090 |
1091 | </Tab>
1092 | <Tab value="TypeScript">
1093 |
1094 | ```typescript
1095 | const res = await fetch('http://localhost:7777/lume/config/locations');
1096 | console.log(await res.json());
1097 | ```
1098 |
1099 | </Tab>
1100 | </Tabs>
1101 |
1102 | ```json
1103 | [
1104 | {
1105 | "name": "default",
1106 | "path": "~/.lume/vms",
1107 | "isDefault": true
1108 | },
1109 | {
1110 | "name": "ssd",
1111 | "path": "/Volumes/SSD/lume/vms",
1112 | "isDefault": false
1113 | }
1114 | ]
1115 | ```
1116 |
1117 | ### Add VM Storage Location
1118 |
1119 | Add a new VM storage location.
1120 |
1121 | `POST: /lume/config/locations`
1122 |
1123 | #### Parameters
1124 |
1125 | | Name | Type | Required | Description |
1126 | | ---- | ------ | -------- | ---------------------------- |
1127 | | name | string | Yes | Storage location name |
1128 | | path | string | Yes | File system path for storage |
1129 |
1130 | #### Example Request
1131 |
1132 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
1133 | <Tab value="Curl">
1134 |
1135 | ```bash
1136 | curl --connect-timeout 6000 \
1137 | --max-time 5000 \
1138 | -X POST \
1139 | -H "Content-Type: application/json" \
1140 | -d '{
1141 | "name": "ssd",
1142 | "path": "/Volumes/SSD/lume/vms"
1143 | }' \
1144 | http://localhost:7777/lume/config/locations
1145 | ```
1146 |
1147 | </Tab>
1148 | <Tab value="Python">
1149 |
1150 | ```python
1151 | import requests
1152 |
1153 | payload = {
1154 | "name": "ssd",
1155 | "path": "/Volumes/SSD/lume/vms"
1156 | }
1157 | r = requests.post("http://localhost:7777/lume/config/locations", json=payload, timeout=50)
1158 | print(r.json())
1159 | ```
1160 |
1161 | </Tab>
1162 | <Tab value="TypeScript">
1163 |
1164 | ```typescript
1165 | const payload = {
1166 | name: 'ssd',
1167 | path: '/Volumes/SSD/lume/vms',
1168 | };
1169 | const res = await fetch('http://localhost:7777/lume/config/locations', {
1170 | method: 'POST',
1171 | headers: { 'Content-Type': 'application/json' },
1172 | body: JSON.stringify(payload),
1173 | });
1174 | console.log(await res.json());
1175 | ```
1176 |
1177 | </Tab>
1178 | </Tabs>
1179 |
1180 | ### Remove VM Storage Location
1181 |
1182 | Remove a VM storage location.
1183 |
1184 | `DELETE: /lume/config/locations/:name`
1185 |
1186 | #### Example Request
1187 |
1188 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
1189 | <Tab value="Curl">
1190 |
1191 | ```bash
1192 | curl --connect-timeout 6000 \
1193 | --max-time 5000 \
1194 | -X DELETE \
1195 | http://localhost:7777/lume/config/locations/ssd
1196 | ```
1197 |
1198 | </Tab>
1199 | <Tab value="Python">
1200 |
1201 | ```python
1202 | import requests
1203 |
1204 | r = requests.delete("http://localhost:7777/lume/config/locations/ssd", timeout=50)
1205 | print(r.status_code)
1206 | ```
1207 |
1208 | </Tab>
1209 | <Tab value="TypeScript">
1210 |
1211 | ```typescript
1212 | const res = await fetch('http://localhost:7777/lume/config/locations/ssd', {
1213 | method: 'DELETE',
1214 | });
1215 | console.log(res.status);
1216 | ```
1217 |
1218 | </Tab>
1219 | </Tabs>
1220 |
1221 | ### Set Default VM Storage Location
1222 |
1223 | Set a storage location as the default.
1224 |
1225 | `POST: /lume/config/locations/default/:name`
1226 |
1227 | #### Example Request
1228 |
1229 | <Tabs groupId="language" persist items={['Curl', 'Python', 'TypeScript']}>
1230 | <Tab value="Curl">
1231 |
1232 | ```bash
1233 | curl --connect-timeout 6000 \
1234 | --max-time 5000 \
1235 | -X POST \
1236 | http://localhost:7777/lume/config/locations/default/ssd
1237 | ```
1238 |
1239 | </Tab>
1240 | <Tab value="Python">
1241 |
1242 | ```python
1243 | import requests
1244 |
1245 | r = requests.post("http://localhost:7777/lume/config/locations/default/ssd", timeout=50)
1246 | print(r.json())
1247 | ```
1248 |
1249 | </Tab>
1250 | <Tab value="TypeScript">
1251 |
1252 | ```typescript
1253 | const res = await fetch('http://localhost:7777/lume/config/locations/default/ssd', {
1254 | method: 'POST',
1255 | });
1256 | console.log(await res.json());
1257 | ```
1258 |
1259 | </Tab>
1260 | </Tabs>
1261 |
```
--------------------------------------------------------------------------------
/libs/lume/src/Server/Handlers.swift:
--------------------------------------------------------------------------------
```swift
1 | import ArgumentParser
2 | import Foundation
3 | import Virtualization
4 |
5 | @MainActor
6 | extension Server {
7 | // MARK: - VM Management Handlers
8 |
9 | func handleListVMs(storage: String? = nil) async throws -> HTTPResponse {
10 | do {
11 | let vmController = LumeController()
12 | let vms = try vmController.list(storage: storage)
13 | return try .json(vms)
14 | } catch {
15 | print(
16 | "ERROR: Failed to list VMs: \(error.localizedDescription), storage=\(String(describing: storage))"
17 | )
18 | return .badRequest(message: error.localizedDescription)
19 | }
20 | }
21 |
22 | func handleGetVM(name: String, storage: String? = nil) async throws -> HTTPResponse {
23 | print("Getting VM details: name=\(name), storage=\(String(describing: storage))")
24 |
25 | do {
26 | let vmController = LumeController()
27 | print("Created VM controller, attempting to get VM")
28 | let vm = try vmController.get(name: name, storage: storage)
29 | print("Successfully retrieved VM")
30 |
31 | // Check for nil values that might cause crashes
32 | if vm.vmDirContext.config.macAddress == nil {
33 | print("ERROR: VM has nil macAddress")
34 | return .badRequest(message: "VM configuration is invalid (nil macAddress)")
35 | }
36 | print("MacAddress check passed")
37 |
38 | // Log that we're about to access details
39 | print("Preparing VM details response")
40 |
41 | // Print the full details object for debugging
42 | let details = vm.details
43 | print("VM DETAILS: \(details)")
44 | print(" name: \(details.name)")
45 | print(" os: \(details.os)")
46 | print(" cpuCount: \(details.cpuCount)")
47 | print(" memorySize: \(details.memorySize)")
48 | print(" diskSize: \(details.diskSize)")
49 | print(" display: \(details.display)")
50 | print(" status: \(details.status)")
51 | print(" vncUrl: \(String(describing: details.vncUrl))")
52 | print(" ipAddress: \(String(describing: details.ipAddress))")
53 | print(" locationName: \(details.locationName)")
54 |
55 | // Serialize the VM details
56 | print("About to serialize VM details")
57 | let response = try HTTPResponse.json(vm.details)
58 | print("Successfully serialized VM details")
59 | return response
60 |
61 | } catch {
62 | // This will catch errors from both vmController.get and the json serialization
63 | print("ERROR: Failed to get VM details: \(error.localizedDescription)")
64 | return .badRequest(message: error.localizedDescription)
65 | }
66 | }
67 |
68 | func handleCreateVM(_ body: Data?) async throws -> HTTPResponse {
69 | guard let body = body,
70 | let request = try? JSONDecoder().decode(CreateVMRequest.self, from: body)
71 | else {
72 | return HTTPResponse(
73 | statusCode: .badRequest,
74 | headers: ["Content-Type": "application/json"],
75 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
76 | )
77 | }
78 |
79 | do {
80 | let sizes = try request.parse()
81 | let vmController = LumeController()
82 | try await vmController.create(
83 | name: request.name,
84 | os: request.os,
85 | diskSize: sizes.diskSize,
86 | cpuCount: request.cpu,
87 | memorySize: sizes.memory,
88 | display: request.display,
89 | ipsw: request.ipsw,
90 | storage: request.storage
91 | )
92 |
93 | return HTTPResponse(
94 | statusCode: .ok,
95 | headers: ["Content-Type": "application/json"],
96 | body: try JSONEncoder().encode([
97 | "message": "VM created successfully", "name": request.name,
98 | ])
99 | )
100 | } catch {
101 | return HTTPResponse(
102 | statusCode: .badRequest,
103 | headers: ["Content-Type": "application/json"],
104 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
105 | )
106 | }
107 | }
108 |
109 | func handleDeleteVM(name: String, storage: String? = nil) async throws -> HTTPResponse {
110 | do {
111 | let vmController = LumeController()
112 | try await vmController.delete(name: name, storage: storage)
113 | return HTTPResponse(
114 | statusCode: .ok, headers: ["Content-Type": "application/json"], body: Data())
115 | } catch {
116 | return HTTPResponse(
117 | statusCode: .badRequest, headers: ["Content-Type": "application/json"],
118 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription)))
119 | }
120 | }
121 |
122 | func handleCloneVM(_ body: Data?) async throws -> HTTPResponse {
123 | guard let body = body,
124 | let request = try? JSONDecoder().decode(CloneRequest.self, from: body)
125 | else {
126 | return HTTPResponse(
127 | statusCode: .badRequest,
128 | headers: ["Content-Type": "application/json"],
129 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
130 | )
131 | }
132 |
133 | do {
134 | let vmController = LumeController()
135 | try vmController.clone(
136 | name: request.name,
137 | newName: request.newName,
138 | sourceLocation: request.sourceLocation,
139 | destLocation: request.destLocation
140 | )
141 |
142 | return HTTPResponse(
143 | statusCode: .ok,
144 | headers: ["Content-Type": "application/json"],
145 | body: try JSONEncoder().encode([
146 | "message": "VM cloned successfully",
147 | "source": request.name,
148 | "destination": request.newName,
149 | ])
150 | )
151 | } catch {
152 | return HTTPResponse(
153 | statusCode: .badRequest,
154 | headers: ["Content-Type": "application/json"],
155 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
156 | )
157 | }
158 | }
159 |
160 | // MARK: - VM Operation Handlers
161 |
162 | func handleSetVM(name: String, body: Data?) async throws -> HTTPResponse {
163 | guard let body = body,
164 | let request = try? JSONDecoder().decode(SetVMRequest.self, from: body)
165 | else {
166 | return HTTPResponse(
167 | statusCode: .badRequest,
168 | headers: ["Content-Type": "application/json"],
169 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
170 | )
171 | }
172 |
173 | do {
174 | let vmController = LumeController()
175 | let sizes = try request.parse()
176 | try vmController.updateSettings(
177 | name: name,
178 | cpu: request.cpu,
179 | memory: sizes.memory,
180 | diskSize: sizes.diskSize,
181 | display: sizes.display?.string,
182 | storage: request.storage
183 | )
184 |
185 | return HTTPResponse(
186 | statusCode: .ok,
187 | headers: ["Content-Type": "application/json"],
188 | body: try JSONEncoder().encode(["message": "VM settings updated successfully"])
189 | )
190 | } catch {
191 | return HTTPResponse(
192 | statusCode: .badRequest,
193 | headers: ["Content-Type": "application/json"],
194 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
195 | )
196 | }
197 | }
198 |
199 | func handleStopVM(name: String, storage: String? = nil) async throws -> HTTPResponse {
200 | Logger.info(
201 | "Stopping VM", metadata: ["name": name, "storage": String(describing: storage)])
202 |
203 | do {
204 | Logger.info("Creating VM controller", metadata: ["name": name])
205 | let vmController = LumeController()
206 |
207 | Logger.info("Calling stopVM on controller", metadata: ["name": name])
208 | try await vmController.stopVM(name: name, storage: storage)
209 |
210 | Logger.info(
211 | "VM stopped, waiting 5 seconds for locks to clear", metadata: ["name": name])
212 |
213 | // Add a delay to ensure locks are fully released before returning
214 | for i in 1...5 {
215 | try? await Task.sleep(nanoseconds: 1_000_000_000)
216 | Logger.info("Lock clearing delay", metadata: ["name": name, "seconds": "\(i)/5"])
217 | }
218 |
219 | // Verify the VM is really in a stopped state
220 | Logger.info("Verifying VM is stopped", metadata: ["name": name])
221 | let vm = try? vmController.get(name: name, storage: storage)
222 | if let vm = vm, vm.details.status == "running" {
223 | Logger.info(
224 | "VM still reports as running despite stop operation",
225 | metadata: ["name": name, "severity": "warning"])
226 | } else {
227 | Logger.info(
228 | "Verification complete: VM is in stopped state", metadata: ["name": name])
229 | }
230 |
231 | Logger.info("Returning successful response", metadata: ["name": name])
232 | return HTTPResponse(
233 | statusCode: .ok,
234 | headers: ["Content-Type": "application/json"],
235 | body: try JSONEncoder().encode(["message": "VM stopped successfully"])
236 | )
237 | } catch {
238 | Logger.error(
239 | "Failed to stop VM",
240 | metadata: [
241 | "name": name,
242 | "error": error.localizedDescription,
243 | "storage": String(describing: storage),
244 | ])
245 | return HTTPResponse(
246 | statusCode: .badRequest,
247 | headers: ["Content-Type": "application/json"],
248 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
249 | )
250 | }
251 | }
252 |
253 | func handleRunVM(name: String, body: Data?) async throws -> HTTPResponse {
254 | Logger.info("Running VM", metadata: ["name": name])
255 |
256 | // Log the raw body data if available
257 | if let body = body, let bodyString = String(data: body, encoding: .utf8) {
258 | Logger.info("Run VM raw request body", metadata: ["name": name, "body": bodyString])
259 | } else {
260 | Logger.info("No request body or could not decode as string", metadata: ["name": name])
261 | }
262 |
263 | do {
264 | Logger.info("Creating VM controller and parsing request", metadata: ["name": name])
265 | let request =
266 | body.flatMap { try? JSONDecoder().decode(RunVMRequest.self, from: $0) }
267 | ?? RunVMRequest(
268 | noDisplay: nil, sharedDirectories: nil, recoveryMode: nil, storage: nil)
269 |
270 | Logger.info(
271 | "Parsed request",
272 | metadata: [
273 | "name": name,
274 | "noDisplay": String(describing: request.noDisplay),
275 | "sharedDirectories": "\(request.sharedDirectories?.count ?? 0)",
276 | "storage": String(describing: request.storage),
277 | ])
278 |
279 | Logger.info("Parsing shared directories", metadata: ["name": name])
280 | let dirs = try request.parse()
281 | Logger.info(
282 | "Successfully parsed shared directories",
283 | metadata: ["name": name, "count": "\(dirs.count)"])
284 |
285 | // Start VM in background
286 | Logger.info("Starting VM in background", metadata: ["name": name])
287 | startVM(
288 | name: name,
289 | noDisplay: request.noDisplay ?? false,
290 | sharedDirectories: dirs,
291 | recoveryMode: request.recoveryMode ?? false,
292 | storage: request.storage
293 | )
294 | Logger.info("VM start initiated in background", metadata: ["name": name])
295 |
296 | // Return response immediately
297 | return HTTPResponse(
298 | statusCode: .accepted,
299 | headers: ["Content-Type": "application/json"],
300 | body: try JSONEncoder().encode([
301 | "message": "VM start initiated",
302 | "name": name,
303 | "status": "pending",
304 | ])
305 | )
306 | } catch {
307 | Logger.error(
308 | "Failed to run VM",
309 | metadata: [
310 | "name": name,
311 | "error": error.localizedDescription,
312 | ])
313 | return HTTPResponse(
314 | statusCode: .badRequest,
315 | headers: ["Content-Type": "application/json"],
316 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
317 | )
318 | }
319 | }
320 |
321 | // MARK: - Image Management Handlers
322 |
323 | func handleIPSW() async throws -> HTTPResponse {
324 | do {
325 | let vmController = LumeController()
326 | let url = try await vmController.getLatestIPSWURL()
327 | return HTTPResponse(
328 | statusCode: .ok,
329 | headers: ["Content-Type": "application/json"],
330 | body: try JSONEncoder().encode(["url": url.absoluteString])
331 | )
332 | } catch {
333 | return HTTPResponse(
334 | statusCode: .badRequest,
335 | headers: ["Content-Type": "application/json"],
336 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
337 | )
338 | }
339 | }
340 |
341 | func handlePull(_ body: Data?) async throws -> HTTPResponse {
342 | guard let body = body,
343 | let request = try? JSONDecoder().decode(PullRequest.self, from: body)
344 | else {
345 | return HTTPResponse(
346 | statusCode: .badRequest,
347 | headers: ["Content-Type": "application/json"],
348 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
349 | )
350 | }
351 |
352 | do {
353 | let vmController = LumeController()
354 | try await vmController.pullImage(
355 | image: request.image,
356 | name: request.name,
357 | registry: request.registry,
358 | organization: request.organization,
359 | storage: request.storage
360 | )
361 |
362 | return HTTPResponse(
363 | statusCode: .ok,
364 | headers: ["Content-Type": "application/json"],
365 | body: try JSONEncoder().encode([
366 | "message": "Image pulled successfully",
367 | "image": request.image,
368 | "name": request.name ?? "default",
369 | ])
370 | )
371 | } catch {
372 | return HTTPResponse(
373 | statusCode: .badRequest,
374 | headers: ["Content-Type": "application/json"],
375 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
376 | )
377 | }
378 | }
379 |
380 | func handlePruneImages() async throws -> HTTPResponse {
381 | do {
382 | let vmController = LumeController()
383 | try await vmController.pruneImages()
384 | return HTTPResponse(
385 | statusCode: .ok,
386 | headers: ["Content-Type": "application/json"],
387 | body: try JSONEncoder().encode(["message": "Successfully removed cached images"])
388 | )
389 | } catch {
390 | return HTTPResponse(
391 | statusCode: .badRequest,
392 | headers: ["Content-Type": "application/json"],
393 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
394 | )
395 | }
396 | }
397 |
398 | func handlePush(_ body: Data?) async throws -> HTTPResponse {
399 | guard let body = body,
400 | let request = try? JSONDecoder().decode(PushRequest.self, from: body)
401 | else {
402 | return HTTPResponse(
403 | statusCode: .badRequest,
404 | headers: ["Content-Type": "application/json"],
405 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
406 | )
407 | }
408 |
409 | // Trigger push asynchronously, return Accepted immediately
410 | Task.detached { @MainActor @Sendable in
411 | do {
412 | let vmController = LumeController()
413 | try await vmController.pushImage(
414 | name: request.name,
415 | imageName: request.imageName,
416 | tags: request.tags,
417 | registry: request.registry,
418 | organization: request.organization,
419 | storage: request.storage,
420 | chunkSizeMb: request.chunkSizeMb,
421 | verbose: false, // Verbose typically handled by server logs
422 | dryRun: false, // Default API behavior is likely non-dry-run
423 | reassemble: false // Default API behavior is likely non-reassemble
424 | )
425 | print(
426 | "Background push completed successfully for image: \(request.imageName):\(request.tags.joined(separator: ","))"
427 | )
428 | } catch {
429 | print(
430 | "Background push failed for image: \(request.imageName):\(request.tags.joined(separator: ",")) - Error: \(error.localizedDescription)"
431 | )
432 | }
433 | }
434 |
435 | return HTTPResponse(
436 | statusCode: .accepted,
437 | headers: ["Content-Type": "application/json"],
438 | body: try JSONEncoder().encode([
439 | "message": AnyEncodable("Push initiated in background"),
440 | "name": AnyEncodable(request.name),
441 | "imageName": AnyEncodable(request.imageName),
442 | "tags": AnyEncodable(request.tags),
443 | ])
444 | )
445 | }
446 |
447 | func handleGetImages(_ request: HTTPRequest) async throws -> HTTPResponse {
448 | let pathAndQuery = request.path.split(separator: "?", maxSplits: 1)
449 | let queryParams =
450 | pathAndQuery.count > 1
451 | ? pathAndQuery[1]
452 | .split(separator: "&")
453 | .reduce(into: [String: String]()) { dict, param in
454 | let parts = param.split(separator: "=", maxSplits: 1)
455 | if parts.count == 2 {
456 | dict[String(parts[0])] = String(parts[1])
457 | }
458 | } : [:]
459 |
460 | let organization = queryParams["organization"] ?? "trycua"
461 |
462 | do {
463 | let vmController = LumeController()
464 | let imageList = try await vmController.getImages(organization: organization)
465 |
466 | // Create a response format that matches the CLI output
467 | let response = imageList.local.map {
468 | [
469 | "repository": $0.repository,
470 | "imageId": $0.imageId,
471 | ]
472 | }
473 |
474 | return HTTPResponse(
475 | statusCode: .ok,
476 | headers: ["Content-Type": "application/json"],
477 | body: try JSONEncoder().encode(response)
478 | )
479 | } catch {
480 | return HTTPResponse(
481 | statusCode: .badRequest,
482 | headers: ["Content-Type": "application/json"],
483 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
484 | )
485 | }
486 | }
487 |
488 | // MARK: - Config Management Handlers
489 |
490 | func handleGetConfig() async throws -> HTTPResponse {
491 | do {
492 | let vmController = LumeController()
493 | let settings = vmController.getSettings()
494 | return try .json(settings)
495 | } catch {
496 | return .badRequest(message: error.localizedDescription)
497 | }
498 | }
499 |
500 | struct ConfigRequest: Codable {
501 | let homeDirectory: String?
502 | let cacheDirectory: String?
503 | let cachingEnabled: Bool?
504 | }
505 |
506 | func handleUpdateConfig(_ body: Data?) async throws -> HTTPResponse {
507 | guard let body = body,
508 | let request = try? JSONDecoder().decode(ConfigRequest.self, from: body)
509 | else {
510 | return HTTPResponse(
511 | statusCode: .badRequest,
512 | headers: ["Content-Type": "application/json"],
513 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
514 | )
515 | }
516 |
517 | do {
518 | let vmController = LumeController()
519 |
520 | if let homeDir = request.homeDirectory {
521 | try vmController.setHomeDirectory(homeDir)
522 | }
523 |
524 | if let cacheDir = request.cacheDirectory {
525 | try vmController.setCacheDirectory(path: cacheDir)
526 | }
527 |
528 | if let cachingEnabled = request.cachingEnabled {
529 | try vmController.setCachingEnabled(cachingEnabled)
530 | }
531 |
532 | return HTTPResponse(
533 | statusCode: .ok,
534 | headers: ["Content-Type": "application/json"],
535 | body: try JSONEncoder().encode(["message": "Configuration updated successfully"])
536 | )
537 | } catch {
538 | return HTTPResponse(
539 | statusCode: .badRequest,
540 | headers: ["Content-Type": "application/json"],
541 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
542 | )
543 | }
544 | }
545 |
546 | func handleGetLocations() async throws -> HTTPResponse {
547 | do {
548 | let vmController = LumeController()
549 | let locations = vmController.getLocations()
550 | return try .json(locations)
551 | } catch {
552 | return .badRequest(message: error.localizedDescription)
553 | }
554 | }
555 |
556 | struct LocationRequest: Codable {
557 | let name: String
558 | let path: String
559 | }
560 |
561 | func handleAddLocation(_ body: Data?) async throws -> HTTPResponse {
562 | guard let body = body,
563 | let request = try? JSONDecoder().decode(LocationRequest.self, from: body)
564 | else {
565 | return HTTPResponse(
566 | statusCode: .badRequest,
567 | headers: ["Content-Type": "application/json"],
568 | body: try JSONEncoder().encode(APIError(message: "Invalid request body"))
569 | )
570 | }
571 |
572 | do {
573 | let vmController = LumeController()
574 | try vmController.addLocation(name: request.name, path: request.path)
575 |
576 | return HTTPResponse(
577 | statusCode: .ok,
578 | headers: ["Content-Type": "application/json"],
579 | body: try JSONEncoder().encode([
580 | "message": "Location added successfully",
581 | "name": request.name,
582 | "path": request.path,
583 | ])
584 | )
585 | } catch {
586 | return HTTPResponse(
587 | statusCode: .badRequest,
588 | headers: ["Content-Type": "application/json"],
589 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
590 | )
591 | }
592 | }
593 |
594 | func handleRemoveLocation(_ name: String) async throws -> HTTPResponse {
595 | do {
596 | let vmController = LumeController()
597 | try vmController.removeLocation(name: name)
598 | return HTTPResponse(
599 | statusCode: .ok,
600 | headers: ["Content-Type": "application/json"],
601 | body: try JSONEncoder().encode(["message": "Location removed successfully"])
602 | )
603 | } catch {
604 | return HTTPResponse(
605 | statusCode: .badRequest,
606 | headers: ["Content-Type": "application/json"],
607 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
608 | )
609 | }
610 | }
611 |
612 | func handleSetDefaultLocation(_ name: String) async throws -> HTTPResponse {
613 | do {
614 | let vmController = LumeController()
615 | try vmController.setDefaultLocation(name: name)
616 | return HTTPResponse(
617 | statusCode: .ok,
618 | headers: ["Content-Type": "application/json"],
619 | body: try JSONEncoder().encode(["message": "Default location set successfully"])
620 | )
621 | } catch {
622 | return HTTPResponse(
623 | statusCode: .badRequest,
624 | headers: ["Content-Type": "application/json"],
625 | body: try JSONEncoder().encode(APIError(message: error.localizedDescription))
626 | )
627 | }
628 | }
629 |
630 | // MARK: - Log Handlers
631 |
632 | func handleGetLogs(type: String?, lines: Int?) async throws -> HTTPResponse {
633 | do {
634 | let logType = type?.lowercased() ?? "all"
635 | let infoPath = "/tmp/lume_daemon.log"
636 | let errorPath = "/tmp/lume_daemon.error.log"
637 |
638 | let fileManager = FileManager.default
639 | var response: [String: String] = [:]
640 |
641 | // Function to read log files
642 | func readLogFile(path: String) -> String? {
643 | guard fileManager.fileExists(atPath: path) else {
644 | return nil
645 | }
646 |
647 | do {
648 | let content = try String(contentsOfFile: path, encoding: .utf8)
649 |
650 | // If lines parameter is provided, return only the specified number of lines from the end
651 | if let lineCount = lines {
652 | let allLines = content.components(separatedBy: .newlines)
653 | let startIndex = max(0, allLines.count - lineCount)
654 | let lastLines = Array(allLines[startIndex...])
655 | return lastLines.joined(separator: "\n")
656 | }
657 |
658 | return content
659 | } catch {
660 | return "Error reading log file: \(error.localizedDescription)"
661 | }
662 | }
663 |
664 | // Get logs based on requested type
665 | if logType == "info" || logType == "all" {
666 | response["info"] = readLogFile(path: infoPath) ?? "Info log file not found"
667 | }
668 |
669 | if logType == "error" || logType == "all" {
670 | response["error"] = readLogFile(path: errorPath) ?? "Error log file not found"
671 | }
672 |
673 | return try .json(response)
674 | } catch {
675 | return .badRequest(message: error.localizedDescription)
676 | }
677 | }
678 |
679 | // MARK: - Private Helper Methods
680 |
681 | nonisolated private func startVM(
682 | name: String,
683 | noDisplay: Bool,
684 | sharedDirectories: [SharedDirectory] = [],
685 | recoveryMode: Bool = false,
686 | storage: String? = nil
687 | ) {
688 | Logger.info(
689 | "Starting VM in detached task",
690 | metadata: [
691 | "name": name,
692 | "noDisplay": "\(noDisplay)",
693 | "recoveryMode": "\(recoveryMode)",
694 | "storage": String(describing: storage),
695 | ])
696 |
697 | Task.detached { @MainActor @Sendable in
698 | Logger.info("Background task started for VM", metadata: ["name": name])
699 | do {
700 | Logger.info("Creating VM controller in background task", metadata: ["name": name])
701 | let vmController = LumeController()
702 |
703 | Logger.info(
704 | "Calling runVM on controller",
705 | metadata: [
706 | "name": name,
707 | "noDisplay": "\(noDisplay)",
708 | ])
709 | try await vmController.runVM(
710 | name: name,
711 | noDisplay: noDisplay,
712 | sharedDirectories: sharedDirectories,
713 | recoveryMode: recoveryMode,
714 | storage: storage
715 | )
716 | Logger.info("VM started successfully in background task", metadata: ["name": name])
717 | } catch {
718 | Logger.error(
719 | "Failed to start VM in background task",
720 | metadata: [
721 | "name": name,
722 | "error": error.localizedDescription,
723 | ])
724 | }
725 | }
726 | Logger.info("Background task dispatched for VM", metadata: ["name": name])
727 | }
728 | }
729 |
```
--------------------------------------------------------------------------------
/blog/build-your-own-operator-on-macos-2.md:
--------------------------------------------------------------------------------
```markdown
1 | # Build Your Own Operator on macOS - Part 2
2 |
3 | _Published on April 27, 2025 by Francesco Bonacci_
4 |
5 | In our [previous post](build-your-own-operator-on-macos-1.md), we built a basic Computer-Use Operator from scratch using OpenAI's `computer-use-preview` model and our [cua-computer](https://pypi.org/project/cua-computer) package. While educational, implementing the control loop manually can be tedious and error-prone.
6 |
7 | In this follow-up, we'll explore our [cua-agent](https://pypi.org/project/cua-agent) framework - a high-level abstraction that handles all the complexity of VM interaction, screenshot processing, model communication, and action execution automatically.
8 |
9 | <div align="center">
10 | <video src="https://github.com/user-attachments/assets/0be7e3e3-eead-4646-a4a3-5bb392501ee7" width="600" controls></video>
11 | </div>
12 |
13 | ## What You'll Learn
14 |
15 | By the end of this tutorial, you'll be able to:
16 |
17 | - Set up the `cua-agent` framework with various agent loop types and model providers
18 | - Understand the different agent loop types and their capabilities
19 | - Work with local models for cost-effective workflows
20 | - Use a simple UI for your operator
21 |
22 | **Prerequisites:**
23 |
24 | - Completed setup from Part 1 ([lume CLI installed](https://github.com/trycua/cua?tab=readme-ov-file#option-2-full-computer-use-agent-capabilities), macOS CUA image already pulled)
25 | - Python 3.10+. We recommend using Conda (or Anaconda) to create an ad hoc Python environment.
26 | - API keys for OpenAI and/or Anthropic (optional for local models)
27 |
28 | **Estimated Time:** 30-45 minutes
29 |
30 | ## Introduction to cua-agent
31 |
32 | The `cua-agent` framework is designed to simplify building Computer-Use Agents. It abstracts away the complex interaction loop we built manually in Part 1, letting you focus on defining tasks rather than implementing the machinery. Among other features, it includes:
33 |
34 | - **Multiple Provider Support**: Works with OpenAI, Anthropic, UI-Tars, local models (via Ollama), or any OpenAI-compatible model (e.g. LM Studio, vLLM, LocalAI, OpenRouter, Groq, etc.)
35 | - **Flexible Loop Types**: Different implementations optimized for various models (e.g. OpenAI vs. Anthropic)
36 | - **Structured Responses**: Clean, consistent output following the OpenAI Agent SDK specification we touched on in Part 1
37 | - **Local Model Support**: Run cost-effectively with locally hosted models (Ollama, LM Studio, vLLM, LocalAI, etc.)
38 | - **Gradio UI**: Optional visual interface for interacting with your agent
39 |
40 | ## Installation
41 |
42 | Let's start by installing the `cua-agent` package. You can install it with all features or selectively install only what you need.
43 |
44 | From your python 3.10+ environment, run:
45 |
46 | ```bash
47 | # For all features
48 | pip install "cua-agent[all]"
49 |
50 | # Or selectively install only what you need
51 | pip install "cua-agent[openai]" # OpenAI support
52 | pip install "cua-agent[anthropic]" # Anthropic support
53 | pip install "cua-agent[uitars]" # UI-Tars support
54 | pip install "cua-agent[omni]" # OmniParser + VLMs support
55 | pip install "cua-agent[ui]" # Gradio UI
56 | ```
57 |
58 | ## Setting Up Your Environment
59 |
60 | Before running any code examples, let's set up a proper environment:
61 |
62 | 1. **Create a new directory** for your project:
63 |
64 | ```bash
65 | mkdir cua-agent-tutorial
66 | cd cua-agent-tutorial
67 | ```
68 |
69 | 2. **Set up a Python environment** using one of these methods:
70 |
71 | **Option A: Using conda command line**
72 |
73 | ```bash
74 | # Using conda
75 | conda create -n cua-agent python=3.10
76 | conda activate cua-agent
77 | ```
78 |
79 | **Option B: Using Anaconda Navigator UI**
80 | - Open Anaconda Navigator
81 | - Click on "Environments" in the left sidebar
82 | - Click the "Create" button at the bottom
83 | - Name your environment "cua-agent"
84 | - Select Python 3.10
85 | - Click "Create"
86 | - Once created, select the environment and click "Open Terminal" to activate it
87 |
88 | **Option C: Using venv**
89 |
90 | ```bash
91 | python -m venv cua-env
92 | source cua-env/bin/activate # On macOS/Linux
93 | ```
94 |
95 | 3. **Install the cua-agent package**:
96 |
97 | ```bash
98 | pip install "cua-agent[all]"
99 | ```
100 |
101 | 4. **Set up your API keys as environment variables**:
102 |
103 | ```bash
104 | # For OpenAI models
105 | export OPENAI_API_KEY=your_openai_key_here
106 |
107 | # For Anthropic models (if needed)
108 | export ANTHROPIC_API_KEY=your_anthropic_key_here
109 | ```
110 |
111 | 5. **Create a Python file or notebook**:
112 |
113 | **Option A: Create a Python script**
114 |
115 | ```bash
116 | # For a Python script
117 | touch cua_agent_example.py
118 | ```
119 |
120 | **Option B: Use VS Code notebooks**
121 | - Open VS Code
122 | - Install the Python extension if you haven't already
123 | - Create a new file with a `.ipynb` extension (e.g., `cua_agent_tutorial.ipynb`)
124 | - Select your Python environment when prompted
125 | - You can now create and run code cells in the notebook interface
126 |
127 | Now you're ready to run the code examples!
128 |
129 | ## Understanding Agent Loops
130 |
131 | If you recall from Part 1, we had to implement a custom interaction loop to interact with the compute-use-preview model.
132 |
133 | In the `cua-agent` framework, an **Agent Loop** is the core abstraction that implements the continuous interaction cycle between an AI model and the computer environment. It manages the flow of:
134 |
135 | 1. Capturing screenshots of the computer's state
136 | 2. Processing these screenshots (with or without UI element detection)
137 | 3. Sending this visual context to an AI model along with the task instructions
138 | 4. Receiving the model's decisions on what actions to take
139 | 5. Safely executing these actions in the environment
140 | 6. Repeating this cycle until the task is complete
141 |
142 | The loop handles all the complex error handling, retries, context management, and model-specific interaction patterns so you don't have to implement them yourself.
143 |
144 | While the core concept remains the same across all agent loops, different AI models require specialized handling for optimal performance. To address this, the framework provides 4 different agent loop implementations, each designed for different computer-use modalities.
145 | | Agent Loop | Supported Models | Description | Set-Of-Marks |
146 | |:-----------|:-----------------|:------------|:-------------|
147 | | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA Preview model | Not Required |
148 | | `AgentLoop.ANTHROPIC` | • `claude-sonnet-4-5-20250929`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use Beta Tools | Not Required |
149 | | `AgentLoop.UITARS` | • `ByteDance-Seed/UI-TARS-1.5-7B` | Uses ByteDance's UI-TARS 1.5 model | Not Required |
150 | | `AgentLoop.OMNI` | • `claude-sonnet-4-5-20250929`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama or OpenAI-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
151 |
152 | Each loop handles the same basic pattern we implemented manually in Part 1:
153 |
154 | 1. Take a screenshot of the VM
155 | 2. Send the screenshot and task to the AI model
156 | 3. Receive an action to perform
157 | 4. Execute the action
158 | 5. Repeat until the task is complete
159 |
160 | ### Why Different Agent Loops?
161 |
162 | The `cua-agent` framework provides multiple agent loop implementations to abstract away the complexity of interacting with different CUA models. Each provider has unique API structures, response formats, conventions and capabilities that require specialized handling:
163 |
164 | - **OpenAI Loop**: Uses the Responses API with a specific `computer_call_output` format for sending screenshots after actions. Requires handling safety checks and maintains a chain of requests using `previous_response_id`.
165 |
166 | - **Anthropic Loop**: Implements a [multi-agent loop pattern](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use#understanding-the-multi-agent-loop) with a sophisticated message handling system, supporting various API providers (Anthropic, Bedrock, Vertex) with token management and prompt caching capabilities.
167 |
168 | - **UI-TARS Loop**: Requires custom message formatting and specialized parsing to extract actions from text responses using a "box token" system for UI element identification.
169 |
170 | - **OMNI Loop**: Uses [Microsoft's OmniParser](https://github.com/microsoft/OmniParser) to create a [Set-of-Marks (SoM)](https://arxiv.org/abs/2310.11441) representation of the UI, enabling any vision-language model to interact with interfaces without specialized UI training.
171 |
172 | - **AgentLoop.OMNI**: The most flexible option that works with virtually any vision-language model including local and open-source ones. Perfect for cost-effective development or when you need to use models without native computer-use capabilities.
173 |
174 | These abstractions allow you to easily switch between providers without changing your application code. All loop implementations are available in the [cua-agent GitHub repository](https://github.com/trycua/cua/tree/main/libs/python/agent).
175 |
176 | Choosing the right agent loop depends not only on your API access and technical requirements but also on the specific tasks you need to accomplish. To make an informed decision, it's helpful to understand how these underlying models perform across different computing environments – from desktop operating systems to web browsers and mobile interfaces.
177 |
178 | ## Computer-Use Model Capabilities
179 |
180 | The performance of different Computer-Use models varies significantly across tasks. These benchmark evaluations measure an agent's ability to follow instructions and complete real-world tasks in different computing environments.
181 |
182 | | Benchmark type | Benchmark | UI-TARS-1.5 | OpenAI CUA | Claude 3.7 | Previous SOTA | Human |
183 | | ---------------- | ------------------------------------------------------------------ | ----------- | ---------- | ---------- | --------------- | ----- |
184 | | **Computer Use** | [OSworld](https://arxiv.org/abs/2404.07972) (100 steps) | **42.5** | 36.4 | 28 | 38.1 (200 step) | 72.4 |
185 | | | [Windows Agent Arena](https://arxiv.org/abs/2409.08264) (50 steps) | **42.1** | - | - | 29.8 | - |
186 | | **Browser Use** | [WebVoyager](https://arxiv.org/abs/2401.13919) | 84.8 | **87** | 84.1 | 87 | - |
187 | | | [Online-Mind2web](https://arxiv.org/abs/2504.01382) | **75.8** | 71 | 62.9 | 71 | - |
188 | | **Phone Use** | [Android World](https://arxiv.org/abs/2405.14573) | **64.2** | - | - | 59.5 | - |
189 |
190 | ### When to Use Each Loop
191 |
192 | - **AgentLoop.OPENAI**: Choose when you have OpenAI Tier 3 access and need the most capable computer-use agent for web-based tasks. Uses the same [OpenAI Computer-Use Loop](https://platform.openai.com/docs/guides/tools-computer-use) as Part 1, delivering strong performance on browser-based benchmarks.
193 |
194 | - **AgentLoop.ANTHROPIC**: Ideal for users with Anthropic API access who need strong reasoning capabilities with computer-use abilities. Works with `claude-sonnet-4-5-20250929` and `claude-3-7-sonnet-20250219` models following [Anthropic's Computer-Use tools](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use#understanding-the-multi-agent-loop).
195 |
196 | - **AgentLoop.UITARS**: Best for scenarios requiring more powerful OS/desktop, and latency-sensitive automation, as UI-TARS-1.5 leads in OS capabilities benchmarks. Requires running the model locally or accessing it through compatible endpoints (e.g. on Hugging Face).
197 |
198 | - **AgentLoop.OMNI**: The most flexible option that works with virtually any vision-language model including local and open-source ones. Perfect for cost-effective development or when you need to use models without native computer-use capabilities.
199 |
200 | Now that we understand the capabilities and strengths of different models, let's see how easy it is to implement a Computer-Use Agent using the `cua-agent` framework. Let's look at the implementation details.
201 |
202 | ## Creating Your First Computer-Use Agent
203 |
204 | With the `cua-agent` framework, creating a Computer-Use Agent becomes remarkably straightforward. The framework handles all the complexities of model interaction, screenshot processing, and action execution behind the scenes. Let's look at a simple example of how to build your first agent:
205 |
206 | **How to run this example:**
207 |
208 | 1. Create a new file named `simple_task.py` in your text editor or IDE (like VS Code, PyCharm, or Cursor)
209 | 2. Copy and paste the following code:
210 |
211 | ```python
212 | import asyncio
213 | from computer import Computer
214 | from agent import ComputerAgent
215 |
216 | async def run_simple_task():
217 | async with Computer() as macos_computer:
218 | # Create agent with OpenAI loop
219 | agent = ComputerAgent(
220 | model="openai/computer-use-preview",
221 | tools=[macos_computer]
222 | )
223 |
224 | # Define a simple task
225 | task = "Open Safari and search for 'Python tutorials'"
226 |
227 | # Run the task and process responses
228 | async for result in agent.run(task):
229 | print(f"Action: {result.get('text')}")
230 |
231 | # Run the example
232 | if __name__ == "__main__":
233 | asyncio.run(run_simple_task())
234 | ```
235 |
236 | 3. Save the file
237 | 4. Open a terminal, navigate to your project directory, and run:
238 |
239 | ```bash
240 | python simple_task.py
241 | ```
242 |
243 | 5. The code will initialize the macOS virtual machine, create an agent, and execute the task of opening Safari and searching for Python tutorials.
244 |
245 | You can also run this in a VS Code notebook:
246 |
247 | 1. Create a new notebook in VS Code (.ipynb file)
248 | 2. Copy the code into a cell (without the `if __name__ == "__main__":` part)
249 | 3. Run the cell to execute the code
250 |
251 | You can find the full code in our [notebook](https://github.com/trycua/cua/blob/main/notebooks/blog/build-your-own-operator-on-macos-2.ipynb).
252 |
253 | Compare this to the manual implementation from Part 1 - we've reduced dozens of lines of code to just a few. The cua-agent framework handles all the complex logic internally, letting you focus on the overarching agentic system.
254 |
255 | ## Working with Multiple Tasks
256 |
257 | Another advantage of the cua-agent framework is easily chaining multiple tasks. Instead of managing complex state between tasks, you can simply provide a sequence of instructions to be executed in order:
258 |
259 | **How to run this example:**
260 |
261 | 1. Create a new file named `multi_task.py` with the following code:
262 |
263 | ```python
264 | import asyncio
265 | from computer import Computer
266 | from agent import ComputerAgent
267 |
268 | async def run_multi_task_workflow():
269 | async with Computer() as macos_computer:
270 | agent = ComputerAgent(
271 | model="anthropic/claude-sonnet-4-5-20250929",
272 | tools=[macos_computer]
273 | )
274 |
275 | tasks = [
276 | "Open Safari and go to github.com",
277 | "Search for 'trycua/cua'",
278 | "Open the repository page",
279 | "Click on the 'Issues' tab",
280 | "Read the first open issue"
281 | ]
282 |
283 | for i, task in enumerate(tasks):
284 | print(f"\nTask {i+1}/{len(tasks)}: {task}")
285 | async for result in agent.run(task):
286 | # Print just the action description for brevity
287 | if result.get("text"):
288 | print(f" → {result.get('text')}")
289 | print(f"✅ Task {i+1} completed")
290 |
291 | if __name__ == "__main__":
292 | asyncio.run(run_multi_task_workflow())
293 | ```
294 |
295 | 2. Save the file
296 | 3. Make sure you have set your Anthropic API key:
297 | ```bash
298 | export ANTHROPIC_API_KEY=your_anthropic_key_here
299 | ```
300 | 4. Run the script:
301 | ```bash
302 | python multi_task.py
303 | ```
304 |
305 | This pattern is particularly useful for creating workflows that navigate through multiple steps of an application or process. The agent maintains visual context between tasks, making it more likely to successfully complete complex sequences of actions.
306 |
307 | ## Understanding the Response Format
308 |
309 | Each action taken by the agent returns a structured response following the OpenAI Agent SDK specification. This standardized format makes it easy to extract detailed information about what the agent is doing and why:
310 |
311 | ```python
312 | async for result in agent.run(task):
313 | # Basic information
314 | print(f"Response ID: {result.get('id')}")
315 | print(f"Response Text: {result.get('text')}")
316 |
317 | # Detailed token usage statistics
318 | usage = result.get('usage')
319 | if usage:
320 | print(f"Input Tokens: {usage.get('input_tokens')}")
321 | print(f"Output Tokens: {usage.get('output_tokens')}")
322 |
323 | # Reasoning and actions
324 | for output in result.get('output', []):
325 | if output.get('type') == 'reasoning':
326 | print(f"Reasoning: {output.get('summary', [{}])[0].get('text')}")
327 | elif output.get('type') == 'computer_call':
328 | action = output.get('action', {})
329 | print(f"Action: {action.get('type')} at ({action.get('x')}, {action.get('y')})")
330 | ```
331 |
332 | This structured format allows you to:
333 |
334 | - Log detailed information about agent actions
335 | - Provide real-time feedback to users
336 | - Track token usage for cost monitoring
337 | - Access the reasoning behind decisions for debugging or user explanation
338 |
339 | ## Using Local Models with OMNI
340 |
341 | One of the most powerful features of the framework is the ability to use local models via the OMNI loop. This approach dramatically reduces costs while maintaining acceptable reliability for many agentic workflows:
342 |
343 | **How to run this example:**
344 |
345 | 1. First, you'll need to install Ollama for running local models:
346 | - Visit [ollama.com](https://ollama.com) and download the installer for your OS
347 | - Follow the installation instructions
348 | - Pull the Gemma 3 model:
349 | ```bash
350 | ollama pull gemma3:4b-it-q4_K_M
351 | ```
352 |
353 | 2. Create a file named `local_model.py` with this code:
354 |
355 | ```python
356 | import asyncio
357 | from computer import Computer
358 | from agent import ComputerAgent
359 |
360 | async def run_with_local_model():
361 | async with Computer() as macos_computer:
362 | agent = ComputerAgent(
363 | model="omniparser+ollama_chat/gemma3",
364 | tools=[macos_computer]
365 | )
366 |
367 | task = "Open the Calculator app and perform a simple calculation"
368 |
369 | async for result in agent.run(task):
370 | print(f"Action: {result.get('text')}")
371 |
372 | if __name__ == "__main__":
373 | asyncio.run(run_with_local_model())
374 | ```
375 |
376 | 3. Run the script:
377 | ```bash
378 | python local_model.py
379 | ```
380 |
381 | You can also use other local model servers with the OAICOMPAT provider, which enables compatibility with any API endpoint following the OpenAI API structure:
382 |
383 | ```python
384 | agent = ComputerAgent(
385 | model=LLM(
386 | provider=LLMProvider.OAICOMPAT,
387 | name="gemma-3-12b-it",
388 | provider_base_url="http://localhost:1234/v1" # LM Studio endpoint
389 | ),
390 | tools=[macos_computer]
391 | )
392 | ```
393 |
394 | Common local endpoints include:
395 |
396 | - LM Studio: `http://localhost:1234/v1`
397 | - vLLM: `http://localhost:8000/v1`
398 | - LocalAI: `http://localhost:8080/v1`
399 | - Ollama with OpenAI compat: `http://localhost:11434/v1`
400 |
401 | This approach is perfect for:
402 |
403 | - Development and testing without incurring API costs
404 | - Offline or air-gapped environments where API access isn't possible
405 | - Privacy-sensitive applications where data can't leave your network
406 | - Experimenting with different models to find the best fit for your use case
407 |
408 | ## Deploying and Using UI-TARS
409 |
410 | UI-TARS is ByteDance's Computer-Use model designed for navigating OS-level interfaces. It shows excellent performance on desktop OS tasks. To use UI-TARS, you'll first need to deploy the model.
411 |
412 | ### Deployment Options
413 |
414 | 1. **Local Deployment**: Follow the [UI-TARS deployment guide](https://github.com/bytedance/UI-TARS/blob/main/README_deploy.md) to run the model locally.
415 |
416 | 2. **Hugging Face Endpoint**: Deploy UI-TARS on Hugging Face Inference Endpoints, which will give you a URL like:
417 | `https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1`
418 |
419 | 3. **Using with cua-agent**: Once deployed, you can use UI-TARS with the cua-agent framework:
420 |
421 | ```python
422 | agent = ComputerAgent(
423 | model=LLM(
424 | provider=LLMProvider.OAICOMPAT,
425 | name="tgi",
426 | provider_base_url="https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1"
427 | ),
428 | tools=[macos_computer]
429 | )
430 | ```
431 |
432 | UI-TARS is particularly useful for desktop automation tasks, as it shows the highest performance on OS-level benchmarks like OSworld and Windows Agent Arena.
433 |
434 | ## Understanding Agent Responses in Detail
435 |
436 | The `run()` method of your agent yields structured responses that follow the OpenAI Agent SDK specification. This provides a rich set of information beyond just the basic action text:
437 |
438 | ```python
439 | async for result in agent.run(task):
440 | # Basic ID and text
441 | print("Response ID:", result.get("id"))
442 | print("Response Text:", result.get("text"))
443 |
444 | # Token usage statistics
445 | usage = result.get("usage")
446 | if usage:
447 | print("\nUsage Details:")
448 | print(f" Input Tokens: {usage.get('input_tokens')}")
449 | if "input_tokens_details" in usage:
450 | print(f" Input Tokens Details: {usage.get('input_tokens_details')}")
451 | print(f" Output Tokens: {usage.get('output_tokens')}")
452 | if "output_tokens_details" in usage:
453 | print(f" Output Tokens Details: {usage.get('output_tokens_details')}")
454 | print(f" Total Tokens: {usage.get('total_tokens')}")
455 |
456 | # Detailed reasoning and actions
457 | outputs = result.get("output", [])
458 | for output in outputs:
459 | output_type = output.get("type")
460 | if output_type == "reasoning":
461 | print("\nReasoning:")
462 | for summary in output.get("summary", []):
463 | print(f" {summary.get('text')}")
464 | elif output_type == "computer_call":
465 | action = output.get("action", {})
466 | print("\nComputer Action:")
467 | print(f" Type: {action.get('type')}")
468 | print(f" Position: ({action.get('x')}, {action.get('y')})")
469 | if action.get("text"):
470 | print(f" Text: {action.get('text')}")
471 | ```
472 |
473 | This detailed information is invaluable for debugging, logging, and understanding the agent's decision-making process in an agentic system. More details can be found in the [OpenAI Agent SDK Specification](https://platform.openai.com/docs/guides/responses-vs-chat-completions).
474 |
475 | ## Building a Gradio UI
476 |
477 | For a visual interface to your agent, the package also includes a Gradio UI:
478 |
479 | **How to run the Gradio UI:**
480 |
481 | 1. Create a file named `launch_ui.py` with the following code:
482 |
483 | ```python
484 | from agent.ui.gradio.app import create_gradio_ui
485 |
486 | # Create and launch the UI
487 | if __name__ == "__main__":
488 | app = create_gradio_ui()
489 | app.launch(share=False) # Set share=False for local access only
490 | ```
491 |
492 | 2. Install the UI dependencies if you haven't already:
493 |
494 | ```bash
495 | pip install "cua-agent[ui]"
496 | ```
497 |
498 | 3. Run the script:
499 |
500 | ```bash
501 | python launch_ui.py
502 | ```
503 |
504 | 4. Open your browser to the displayed URL (usually http://127.0.0.1:7860)
505 |
506 | **Creating a Shareable Link (Optional):**
507 |
508 | You can also create a temporary public URL to access your Gradio UI from anywhere:
509 |
510 | ```python
511 | # In launch_ui.py
512 | if __name__ == "__main__":
513 | app = create_gradio_ui()
514 | app.launch(share=True) # Creates a public link
515 | ```
516 |
517 | When you run this, Gradio will display both a local URL and a public URL like:
518 |
519 | ```
520 | Running on local URL: http://127.0.0.1:7860
521 | Running on public URL: https://abcd1234.gradio.live
522 | ```
523 |
524 | **Security Note:** Be cautious when sharing your Gradio UI publicly:
525 |
526 | - The public URL gives anyone with the link full access to your agent
527 | - Consider using basic authentication for additional protection:
528 | ```python
529 | app.launch(share=True, auth=("username", "password"))
530 | ```
531 | - Only use this feature for personal or team use, not for production environments
532 | - The temporary link expires when you stop the Gradio application
533 |
534 | This provides:
535 |
536 | - Model provider selection
537 | - Agent loop selection
538 | - Task input field
539 | - Real-time display of VM screenshots
540 | - Action history
541 |
542 | ### Setting API Keys for the UI
543 |
544 | To use the UI with different providers, set your API keys as environment variables:
545 |
546 | ```bash
547 | # For OpenAI models
548 | export OPENAI_API_KEY=your_openai_key_here
549 |
550 | # For Anthropic models
551 | export ANTHROPIC_API_KEY=your_anthropic_key_here
552 |
553 | # Launch with both keys set
554 | OPENAI_API_KEY=your_key ANTHROPIC_API_KEY=your_key python launch_ui.py
555 | ```
556 |
557 | ### UI Settings Persistence
558 |
559 | The Gradio UI automatically saves your configuration to maintain your preferences between sessions:
560 |
561 | - Settings like Agent Loop, Model Choice, Custom Base URL, and configuration options are saved to `.gradio_settings.json` in the project's root directory
562 | - These settings are loaded automatically when you restart the UI
563 | - API keys entered in the custom provider field are **not** saved for security reasons
564 | - It's recommended to add `.gradio_settings.json` to your `.gitignore` file
565 |
566 | ## Advanced Example: GitHub Repository Workflow
567 |
568 | Let's look at a more complex example that automates a GitHub workflow:
569 |
570 | **How to run this advanced example:**
571 |
572 | 1. Create a file named `github_workflow.py` with the following code:
573 |
574 | ```python
575 | import asyncio
576 | import logging
577 | from computer import Computer
578 | from agent import ComputerAgent
579 |
580 | async def github_workflow():
581 | async with Computer(verbosity=logging.INFO) as macos_computer:
582 | agent = ComputerAgent(
583 | model="openai/computer-use-preview",
584 | save_trajectory=True, # Save screenshots for debugging
585 | only_n_most_recent_images=3, # Only keep last 3 images in context
586 | verbosity=logging.INFO,
587 | tools=[macos_computer]
588 | )
589 |
590 | tasks = [
591 | "Look for a repository named trycua/cua on GitHub.",
592 | "Check the open issues, open the most recent one and read it.",
593 | "Clone the repository in users/lume/projects if it doesn't exist yet.",
594 | "Open the repository with Cursor (on the dock, black background and white cube icon).",
595 | "From Cursor, open Composer if not already open.",
596 | "Focus on the Composer text area, then write and submit a task to help resolve the GitHub issue.",
597 | ]
598 |
599 | for i, task in enumerate(tasks):
600 | print(f"\nExecuting task {i+1}/{len(tasks)}: {task}")
601 | async for result in agent.run(task):
602 | print(f"Action: {result.get('text')}")
603 | print(f"✅ Task {i+1}/{len(tasks)} completed")
604 |
605 | if __name__ == "__main__":
606 | asyncio.run(github_workflow())
607 | ```
608 |
609 | 2. Make sure your OpenAI API key is set:
610 |
611 | ```bash
612 | export OPENAI_API_KEY=your_openai_key_here
613 | ```
614 |
615 | 3. Run the script:
616 |
617 | ```bash
618 | python github_workflow.py
619 | ```
620 |
621 | 4. Watch as the agent completes the entire workflow:
622 | - The agent will navigate to GitHub
623 | - Find and investigate issues in the repository
624 | - Clone the repository to the local machine
625 | - Open it in Cursor
626 | - Use Cursor's AI features to work on a solution
627 |
628 | This example:
629 |
630 | 1. Searches GitHub for a repository
631 | 2. Reads an issue
632 | 3. Clones the repository
633 | 4. Opens it in an IDE
634 | 5. Uses AI to write a solution
635 |
636 | ## Comparing Implementation Approaches
637 |
638 | Let's compare our manual implementation from Part 1 with the framework approach:
639 |
640 | ### Manual Implementation (Part 1)
641 |
642 | - Required writing custom code for the interaction loop
643 | - Needed explicit handling of different action types
644 | - Required direct management of the OpenAI API calls
645 | - Around 50-100 lines of code for basic functionality
646 | - Limited to OpenAI's computer-use model
647 |
648 | ### Framework Implementation (Part 2)
649 |
650 | - Abstracts the interaction loop
651 | - Handles all action types automatically
652 | - Manages API calls internally
653 | - Only 10-15 lines of code for the same functionality
654 | - Works with multiple model providers
655 | - Includes UI capabilities
656 |
657 | ## Conclusion
658 |
659 | The `cua-agent` framework transforms what was a complex implementation task into a simple, high-level interface for building Computer-Use Agents. By abstracting away the technical details, it lets you focus on defining the tasks rather than the machinery.
660 |
661 | ### When to Use Each Approach
662 |
663 | - **Manual Implementation (Part 1)**: When you need complete control over the interaction loop or are implementing a custom solution
664 | - **Framework (Part 2)**: For most applications where you want to quickly build and deploy Computer-Use Agents
665 |
666 | ### Next Steps
667 |
668 | With the basics covered, you might want to explore:
669 |
670 | - Customizing the agent's behavior with additional parameters
671 | - Building more complex workflows spanning multiple applications
672 | - Integrating your agent into other applications
673 | - Contributing to the open-source project on GitHub
674 |
675 | ### Resources
676 |
677 | - [cua-agent GitHub repository](https://github.com/trycua/cua/tree/main/libs/python/agent)
678 | - [Agent Notebook Examples](https://github.com/trycua/cua/blob/main/notebooks/agent_nb.ipynb)
679 | - [OpenAI Agent SDK Specification](https://platform.openai.com/docs/api-reference/responses)
680 | - [Anthropic API Documentation](https://docs.anthropic.com/en/api/getting-started)
681 | - [UI-TARS GitHub](https://github.com/ByteDance/UI-TARS)
682 | - [OmniParser GitHub](https://github.com/microsoft/OmniParser)
683 |
```