The next step is a persistent assistant session that can plan, act, inspect results, and adapt across your applications and machines. You should be able to keep talking, change direction, or leave while the work continues.
19 September 2026 · Primary-source research and inspection of the current working tree, based on commit 2870733fd0 with uncommitted changes. Architecture proposal, not an implemented or benchmarked capability.
The product promise
"Tell Circe what you want accomplished. It uses the context and capabilities you have granted, carries the work through, and involves you when your judgment is needed."
FRIDAY is a useful interaction reference: shared context, continuity, initiative within an assignment, and the ability to interrupt. A particular workflow, including file transfer, is only one test of that promise.
The first release should earn that experience within a declared capability set. Universal competence is a research ambition. Owning the machines does not remove application permissions, operating-system restrictions, or model errors.
What the research supports
Implemented reference
UFO³: coordination across devices
Microsoft's open-source system separates a reasoning agent from an executor that schedules device work and responds to results. Its paper reports 70.9% complete-task success on 55 cross-device tasks across five machines. That is an authors' benchmark result, not a reliability estimate for Circe. Paper, revised March 2026.
Use explicit dependencies, device capabilities, and result-driven replanning. Start with one request owner and a few sequential steps; Circe does not need the entire UFO scheduling framework.
Implemented reference
UFO²: applications need several control paths
UFO² combines Windows UI Automation, visual parsing, and application APIs. It separates application execution from cross-application coordination. Paper · Hybrid action implementation.
Adopt the combination of structured APIs and UI interaction. Do not make every operation pay for a screenshot, model call, and mouse movement.
Experimental platform
WebMCP: websites describe callable actions
Chrome documents an origin trial from Chrome 149 and a local testing flag. Sites can expose JavaScript tools or annotated forms. It requires site participation; discovery happens in the visited site. Chrome documentation, updated August 2026.
Add WebMCP as an optional browser capability. Ordinary browsing must work when it is absent. This is neither the machine mesh nor Circe's durable task runtime.
Implemented reference
OpenClaw: a resident service and remote capabilities
OpenClaw's gateway owns sessions and connects clients and capability-advertising nodes over WebSocket. Its browser tooling distinguishes a managed profile from an attached signed-in session. Architecture · Browser.
Keep Circe's existing identity, routing, and lifecycle. A future OpenClaw integration can be an explicitly delegated executor. Importing a second gateway now would create overlapping ownership.
A recent Unified Agent preprint argues for carrying compact interaction state across devices and time. It is useful evidence for designing context explicitly; it is not proof of a generally reliable personal assistant.
Where Circe is constrained today
These are source observations in the inspected working tree. Some design documents still describe preserving an upstream boundary; the current repository instructions supersede that premise. The execution engine is Circe code and can be changed.
Observed behavior
Consequence
Evidence
Compound routing rejects steps naming different nodes.
A request cannot naturally compose execution across machines.
The autonomous surface loop has a default 24-step budget, six action choices, and one supplied typeText value.
It is a bounded screen operator, not a general planner that can produce several different field values, discover tools, or revise a multi-application approach.
packages/circe-core/src/computerUse.ts:314
That loop accepts model-selected "done" after at least one applied action.
An action count is not a goal check. It also rejects a correctly observed already-satisfied goal with zero actions.
packages/circe-core/src/computerUse.ts:381
The autonomous desktop mission accepts Linux backends only. Lower-level macOS and Windows input helpers exist separately.
Raw desktop support must not be confused with cross-platform autonomous operation.
Retain qualified node/project/task identities, authoritative host validation, typed clarification, authenticated routes, provider adapters, event storage, and the input queue's release guarantees. Extend these owners instead of building a competing assistant stack.
The architecture I would build
One assistant session owns the conversation and references ongoing requests. Each request has one durable owner on an existing execution-capable node. The provider plans and revises work; deterministic host code resolves targets, checks authority, schedules execution, and records outcomes.
You: voice, text, a selected object, or a shared screenFull / Controller desktop · web · mobile. Origin device is explicit.
↓
Assistant session on its assigned home nodeConversation · current referents · preferences · ongoing request references. Voice connection may close without ending the work.
↓
Circe's existing orchestration engine, extendedOrdinary provider adapter proposes next actions → host validates → durable operation dispatch → observation and verification → next decision.
Node BLocal projects and files Existing coding providers Its own app capabilities
Other devicesOnly advertised capabilities Mobile-specific integrations Future external executors
↑ real results, blockers, and changed state
Shared presentationShort spoken updates when useful · result cards · preview or application handoff · inspectable activity · stop and correction.
A fast conversation and a durable execution loop
The conversational layer submits a request and receives its accepted identity promptly. Work continues independently. A later "change that" binds to that identity and revises the request. "Stop" enters host cancellation directly; it does not wait for the planning model to finish thinking.
Realtime tool support varies across models. For example, Google's documentation distinguishes synchronous and asynchronous live tool calling. Circe should implement background work in its own runtime, so conversation responsiveness does not depend on one voice model's behavior. Live tool-calling documentation.
Keep the small classifier for navigation and grounding. Give open-ended planning to an ordinary provider session with discoverable tools. Generated text and reasoning can be flexible while target identity and execution authority remain deterministic.
Reuse a running provider session across actions and retrieve only the tools and context relevant to its next step. Avoid starting a CLI or loading every tool schema for each click. If the capability boundary cannot enforce the required limits on a provider's shell or other execution path, that provider must not receive the restricted unattended mode.
Context is evidence with an origin
Track a small set of named devices, selected objects, recent artifacts, active applications, and ongoing requests. Each observed fact carries its owning node, source, time, and freshness. Preferences are separately inspectable, correctable, and removable. A stale browser tab or remembered filename is a candidate to check, never authority to act.
"This", "here", and "the other one" resolve against explicit selection and recent interaction. The screen you are viewing, the machine you are speaking from, and the node executing work are separate fields. Observation must not silently authorize a target.
Start with event-driven capability and task updates, plus on-demand app observations. Avoid indexing every screen or running a model continuously while idle. Owning the runtime also does not imply local inference: cloud providers still receive the context supplied to them.
Minimal contracts and ownership
CapabilityRef = node + adapter + resource/session + operation + version
Observation = resource ref + source + observed time + revision + bounded data
Operation = request ref + operation id + target + arguments
+ grant + expected revision + deadline
Result = applied | already-satisfied | blocked | failed | effect-unknown
+ observed output + supported cancellation/compensation
These are proposed meanings, not current API names. Add them to existing contracts and event storage. A request keeps its objective, constraints, linked operations, pending input, and next unresolved step. Store authoritative operation results at the executing node; the request owner stores references and orchestration decisions. Do not copy every provider lifecycle into a second workflow database.
A new assistant session type must remove the mandatory project assumption across contracts, persistence, server, web, desktop, mobile, and each provider adapter. Coding operations still reference real projects. Where a CLI needs a working directory, provide a private session workspace without creating a fake user project.
How browser and desktop control should work
Prefer the most precise available operation with a verifiable result. That often means an application API or MCP tool, then WebMCP on a participating site, then DOM/accessibility interaction, then vision and input for surfaces without useful structure. This is a selection policy, not an automatic retry ladder after a write may already have happened.
The browser bridge belongs on the machine with the browser
Build a Chromium extension paired through native messaging to the existing Circe node. It carries tab, frame, document, origin, and account-context references. It can expose permitted page observations and actions through the same node capability boundary. Native messaging is an established extension-to-native transport; it is not itself an authorization policy. Chrome native messaging.
DOM actions should use fresh, scoped element references and actionability checks. Chrome's debugger API is one possible privileged CDP bridge; managed automation sessions can use Playwright locators. An attached personal browser and an isolated managed browser must remain explicit choices with distinct sessions. Never assume their logins are interchangeable. Debugger API · Playwright locators.
WebMCP sits behind a versioned adapter. The September draft is still a Community Group report, not a W3C Standard. Chrome's September documentation uses document.modelContext, including discovery, execution, and tool-change events. Pin and probe the installed browser instead of copying an older navigator.modelContext tutorial. Current draft · Current API documentation.
The extension must validate tab/frame identity at invocation, invalidate handles on navigation, preserve origin restrictions, and report a lost result as uncertain when a write might have occurred. The extension-to-page WebMCP bridge needs an implementation spike on the chosen Chrome build; the standard does not provide a universal remote MCP endpoint.
Platform
Primary control path
Circe decision
Linux desktop
AT-SPI elements and actions; Wayland RemoteDesktop/ScreenCast portals for permitted input and capture. The portal recommends EIS/libei for input. AT-SPI · Portal.
Extend the existing accessibility adapter. Add a portal-backed Wayland route; probe compositor and permission support. Keep other working backends as explicit capabilities.
Windows
UI Automation patterns for invoke, value, selection, and windows; application APIs where available. Microsoft documentation.
Add semantic observation/action support above existing input helpers. Elevation and session boundaries remain real limits.
macOS
Accessibility elements/actions, supported application automation, and permitted capture/input. Apple accessibility model.
Add AX observation and element actions. A raw click helper alone is not a semantic desktop agent.
Android
Intents and app integrations; explicitly enabled accessibility services can inspect and act on UI. Android documentation.
Treat general device automation as a separate feasibility/distribution workstream. Simulator tooling is not evidence that a shipped phone agent has equivalent access.
iPhone / iPad
Circe companion capabilities and user-configured integration flows. App Intents expose app actions through system experiences such as Shortcuts. AppIntent.
Start as a controller and an explicitly limited capability endpoint. App Intents do not imply that Circe can enumerate and remotely execute every other app's actions.
For custom canvases, games, or poorly exposed controls, add a visual grounding adapter that binds candidate regions to a current frame. OmniParser is one reference for screen parsing, not a guarantee of correct targeting. A visual action remains probabilistic; require a fresh observation and verify its effect.
Desktop ownership must span observation and action, not just individual key events. If the user takes over or the target window changes, pause and re-observe. Keep one writer per shared desktop session. Independent API operations can run concurrently; two agents must not compete for the same mouse.
What makes it feel like one assistant
Keep a request alive through time, interruption, and changing context. A long chat transcript is not sufficient. The runtime must know which work remains, which results are established, and what your latest correction changed.
One assigned owner
Use an existing Full or Headless execution node as the session's home. Full and Controller retain speech and control; Controller delegates execution. The home stores the assistant session and runs ordinary provider work. This is a role of an existing node, not a new gateway, launcher, or mesh identity.
If it goes offline, already accepted remote operations can finish locally. New coordination waits for the same owner to return. Automatic failover and ownership election are deliberately outside the first milestone.
Delegation with a real boundary
Existing client pairing is not automatically permission for node A to operate node B. Extend authenticated node routes with target-issued, scoped delegation tied to the originating user and request. Keep provider and application credentials at their owning node. Revoking delegation blocks future operations.
The home dispatches references, authorized arguments, and only needed content. The relay remains a connection broker. It does not become the assistant's state owner.
Resume after a lost connection
Record a stable operation identity and intent before dispatch. The target records acceptance and result. On reconnect, query that identity before retrying. Use application idempotency keys where supported.
A GUI click or external write may succeed before its response is lost. That is effect-unknown: inspect the target state before doing anything else. Never promise exactly-once effects from network retries alone.
Correction and cancellation
A correction changes the request revision; stale planned operations fail validation before dispatch. Already accepted actions may still finish. A stop prevents new work, requests cancellation of active work, and reports the actual result.
Stopping speech, stopping a request, and undoing an effect are different operations. Only offer undo when the adapter has a real compensation action. An offline target means cancellation is pending or unknown, not completed.
MCP's optional Tasks extension supports durable handles, status, later input, and cooperative cancellation. It is useful at integration boundaries; support must be negotiated. Circe should keep its existing event subscriptions internally and adapt external polling only where required. Current Tasks extension.
Tool metadata, page text, and accessibility labels remain untrusted observations. A schema-valid action can still be the wrong action. Host policy must bind effects to the user's request, account, resource, and grant. Unknown tools should not receive broad unattended authority merely because their descriptions claim they are harmless. WebMCP security guidance · MCP annotation limits.
Authorize a useful scope once and retain it for the assignment. Routine authorized steps should proceed without repeated confirmation. Ask when the target or intent is ambiguous, the required scope changes, or an effect needs authority the user has not granted. The model must not satisfy its own human approval through browser or desktop automation.
The interface follows the work
Keep conversation available while work runs. Show the relevant document, application, comparison, preview, or choice when it helps. The detailed coding UI stays accessible for coding tasks. The default assistant surface shows what is being done, where it is happening, and what needs your attention.
Acknowledgment means accepted. Progress means an observed change. Completion means the requested outcome was checked. Silence is appropriate when nothing useful has changed.
Explore the proposed behavior
Illustrative interaction states. These buttons do not run an agent or control a device.
You: Get the demo ready on the workstation. Check it against the feedback I have open here.
Circe: I'll use the selected feedback and the workstation's demo project. You can keep working here.
Runtime: binds the explicit selection on the origin device and the named workstation, records the request, then launches ordinary provider work against the real project. Ambiguous projects require a choice.
Proactivity begins with explicit standing instructions: "when this export finishes, check it and prepare a draft." Register a bounded event trigger with scope, expiry, and cancellation. It should wake the same request runtime, rather than create a second always-thinking assistant. Broad inferred habits can wait.
The next step: a persistent operator for your existing machines
From any Circe client, give one request spanning supported apps and two enrolled nodes. Circe chooses the necessary operations, continues after you close the conversation, accepts corrections, and returns a checked result.
Scope the first complete milestone to the existing Linux desktop path, one Chromium browser bridge, existing coding/file execution, two enrolled nodes, and phone/web/desktop control of the same requests. macOS and Windows semantic adapters, broad mobile execution, and external agent systems can follow behind the same contracts.
Make one ongoing request real. Add an assistant session independent of a project and extend the existing engine with durable operation identities, pending input, cancellation, and verification. Run one local request from conversation through a real capability and back. Do not build a scheduler framework first.
Give the executor useful app access. Connect the Chromium bridge, existing desktop adapter, and existing provider/file tools to a discoverable capability catalog. Generate per-operation arguments through the provider. Preserve host validation. Demonstrate a non-coding request that crosses a website and a desktop application.
Carry the same request across two nodes. Add explicit delegated authority and remote operation recovery. Reuse current identities and routes. Show cross-node status, correction, stop, and resumption from another client.
Add WebMCP behind a feature gate. Prove discovery, invocation, navigation invalidation, and result inspection on a controlled participating site. Re-run the request with WebMCP unavailable through the ordinary browser path. The release must not depend on broad WebMCP adoption.
Let an event continue assigned work. One explicit wait condition, such as a build or export completing, resumes the same request. Surface results from durable state; avoid replaying speech after reconnection.
The first engineering cut
A durable operator request that invokes an existing capability, receives a real result, and remains inspectable after the client disconnects. It must support cancellation and a result check before expanding the tool catalog. In parallel conceptually, the browser bridge can be a small compatibility spike; it should not force a rewrite of voice or providers.
The material migration is removing project-only session assumptions across the existing engine. Do that explicitly. A hidden "personal assistant project" would preserve the constraint in a different place.
Where the implementation belongs
Owner
Change
packages/contracts
Assistant context, qualified capabilities and operations, scoped delegation, verification and uncertain outcomes. Schema changes shared by every client.
Generalize session scope and extend existing durable execution. Keep one lifecycle authority; preserve provider success even if optional checkpoint bookkeeping fails.
packages/circe-core apps/server/src/circe
Grounding, operation policy, conversation-to-request binding, adaptive execution integration. Retire the bounded selector as the sole autonomous planner.
One resident lifecycle, native helper integration, assistant surface, result views, and consistent stop/approval state. Keep each preset's execution restrictions.
All provider adapters
Codex, Claude, Cursor, Grok, and OpenCode receive an explicit capability decision. Text-only providers can use semantic tools; image support is advertised separately. No provider-specific shortcut around policy.
What would prove the step works
Use several kinds of work so the architecture is not designed around one demo: prepare an app for a meeting, compare browser material into a local document, change and verify a coding project, continue work across two machines, and finish a request after an explicit event.
Proposed pilot gate, not measured results
Choose 20 requests across those five categories before tuning the prompts. Run each three times with fresh state and ordinary variations. Target at least 54 of 60 checked outcomes without a person explaining the next click. Record necessary clarification separately from manual rescue. Report elapsed time, model cost, and interventions alongside success.
Every run must preserve correct target identity, authorized effects, and truthful status. A failure may be a clear blocker or uncertain outcome; it must never become a false success. Use independent app or artifact state checks where available, and label visual-only verification accordingly.
Inject a closed tab, expired login, changed element, permission revocation, target disconnect, coordinator restart, duplicate delivery, lost response after a write, user mouse takeover, and mid-run correction. Confirm that the runtime does not duplicate uncertain effects or dispatch stale work.
Measure stop acknowledgment separately from actual cessation. No new action should start after cancellation is accepted at its executing node. In-flight calls may finish; disconnected nodes must show an unresolved stop. Test Full, Controller, and Headless capability boundaries.
OSWorld provides a useful precedent for checking real application state instead of trusting an agent's final sentence. Its site points to a newer 2.0 benchmark; this proposal does not claim a current leaderboard comparison.
Defer automatic failover, a universal personal knowledge graph, uncontrolled background initiative, a plugin marketplace, and importing another complete agent framework. Each should solve a demonstrated limit after the first persistent operator is useful.
Sources and confidence
All links were inspected on 19 September 2026. External capability claims above come from primary documentation or authors' research. Architectural choices are recommendations. No live browser, desktop, mobile, or distributed execution was tested for this briefing.
Android accessibility service and Apple AppIntent. Limited mobile building blocks. Android distribution policy and a specific iOS integration path remain follow-up investigations.
Unified Agent. August 2026 preprint; used for its carried-state design idea, not as a validated production dependency.
Published with the Abbyplan workflow. This briefing changes no Circe source, live process, or user data.