Make voice a reliable remote control

The decision I would make

This is a focused replacement of conversation ownership, control planning, and device execution. Existing node identity, authenticated connections, project/task catalogs, provider adapters, and persistence remain useful. Adding more phrase rules or a more capable model to the present dispatch paths will leave the main failure modes intact.

Scope: recommendations, not an implemented fix. No live browser or desktop was driven for this review. Existing tests were not used as evidence of reliability. The inspected working tree contains concurrent uncommitted fixes atop e0f1301ccb1ae2860442d60c68c4f4e058fd9280; source locators below describe that working snapshot, not a clean commit.

What the code actually does

Six failures in ownership and meaning

The updated web code has already removed the fallback that recast a failed browser mission as coding-provider work. Preserve that change. Historical YouTube messages show preview-versus-Chrome confusion; they do not establish that every older fallback is still present.

Replacement design

One conversation owner; execution stays on its device

The conversation record

An Interaction stores the current goal, input revision, pending question, resolved references, and active operation reference. A question includes its kind, missing slot, known arguments, and owner. It exists before Circe asks it aloud. An answer is interpreted against this state before any project/task classification.

Interaction
  ownerNode + interactionId + revision
  goal: weather | browser | nativeApp | coding | conversation
  pending: question { slot, knownArguments, choices? } | approval | none
  target: qualified device/app reference | unresolved
  operation: qualified operation reference | none

Input relation: answer | correction | new request | cancel

Use the existing server-side pending-state machinery and persistence conventions as the starting point. Generalize their ownership across direct assistant actions. Do not require a fake “assistant project.” Coding still links to an ordinary project-qualified provider task. There is no need to make every provider thread projectless before weather and browser operations can work.

Every client submits the same typed input and renders the resulting state. Clients may cache it, but they do not separately decide what “Ahmedabad” or “yes” resumes. A revision check handles two devices answering the same question. If the owner disconnects, show that fact; do not silently create a replacement interaction on another node.

The device operation

An Operation belongs to the target node and records acceptance, status, cancellation, and the latest effect/result evidence. The interaction coordinates intent; the operation owns physical effects. Reuse the existing SQLite/event projection infrastructure and authenticated node transport. These two records have different jobs; neither requires a new general orchestration engine.

TargetRef = node + desktopSession + application
BrowserTarget = TargetRef + connectorProfile + window + tab

Operation = id + interactionRevision + target + authorizedScope
          + goal + expectedOutcome + status

Step = observationRef + groundedAction + typedArguments
Result = applied | notApplied | unknown | blocked

Operation status includes needsInput, running, verifying,
completed, stopped, blocked, and outcomeUnknown.

Persist operation acceptance before execution and deduplicate retries by operation identity. After a crash or lost response, inspect the existing operation and app state. A crash between a click and its result cannot be solved by claiming “exactly once.” An uncertain effect stays uncertain until it is observed; irreversible actions are not blindly retried.

Keep the public interface small: submit input, interrupt an operation, read state, subscribe to changes. Put device selection, clarification routing, and completion rules behind this interface. Stop must bypass the normal input queue. If the target is offline, distinguish “stop requested” from “stop confirmed.”

Speech follows that state

Introduce a voice-provider adapter that produces finalized utterances with stable host IDs and revisions. The live connection remains live; no push-to-talk is introduced. Do not finalize a request on the first nonempty transcript fragment or because the assistant started speaking. The adapter must map the actual provider's endpointing/delegation protocol into this contract, with one admission path.

Barge-in immediately stops playback. An explicit “stop” cancels the current operation; a correction updates the goal and invalidates future stale steps. These are separate events. A short backchannel does not mark a delegated operation finished.

The host owns factual task progress and pending questions. The voice model can speak naturally and handle conversation, but cannot invent completion or a new unregistered question for an active operation. If the current voice API cannot support this separation, replace that adapter after a measured protocol comparison. A model swap alone is not the proposed fix.

Use existing task/operation state for the UI and bounded live presentation to the origin interaction. Reconnect reads current state; it does not replay old speech. No second delivery ledger, acknowledgment election, or speech replay engine is needed.

How TypeSafe becomes more useful

TypeSafe evaluates typed questions against supplied state. Questions in the same call are evaluated independently. That is useful for independent judgments, but it cannot make a later question depend on an earlier answer inside the same batch. TypeSafe documentation.

1. Resolve the input's relationship first. Supply the active goal, the actual pending slot, and recent relevant results. Choose answer, correction, new request, or cancel. Deterministic state decides which transitions are legal.
2. Resolve authority from real catalogs. Select the qualified device, app, profile, task, or project only when needed. Keep ambiguous candidates explicit. A high confidence score never grants permission.
3. Resolve content separately. Search text is data. Extract a validated source span with token boundaries and bounded length, preserving the user's words. Use hierarchical boundary selection or a schema-validated extraction proposal. Do not truncate an enumeration before the intended phrase is representable.
4. Select an action in context. Offer actions supported by the selected app and current observation. Then request dependent parameters, or offer complete valid action candidates. Independent questions can still be batched.
5. Use uncertainty to recover. Low confidence about the target means observe more, ask, or replan. It must not mean “type into whichever field has focus.”

For unfamiliar multi-step work, allow an ordinary provider adapter to propose a plan or help recover from a stall. The host validates the plan and grounds each executable action. This is optional planning through existing provider infrastructure, not a separate Circe-only execution path. Full nodes can plan locally; Controller nodes route provider planning to an authorized execution node.

Keep common lookups and simple navigation on short typed paths. Avoid a heavyweight coding CLI turn for every click. A plan should carry progress, resolved arguments, and checked outcomes across steps; a short history of action descriptions is insufficient.

Measure candidate coverage separately from classifier accuracy. Calibrate abstention on real utterances, including long phrasing and corrections. Type validity prevents malformed choices. It does not prove intent, permissions, or task completion.

Existing apps, visibly controlled

Browser control needs a browser connection

My choice is a Chrome extension in the user's existing profile, connected by native messaging to Circe's existing desktop runtime. It identifies tabs and exposes scoped DOM/accessibility inspection and actions. The remote phone communicates with the target Circe node over the existing authenticated mesh; it never connects to a public debugging port.

The extension debugger API can attach by tab ID and supports the DOM, accessibility, and input domains needed for this approach. Account for frame contexts, out-of-process iframes, navigation, target closure, and user detachment. These are connector lifecycle states. Chrome debugger API.

Native messaging provides the local extension-to-host channel with an explicit extension allowlist. Package its small connector with Circe. It should connect to the existing desktop owner through authenticated local IPC, without a second tray app, launcher, or updater. Chrome native messaging.

Do not build this around restarting normal Chrome with a default-profile debugging port. Chrome 136 changed those switches to require a nonstandard data directory. That conflicts with the user's request to use their existing signed-in profile. Chrome's debugging change.

Pair each connector instance with an explicit user-visible profile label. Treat tab IDs as session-scoped; invalidate handles after reconnect. Default to visible control of the chosen tab. Keep preview-browser work as an explicit different choice. The first release should promise Chrome on three OSes, not every browser; Edge, Firefox, and Safari require separate compatibility decisions.

Native control uses one contract and three adapters

Platform references: Wayland RemoteDesktop portal; Apple AXUIElement and ScreenCaptureKit; Windows UI Automation patterns and security boundaries. Platform mechanisms are documented; Circe adapter feasibility and coverage still require real-device trials.

One execution loop regardless of observation method

Prefer authorized app APIs when they accomplish the requested visible effect, browser DOM/accessibility for browser work, and native accessibility for native apps. If an app exposes only a canvas, bounded visual control may be necessary. Keep that fallback inside the same targeted operation, with the same cancellation and verification. A screenshot should not cause a handoff to a new task that forgets the goal.

Scope observations to the selected application and relevant container. Expand or page the tree when necessary instead of truncating the whole desktop to its first 60 elements. Bind handles to a session and observation generation; refresh stale handles before mutation.

Serialize competing physical-input sequences per desktop session. Browser operations on isolated tabs can have narrower ownership if they do not steal shared focus. Pause on human takeover. Check cancellation and the interaction revision immediately before every mutation. Release held keys/buttons when stopping.

Advertise both policy and observed readiness: preset permits control; adapter is supported; user session is connected; permission is granted; target app/profile exists. Full and Controller retain their current roles. Headless remains background execution. No silent switch to another desktop when readiness changes.

Required product capability

Proper desktop use is part of the first release

Circe must operate the user's existing desktop session from voice, phone, or another computer. Native app control shares the same conversation and operation owner as browser control. The first Linux release must complete a native task and a task spanning multiple apps; browser-only success does not meet the requirement.

A desktop session owns the moving parts

The target node's desktop runtime owns session readiness, window/app discovery, focus, input sequencing, display geometry, native observation and optional capture. The planner receives a bounded view of the relevant app, including its active dialog and supported actions. It does not assemble a desktop from unrelated screenshot, keyboard and window tools.

Desktop session: node + logged-in session + session generation
Window: session + app instance + window identity
Observation: window + revision + focused element + modal dialog
Action: observation + target element or explicit input target
Transition: source app + intended destination + observed result

Bind each step to one window while allowing the operation to move between authorized apps on the same device. A Save dialog temporarily owns input; a notification or unrelated foreground window does not become the target. Re-resolve references after an app restarts. A stale handle must never silently become a coordinate click.

Coordinates, when necessary, are local to an observed display/window with a recorded scale and geometry revision. A monitor move, resize or scale change invalidates them. Use semantic actions and text/value interfaces where available; AT-SPI exposes application, state, action, selection and editable-text interfaces for this purpose. AT-SPI accessible objects.

Visual control is an explicit part of the same operation

Some native apps expose poor accessibility data. The runtime may inspect a fresh window image and use grounded pointer actions there, retaining the exact device, goal, grant and stop behavior. Bound attempts and detect lack of progress. If neither semantic nor visual observation is sufficient, report the blocked step. Do not switch to an unrelated provider conversation.

Screenshot capture, continuous viewing and input control are separate capabilities. The voice controller need not continuously stream the desktop to act. A phone live view is a subscribed presentation of the target session; closing it does not restart the task. Human input pauses automated input until an explicit resume, and cancellation releases held keys and buttons.

The acceptance task

From the phone: “On my desktop, open the notes file I selected, change its heading to Meeting notes, save it, and show it in the file manager.” Circe must retain the selected file, activate the correct editor, modify the requested heading, handle any save dialog, verify the saved contents using an authorized read or reopen, and reveal the same file in the file manager.

Repeat with a second document/window open, an unexpected dialog, an unsaved-change prompt, a moved window, human mouse takeover and a disconnect after Save. Include a browser-to-native file-picker task as well. These are release requirements, not claims that arbitrary apps already work.

Linux AT-SPI/Wayland or X11, macOS AX and Windows UIA implement this same contract. Each advertises actual operations and limitations. Keep login screens, locked sessions and elevated system dialogs outside the initial promise unless a separately validated capability supports them.

What the user experience should be

These are proposed trajectories, not recordings of working software.

Implementation order

Deliver complete paths, then remove their old owners

The previous next-interface briefing describes the wider product direction. This plan narrows the first release to the failures observed here. Do not start with a universal personal knowledge graph, a new agent framework, or adapters for dozens of apps.

Concrete ownership changes

Roll out by interaction capability/version. A new interaction has exactly one routing owner; do not run old and new dispatch together. Older clients receive an explicit unsupported-version result for a new control flow. Preserve old durable task state and shipped migration IDs. Keep a rollback for new admissions without pretending an already accepted operation vanished.

Acceptance

Prove it on real devices and real language

Use held-out spoken requests and independently inspected app state. The engineer should not rescue a failed run by rewriting the prompt into the one phrase the implementation accepts. Record task outcome separately from the assistant's own success statement.

Before tuning, select at least 20 utterance variations across these scenarios and retain unseen variants for acceptance. Record correct completion, necessary clarification, manual rescue, wrong target, duplicate effect and false success separately. Require zero observed wrong-target actions, duplicate effects and false completions in the release set; that is a release gate, not a proof that failures are impossible.

Measure end-of-speech to admission, admission to first useful response, first action, verified result, and stop-to-cessation at p50/p95. Also count model calls, observation sizes and idle CPU/network use. Do not publish a latency promise from a model vendor's benchmark. Keep disabled voice idle and wake subscriptions from events.

This review ran a direct candidate-generation probe and inspected production source/history. It did not validate a microphone, connector install, physical app action, network recovery or OS permission dialog. Those are the implementation's acceptance work.

What should stay deliberately small

Keep one authenticated node directory, one desktop identity, and ordinary provider adapters. Store explicit removable preferences such as “my desktop” and “use this Chrome profile.” Do not use unlimited transcript memory as routing authority. Screen content is task data and cannot rewrite permissions or the selected device.

The initial remote-control promise should cover paired, available desktops with an active user session and granted permissions. Waking machines, controlling login screens and broad unattended elevation are separate capabilities. The platform adapters must expose those limits accurately.

The first milestone is concrete: from a phone, complete a weather clarification, a YouTube search in existing desktop Chrome, and an editor-to-file-manager task on the Linux desktop; then stop, correct and reconnect without losing the goal. Build that whole path before claiming a smarter general assistant.

Evidence and source notes

Source inspection and documentation research: 20 September 2026. Repository paths above are locators in the inspected Circe working tree; concurrent edits can move line numbers. Historical user-message findings are summarized without publishing private conversation dumps. Architecture and rollout choices are recommendations.

External claims use linked primary documentation near the relevant claim. Browser extension lifecycle and OS accessibility support require installed-version validation. No comparative claim about model quality, cost, or current benchmark leadership is made here.

Published with the Abbyplan workflow. This investigation changes no Circe source, process, test, or user database.