V2UI: A Framework for Governed Voice-Directed Interfaces

A static dashboard answers questions selected before the user arrives. A conversational interface answers the present question, but often dissolves the answer into a transcript. Voice-to-UI, or V2UI, defines a third form: let a person state the information or control they need, then let the host application move to—or construct—the appropriate visual state.

The word construct needs a careful boundary. V2UI does not give a voice model an open browser, a DOM selector, or permission to emit executable frontend code. The model interprets intent and proposes typed declarations. The application owns the available destinations, data tools, component catalog, validation, persistence, rendering, and effects. Voice directs the interface; it does not inherit the interface’s authority.

This paper names and generalizes an architecture developed through a working reference implementation. That implementation demonstrates that every stage can be built: live voice input, contextual intent routing, declared navigation, asynchronous surface production, schema validation, persistent workspaces, reconnect replay, native rendering, and interaction after generation. The framework is extracted from those working contracts rather than proposed only as a diagram.

V2UI is nevertheless an architectural and interaction framework, not yet a standalone software framework, package, protocol, or standard. It defines responsibilities, boundaries, state transitions, and testable invariants. A future SDK could package those responsibilities; a future specification could standardize their interfaces. Neither is required for the present architectural claim.

The preceding field paper, Voice as the Orchestrator Layer, studied voice at the work boundary: capture a software brief, admit bounded execution, and return evidence. V2UI studies voice at the presentation boundary: choose where the user is looking, construct the visual state they need, and preserve that state long enough to remain useful.

Why voice needs a visual state

Speech is effective for expressing goals, constraints, comparisons, and follow-up questions. It is a poor medium for retaining dense state. A spoken answer disappears in time; a transcript preserves words but not necessarily the relationships, controls, provenance, or spatial grouping needed for continued work.

Conventional dashboards solve the persistence problem by fixing the interface in advance. Their measures, filters, charts, alerts, permissions, and layout can be reviewed before use. That stability becomes friction when the useful question changes faster than the dashboard. The user searches through existing pages, asks someone to add a view, exports data into another tool, or accepts a prose answer that cannot be monitored or manipulated.

Natural-language visualization research has long shortened the path from a question to a view. DataTone translated natural-language queries into data and visual specifications, then exposed ambiguity widgets where a user could correct uncertain interpretations. NL4DV returned analytic specifications containing attributes, tasks, and candidate Vega-Lite views. Both lines of work establish a useful separation: language describes analytical intent; another system constructs the visualization.

Generative UI extends that separation from a single chart to an interface. A request may need a heading, current values, a trend, a comparison table, a status badge, and an action. The response can be assembled for the question while still using components, interaction rules, and visual language that the host application already trusts.

V2UI adds persistence and control to that pattern. Its output is a durable surface, not a disposable answer bubble. A surface has a stable identity, workspace ownership, component structure, bound data, layout, provenance, and lifecycle. It can be refreshed, updated through a narrow data patch, arranged beside other surfaces, moved, removed, or replayed after a reconnect. The visual workspace becomes working memory for the conversation.

What V2UI is—and is not

V2UI converts spoken intent into one of two bounded outcomes:

1. a transition within the host application’s declared state machine; or
2. a validated mutation to persistent declarative interface state.

That definition makes V2UI broader than speech recognition and narrower than general computer use. Recognition supplies words or semantic input. V2UI resolves that input against live application context, selects an allowed effect, and carries it through an application-owned contract. General computer use may inspect pixels, move a pointer, or operate an unfamiliar website. V2UI instead acts within a host that explicitly publishes its routes, objects, components, and effects.

The framework contains ten logical responsibilities. They can be separate services, modules in one process, or functions within an application; V2UI does not prescribe deployment topology.

Responsibility	Purpose	Required invariant
Voice adapter	Streams speech, transcripts, interruption, and session state.	Listening and interruption remain visible and controllable.
Context resolver	Supplies the active destination, workspace, selected objects, and caller scope.	Context is refreshed at the turn boundary rather than assumed from stale dialogue.
Effect router	Maps language to declared typed operations.	The model can select a contract but cannot invent a capability.
Navigation controller	Applies host-defined destination changes.	No arbitrary URL, DOM selector, or script enters the contract.
Surface producer	Acquires evidence and proposes component structure plus data.	Retrieval authority remains separate from visual composition.
Declaration adapter	Encodes a surface in a host-supported declarative format.	The output names trusted components rather than executable code.
Validator and publisher	Materializes, checks, repairs, and atomically publishes surface state.	Unvalidated generations never become authoritative UI state.
Surface registry	Stores identity, workspace ownership, generations, layout, and provenance.	Mutations address canonical objects, not conversational guesses.
Host renderer	Maps declarations to native components and the host design system.	Unknown or malformed components fail visibly and locally.
Action bridge	Returns voice and component interactions to typed application effects.	Rendering a control does not silently grant the authority to execute it.

The framework is provider-neutral. A streaming multimodal model, a speech-to-text pipeline plus language model, or another real-time voice runtime can implement the voice adapter and router. V2UI is also transport-neutral: WebSocket, SSE, or another ordered channel can carry state. Its declaration adapter can target any format that preserves the trusted-component and validation boundaries.

The reference profile in this paper uses A2UI because A2UI already defines declarative, streaming, catalog-backed interfaces that render natively without executing arbitrary generated code. A2UI v0.9.1 is the current production release, and a 1.0 release candidate has been published. V2UI does not replace or extend the A2UI wire protocol. It governs the human-voice-to-application path around it:

human voice
    ↓
V2UI · context, intent, authority, navigation, persistence
    ↓
A2UI profile · declarative surface structure and data
    ↓
host-owned renderer, components, and actions

Two control planes, one authority boundary

“Control the application” and “build a dashboard” sound similar when both begin with speech, but they are different system effects.

Navigation selects a destination the host already knows. It should be fast, scoped to the requesting client unless the product explicitly says otherwise, reversible through ordinary history, and unable to manufacture a new destination. Composition creates or changes persistent interface state. It may require data retrieval, model generation, validation, and a visible pending state. It should survive the voice turn and carry an explicit sharing scope.

V2UI therefore defines two control planes:

• The navigation plane converts a spoken destination into a closed, application-defined transition such as navigate(destinationId, workspaceId?). Application code validates the identifiers and applies the same state transition used by ordinary visual navigation.
• The composition plane converts an explicit visual need into a bounded surface operation: create, update, remove, investigate, arrange, or refresh. An asynchronous producer can retrieve evidence, compose a declaration and data model, validate the result, commit it to a registry, and stream it to the renderer.

Figure 1 · The two V2UI control planes

Scroll horizontally to compare the two planes →

Keeping the planes separate prevents subtle failures. A slow surface task does not delay a clear navigation request. Saying “open the finance workspace” does not create that workspace if it does not exist. A client-scoped navigation event does not accidentally become shared state, while a persistent surface mutation cannot masquerade as a harmless view transition. The spoken acknowledgement can remain short because the interface change is itself the feedback.

The navigation plane controls declared application state

V2UI navigation is not general browser automation. The host publishes a closed set of destination identifiers and the optional objects that may qualify them. Unknown destinations fail validation. Unknown workspaces or resources return a resolution error. The command schema does not accept a URL, CSS selector, DOM path, script, or arbitrary view name.

This is the same authority boundary used by model function calling: a model can select a declared function and propose its arguments, but the application executes the function. A voice runtime therefore behaves as a semantic adapter for an application state machine, not as the state machine itself.

A generic client-scoped event could look like:

{
  "type": "interface_control",
  "action": "navigate",
  "destination": "analytics",
  "workspace": "research"
}

The names are illustrative rather than a V2UI wire specification. The important properties are structural: destination is enumerated by the host, workspace is canonically resolved, scope is explicit, and the client uses the same router and history model as a visual navigation action.

Workspace lifecycle belongs to a separate effect contract. Creating, switching, listing, renaming, or deleting a workspace changes durable application state. Creation should not imply navigation unless that combined effect is explicitly defined. A protected default workspace can refuse deletion; a non-empty workspace can require its surfaces to be moved or removed. A move accepts a canonical surface identifier and an existing destination.

Spoken intention	Logical contract	Application boundary
“Open analytics.”	navigate(analytics)	Destination must be declared; scope is the requesting client.
“Show the research workspace.”	navigate(dashboard, research)	Workspace must already exist.
“Create a workspace called research.”	manageWorkspace(create, research)	Explicit persistent mutation; creation does not imply navigation.
“Move the latency view to operations.”	resolveSurface → moveSurface(surface_17, operations)	Exact surface and existing workspace required.
“Open this URL and press approve.”	No matching V2UI contract	Arbitrary browsing and DOM actions remain outside the framework.

This is the defensible meaning of voice-directed application control: language selects a known transition in the host state machine. Extending that vocabulary is an application design and security decision, not something a model can do by improvising a new command.

The composition plane produces declarations, not code

The composition plane turns a visual need into persistent declarative state. It does not ask the live voice model to return markup during the audio turn. The voice path admits a bounded operation and can then release the conversation while a producer obtains evidence, assembles a candidate surface, and validates it.

In the A2UI v0.9.1 reference profile, four server-to-client message forms carry the surface:

1. createSurface establishes a surface identifier and component catalog.
2. updateComponents supplies or replaces the flat component adjacency list.
3. updateDataModel supplies or patches values bound into those components.
4. deleteSurface removes the surface.

The separation between structure and data matters. A line chart can remain a line chart while a small data-model patch replaces its latest points. The system does not need to regenerate the entire surface, and the generator is less likely to change layout while merely refreshing values.

The host owns the catalog. A basic profile might offer text, rows, columns, lists, tabs, buttons, and inputs. An analytical extension might add statistics, line and bar charts, gauges, progress, key-value lists, badges, and data tables. The generator can select one of those semantics and bind data. It cannot replace the chart library, import a remote script, escape the host theme, or invent a component the renderer has not registered.

V2UI framework boundary

A small set of invariants carries a flexible interface

2 Control planes

Immediate declared navigation and asynchronous persistent composition.

4 A2UI message forms

The profile's server-to-client forms create a surface, update structure or data, and delete it.

2× Validation boundaries

The publisher protects durable state; the renderer defensively protects the client.

Declarative output does not make generation automatically correct. A2UI’s prompt-first protocol family requires post-generation validation and correction. The safety improvement is that malformed output can be rejected as data before it becomes executable application code.

Publish each surface as a governed transaction

A composition request should carry an operation, visual brief, optional canonical target, evidence domain, sharing scope, and optional refresh policy. The framework does not require those exact field names, but it requires those decisions to be explicit before persistent state changes.

For creation, the publisher reserves a stable surface identifier and layout position, then emits a pending state. The user may see a loading skeleton, but the reserved surface is not yet authoritative rendered state. The asynchronous transaction then follows eight stages:

1. Resolve scope. Bind the request to the caller, client, workspace, permissions, and intended operation.
2. Acquire evidence. Source-specific tools retrieve authorized data and reduce it into a bounded brief with provenance.
3. Compose structure. A structural producer receives the visual brief and evidence. It selects catalog components and places mutable values in a data model. It does not acquire new data on its own.
4. Materialize effective state. Creation produces a complete surface; updates merge against the existing generation and prefer narrow data patches when structure remains suitable.
5. Validate declarations. The publisher checks the protocol envelope, component schemas, bindings, identifiers, and operation order.
6. Repair within a fixed budget. Invalid generated output may receive a bounded correction attempt containing the validation error. Repeated failure becomes a deterministic, valid, visible error state.
7. Commit atomically. Only valid effective state enters the registry as the next generation.
8. Render and observe. The publisher emits ordered messages and task status; the host renders the generation and records failures.

This sequence is transaction-like even when it spans model calls and a streaming transport. The user sees a reservation followed by either a valid committed surface or a valid visible failure. Durable state never has to treat an unvalidated model response as authoritative.

Evidence and structure need separate ownership. A component composer should not invent live values merely because a plausible chart requires them. If retrieval fails and the product deliberately chooses an illustrative fallback, every generated value must be labeled as sample data. Otherwise the system should render the failure or preserve the last known generation with an explicit stale state.

Refresh is another governed mutation, not a timer attached casually to a model prompt. The host should bound the minimum and maximum interval, associate the refresh with the surface owner and data authority, coalesce overlapping work, and stop the refresher when the surface is removed. Scheduled refresh should prefer data-model patches and preserve the approved structure.

Durable surfaces become interface memory

A voice-directed surface becomes useful when the user can leave and return to it. V2UI therefore treats interface state as a first-class record. A minimal registry entry contains:

surface identity       stable ID + generation
ownership              workspace + sharing scope + caller policy
presentation           catalog profile + components + bound data + layout
evidence               source pointers + provenance + freshness
lifecycle              pending · active · stale · failed · removed
history                created and updated timestamps + operation metadata

The registry must distinguish view state from shared state. The active workspace or selected destination can be local to one client. Creating, updating, moving, arranging, or removing a shared surface is a durable application mutation. Every transport frame carries enough ownership metadata for clients to project only the state they should display.

Reconnect requires a complete bootstrap, not conversational reconstruction. The publisher can rebuild each stored surface as an idempotent declaration sequence, then send workspace layouts. Starting a new voice session may reset transient conversation state while preserving the UI artifacts already committed. This is an important property: the conversation is an input channel; it is not the system of record for the interface.

Persistence changes how pronouns should be resolved. “Refresh that view” is not safe to interpret from dialogue alone. The resolver first reads the current surface inventory and workspace context. A canonical ID or unique exact title may resolve automatically. A fuzzy label produces candidates and a clarification. Similar surfaces in different workspaces remain distinct because ownership is part of their identity.

This follows a lesson from mixed-initiative visualization research: natural language does not remove ambiguity. The system must surface ambiguity at the point where a decision matters. In a persistent V2UI workspace, the highest-risk ambiguity is often not which chart to choose but which durable object the user intends to mutate.

Voice and direct manipulation share one state machine

A generated surface that cannot be manipulated is only a formatted answer. V2UI closes the loop through voice and native component actions.

Voice may update, refresh, move, arrange, investigate, or remove a canonical surface. Arrangement receives the complete live inventory and returns a layout containing each eligible surface exactly once with a valid order and span. It cannot invent, duplicate, or omit surface identities.

The rendered surface may also contain buttons and inputs. The renderer converts a component interaction into a typed action containing the action name, source component, surface identity, and context. The host resolves that action through the same authority boundary used by voice. A resulting surface update passes through evidence acquisition, validation, persistence, and rendering again.

spoken request ─┐
                ├─> typed effect ─> validated surface state ─> native renderer
surface action ─┘                                            │
                  <────────── typed action event ─────────────┘

This is more robust than asking a model to simulate a mouse. A declared button carries an application-defined action. A spoken command resolves an application-defined object and effect. Both paths cross the same policy boundary, and neither requires knowledge of the rendered DOM.

Rendering a control must not authorize its effect. A surface may display an “approve” button because the catalog supports one; the host still decides whether the caller can approve this object now, whether confirmation is needed, and what evidence must be recorded. Interface generation and effect authority remain orthogonal.

A provider-neutral reference architecture

The complete V2UI path begins at a visible voice boundary in the host application. Each turn refreshes destination, workspace, selection, and caller context. Simple questions can remain spoken. Clear navigation takes the immediate client-scoped plane. Explicit visual requests enter the asynchronous composition plane.

The application-owned control boundary is central. It owns operation schemas, destination enums, canonical surface resolution, workspace lifecycle, refresh bounds, source capabilities, and authorization. The producer separates evidence retrieval from structural composition. A declaration adapter maps the candidate to the selected UI profile. A publisher validates before the registry commits. An ordered transport preserves generation and ownership. The host renderer validates defensively and maps only trusted catalog components into native UI.

Figure 2 · The V2UI reference architecture

Scroll to inspect the architecture →

The architecture deliberately contains two validation boundaries. The publisher must not commit an invalid surface because it owns durable state and may broadcast to multiple clients. The renderer must still distrust wire input so a client does not crash or render an unsupported component when the publisher, transport, or stored state is wrong. Unknown components become visible warnings; rendering faults stay inside a surface error boundary.

The working reference implementation uses a real-time model with function calling, an in-process orchestrator, an A2UI v0.9.1 declaration adapter, a durable registry, ordered WebSocket frames, and a React renderer with a trusted extended catalog. That implementation establishes feasibility. None of those provider or framework choices is a normative dependency of V2UI itself.

Preserve the boundary as capability grows

A useful destination vocabulary and component catalog will expand. The correct question is not whether a model can generate a richer interface; it is which new semantic capability the host application is willing to implement and govern.

Requested capability	Framework policy	Reason
Navigate to a declared destination	Allow after schema and identifier validation	The state transition and scope are host-owned.
Select an existing workspace	Allow after canonical resolution	The workspace is a known application object.
Create a catalog-backed surface	Allow after evidence acquisition and full validation	Output is declarative and rendered through trusted components.
Patch bound surface data	Prefer over structural regeneration	It preserves layout and narrows the mutation.
Trigger a declared surface action	Route through a typed effect contract	The component proposes an action; the host authorizes it.
Generate arbitrary HTML, CSS, or JavaScript	Reject	It bypasses the component catalog and execution boundary.
Navigate to an arbitrary URL or click a DOM selector	Reject from the V2UI core	It turns declared application control into open-ended browser automation.
Perform a consequential external action	Require a separate tool, authorization, and confirmation policy	Rendering a control must not grant its effect.
Resolve multiple plausible target surfaces	Clarify	A visually reasonable guess can mutate the wrong durable object.

Catalog components should remain semantic. A DataTable communicates tabular data; a Button communicates an action. A hypothetical component that accepts arbitrary markup or script would collapse the boundary even if it were technically registered. Catalog extension review therefore resembles adding a backend tool: define its input schema, data binding, accessible behavior, authorization, failure state, observability, and tests.

Accessibility is a renderer responsibility, not an assumed property of model output. Every generated surface needs keyboard reachability, visible focus, appropriate roles and labels, text alternatives for meaningful graphics, color-independent status, and a readable order when a multi-column layout collapses. Speech cannot be the only way to create or manipulate a workspace. The host must expose equivalent navigation and controls for users who cannot or do not want to speak.

Privacy belongs at the voice and data boundaries. A session may include screen context, workspace names, source identifiers, and spoken data. Listening state, retention, source authorization, redaction, and session termination need explicit product rules. Declarative UI prevents arbitrary client code; it does not decide which data should have been retrieved or displayed.

Evaluate the framework, not the demonstration

A convincing demonstration asks for a chart and receives one. A useful evaluation tests whether the right object, evidence, structure, and effect survive ambiguity, failure, interruption, and time.

The first controlled study should compare three conditions on the same tasks and data:

1. a fixed visual interface with ordinary pointer and keyboard navigation;
2. typed natural-language intent routed through the same V2UI contracts;
3. spoken intent routed through those contracts.

Separating typed and spoken language reveals whether an improvement comes from the intent abstraction or the audio modality. All conditions should use the same component catalog, evidence permissions, producer, model tier, and network environment.

Representative tasks include finding an existing destination, creating a workspace, constructing current-value and trend surfaces, refreshing data without changing structure, updating a surface with ambiguous wording, arranging several surfaces, moving duplicate-titled surfaces across workspaces, recovering after a reconnect, and using a generated control.

Measure the full interaction rather than speech recognition alone:

• time from intent onset to the correct visible destination;
• time to pending state and time to a useful committed surface;
• correct destination, workspace, surface, evidence source, and visualization;
• clarification rate and wrong-target mutation rate;
• first-pass schema-valid generation, repair rate, and visible fallback rate;
• factual agreement between displayed values and authorized evidence;
• structural stability during data-only refreshes;
• replay fidelity after reload or disconnect;
• successful completion by keyboard and assistive technology;
• user workload, trust calibration, and willingness to retain the surface.

Safety acceptance criteria should remain absolute even when performance is good: zero unvalidated surfaces committed, zero arbitrary URL or selector execution, zero silent cross-workspace target guesses, zero unsupported component execution, zero external effect authorized merely because a control was rendered, and zero fabricated values presented as retrieved facts.

The working implementation supplies instrumentation points for operation status, surface identity, validation failure, repair, publication, layout, replay, and component actions. It does not yet supply controlled participant results. Until that study is run, V2UI supports claims about architectural feasibility, boundedness, and observable behavior—not a claim that voice is faster or that adaptive surfaces improve decisions.

What V2UI establishes

V2UI demonstrates that voice can shape an application without becoming an unrestricted browser agent. Its reusable pattern has four parts:

1. Use a small typed vocabulary for application control. Language selects declared destinations and canonical objects; application code performs the state transition.
2. Use a declarative profile for generated UI. A producer describes surfaces through a trusted component catalog rather than shipping executable frontend code.
3. Treat a generated surface as governed state. Evidence, validation, bounded repair, atomic publication, ownership, provenance, and replay belong to the framework rather than cleanup around a model response.
4. Unify voice and direct manipulation after rendering. Both resolve typed objects and enter the same application-owned effect path.

The contribution is not the invention of speech recognition, natural-language visualization, function calling, or A2UI. It is the composition of those capabilities into an authority-preserving architecture for navigation and persistent adaptive interfaces. Naming that composition makes its boundaries reviewable and its implementations comparable.

This middle ground preserves the best property of a dashboard—visible, manipulable, durable state—while removing the assumption that every useful view must be designed before the question is asked. It preserves the best property of voice—low-friction expression of intent—without asking speech to carry exact layout or execution details.

The resulting division of labor is simple: the human states the current need; the voice runtime interprets it; V2UI resolves and governs the effect; a declaration profile describes the visual state; the host validates, renders, and owns it; the user verifies and continues. Voice directs the interface. Code retains authority.

References

1. A2UI Project, A2UI Protocol v0.9.1, current production specification, 2026.
2. Google A2UI Team, “A2UI v0.9: The New Standard for Portable, Framework-Agnostic Generative UI”, Google Developers Blog, 17 April 2026.
3. Google AI for Developers, Tool use with the Live API, 2026.
4. Google AI for Developers, Function calling with the Gemini API, 2026.
5. Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G. Karahalios, “DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization”, Proceedings of UIST 2015, pp. 489–500.
6. Arpit Narechania, Arjun Srinivasan, and John Stasko, “NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries”, IEEE Transactions on Visualization and Computer Graphics, 2021.
7. Fancy Kong et al., “Macaron-A2UI: A Model for Generative UI in Personal Agents”, arXiv:2605.24830, 2026.
8. World Wide Web Consortium, Web Content Accessibility Guidelines (WCAG) 2.2, W3C Recommendation, 2023.

Suggested citation