Embodied Agentic Runtime

Bringing
Intelligence
to Reality

Embodied Agentic Runtime is the layer that brings AI systems out of the lab and into real environments, where they can act, learn from what happens, and generate the data needed to improve over time.

Goal → Plan → Act → Verify Multi-agent runtime Grounded interaction data Step-level tracing

View Live Demo

Shared World State Verified Execution

Overview

This runs in the real world, not just in a simulated workflow.

One language request turns into planning, perception, manipulation, and verification steps in a live environment.

What To Notice

You can see each agent's role in the log.

Planner, perception, navigation, manipulation, and supervisor each show up in the trace instead of being hidden behind one controller.

Closed Loop

The system checks the scene again before it commits.

It looks again, adjusts alignment, and updates the next move before a grasp or handoff is accepted.

Verification

The task is only done when the checks pass.

A run is marked successful only after the postconditions pass, which keeps execution supervised and easy to review.

Goal

A plain-language request turns into a visible execution trace.

The demo starts with a task instruction, then shows how it is planned, carried out, and checked step by step.

Plain-language request → step-by-step execution

Trace

The run stays readable while it happens.

Tool calls, agents, recoveries, and timing stay visible during the job instead of only showing up afterward.

See who acted, when they acted, and why

Data

Each run leaves behind useful interaction data.

The same trace that supervises the runtime also captures the records needed to improve models over time.

Plans, actions, video, outcomes, and interventions stay linked

The Thesis

Simulation is not where intelligence proves itself.

It becomes useful when it can act in the world, deal with outcomes, and improve from data gathered through physical interaction. Embodied Agentic Runtime is built for the first two steps: deploy in the real world, then collect interaction data at the scale needed to make the next version better.

01 · bring to real life

The first step is safe, supervised execution.

The runtime brings intelligence into physical environments with explicit plans, verification, and human oversight.

02 · collect + improve

Every run creates data for the next training loop.

Tasks, perception, actions, outcomes, recoveries, and operator interventions are captured at scale to support better models and more autonomy.

Why This Matters

Traditional automation breaks easily. Adaptive systems handle change.

Hard-coded sequences fail when the scene shifts, tolerances drift, or handoff conditions change. This approach breaks work into readable steps, keeps the scene in context, and recovers instead of failing silently.

System Overview

Everything is visible in one dashboard.

Agents, workflow graphs, digital twin state, process monitoring, and live video all sit in the same interface.

01 · Agent Team

Robot arms, mobile bases, and shared context stay in one runtime.

Different hardware and support services expose clear capabilities without splitting the orchestration layer apart.

02 · Task DAG

The job flow stays visible while work is in progress.

The graph shows ordering, live state, and recovery points instead of hiding execution inside opaque controllers.

03 · Human In The Loop

Pause, modify, and override without leaving the workflow.

Supervisor controls keep intervention close to the running job instead of burying it in a separate panel.

04 · Twin + Stream

3D state stays in sync with live video for planning and verification.

Each step is grounded in both the digital twin and the camera feed, so the run stays easy to follow.

How We Did It

Five layers make the runtime work in the real world.

Multi-agent coordination, step-by-step execution, shared world state, typed tools, and supervision turn a deployment into a system you can actually trust.

01 · Multi-agent roles

02 · Goal → plan → act → verify

03 · Shared digital twin

04 · Unified tool registry

05 · Safety + supervision

04 · Registry

Typed tools

One tool registry covers robot arms, AGVs, sensors, and services.

Hardware is exposed as typed tools with schemas, constraints, and capabilities, so backends can change without rewriting agent logic.

03 · Twin

World state

A live digital twin keeps world state shared.

Agents query poses, plan motion, and verify placements against a shared scene representation that stays live during execution.

02 · Trace

Goal → Plan → Act → Verify

Execution stays step-by-step, iterative, and auditable.

The system runs in readable steps with closed-loop checks and recovery instead of fire-and-forget playback.

05 · Safety

Control plane

Safety and supervision stay in the control plane.

Force and speed limits, postcondition checks, retries, and human override paths keep execution supervised and repeatable.

01 · Roles

Multi-agent runtime

Planner, perception, navigation, manipulation, and supervisor work together in one runtime.

Specialized roles collaborate through structured tool calls instead of being forced into a single controller.

Skills

Complex robotic skills stay readable when they run.

Skills are defined with clear specs, checks, and fallbacks, and the runtime pairs those definitions with a trace that shows which agent did what, when, and why.

Task Query

input

>"Move cassette from Shelf A → Inspect seal quality → Regrasp if needed → Deliver to Conveyor 3."

Execution Trace

trace · 9 steps · 58.4s

STEP 1  Parse task            planner          0.0s
STEP 2  Query world state     supervisor       1.3s
STEP 3  Navigate to shelf     nav              9.8s
STEP 4  Detect cassette       perception       13.7s
STEP 5  Pick + force check    manipulation     21.4s
STEP 6  Inspect seal region   perception       29.6s
STEP 7  Regrasp for alignment manip + planner  38.9s
STEP 8  Deliver to conveyor   nav + manipulation51.0s
STEP 9  Verify postcondition  supervisor       58.4s

pick-with-alignment

Recover grasp quality when the initial pickup is unstable or offset.

name: pick-with-alignment
agents: [perception, manipulation, supervisor]
stages:
  - detect_candidates(top_k=5)
  - score_grasps(force_limit_n=18)
  - pick(best)
  - regrasp(if tilt_deg > 7)
fallback: "request_new_view"

visual-qa-gate

Route the object to rework or quarantine based on visual QA and tolerance checks.

name: visual-qa-gate
tools: [rgbd_detector, visual_qa, reject_gate]
checks:
  seal_gap_mm: 0.8
  label_present: true
  surface_glare: "auto_compensate"
on_fail: "quarantine_bin"

conveyor-intercept

Synchronize mobile base timing with a downstream handoff target.

name: conveyor-intercept
agents: [nav, supervisor]
params:
  target_lane: 3
  eta_slack_s: 2.5
  geofence: "conveyor_zone"
retry: "replan_if_lane_blocked"

multi-robot-transfer

Coordinate arm, AGV, and supervisor checkpoints for a reliable handoff.

name: multi-robot-transfer
sequence:
  - approach(sync_pose=true)
  - micro_align(vision=true)
  - transfer(force_limit_n=12)
  - verify_release()
abort_if: "postcondition_timeout"

Data Flywheel

Built for teams doing real deployments.

For robotics teams moving from demos to production, integrators coordinating mixed hardware, and embodied AI teams that need real-world data at scale. The runtime brings robots into real environments, verifies each interaction, and turns every run into data the next version can learn from.