SO-101 上的 π0.5Running π0.5 on a Real SO-101: Remote Inference and Action Handoff

π0.5 是 Physical Intelligence 提出的视觉—语言—动作（VLA）模型。它把相机画面、自然语言指令和机器人状态放进同一个策略，直接预测机器人接下来该怎样动作。相比上一篇使用的 ACT，π0.5 不只学习画面到动作的映射，还把语言作为任务条件，希望同一个模型能处理更多机器人和场景。

这次没有测试 π0.5 的通用能力，而是先把它用在一个固定任务上：读取腕部相机画面、SO-101 的关节状态和一句任务指令，再预测接下来的一段关节动作。模型变大以后，问题也从“能不能学会”变成了“远端模型怎样持续控制本地真机”。

任务仍然没变：让 SO-101 把白色方框里的蓝色胶带卷抓起来，搬到旁边的白色盒子里。这样做是为了保留 ACT 的参照，不把换模型、换任务和换场景三个变量混在一起。

这一轮真正跑通的不是一个离线 Demo，而是一条跨机器的真机闭环：Home Linux 负责相机和机械臂，LAB-2 上的 RTX 3090 负责 π0.5 推理，两端通过网络交换观测和动作。

RTC 是什么

RTC 是 Real-Time Chunking（实时动作分块）。它处理的是这样一个问题：模型还在计算下一段动作时，机器人该怎么继续动，以及新旧两段动作怎样接上。这里讲的是本项目采用的推理时 RTC，不是后来需要配合训练的变体。

动作块就是模型一次预测的一串未来动作。例如一次返回 50 个关节目标，机器人按顺序执行。分块能减少调用模型的次数，但如果等这 50 步走完才请求下一块，机械臂就会停下来等模型。提前请求可以把计算和运动重叠起来，却还需要处理交接处的目标跳变。

RTC 会在生成新块时参考旧块还没执行完的动作。推理等待期间必然执行的那一段，作为新预测的前缀约束；后面还有调整余地的动作，则允许结合新观测更新。这种“保留已确定部分、补出后续动作”的方法叫 inpainting（补全），这里补的是动作序列。参见 Physical Intelligence 的 RTC 说明 和 LeRobot RTC 文档。

用 50 步动作理解交接

下面用 30 Hz 控制、每块 50 步举例。假设执行了约 20 步就请求下一块，队列还剩约 30 步；再假设从发出请求到拿到结果需要 0.5 秒，这期间机械臂还会执行约 15 步。这里的 0.5 秒用于说明时间关系，不是新增的测试结果。

旧块：执行约 20 步 → 继续执行约 15 步 → 交接
新块：             采集观测并开始推理 → 结果到达
                  按同一时间轴约束前缀，交接时使用尚未过期的动作

新块到达时，不能再从对应旧观测时刻的第一个动作开始执行，因为那段时间已经过去。也不能让模型完全忽略机器人在等待期间的运动。RTC 生成约束与客户端的时间对齐需要一起工作。

第一块还没有上一块可参考，因此旧动作前缀的约束从第二块才开始发挥作用。在本项目的服务端代码里，prev_chunk_left_over 传入按观测步数对齐后的旧块剩余动作，inference_delay 表示估计延迟对应的控制步数，execution_horizon 控制这次约束的范围。配置中的 rtc_execution_horizon=30 是步数上限，绝不是“从第 30 个动作块开始”。

RTC 和加权平均有什么区别

后文的“冻结 RTC”是保留的实验配置名称：服务端使用 RTC，客户端采用 latest_only，不做新旧目标平均；“加权 RTC”则在服务端 RTC 之外，又加了 weighted_average。不能把加权平均本身当成 RTC。两个分别合理的动作，平均后也未必适合当前接触阶段，例如接近物体和闭合夹爪对时机的要求就不同。

面试时，我会这样概括：RTC 让机器人执行旧动作块的同时生成下一块，并把等待期间已经确定要执行的动作作为生成约束，使新预测能接上正在发生的运动。它缓解推理延迟和块间不连续，但不会让模型本身算得更快，也不保证抓取成功。 下面的实验仍需要分别检查执行频率、交接跳变和任务是否完成。

先把数据扩到 50 条

ACT 的 9 条示教证明了链路可行，但直接拿来训练 π0.5 还是太少。第二批数据扩到 50 Episodes、约 18,000 Frames，仍然只做蓝色胶带到白色盒子的单任务，指令固定为：

Pick up the blue tape roll and place it in the white tray.

采集时没有刻意追求复杂语言，而是先保证动作边界、物体复位、腕部画面和任务文本稳定。50 条轨迹按不同起始区域分组，避免胶带每次都出现在完全相同的位置。模型最终使用 30K Checkpoint；这里的“语言条件”只代表策略收到了任务句子，不等于已经证明它能理解任意新指令。

为什么要做远程 Policy Server

机械臂所在的 Home Linux 负责 USB 相机、串口和 30 Hz 控制，但不适合同时承担 π0.5 推理。于是把系统拆成两部分：

机器人端：采集图像和关节状态 → 推理端：RTX 3090 生成动作块 → 机器人端：按节拍执行并继续上传新观测。

正式下发动作之前，先跑 Shadow Mode。它会把真实观测发送到 3090，打印预测动作和当前状态的差值，但不会给机械臂发送任何指令。只有图像尺寸、关节顺序、动作范围和网络往返都通过检查，才进入短时真机测试。

8 月 14 日第一次完整跑通时，真机连续执行了 360 个动作周期，控制循环稳定在 30 Hz，deadline_misses=0。图像上传平均 103 ms、P95 为 149 ms，服务器日志中的模型调用约 400–430 ms，其计时包围 predict_action_chunk()，不包含图像上传和动作回传。客户端从发出观测到收到动作块的总等待还包含通信及其他处理，不能把这两个口径混用。20° 的单周期关节保护在 360 帧中修正了 11 帧异常目标。

数字看上去很稳，视频里的机械臂却仍有明显顿挫。这件事很关键：30 Hz 没有丢 Deadline，不代表动作本身连续。

对比动作块的交接方式

π0.5 每次返回的是一段未来动作，而不是一个孤立控制点。新动作块到达时，旧动作块还可能没有执行完。怎样交接，直接决定机械臂是停下来等、突然跳到新目标，还是在两段预测之间做融合。

最后保留了三条路线做同场景对比：

这里不能把 1/1 或 1/2 写成“成功率”。样本只有五轮，只能说明这五次录像里发生了什么。两次失败都出现在接触和夹紧阶段；只要胶带被可靠夹住，后面的搬运和释放都能完成。

先看非 RTC 的一次完整成功。它会等待新的动作块，因此节拍不如 RTC 连续，但这组观测里的关节跳变最小。

π0.5 is a vision-language-action (VLA) model developed by Physical Intelligence. It places camera images, natural-language instructions and robot state inside one policy, then directly predicts what the robot should do next. Compared with the ACT baseline in the previous article, π0.5 also uses language as a task condition and is designed to work across a broader range of robots and scenes.

This experiment did not test the model’s general capabilities. It first applied π0.5 to one fixed task, using the wrist-camera image, SO-101 joint state and a single instruction to predict a chunk of future actions. Once the model became larger, the main question shifted from “can it learn?” to “how can a remote model keep controlling a local robot?”

The task stayed unchanged: SO-101 had to pick up the blue tape roll inside the marked square and place it in the nearby white box. Keeping the same task preserved the ACT baseline instead of changing the model, task and scene at once.

This was not an offline demo. It was a closed loop across two machines: Home Linux handled the wrist camera and robot, while the RTX 3090 on LAB-2 ran π0.5 inference. Observations and action chunks travelled between them over the network.

What RTC does

RTC means Real-Time Chunking. It addresses how a robot keeps moving while a model computes its next action chunk, and how the two chunks connect. This project uses inference-time RTC, rather than a later training-time variant.

An action chunk is a sequence of future actions predicted in one call, such as 50 joint targets. Waiting until all 50 have executed before requesting another chunk creates a pause. Requesting early overlaps inference with execution, but a new prediction can still disagree with the motion already underway.

RTC conditions generation on the unexecuted portion of the previous chunk. Actions committed during the inference delay constrain the new prefix; later actions have more freedom to respond to fresh observations. This is action-sequence inpainting: preserve the committed part and generate the continuation. See Physical Intelligence’s RTC explanation and the LeRobot RTC documentation.

A 50-step example

Consider 50-step chunks at 30 Hz. If the next request starts after roughly 20 steps, about 30 remain. Suppose the request takes 0.5 seconds to return: another 15 steps execute during that interval. The 0.5-second delay is an illustrative assumption, not an additional measurement.

The arriving chunk must be aligned to the current execution time; its first entries refer to time that has already passed. Generation constraints and client-side time alignment must work together. The first chunk has no predecessor, so previous-chunk prefix guidance starts with chunk two.

In this project’s server, prev_chunk_left_over provides the previous suffix aligned to the observation timestep, inference_delay estimates latency in control steps, and execution_horizon controls the current constraint span. The configured rtc_execution_horizon=30 is a step limit, not an instruction to begin at chunk 30.

RTC versus weighted averaging

Asynchronous requests overlap inference with execution. Server-side RTC constrains the new chunk while it is being generated. Client-side weighted averaging combines targets only after generation. These are distinct operations.

The experiment label “frozen RTC” refers to the retained server-RTC configuration with latest_only on the client, without averaging targets. “Weighted RTC” adds client-side weighted_average to server RTC. Averaging is not itself RTC, and two individually valid actions can average into an unsuitable action during contact or gripper closure.

My short explanation is: RTC generates the next chunk while the robot executes the current one, conditioning generation on actions committed during the delay so the continuation agrees with ongoing motion. It addresses waiting and chunk discontinuity; it does not make inference itself faster or guarantee a successful grasp. Execution timing, target jumps and task completion still need separate checks in the experiments below.

Expanding the dataset to 50 demonstrations

The nine ACT demonstrations proved that the pipeline worked, but they were too few for the next experiment. The π0.5 dataset was expanded to 50 episodes and about 18,000 frames, still focused on the single blue-tape-to-white-box task. The instruction was fixed as:

Pick up the blue tape roll and place it in the white tray.

The priority was not linguistic variety. Episode boundaries, object resets, wrist images and task text had to remain consistent. The 50 trajectories were grouped across different starting regions so that the tape did not always appear at exactly the same position. The deployed policy used the 30K checkpoint. “Language-conditioned” here only means that the task sentence was provided to the policy; it does not demonstrate open-ended instruction following.

Why the policy ran on a remote server

Home Linux handled the USB camera, serial devices and 30 Hz control loop, but it was not the right place to run π0.5 inference. The system was therefore split into two parts:

Robot endpoint: capture image and joint state → inference endpoint: generate an action chunk on the RTX 3090 → robot endpoint: execute it on schedule and upload the next observation.

Before any action was sent, Shadow Mode forwarded real observations to the 3090 and printed the difference between the predicted actions and current state without commanding the arm. Real-robot tests started only after the image shape, joint order, action range and network path had passed inspection.

The first complete rollout on August 14 executed 360 action cycles at a stable 30 Hz with deadline_misses=0. Image upload averaged 103 ms, with a P95 of 149 ms, while the server-side model call took roughly 400–430 ms. That timer surrounds predict_action_chunk() and excludes image upload and action return. The client-observed wait also includes communication and other processing; these timing scopes are different. A 20-degree per-cycle joint guard corrected 11 of the 360 commanded frames.

The timing numbers looked stable, but the arm still showed visible jerkiness. That distinction mattered: meeting every 30 Hz deadline did not guarantee continuous action targets.

Comparing action-chunk handoff strategies

π0.5 returns a sequence of future actions rather than one isolated control point. When a new chunk arrives, the previous chunk may still be executing. The handoff policy determines whether the arm pauses, jumps to a new target or blends two predictions.

Three routes were kept for a same-scene comparison:

These are five recorded trials, not statistically meaningful success rates. Both failures happened during contact and closure. Once the tape was securely grasped, carrying and release completed normally.

The first video shows a complete non-RTC success. It waits for the next action chunk, so the timing is less continuous than RTC, but it produced the smallest joint jumps in this observation set.

冻结 RTC 能把执行频率保持在 30 Hz。下面是其中一次成功，但同样参数的另一轮没有可靠夹住胶带。逐帧回看后，失败不是发生在搬运途中，而是接触阶段就已经错过了夹持位置。

Frozen RTC maintained a 30 Hz execution rate. The next video is one successful trial, while another trial with the same route failed to secure the tape. Frame-by-frame review showed that the failure happened at contact, not during transport.

加权 RTC 的想法是把新旧动作块做融合，避免整段切换。但在这组数据里，它仍出现了较大的单周期目标变化，也有一轮在夹紧前失败。一次成功录像不能证明加权就更好。

Weighted RTC blended old and new chunks to avoid an abrupt replacement. In this observation set, however, it still produced large one-cycle target changes and failed once before secure closure. One successful video was not enough to show that weighting was better.

2026-09-17 补测：图片上传到动作回传有多慢

这次用原 30K 权重、8 次去噪和同一张示教图像，从 Mac 经阿里云 SSH 通道把 RGB 图像、关节状态和任务文本发到 3090，再接收完整的 50×6 动作数组。每组测 7 次，排除第一轮，下面是剩余 6 次的中位数。

完整往返从 Mac 准备图像字节开始，到解析完返回动作结束，包含上传、服务器处理和回传；不含提前读取和缩放图片、实际拍照、模型加载及电机执行。模型计时用 CUDA 同步确认计算完成。模型首次加载约 178 秒，加载后首次推理约 3.54 秒，单独记录，未混入表中。

这次用了原服务端的推理处理流程，但传输接口是 SSH 字节流，不是机器人端原有的 gRPC，因此不能把结果直接当作 Home Linux 链路的复测。它说明当前路径上的完整等待约为半秒，模型计算占主要部分；约 30 ms 的短消息网络往返不能代替真实图像请求的总时间。由于没有同期旧线路对照，本次也不能量化阿里云优化前后总等待缩短了多少，更不能据此确认历史抓取失败的主因。

分开检查等待、目标跳变和抓取失败

把日志和视频对齐后，三条路线暴露的是两类不同问题。非 RTC 的不连续主要来自“一个动作块执行完后等待下一块”；RTC 的抽动则更多来自“新预测块的目标和正在执行的目标差得太远”。前者是等待，后者是跳变，不能用同一个补丁解决。

冻结 RTC 的成功配置因此被保留下来作为可回退基线，而不是继续覆盖。下一步要比较的是肩、肘单周期变化、方向反转、目标—实测跟踪误差和限幅次数，同时继续记录任务有没有真的完成。只看动作顺滑、不看抓取成功，同样会把问题带偏。

到这里，π0.5 已经完成了蓝色胶带到白色盒子的远程真机闭环，但“跑通”和“足够丝滑”仍是两件事。后者需要更稳定的动作交接，也需要在仿真里把物体、夹爪和接触关系看得更清楚。这也是下一篇开始搭 Isaac Sim 数字孪生的原因。

和 ACT 那篇不同，π0.5 到这里还不能算收尾。目前只能说第一版远程真机闭环已经跑通，动作平滑度、不同起点下的稳定性、重复测试以及语言条件下的泛化都还没有充分验证。后续还会继续补充数据、优化动作交接并扩大测试轮次，新的结果也会继续记录在这个项目里。

September 17, 2026 retest: image upload to action return

Using the original 30K checkpoint, eight denoising steps and one recorded observation, the Mac sent RGB bytes, joint state and task text through the Aliyun SSH route to the RTX 3090 and received all 50×6 action values. Each group had seven requests; the first was excluded. The following medians use the remaining six.

End-to-end timing starts before image-byte serialization on the Mac and ends after parsing the returned actions. It includes upload, server processing and return, but excludes prior image loading/resizing, camera capture, model loading and motor execution. Model timing uses CUDA synchronization. Initial model loading took about 178 seconds; the first inference after loading took about 3.54 seconds. Both were recorded separately.

The test reused the original server processing path through an SSH byte-stream adapter, not the robot’s original gRPC transport. It therefore does not retest the Home Linux connection. On this measured route, the complete wait was roughly half a second and model computation accounted for most of it. A roughly 30 ms short-message round trip is not the image-request latency. Without a contemporaneous old-route comparison, this test cannot quantify the improvement from Aliyun or establish the cause of historical grasp failures.

Separating waiting, target jumps and grasp failures

Aligning logs with video revealed two different discontinuities. Non-RTC motion paused because one chunk finished before the next arrived. RTC motion jerked because a new predicted target could be far from the action already being executed. One problem was waiting; the other was target discontinuity. They required different fixes.

The successful frozen-RTC configuration was preserved as a rollback baseline instead of being overwritten. The next comparison needs to track per-cycle shoulder and elbow change, direction reversals, target-to-measured tracking error and safety clamps, while still recording whether the task actually succeeds. Smooth motion without a reliable grasp would not be an improvement.

At this point, π0.5 had completed the remote real-robot loop from blue tape to white box. But “working” and “smooth enough” were still separate milestones. The latter needs a better action handoff and a clearer view of the object, gripper and contact geometry. That is where the Isaac Sim digital-twin work in the next article begins.

Unlike the ACT work, the π0.5 project is not finished here. The current result only shows that the first remote real-robot loop can run. Motion smoothness, stability across starting positions, repeated evaluation and generalization under language conditioning still need substantially more testing. The next stage will add data, improve action handoff and expand the evaluation, with new results recorded as the project continues.