HiRe-MoMa

Mobile Manipulation (MoMa) is a cornerstone of autonomous robotic systems, yet real-world fine-tuning of such tasks remains a formidable challenge due to their high-dimensional action spaces and the intricate coupling between heterogeneous modules. In this paper, we propose HiRe-MoMa (Hierarchical Residual Mobile Manipulation), a systematic framework for stable and sample-efficient real-world fine-tuning of complex MoMa systems. At its core, HiRe-MoMa adopts a Hierarchical Residual Architecture that fine-tunes a frozen base policy. A Policy-Agnostic Residual learns additive corrections atop the base policy, while Hierarchical Decoupling splits the residual into mobile-base and manipulator-arm heads with unidirectional conditioning, mitigating gradient interference and aligning corrections with the platform's physical hierarchy. To improve sample efficiency in data-expensive real-world settings, we integrate Expert-Guided Value Alignment, which leverages sparse human interventions with under 20% training coverage to reshape the critic's value landscape, effectively guiding autonomous exploration through expert priors. Extensive real-world experiments on the TIAGo++ platform demonstrate that HiRe-MoMa consistently enhances task performance with only around 30 minutes of online interaction, regardless of the base policy architecture. To our knowledge, this is among the first frameworks to enable efficient on-robot fine-tuning for mobile manipulation in the real world, providing a practical paradigm for deploying complex robotic systems in dynamic, human-centric environments.

Fine-tuning a mobile manipulator in the real world is hard: the action space is high-dimensional, and the base and arm are tightly coupled, so a small error in the base's approach becomes a large pose error at the gripper. HiRe-MoMa keeps the pre-trained base policy frozen and learns a residual on top of it. Training runs asynchronously: a central learner updates the residual while a rollout worker drives the robot under shared control with a human expert.

Policy-agnostic residual. A residual policy learns additive corrections on top of the frozen base policy, so the same procedure works for any base policy (Diffusion Policy, ACT, ...) without needing its log-likelihood. It is trained off-policy with a sparse success reward, and the correction is bounded to stay close to the base trajectory:

$$\mathbf{a}^{\text{exe}}_t = \mathbf{a}^{\text{base}}_t + \pi^{\text{res}}_\theta(\mathbf{o}_t, \mathbf{a}^{\text{base}}_t), \qquad \|\mathbf{a}^{\text{res}}\| \le \epsilon.$$

Hierarchical decoupling. Instead of one residual over the full action space, HiRe-MoMa uses separate heads for the mobile base and the manipulator. The arm head is conditioned on the base correction, following the platform's physical hierarchy: where the arm should reach depends on where the base goes, not the reverse.

$$\begin{aligned} \mathbf{a}^{\text{res}}_{t,\text{mo}} &= \pi^{\text{res}}_{\theta_{\text{mo}}}(\mathbf{o}_t, \mathbf{a}^{\text{base}}_{t,\text{mo}}), \\ \mathbf{a}^{\text{res}}_{t,\text{ma}} &= \pi^{\text{res}}_{\theta_{\text{ma}}}(\mathbf{a}^{\text{res}}_{t,\text{mo}}, \mathbf{o}_t, \mathbf{a}^{\text{base}}_{t,\text{ma}}). \end{aligned}$$

Expert-guided value alignment. A human can take over at any step through VR teleoperation, and these interventions are kept in an expert buffer. Rather than imitating them directly, which is sensitive to teleoperation noise, an auxiliary loss shapes the critic to prefer the human's action \(\mathbf{a}^h_t\) over the policy's:

$$\mathcal{L}_{\text{align}} = \lambda_{\text{imi}}\, \mathbb{E}\left[\log\left(1 + \exp\left(Q_\phi(\mathbf{o}_t, \mathbf{a}^{\text{res}}_t) - Q_\phi(\mathbf{o}_t, \mathbf{a}^h_t)\right)\right)\right].$$

All experiments run on a TIAGo++ with the torso and head fixed, giving a 17-dimensional action space (a 3-DoF mobile base and two 7-DoF arms). The policy observes one front RGB camera and proprioception, and the reward is a sparse binary success signal. Human interventions come from a Meta Quest 3 teleoperation interface. We evaluate three tasks of increasing difficulty, each over 20 trials with randomized initialization.

All methods fine-tune the same frozen Diffusion Policy (DP). +HG-DAgger learns the residual by regression on human corrections; +Flat Residual (the HIL-SERL residual) uses a single residual head over the whole action space. HiRe-MoMa improves both success rate and execution efficiency on every task.

HiRe-MoMa makes no assumptions about the base policy. With ACT, a Transformer-based policy, as the base, it improves the average success rate by 94%. The final performance is still bounded by the base policy's initial coverage.

Starting from the frozen DP, each column adds one component on top of the previous one (base → fine-tuned). Splitting the residual into base and arm heads gives the largest gain, raising the average success rate from 50.0% to 85.0%. Adding value alignment brings it to 92.5% and shortens episodes further, while requiring fewer human interventions during training.