Reinforcement Learning Drives Magnetic Soft Robot Control
Magnetic soft robots (MSRs) have emerged as a promising class of devices in biomedical and precision engineering due to their deformability, dexterity, and ability to be actuated without physical contact. Traditional approaches to designing their locomotion have relied heavily on trial-and-error heuristics, often requiring years of iterative refinement. While this has yielded versatile robots, it has also highlighted the inefficiency of manual gait design for systems governed by complex nonlinear mechanics.
In rigid robotics, deep reinforcement learning (RL) has transformed control strategies, enabling autonomous adaptation without explicit human guidance. By abstracting control objectives into reward functions, RL agents learn optimal actions through interaction with their environment. In MSR research, prior applications of machine learning have optimized parameters within fixed actuation patterns, but none have generated complete actuation strategies from scratch.
The work presented here addresses this gap by developing a computationally efficient simulation environment tailored for MSRs, integrating magnetic torques and a novel dissipation model into the Cosserat rod framework. This model captures tension, bending, shear, and torsion while allowing arbitrary magnetization patterns without additional computational cost. The dissipation model, based on relative velocity between non-adjacent nodes, reduces vibration during large deformations and contact events, preserving realistic rigid-body motion.
Simulation outputs—position, orientation, velocity—are distilled into state variables for RL agents, structured as Markov decision processes. Angular quantities and external magnetic field parameters in polar coordinates form the core of the state representation, supplemented by contact indicators that encode ground interaction strength. Actions are defined as incremental changes to magnetic field components, smoothing transitions and mitigating inductive effects. A simple reward function proportional to forward displacement guides learning.
The TD3 algorithm was chosen for its stability and ability to output continuous actions. TD3’s actor-critic architecture uses twin critic networks to reduce overestimation bias, target policy smoothing to improve generalization, and delayed policy updates to stabilize learning. Agents are trained in simulation, with experiences stored in replay buffers and sampled randomly to prevent overfitting. Once trained, control policies generate magnetic field sequences that are applied to real MSRs via Helmholtz coils, ensuring uniform fields and avoiding unmodeled gradient forces.
Under small field amplitudes (4 mT), robots with differing magnetization patterns converged on similar crawling gaits reminiscent of inchworm locomotion, arching and stretching in cycles while maintaining ground contact at both ends. Despite not being informed of magnetization patterns, RL agents produced field sequences that, when rotated, yielded equivalent magnetic torques, demonstrating mechanical coherence in learned strategies.
With larger field amplitudes (10 mT), diversity in learned gaits increased. For one magnetization pattern, agents discovered both a three-phase arch-roll-unfold gait and a continuous rolling gait. The latter achieved higher average rewards but exhibited more velocity fluctuation. For the second magnetization pattern, large deflections induced magnetic attraction between opposite poles at the robot’s ends, causing failure cases in experiments not predicted by the model.
Analysis of Q-value distributions revealed that actors generally selected actions with the highest critic-estimated values, though occasional deviations occurred due to the continuous nature of the action space and function approximation limits. Suboptimal actions sometimes still contributed to task success, reflecting RL’s capacity to navigate complex control landscapes.
The simulation method was validated against finite element models and experiments, capturing deformation patterns accurately while offering significant computational speed gains when using approximate physical parameters. This efficiency enabled training multiple agents in parallel, refining policies in more accurate simulations before deployment.
Fabricated MSRs consisted of an Ecoflex 00-10 polymer matrix loaded with neodymium magnetic powder, magnetized post-curing to program internal profiles. The actuation system translated RL-generated field sequences into coil currents via microcontroller and DAC modules, with regular calibration ensuring repeatable field generation.
This work represents the first demonstration of RL-generated control strategies directly actuating real MSRs without human-designed gaits. By abstracting the inverse design of magnetic fields into a learning problem, it opens pathways to autonomous control in increasingly complex MSR applications, where manual torque design becomes intractable. The approach’s generality, from feature extraction to simulation fidelity, offers a foundation for future research integrating more advanced models, feedback sensing, and expanded motion control.
