Hierarchical Reinforcement Learning Boosts UAV Swarm Combat
Unmanned aerial vehicles have long been valued for their low cost, maneuverability, concealment, and resilience in harsh environments. While single UAVs can perform reconnaissance or strike missions, their limited detection range and payload capacity make them less effective in complex engagements. Coordinated swarms, communicating in real time, expand situational awareness, enable distributed task allocation, and improve survivability. In contested airspace, such swarm-versus-swarm encounters form a multiagent system with dynamic cooperation and competition.
Traditional swarm confrontation methods—differential games, expert systems, and guidance laws—perform adequately in small, predictable scenarios but falter in large-scale, uncertain environments. Reinforcement learning (RL) offers a model-free alternative, optimizing policies via reward signals rather than explicit environmental models. However, single-agent RL algorithms such as DQN or DDPG struggle in multiagent contexts due to exponentially growing state-action spaces and nonstationarity from concurrent policy updates.
Multiagent RL (MARL) approaches like MADDPG, DIAL, and ATOC extend single-agent methods to handle multiple actors, often using centralized training with decentralized execution (CTDE). Yet sparse rewards in UAV combat make learning effective cooperation difficult, and large joint action spaces slow convergence. Addressing these challenges, the hierarchical multiagent deep deterministic policy gradient (h-MADDPG) framework introduces temporal abstraction and macro actions derived from human expertise.
In h-MADDPG, a high-level policy selects discrete macro actions—Cruise, Chase, Escape, Fire, Support—at lower frequency, while a low-level policy executes continuous primitive actions every time step. Macro actions condense decision-making, reduce the search space, and align agent behavior with tactical principles: maintain safe spacing, assist allies under threat, and coordinate attacks to maximize damage while minimizing losses. Pretraining low-level policies ensures agents can reliably perform macro actions before high-level training begins.
The simulated confrontation environment models red and blue homogeneous UAV swarms, constrained by position, velocity, acceleration, and obstacle avoidance. Each UAV’s state includes position, speed, heading, and attack zone. Combat outcomes depend on relative distance and angle, with rewards assigned for destroying enemies (+1), being destroyed (-1), and winning engagements (+3). The high-level policy in h-MADDPG uses MADDPG to coordinate macro action selection across agents, mitigating inconsistent choices.
Technical refinements enhance training. Parameter sharing among identical agents improves efficiency. Action masking prevents selection of unavailable actions, while death masking zeroes out states of destroyed UAVs to avoid biasing the critic network. An agent-specific global state representation balances comprehensive situational data with manageable input dimensions, aiding convergence.
Experiments compared h-MADDPG with independent DDPG (i-DDPG) and MADDPG in 3 vs. 3, 5 vs. 5, and 10 vs. 10 scenarios. In small engagements, h-MADDPG’s winning rate was close to the others—87% versus 82% for i-DDPG and 66% for MADDPG—due to limited action space. In larger swarms, h-MADDPG excelled: 88% in 5 vs. 5 and 91% in 10 vs. 10, outperforming i-DDPG’s 70% and 57% and MADDPG’s 50% and 59%. The reduced joint macro action space and temporally abstracted decisions allowed faster convergence and better cooperation.
Visualized decision sequences showed coordinated behavior: initial cruising to locate targets, transitioning to chase and fire when advantageous, and executing escape to lure enemies into allies’ attack zones. Obstacle-inclusive environments confirmed robustness, with performance largely maintained except for slight declines in crowded large-scale scenarios where collisions were unavoidable.
Ablation studies isolated the impact of action masking, death masking, and agent-specific global state. Each contributed to higher winning rates and faster learning, particularly in dense 10 vs. 10 battles. Testing different macro action intervals revealed optimal performance at five time steps, balancing responsiveness with reward sparsity mitigation.
By combining human tactical insight with hierarchical MARL, h-MADDPG addresses the curse of dimensionality and sparse rewards in UAV swarm combat. The framework’s design demonstrates how structured decision layers and domain-informed abstractions can unlock scalable, cooperative strategies in complex multiagent aerospace systems.
