Evaluating Collaborative Autonomy of Unmanned Surface Vehicles in Adversarial Maritime Capture-the-Flag Competitions

Core Concepts

The objective of this work is to evaluate multi-agent artificial intelligence methods when deployed on teams of unmanned surface vehicles (USVs) in an adversarial maritime environment using Capture-the-Flag (CTF) style competitions.

Abstract

This paper presents an overview of the Project Aquaticus test-bed, which is a Capture-the-Flag (CTF) style competition involving teams of unmanned surface vehicles (USVs). The objective is to evaluate multi-agent artificial intelligence methods when deployed on these USV teams in an adversarial environment.
The authors describe two main approaches to autonomy in the Aquaticus competition:

Behavior-based autonomy using MOOS-IvP:

The default Pav01 strategy uses a simple attacker and defender role assignment.
More advanced strategies use rule-based mode switching to dynamically adapt roles based on opponent behavior.
These rule-based strategies outperformed the default and deep reinforcement learning (RL) approaches.

Deep Reinforcement Learning using Pyquaticus:

The Pyquaticus environment is a lightweight gymnasium environment for training RL agents to play the Aquaticus CTF game in simulation.
Various RL approaches were explored, including on-policy PPO, off-policy TD3, and options-based hierarchical RL.
The RL agents tended to be overly defensive and struggled to learn effective attacking behaviors.

The results demonstrate that the rule-based, behavior-based autonomy approaches performed better than the deep RL methods in the real-world Aquaticus competitions. However, the authors note that further integration of the Pyquaticus environment with the MOOS-IvP framework and continued research on reward shaping and sim-to-real methodologies may help improve the performance of deep RL in future studies.

Stats

The Aquaticus competitions resulted in a total of 22 games played with over 3.5 hours of gameplay.
The rule-based behavior-based autonomy strategies had a mean of 4 flag grabs and 2 flag captures per game.
The deep RL Pyquaticus agents had a mean of less than 1 flag grab and capture per game.

Quotes

"As the development of experimental deep RL methods continues, the authors expect that the competitive gap between behavior-based autonomy and deep RL will be reduced."
"Further integration of the Pyquaticus gymnasium environment for RL with MOOS-IvP in terms of configuration and control schema will allow for more competitive CTF games in future studies."

Key Insights Distilled From

Evaluating Collaborative Autonomy in Opposed Environments using Maritime Capture-the-Flag Competitions

by Jordan Beaso... at arxiv.org 04-29-2024

https://arxiv.org/pdf/2404.17038.pdf

Deeper Inquiries

How can the reward shaping and training optimization of the deep RL approaches be improved to better incentivize aggressive and cooperative behaviors in the Aquaticus CTF game

To enhance the reward shaping and training optimization of deep RL approaches in the Aquaticus CTF game, several strategies can be implemented. Firstly, adjusting the reward function to provide more immediate and dense rewards for aggressive and cooperative behaviors can incentivize agents to take more decisive actions. For example, increasing the reward for successfully capturing an opponent's flag or effectively defending their own flag can encourage proactive gameplay. Additionally, incorporating a curriculum learning approach can help guide the training process towards more desirable behaviors by gradually increasing the complexity of tasks and rewards.
Moreover, introducing intrinsic motivation mechanisms, such as curiosity-driven exploration or novelty rewards, can encourage agents to explore different strategies and behaviors, leading to a more diverse and adaptive gameplay. By rewarding agents for exploring new tactics or achieving novel objectives, the training process can become more robust and versatile.
Furthermore, implementing a form of reward shaping that focuses on collaborative behaviors can promote teamwork and coordination among agents. By rewarding actions that support teammates, communicate effectively, or strategically switch roles based on the game state, the deep RL models can learn to work together more efficiently and effectively.

What are the potential benefits and drawbacks of combining behavior-based and deep RL approaches, such as using the behavior-based methods as a foundation and then fine-tuning with deep RL

Combining behavior-based and deep RL approaches in the context of autonomous systems like the Aquaticus CTF game can offer a range of benefits and drawbacks. One potential benefit is leveraging the strengths of both methods to create a more robust and adaptive system. Behavior-based methods provide a structured and rule-based foundation for decision-making, ensuring safety and reliability in known scenarios. Deep RL, on the other hand, offers the flexibility to learn complex behaviors and adapt to dynamic environments through trial and error.
By using behavior-based methods as a foundation and then fine-tuning with deep RL, the system can benefit from the structured decision-making of behavior trees while also incorporating the learning capabilities of deep reinforcement learning. This hybrid approach can lead to more efficient and effective autonomous behaviors, especially in scenarios where a combination of rule-based and learned behaviors is required.
However, a potential drawback of this combination is the complexity of integrating two different approaches, which can lead to challenges in training, debugging, and maintaining the system. Balancing the rule-based constraints with the learned behaviors to ensure safety and performance can be a delicate process that requires careful tuning and validation.

How can the Aquaticus test-bed be extended to incorporate more realistic environmental factors, such as dynamic obstacles, changing weather conditions, and sensor uncertainty, to further stress-test the autonomy approaches

Expanding the Aquaticus test-bed to incorporate more realistic environmental factors can provide a more comprehensive evaluation of autonomy approaches. To stress-test the autonomy methods in dynamic and uncertain conditions, introducing factors such as dynamic obstacles, changing weather conditions, and sensor uncertainty can be beneficial.
One approach is to simulate dynamic obstacles that move unpredictably within the game environment, forcing agents to adapt their strategies in real-time to navigate around these obstacles. This can test the agility and responsiveness of the autonomy algorithms under challenging conditions.
Introducing changing weather conditions, such as strong winds or rough sea conditions, can test the robustness of the autonomy approaches to external disturbances. Agents may need to adjust their navigation and control strategies to account for varying environmental factors, enhancing their adaptability and resilience.
Moreover, incorporating sensor uncertainty, such as noisy sensor data or limited field of view, can challenge the perception and decision-making capabilities of the autonomous agents. By simulating realistic sensor limitations, the autonomy approaches can be evaluated in more realistic and challenging scenarios, leading to more reliable and effective autonomous behaviors in real-world applications.