On-policy actor-critic reinforcement learning for multiple unmanned aerial vehicle exploration

Tite: On-policy actor-critic reinforcement learning for multiple unmanned aerial vehicle exploration (DOI)
​​​​​​​Authors:  Ali Moltajaei Farid, Jafar Roshanian, Malek Mouhoub.
Journal: Expert Systems with Applications
​​​​​​​  ​​​​​​​
Abstract: Unmanned aerial vehicles (UAVs) have become increasingly popular in various fields, including precision agriculture, search and rescue, and remote sensing. However, exploring unknown environments remains a significant challenge. This study aims to address this challenge using on-policy reinforcement learning (RL) with proximal policy optimization (PPO) to explore the two-dimensional area of interest with multiple UAVs (multi-rotor and fixed-wing). The UAVs will avoid collision with obstacles and each other and will do the exploration in a distributed manner. The proposed solution includes actor-critic networks that use deep convolutional neural networks (CNN) and long-short-term memory (LSTM) to identify UAVs and areas that have already been covered. Compared to other RL techniques, such as the policy gradient (PG) and the actor-critic asynchronous advantage (A3C), the simulation results demonstrate the superiority of the proposed PPO approach. In addition, the results show that combining LSTM with CNN in critic can improve exploration. Since the proposed exploration has to work in unknown environments, where environments not encountered during training, and we evaluate generalization to such unseen environments. The results showed that the proposed setup can complete the coverage when we have new maps that differ from the trained maps. Finally, we show how tuning hyperparameters may affect overall performance. While the proposed framework is designed for generic UAV types, the main formulation and training are implemented for multi-rotor UAVs; fixed-wing behavior is analyzed separately due to their turning constraints and is not included in the main RL training loop.