Improving Autonomous Driving Testing with Digital Siblings: A Multi-Simulator Approach

Core Concepts

A multi-simulator approach called digital siblings can better predict the failures of DNN-based lane-keeping models in a digital twin, compared to using a single simulator.

Abstract

The paper proposes a novel approach called digital siblings (DSS) to improve simulation-based testing of autonomous driving software. The key idea is to use multiple general-purpose simulators (GPSims) collectively as an ensemble, rather than relying on a single GPSim, to better approximate the behavior of the autonomous vehicle (AV) in a high-fidelity digital twin (DT). The authors focus on testing the lane-keeping component of an AV, implemented using deep neural networks (DNNs). They use two open-source simulators, BeamNG and Udacity, as the digital siblings, and a digital twin of a physical 1:16 scale electric AV as the ground truth. The approach involves the following steps: Training or fine-tuning the DNN lane-keeping model to run on both digital siblings and the digital twin. Generating test scenarios (sequences of road points) for each digital sibling using an evolutionary search algorithm (DeepHyperion). Migrating the test cases across the digital siblings and merging their outcomes to obtain a unified view (digital siblings feature map). Executing the test cases from the digital siblings feature map on the digital twin to obtain the ground truth feature map. Analyzing the correlation between the digital siblings feature map and the digital twin feature map to assess the capability of the digital siblings in predicting the failures of the DNN lane-keeping model on the digital twin. The empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin. The authors discuss the findings and provide recommendations for researchers and software engineers interested in automated testing of autonomous driving software.

Stats

The mean squared error (MSE) between the predicted steering angle and the ground truth steering angle on the digital twin is 0.08 for the models trained on simulated images (MS) and 0.07 for the models trained on pseudo-real images (MR). The success rate of the lane-keeping models on the digital twin is 0.69 for MS and 0.95 for MR.

Quotes

"Simulation-based testing represents an important step to ensure the reliability of autonomous driving software." "Using a single-simulator approach for AV testing might be unreliable, as the testing results are highly dependent on the chosen GPSim." "The ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin."

Key Insights Distilled From

Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing

by Matteo Biagi... at arxiv.org 04-25-2024

https://arxiv.org/pdf/2305.08060.pdf

Deeper Inquiries

How can the digital siblings approach be extended to test other autonomous driving functionalities beyond lane-keeping, such as object detection, path planning, and decision making?

The digital siblings approach can be extended to test other autonomous driving functionalities by adapting the test generation process to focus on the specific requirements of each functionality. For object detection, the test cases can involve scenarios with varying numbers and types of objects in different positions and orientations relative to the vehicle. Path planning testing can include generating complex road layouts with obstacles and challenging navigation scenarios. Decision-making testing can involve creating scenarios where the vehicle needs to make critical choices based on various factors like traffic conditions, pedestrian presence, and road signs. Each functionality would require defining specific metrics and criteria for success or failure, which can be incorporated into the feature maps generated by the digital siblings. By training the driving models on diverse datasets that cover a wide range of scenarios relevant to each functionality, the digital siblings can provide a comprehensive evaluation of the autonomous driving system's performance across different aspects.

How can the digital siblings approach be integrated with other testing techniques, such as real-world testing or hardware-in-the-loop testing, to provide a comprehensive testing strategy for autonomous driving systems?

Integrating the digital siblings approach with real-world testing can enhance the validation process by bridging the gap between simulation and reality. Real-world testing can provide valuable data on how the autonomous driving system performs in actual driving conditions, which can be used to validate the predictions made by the digital siblings. By comparing the results from both simulation-based testing with digital siblings and real-world testing, developers can gain a more holistic understanding of the system's capabilities and limitations. Hardware-in-the-loop testing can also complement the digital siblings approach by incorporating physical components of the autonomous driving system into the simulation environment. This integration allows for testing the interaction between the software and hardware components in a controlled environment before deploying the system on actual vehicles. By combining the insights from digital siblings testing with real-world and hardware-in-the-loop testing, developers can create a robust and comprehensive testing strategy that covers a wide range of scenarios and conditions, ensuring the reliability and safety of autonomous driving systems.