Using Assurance Cases to Improve Confidence in the Correctness of Research Software: An Example from Medical Image Analysis

Core Concepts

Assurance cases can guide verification and validation efforts for research software, ensuring the software delivers correct outputs by systematically documenting evidence for software requirements, design, implementation, and user responsibility.

Abstract

• Bibliographic Information: Smith, S., & Lin, J. (2024). Using Assurance Cases to Guide Verification and Validation of Research Software. arXiv preprint arXiv:2411.03291v1.
• Research Objective: This paper demonstrates how assurance cases (ACs) can be used to guide verification and validation (VnV) efforts for research software, focusing on the correctness of calculations.
• Methodology: The authors present a case study of AortaGeomRecon, a software tool for extracting 3D segmentation of the aorta from medical images. They develop an assurance case for this software, outlining the necessary evidence to support its correctness. This evidence includes domain expert reviews, traceability matrices, comparison to a pseudo-oracle, and documentation of input assumptions and user responsibilities.
• Key Findings: The study highlights that code is not the only artifact requiring VnV. Domain expert reviews, traceability matrices, documentation of assumptions, and user instructions are crucial for building confidence in the software's correctness. The authors also emphasize the importance of distinguishing between user and software responsibilities.
• Main Conclusions: ACs provide a structured and systematic approach to guide VnV activities for research software, ensuring that the software delivers correct outputs. The case study demonstrates the feasibility and benefits of using ACs in a medical image analysis context.
• Significance: This research contributes to the growing body of knowledge on software quality assurance for research software, a domain where traditional software engineering practices are often overlooked.
• Limitations and Future Research: The case study focuses on a specific type of research software and a single quality attribute (correctness). Future research could explore the application of ACs to other domains and quality attributes, such as reliability, performance, and security.

Stats

The Dice coefficient comparison to independent solutions from a pseudo-oracle solution on real images has to be greater than 95%.

Quotes

"Assurance Cases (ACs) provide a structured argument that some property (like correctness, reproducibility, safety, etc) holds."
"The evidence required by an AC can be sizeable, but the AC allows the VnV efforts to be guided by a purpose."
"Our case study highlighted that code is not the only artifact of interest for building confidence."

Key Insights Distilled From

Using Assurance Cases to Guide Verification and Validation of Research Software

by W. Spencer S... at arxiv.org 11-06-2024

https://arxiv.org/pdf/2411.03291.pdf

Deeper Inquiries

How can the development and maintenance of assurance cases be integrated into the iterative and often exploratory nature of research software development?

Integrating assurance cases into the iterative and exploratory nature of research software development requires a flexible and adaptable approach. Here's how it can be achieved:

Iterative Development of Assurance Cases: Instead of treating the assurance case as a monolithic document created at the end, adopt an iterative approach. Start with a high-level assurance case outlining the top-level claims and key arguments. As the software evolves through iterations, refine and expand the assurance case by adding sub-claims, evidence, and justifications. This allows the assurance case to grow organically alongside the software.
Lightweight Documentation: Recognize that detailed formal proofs might not be feasible or necessary in the early stages. Instead, leverage lightweight documentation methods like wikis, Markdown files, or even comments within the code to capture design decisions, assumptions, and testing strategies. These can be progressively formalized as the software matures.
Tool Support: Utilize tools that support the creation, management, and visualization of assurance cases. These tools can help link evidence to claims, track dependencies, and highlight areas requiring further attention. Version control systems can also be used to manage changes to the assurance case alongside the software.
Integration with Agile Practices: Align the development of the assurance case with agile methodologies. During sprint planning, identify relevant claims and evidence to be addressed within the sprint. Include assurance case reviews as part of sprint reviews and retrospectives.
Embrace Evolving Requirements: Acknowledge that requirements in research software are often fluid. Design the assurance case to accommodate changes in requirements by clearly documenting the rationale behind decisions and ensuring traceability between requirements, design, and implementation.
Focus on Criticality: Prioritize assurance efforts based on the criticality of different software components. Focus on building stronger assurance for components directly related to the core scientific computations or those with potential safety or ethical implications.
By adopting these strategies, assurance cases can become living documents that evolve with the research software, providing a valuable tool for managing risks and building confidence throughout the development lifecycle.

While the assurance case methodology emphasizes correctness, could an over-reliance on pre-defined assumptions and expert validation potentially limit the flexibility and adaptability of research software in evolving scientific domains?

You raise a valid concern. While assurance cases are valuable for ensuring correctness based on existing knowledge, an over-reliance on pre-defined assumptions and expert validation can indeed hinder flexibility and adaptability, especially in rapidly evolving scientific domains. Here's how to mitigate this potential limitation:

Explicitly Addressing Limitations: Clearly document the limitations of the assurance case, including the scope of the assumptions made and the potential impact of these assumptions on the software's applicability in different contexts. This transparency allows users to understand the boundaries within which the assurance holds.
Designing for Extensibility: Structure the software and the assurance case in a modular and extensible way. This allows for easier modification and extension of the software without requiring a complete overhaul of the assurance argument. New modules or features can be added with their own associated claims and evidence.
Incorporating Uncertainty: Acknowledge that scientific knowledge is constantly evolving. Instead of relying solely on absolute correctness, incorporate mechanisms to handle uncertainty. This might involve using probabilistic reasoning, sensitivity analysis, or clearly documenting the conditions under which the software's results might be less reliable.
Encouraging Experimentation: Foster a culture of experimentation and exploration within the research team. Encourage testing the software's boundaries and exploring edge cases that might challenge existing assumptions. Document these explorations and their findings to inform future development and refinement of the assurance case.
Regular Review and Update: Establish a process for regularly reviewing and updating the assurance case in light of new scientific discoveries, changes in the software, or feedback from users. This ensures that the assurance case remains relevant and reflects the current state of knowledge.
By striking a balance between ensuring correctness and embracing the evolving nature of scientific research, assurance cases can become valuable tools for supporting both rigor and adaptability in research software development.

If scientific discoveries are fundamentally about challenging existing assumptions, how can assurance cases be designed to accommodate and even encourage the exploration of unexpected results or edge cases that might fall outside initial assumptions?

This is a crucial aspect of applying assurance cases in research where challenging assumptions is inherent to the process. Here's how assurance cases can be designed to accommodate and encourage such exploration:

Framing Assumptions as Hypotheses: Instead of treating assumptions as immutable truths, frame them as testable hypotheses within the assurance case. This encourages a mindset of continuous questioning and validation.
Explicitly Identifying Areas of Uncertainty: Clearly demarcate areas within the software and the assurance case where uncertainty is high or where assumptions are more likely to be challenged. This helps focus exploration efforts on these critical areas.
Designing for Falsification: Structure the assurance case to facilitate the identification and investigation of potential falsifications. This might involve developing test cases specifically designed to challenge assumptions or creating mechanisms for users to easily report unexpected results.
Tracking Anomalies and Deviations: Implement systems for logging and analyzing anomalies or deviations from expected behavior. This data can provide valuable insights into the limitations of existing assumptions and guide the exploration of new hypotheses.
Iterative Refinement Based on Discoveries: Establish a feedback loop where unexpected results or edge cases lead to a reevaluation and refinement of the assurance case. This might involve revising assumptions, adding new claims, or adjusting the level of confidence in certain arguments.
Promoting Open Discussion and Collaboration: Foster an open and collaborative environment where researchers and developers feel comfortable discussing unexpected results and challenging existing assumptions. This open dialogue is essential for identifying and addressing potential weaknesses in the assurance case.
By embracing the iterative and exploratory nature of scientific discovery, assurance cases can evolve from static documents into dynamic tools that support the pursuit of new knowledge while maintaining a focus on rigor and trustworthiness.