Challenges in Deploying Machine Learning Models for Structural Engineering Applications

Core Concepts

Successful development of machine learning models does not necessarily translate into useful solutions that can be deployed for real-world datasets in structural engineering applications.

Abstract

This paper discusses the challenges in deploying machine learning (ML) models for structural engineering applications. Despite the promising performance of ML-based solutions, they are usually only demonstrated as proof-of-concept and are rarely deployed for real-world applications.
The key challenges highlighted in the paper include:

Generalizability beyond the training set:

Overfitting can lead to poor performance on new data due to capturing pseudo-relationships or irrelevant patterns in the training data.
Inadequate training datasets that do not represent the diversity of real-world data can result in poor deployment performance.

Explainability through feature importance:

Feature importance metrics may not accurately reflect the true importance of variables, as they often measure the consequence of randomly permuting a feature rather than removing it entirely.
Underspecification, where multiple distinctive sets of features may equally satisfy the evaluation criteria, can lead to misleading interpretations of feature importance.

The paper presents two illustrative examples using datasets from finite element simulations of cold-formed steel channels and experimental studies on reinforced concrete walls. These examples demonstrate the issues of overfitting, variable omission bias, and underspecification, highlighting the importance of implementing rigorous model validation techniques, careful physics-informed feature selection, and considerations of both model complexity and generalizability for successful deployment of ML models in structural engineering.

Stats

The cold-formed steel channel dataset contains information on 14 features, including channel depth, flange width, stiffener length, thickness, slot dimensions, and material properties.
The reinforced concrete wall dataset contains information on 15 features, including slenderness ratio, shear stress demand, reinforcement details, and axial load ratio.

Quotes

"Fundamentally, almost all ML models (due to their statistical nature, and similar to other well-established approaches such as empirical analysis) capture data association rather than causal relationships."
"Even when ignoring the limitations of ML models in identifying causal relationships, many ML models make it difficult to determine the nature of the association between variables. Such a "black-box" aspect negatively affects user confidence when applying the ML model in deployment."
"The over-reliance on accuracy metrics during model development might be inadequate, or even problematic, for deployment under specific circumstances."

Key Insights Distilled From

Beyond development: Challenges in deploying machine learning models for structural engineering applications

by Mohsen Zaker... at arxiv.org 04-22-2024

https://arxiv.org/pdf/2404.12544.pdf

Deeper Inquiries

How can structural engineers incorporate domain knowledge and causal relationships into the development of ML models to improve their deployment reliability?

Incorporating domain knowledge and causal relationships into the development of ML models is crucial for improving their deployment reliability in structural engineering applications. Here are some key strategies to achieve this:

Feature Engineering: Structural engineers can leverage their domain expertise to select relevant features that have a direct impact on the structural behavior being studied. By carefully choosing input variables based on their knowledge of the system, engineers can ensure that the model captures the most important aspects of the problem.

Physics-Informed Modeling: Integrating physical laws and principles into the ML models can enhance their interpretability and generalizability. By incorporating known relationships between variables based on fundamental principles of structural mechanics, engineers can guide the model to make more accurate predictions.

Causal Inference Techniques: Structural engineers can use causal inference techniques to identify and model causal relationships between variables rather than just correlations. By understanding the causal mechanisms at play in the system, engineers can develop more robust models that are better suited for real-world deployment.

Collaboration with ML Experts: Working closely with machine learning experts can help structural engineers translate their domain knowledge into ML models effectively. ML experts can provide guidance on model selection, feature importance analysis, and validation techniques to ensure that the models are both accurate and reliable in deployment.

By combining domain knowledge with advanced ML techniques, structural engineers can develop models that not only perform well in training but also demonstrate reliability and effectiveness when deployed in real-world structural engineering applications.

How can the potential drawbacks of using feature importance metrics as the sole basis for model interpretation be mitigated, and how can they be complemented with other explainability techniques?

While feature importance metrics can provide valuable insights into the contribution of variables to the model, relying solely on these metrics for model interpretation has several drawbacks. To mitigate these drawbacks and enhance model explainability, structural engineers can take the following steps:

Model-Agnostic Techniques: Instead of relying on model-specific feature importance metrics, engineers can use model-agnostic techniques like permutation importance or Shapley values. These methods provide a global explanation of feature importance irrespective of the model used, offering a more comprehensive understanding of variable contributions.

Interpretability Metrics: Engineers can complement feature importance metrics with interpretability metrics that assess the model's transparency, sparsity, and sensitivity to perturbations. By evaluating these aspects, engineers can ensure that the model is not only accurate but also interpretable and explainable.

Domain-Specific Interpretation: Incorporating domain-specific knowledge and causal relationships into the interpretation process can help validate the findings from feature importance analysis. By cross-referencing the model's predictions with known structural principles, engineers can verify the relevance and significance of the identified features.

Visualization Techniques: Visualizing the relationships between features and the target variable can provide a more intuitive understanding of the model's behavior. Techniques like partial dependence plots, LIME (Local Interpretable Model-agnostic Explanations), and SHAP (SHapley Additive exPlanations) plots can offer valuable insights into how individual features impact the model's predictions.

By combining feature importance metrics with model-agnostic techniques, interpretability metrics, domain-specific knowledge, and visualization techniques, structural engineers can enhance the interpretability and explainability of their ML models, leading to more reliable and trustworthy deployment in structural engineering applications.

How can the structural engineering community collaborate with machine learning experts to develop guidelines and best practices for the successful deployment of ML models in real-world applications?

Collaboration between the structural engineering community and machine learning experts is essential for developing guidelines and best practices for the successful deployment of ML models in real-world applications. Here are some ways in which this collaboration can be fostered:

Interdisciplinary Workshops and Seminars: Organizing workshops and seminars that bring together structural engineers and machine learning experts can facilitate knowledge sharing and collaboration. These events can focus on discussing challenges, sharing insights, and developing guidelines for deploying ML models in structural engineering.

Joint Research Projects: Collaborating on joint research projects that combine expertise from both fields can lead to the development of innovative solutions and best practices for deploying ML models in structural engineering applications. By working together on real-world problems, researchers can identify common challenges and devise effective strategies to address them.

Development of Standards and Protocols: The collaborative effort can involve the development of standards, protocols, and guidelines for the deployment of ML models in structural engineering. By establishing best practices and quality assurance measures, the community can ensure the reliability and effectiveness of ML applications in the field.

Knowledge Exchange Platforms: Creating online platforms or forums where structural engineers and machine learning experts can exchange ideas, share resources, and seek advice can foster a culture of collaboration and continuous learning. These platforms can serve as hubs for discussing challenges, proposing solutions, and disseminating best practices.

Continuous Education and Training: Providing opportunities for continuous education and training in both structural engineering and machine learning can help professionals stay updated on the latest advancements and methodologies. Workshops, webinars, and online courses can equip practitioners with the knowledge and skills needed to deploy ML models effectively in real-world scenarios.

By fostering collaboration, sharing expertise, and working together on research and development initiatives, the structural engineering community and machine learning experts can jointly contribute to the advancement of guidelines and best practices for the successful deployment of ML models in real-world structural engineering applications.