Formative Situations: A Valuable Contribution Type for Research through Design

Core Concepts

Formative situations are a valuable contribution type for Research through Design that can provide extensible design knowledge, complement summative evaluations, and improve the translation of research to industry.

Abstract

The authors argue that the HCI community currently places too much emphasis on summative evaluations of finalized prototypes, while overlooking the value of formative evaluations of mid-fidelity prototypes. They propose "formative situations" as a framework for assessing design artifacts at intermediate stages of the design process.
The key points are:

Formative situations exist between highly controlled lab studies and in-the-wild deployments, matching the provisionality of the prototype.
Formative situations can provide rich, subjective feedback to inform future design iterations, as well as extensible design knowledge for the broader community.
The authors present five criteria for assessing the rigor of formative situations: 1) well-fitted to the RtD questions, 2) drawing on related best practices, 3) appropriate weight compared to prototype provisionality, 4) allowing designer involvement, and 5) enabling active participant roles.
Accepting formative situations as a valid contribution type could improve the translation of research insights to industry practitioners, by providing more actionable, mid-fidelity design knowledge.
The authors call for a shift in the HCI community to recognize formative evaluations as a valuable complement to summative studies, rather than prematurely rushing to conduct summative evaluations.

Stats

"Many of the prototypes that researchers develop are, too, anything but convincing products. They may be wholly or partly faked; if implemented, they may be unstable and lack some expected functionality."
"Empirical" research made up the largest portion of contribution types at the ACM Conference on Human Factors in Computing Systems (CHI) in 2016.

Quotes

"Why do we as a research community seem to value summative evaluation over formative evaluation?"
"Allowing for shareable research contributions from formative evaluations along the way to final designs worthy of summative evaluation could free our community to build more flexibly and extensively upon one another's work."

Key Insights Distilled From

"That's Not Good Science!"

by Raquel B Rob... at arxiv.org 04-03-2024

https://arxiv.org/pdf/2404.01848.pdf

Deeper Inquiries

How might formative situations be leveraged to better connect academic research with industry design and development practices?

Formative situations can serve as a bridge between academic research and industry design by providing valuable insights and knowledge at the mid-fidelity prototyping stage. Academic researchers can use formative evaluations to gather rich, subjective feedback from participants, which can inform the design process and lead to the creation of more practical and usable products. By sharing these formative insights with industry practitioners, researchers can offer actionable design knowledge that can be further developed into commercial products. This exchange of information can help industry professionals understand the design space better and make more informed decisions in product development. Additionally, involving industry stakeholders in formative situations can ensure that academic research aligns with real-world needs and constraints, making the transition from research to industry smoother and more effective.

What are potential drawbacks or limitations of over-emphasizing formative situations as a contribution type, and how can these be mitigated?

One potential drawback of over-emphasizing formative situations as a contribution type is the risk of undervaluing summative evaluations, which provide essential insights into the overall performance and effectiveness of a design. By focusing too much on formative evaluations, researchers may neglect the need for rigorous testing and validation of their prototypes in real-world settings. This could lead to a lack of robust evidence to support the claims and findings of the research. To mitigate this limitation, researchers should strike a balance between formative and summative evaluations, ensuring that both types of assessments are conducted to validate the design and its impact thoroughly. Additionally, clear guidelines and criteria for conducting formative evaluations should be established to maintain the rigor and validity of the research.

In what other domains outside of HCI could the concept of formative situations provide valuable insights and knowledge generation?

The concept of formative situations can be applied to various domains beyond HCI to generate valuable insights and knowledge. In fields like product design and development, formative evaluations can help designers gather feedback on prototypes at intermediate stages, allowing for iterative improvements and better alignment with user needs. In education, formative assessments can provide ongoing feedback to students and teachers, enabling personalized learning experiences and continuous improvement. In healthcare, formative evaluations of medical devices and treatments can lead to more patient-centered solutions and improved outcomes. Overall, the concept of formative situations can be adapted and applied across disciplines to enhance the design, development, and evaluation processes in diverse domains.