The Dynamics of Abandoning Harmful Algorithmic Systems: A Thematic Analysis of Real-World Cases

Core Concepts

Algorithmic systems that inflict societal harms can be abandoned by their proprietors due to public pressure and advocacy efforts, following a common process of six iterative phases: discovery, diagnosis, dissemination, dialogue, decision, and death.

Abstract

The paper presents a thematic analysis of 40 real-world cases where algorithmic systems were called to be abandoned or were successfully abandoned. The authors identify a common process of six iterative phases that characterize the dynamics leading to algorithm abandonment:

Discovery: The existence of the algorithm or its potential/realized harms is made public, initiating widespread critique.

Diagnosis: Critics gather further evidence about the algorithm's behavior and impacts through audits and investigations into the design decisions.

Dissemination: Critics amplify their message to raise public awareness and put pressure on algorithm owners.

Dialogue: Algorithm owners respond to criticism, either denying or accepting the validity of the concerns raised.

Decision: Algorithm owners decide whether to repair or abandon the system, or are forced to abandon it through policy or legal action.

Death: After abandonment, the algorithm's impacts may continue to linger, requiring overriding past decisions and providing reparations, while the technology may also be "reincarnated" in different contexts.

The authors also identify key socio-technical factors that influence the outcome and timeline of abandonment, including the algorithm's user base, value proposition, dependencies, transparency, auditability, visibility of criticism, and the regulatory environment. The findings provide insights for various stakeholders, including algorithm owners, FAccT researchers, and policymakers, on supporting future calls for abandoning harmful algorithmic systems.

Stats

"Algorithms deployed in similar domains resulted in similar harms, e.g., chatbots could demean and stereotype [97, 98], and resource allocation algorithms could result in opportunity loss [87, 143]."
"Several different types of harm often co-occurred for each single algorithmic system."
"Relatively few algorithm owners released public statements with explanations or named reasons why they decided to abandon the algorithm."
"The majority of algorithm owners did not provide financial reparations for the harm caused, with a few notable exceptions."

Quotes

"Grassroots movements, moratorium bans, and boardroom decisions to abandon harmful algorithmic systems continue to grow across the world."
"While some algorithms were abandoned in early ideation phases before they were deployed and some were abandoned within mere hours of harm discovery, other algorithms continue(d) to inflict harm for years."
"Abandoning an algorithm that other components rely on might have a domino effect that requires changing its surrounding socio-technical components."

Key Insights Distilled From

The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment

by Nari Johnson... at arxiv.org 04-23-2024

https://arxiv.org/pdf/2404.13802.pdf

Deeper Inquiries

How can algorithm owners be incentivized or required to provide more transparency about their decision-making processes surrounding the development and deployment of algorithmic systems?

Transparency in the decision-making processes of algorithm owners is crucial for accountability and trust-building with stakeholders. To incentivize or require more transparency, several strategies can be implemented:

Regulatory Mandates: Implementing legal requirements that mandate algorithm owners to disclose information about their systems, including the data used, the design process, and potential impacts. This can be enforced through data protection laws, algorithmic accountability regulations, or industry-specific guidelines.

Transparency Reports: Algorithm owners can be encouraged to publish regular transparency reports detailing their decision-making processes, algorithmic design choices, and any updates or changes made to the system. This can be a proactive measure to build trust with users and stakeholders.

Independent Audits: Encouraging or requiring independent audits of algorithmic systems can provide an external validation of the decision-making processes. These audits can assess the fairness, accountability, and transparency of the algorithms, ensuring that they align with ethical standards.

Stakeholder Engagement: Engaging with stakeholders, including impacted communities, advocacy groups, and experts, can help algorithm owners understand the concerns and perspectives of those affected by their systems. This engagement can lead to more transparent decision-making processes and foster collaboration in addressing algorithmic harms.

Incentive Structures: Creating incentive structures, such as certifications or recognition programs for transparent practices, can motivate algorithm owners to prioritize transparency in their decision-making. Public recognition for transparency efforts can enhance the reputation and credibility of organizations.

Education and Awareness: Promoting awareness about the importance of transparency in algorithmic systems through training programs, workshops, and industry events can help foster a culture of openness and accountability among algorithm owners.

By combining regulatory measures, industry standards, stakeholder engagement, and incentives, algorithm owners can be encouraged to provide greater transparency in their decision-making processes, ultimately leading to more accountable and ethical algorithmic systems.

How can the perspectives and experiences of those most impacted by algorithmic systems be better centered in the design, development, and governance of these technologies?

Centering the perspectives and experiences of those most impacted by algorithmic systems is essential for creating inclusive, ethical, and effective technologies. Here are some strategies to achieve this:

Community Engagement: Actively involve impacted communities in the design and development process through focus groups, co-design sessions, and participatory workshops. This ensures that their needs, values, and concerns are integrated into the technology from the outset.

User-Centered Design: Adopt a user-centered design approach that prioritizes the experiences and feedback of end-users throughout the development lifecycle. Conduct user research, usability testing, and feedback sessions to incorporate diverse perspectives into the design process.

Ethical Considerations: Establish ethical review boards or committees that include representatives from impacted communities to evaluate the potential impacts of algorithmic systems. This ensures that ethical considerations are central to decision-making.

Diversity and Inclusion: Promote diversity and inclusion within development teams to bring a range of perspectives to the table. Encouraging diverse voices can lead to more comprehensive solutions that address the needs of all users.

Transparency and Accountability: Maintain transparency about the decision-making processes, data sources, and algorithmic outcomes to build trust with impacted communities. Establish mechanisms for accountability and redress in case of harm or bias.

Continuous Feedback Loop: Implement a feedback mechanism that allows users to provide ongoing input and suggestions for improvement. Regularly solicit feedback, conduct impact assessments, and iterate on the technology based on user insights.

Empowerment and Education: Empower impacted communities by providing education and training on how algorithmic systems work, their potential impacts, and how to advocate for their rights. This empowers users to engage meaningfully in the governance of technology.

By centering the perspectives and experiences of those most impacted by algorithmic systems, organizations can create more inclusive and responsible technologies that better serve the needs of diverse user groups.

What legal and regulatory frameworks would be most effective in holding algorithm vendors accountable for the harms caused by the systems they develop and sell to organizations?

To hold algorithm vendors accountable for the harms caused by the systems they develop and sell, effective legal and regulatory frameworks are essential. Here are key frameworks that can enhance accountability:

Liability Laws: Implementing liability laws that hold algorithm vendors accountable for the harms caused by their products. This can include strict liability standards, where vendors are held responsible for damages regardless of fault.

Data Protection Regulations: Enforcing data protection regulations that require vendors to adhere to strict data privacy and security standards. This ensures that vendors are accountable for the misuse or mishandling of personal data by their algorithms.

Algorithmic Impact Assessments: Mandating algorithmic impact assessments to evaluate the potential risks and impacts of algorithms before deployment. Vendors would be required to conduct and disclose these assessments to regulatory bodies for review.

Ethical Guidelines and Standards: Establishing industry-specific ethical guidelines and standards that outline best practices for algorithm development and deployment. Vendors must comply with these standards to ensure ethical use of their technologies.

Oversight and Regulatory Bodies: Creating independent oversight bodies or regulatory agencies tasked with monitoring and regulating algorithm vendors. These bodies can investigate complaints, enforce compliance, and impose penalties for non-compliance.

Transparency and Accountability Requirements: Requiring vendors to maintain transparency about their algorithms, including disclosing the underlying processes, data sources, and potential biases. Accountability mechanisms should be in place to address algorithmic harms and provide redress to affected parties.

Enforcement Mechanisms: Implementing robust enforcement mechanisms, such as fines, sanctions, or license revocations, for vendors that violate regulations or cause significant harm through their algorithms. Strong enforcement deters misconduct and ensures compliance with legal standards.

International Collaboration: Promoting international collaboration and harmonization of regulations to address cross-border issues related to algorithmic harms. Cooperation between countries can strengthen accountability measures and facilitate information sharing.

By implementing these legal and regulatory frameworks, algorithm vendors can be held accountable for the harms caused by their systems, promoting responsible and ethical practices in the development and deployment of algorithms.