DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

แนวคิดหลัก

Generative models must consider local socio-cultural contexts, leading to the creation of DOSA, a community-generated dataset of social artifacts from 19 Indian subcultures.

บทคัดย่อ

1. Introduction

Generative models integrated into various applications.
Concerns about cultural nuances encoded in LLMs.
Web-based training data limitations.
2. Related Work

Evaluation of LLMs' personality, values, and ethics.
Leveraging psycho-metric and cultural instruments.
3. Methodology for the Dataset Creation

Combination of Survey and Games with a Purpose (GWAP).
Recruitment process and game mechanics explained.
4. Benchmarking LLMs Cultural Familiarity

Experimental setup with open-source and closed-source models.
Evaluation metrics used for assessing cultural familiarity.
5. Results

Variance in LLMs' familiarity with regional subcultures in India.
Performance differences among LLMs on original vs. expanded artifacts.
6. Discussion & Future Work

Importance of evaluating LLMs at geographic subcultural levels.
Need for future work to investigate other dimensions of culture.
7. Limitations

Missing states/union territories in the dataset.
Language limitations impacting diversity of responses.
8. Ethical Considerations

Institutional review approval obtained.

สถิติ

LLMsは、DOSAデータセットの社会的アーティファクトに対する文化的な理解を評価するために使用されました。

คำพูด

"Culture is a complex societal-level concept."
"Creating these datasets would also help preserve knowledge."

ข้อมูลเชิงลึกที่สำคัญจาก

DOSA

by Agrima Seth,... ที่ arxiv.org 03-25-2024

https://arxiv.org/pdf/2403.14651.pdf

สอบถามเพิ่มเติม

How can participatory research be scaled up to provide more breadth and depth to datasets created?

Participatory research can be scaled up by implementing several strategies:

Community Engagement: Actively involve a diverse range of community members in the research process. This could include reaching out to different demographics, such as age groups, professions, or cultural backgrounds.

Collaborative Platforms: Utilize online platforms or tools that allow for remote participation, making it easier for individuals from various locations to contribute their insights and knowledge.

Capacity Building: Provide training and resources to community members to empower them with the skills needed to participate effectively in the research process.

Network Expansion: Establish partnerships with local organizations, NGOs, or educational institutions that have existing connections within communities. This can help broaden the reach of the research efforts.

Iterative Feedback Loops: Implement feedback mechanisms where participants can review and provide input on the data collected, ensuring accuracy and relevance.

Incentivization: Offer incentives or rewards for participation to encourage continued engagement from community members.

By incorporating these approaches, participatory research can expand its scope and depth while ensuring inclusivity and diversity in dataset creation processes.

What are the implications of LLMs not representing their awareness of social artifacts in their outputs?

The implications of LLMs lacking awareness of social artifacts in their outputs are significant:

Cultural Misrepresentation: Without an understanding of social artifacts specific to different cultures, LLMs may produce outputs that misrepresent or stereotype certain communities.

Communication Breakdown: In multicultural contexts, inaccuracies in representing cultural nuances through language models can lead to misunderstandings and breakdowns in communication between users from diverse backgrounds.

Bias Reinforcement: The absence of cultural awareness in LLM outputs perpetuates biases towards dominant cultures while marginalizing minority or underrepresented groups further reinforcing societal inequalities.

Loss of Cultural Heritage: Failure to acknowledge important social artifacts erases valuable aspects of cultural heritage from linguistic representations generated by LLMs leading to potential loss or dilution of unique traditions over time.

How can technology evaluation benefit from engaging community members using participatory research?

Engaging community members through participatory research offers several benefits for technology evaluation:

Diverse Perspectives: Community involvement ensures a wide range of perspectives are considered during technology evaluation leading to more comprehensive assessments.

2.Contextual Relevance:
By involving community members who understand local socio-cultural contexts intimately,
evaluations become more contextually relevant reflecting real-world usage scenarios accurately.
3Ethical Considerations:
Community engagement helps identify ethical considerations specific
To particular user groups ensuring technologies align with moral values
And societal norms.
4Improved Acceptance:
Engaging end-users fosters a sense Of ownership And acceptance towards evaluated technologies,
Increasing adoption rates And overall success metrics
5**Enhanced Validity:
Incorporating inputs From those directly impacted By technological Solutions enhances The validity
Of evaluations As they reflect actual User experiences And needs accurately
Overall Participatory Research methods offer a holistic approach To Technology Evaluation That prioritizes User-Centric Design principles Leading To More Effective And Ethically Sound Technological Solutions