The Adoption and Evaluation of Generative AI Tools in Digital Humanities Research: Insights from a Survey Study

Core Concepts

Digital humanities scholars are divided in their perceptions of the value of generative AI tools, while their actual usage diversifies across various research tasks.

Abstract

The survey-based study investigates how digital humanities (DH) scholars adopt, practice, and critically evaluate the use of generative AI (GenAI) tools, such as ChatGPT, in their research.
The key findings are:

Usage of GenAI Tools:

DH scholars widely use GenAI tools like ChatGPT, Gemini, and DALL-E in their personal lives.
However, a significant portion (37%) do not use GenAI in their research due to concerns over reliability, ethics, and institutional policies.

Purposes of Using GenAI in Research:

The primary use cases include brainstorming research ideas, coding assistance, and data visualization.
DH scholars prefer to leverage GenAI for exploratory, creative, and interpretive tasks rather than routine activities like writing and editing.

Perceived Benefits and Risks:

Benefits: Enhancing coding, creativity, and productivity; aiding researchers with language barriers.
Risks: Inaccuracies, propagation of misinformation, threats to research integrity and authenticity.

Barriers to Adoption:

Lack of skills, funding, training resources, and computing power to effectively integrate GenAI.
Concerns over the opaqueness and limitations of GenAI tools, hampering their seamless integration.

The findings suggest that DH scholars hold divisive sentiments towards the value of GenAI, while their actual usage diversifies across research tasks. The study provides a basis for further empirical research on the impact of GenAI on the evolution of DH scholarship.

Stats

"AI is not a field I am very interested in exploring at the moment."
"I have yet to explore its potential use in my scholarly work on DH librarianship."
"I haven't had enough time to figure out how to implement it in my research workflows."
"the outputs of generative AI are not reliable"
"the free version of (the GenAI) is unable to provide precise answers or solutions, especially for the … work that requires resources and citations."
"AI is still under development for a graduate or scholar level"
"Generative AI cannot do any task in my research better or faster than I can."
"I mistrust generative AI as a solution, as it manifests biases and discrimination, reproduces power dynamics, especially when used in corporate services, and poses hermetic barriers in transparency."
"if something has to be generated in research, it is by my brain :-)"
"I have not fully mastered the use of AI and therefore prefer to do in-depth research on my own to avoid plagiarism issues."
"the technology itself … is currently too opaque to base research on and it is also mostly too reliant on inaccessible technology from private industry to be of use for open, transparent analysis."
"given the "black box" and fundamentally elusive nature of the algorithms themselves and their inability to be fully explained and therefore the research using them unable to be replicated."

Quotes

"AI is not a field I am very interested in exploring at the moment."
"I mistrust generative AI as a solution, as it manifests biases and discrimination, reproduces power dynamics, especially when used in corporate services, and poses hermetic barriers in transparency."
"if something has to be generated in research, it is by my brain :-)"

Key Insights Distilled From

The collective use and evaluation of generative AI tools in digital humanities research: Survey-based results

by Meredith Ded... at arxiv.org 04-22-2024

https://arxiv.org/pdf/2404.12458.pdf

Deeper Inquiries

How can DH scholars effectively collaborate with AI experts to address the technical and ethical challenges of integrating GenAI tools in their research?

Collaboration between DH scholars and AI experts is crucial for addressing the technical and ethical challenges of integrating GenAI tools in research. DH scholars can benefit from the technical expertise of AI specialists in understanding the capabilities and limitations of GenAI tools. By working closely with AI experts, DH scholars can ensure the proper implementation of AI tools in their research projects, including data preprocessing, model training, and result interpretation.
Ethical considerations are paramount in the use of AI in research, and collaboration with AI experts can help DH scholars navigate complex ethical issues. AI experts can provide guidance on ensuring fairness, transparency, and accountability in the use of GenAI tools, helping researchers mitigate risks such as bias, misinformation, and plagiarism. By engaging in interdisciplinary collaboration, DH scholars can leverage the expertise of AI specialists to develop robust ethical frameworks and guidelines for the responsible use of GenAI in humanities research.

What institutional policies and support structures need to be developed to foster the responsible and transparent use of GenAI in DH research?

Institutional policies and support structures play a critical role in fostering the responsible and transparent use of GenAI in DH research. To promote ethical practices, institutions should establish clear guidelines on the use of AI tools, outlining expectations for transparency, accountability, and data privacy. These policies should address issues such as authorship attribution, data integrity, and the ethical implications of using AI-generated content in research publications.
Support structures should be developed to provide training and resources for DH scholars on the ethical and technical aspects of using GenAI tools. Institutions can offer workshops, seminars, and online courses to educate researchers on best practices for integrating AI in their research projects. Additionally, access to computational resources, AI software, and data repositories should be facilitated to enable researchers to effectively utilize GenAI tools in their work.
Collaboration between institutional ethics committees, research offices, and AI research centers can help establish comprehensive policies and support mechanisms for the responsible use of GenAI in DH research. By fostering a culture of transparency, integrity, and ethical conduct, institutions can ensure that DH scholars engage with GenAI tools in a manner that upholds the highest standards of research ethics and integrity.

How might the integration of GenAI tools reshape the fundamental nature of humanities scholarship, and what new research directions could emerge from this technological shift?

The integration of GenAI tools has the potential to reshape the fundamental nature of humanities scholarship by revolutionizing research methodologies, expanding analytical capabilities, and fostering interdisciplinary collaborations. GenAI tools can automate repetitive tasks, such as data analysis and literature review, allowing researchers to focus on higher-order thinking and creative interpretation. This shift towards automation and augmentation of research processes can lead to increased efficiency, productivity, and innovation in humanities scholarship.
Furthermore, the use of GenAI tools can open up new research directions and interdisciplinary opportunities in the humanities. For example, AI-powered text analysis can uncover hidden patterns and insights in large datasets, enabling researchers to explore complex cultural narratives and historical contexts in novel ways. GenAI tools can also facilitate cross-disciplinary collaborations between humanities scholars and AI experts, leading to innovative research projects that combine humanistic inquiry with computational analysis.
Overall, the integration of GenAI tools in humanities scholarship has the potential to catalyze transformative changes in research practices, methodologies, and outcomes. By embracing AI technologies, DH scholars can embark on new research directions, explore uncharted territories, and push the boundaries of knowledge creation in the digital age.