Exploring Art Collections Through Object Detection: Enhancing Museum Experiences with AI-Powered Browsing

Core Concepts

This research explores how object detection can be leveraged to facilitate new ways of encountering and experiencing art in a museum's digital collection. The authors present the design and evaluation of an interactive application, SMKExplore, that enables users to browse and explore a museum's digital art collection by navigating through objects detected in the paintings.

Abstract

The paper presents a research through design exploration of using object detection to support exploration of a museum's digital art collection. The authors first describe their approach to applying object detection to a collection of 6,750 paintings from the National Gallery of Denmark, using a custom set of 120 object labels. They then present the design and development of SMKExplore, an interactive web application that allows users to browse the collection by navigating through the detected objects.
The evaluation of SMKExplore with 22 museum visitors revealed several key insights:

Representing the collection through detected objects enabled participants to notice recurring motifs and reflect on differences in artistic styles and depictions over time. Exploring the collection this way offered a new perspective compared to a traditional museum visit.

The categorization of objects and ability to filter by specific labels helped participants follow their personal interests and discover unexpected artworks.

Participants generally understood and were not bothered by instances of mislabeling, and instead found them interesting as they prompted reflection on the limitations of machine vision and their own interpretations.

The interactive canvas feature, which allowed participants to create new images using the detected objects, encouraged further exploration of the collection and reflection on compositional aspects of the artworks.

Overall, the study demonstrates the potential of object detection to create novel modes of engaging with and exploring digital art collections, while also highlighting important considerations around label selection, handling of errors, and fostering serendipitous discovery.

Stats

The object detection pipeline identified a total of 109,145 objects across 6,477 of the 6,750 paintings in the museum's digital collection.
After selecting a subset of up to 100 instances per label, the final dataset consisted of 10,775 detected objects on 3,906 paintings.

Quotes

"It was nice to see bikes from different paintings [...] I have never thought about looking at paintings and being like, oh, this is a bike here, and there is also a bike there."
"Wow, there are many of these objects I have never noticed in many of the artworks before."
"I think you just become more thoughtful of what actually is happening in a painting like this and what is present."

Key Insights Distilled From

Algorithmic Ways of Seeing

by Loui... at arxiv.org 03-29-2024

https://arxiv.org/pdf/2403.19174.pdf

Deeper Inquiries

How might object detection be combined with other AI techniques, such as generative models, to enable even more novel ways of engaging with and interpreting art collections?

Combining object detection with generative models can open up new possibilities for engaging with art collections. By using object detection to identify specific elements within artworks, such as objects, themes, or styles, generative models can then be employed to create new interpretations or compositions based on these identified elements. For example, after detecting objects in paintings, generative models like DALL·E can be used to generate new images that combine these objects in unique ways, offering a fresh perspective on the original artworks. This approach can enable users to explore art collections in a more interactive and creative manner, encouraging them to discover connections and patterns that may not be immediately apparent.

What ethical considerations should museums and HCI researchers keep in mind when deploying object detection systems, particularly around issues of bias and misrepresentation?

When deploying object detection systems in museums, it is crucial for both museums and HCI researchers to consider ethical implications related to bias and misrepresentation. Some key considerations include:

Bias in Data: Object detection models can inherit biases present in the training data, leading to inaccurate or unfair representations of certain objects or themes in artworks. It is essential to address and mitigate biases in the training data to ensure more accurate and inclusive results.
Cultural Sensitivity: Museums must be mindful of the cultural and historical context of artworks when applying object detection. Some objects or themes may be sensitive or controversial, requiring careful handling to avoid misrepresentation or offense.
Transparency and Accountability: Transparency in how object detection is used and the limitations of the technology should be communicated to users. Additionally, mechanisms for accountability and addressing errors or biases should be in place.
User Consent and Privacy: Users should be informed about the use of object detection in exploring art collections and have the option to opt-in or out of such experiences. Privacy concerns related to data collection and usage should also be addressed.
Continuous Evaluation and Improvement: Regular evaluation of object detection systems for biases and inaccuracies is essential. Continuous improvement and refinement of the models based on feedback and ethical considerations are crucial.

In what other domains beyond art museums could object detection be leveraged to facilitate exploratory search and serendipitous discovery of digital content?

Object detection can be applied in various domains beyond art museums to facilitate exploratory search and serendipitous discovery of digital content. Some potential domains include:

E-commerce: Object detection can help users discover products based on visual attributes, enabling a more intuitive shopping experience.
Archives and Libraries: Object detection can assist in categorizing and searching through vast collections of historical documents, photographs, and manuscripts.
Fashion and Design: Object detection can aid in identifying trends, styles, and patterns in fashion and design collections, offering inspiration for designers and consumers.
Healthcare: Object detection can be used in medical imaging to assist in the identification of anomalies or specific conditions in scans and X-rays.
Environmental Conservation: Object detection can help in monitoring wildlife populations, identifying species, and tracking environmental changes through image analysis.

By leveraging object detection in these domains, users can explore and discover digital content in a more visual and interactive manner, fostering serendipitous discoveries and enhancing the overall user experience.