A Taxonomy of Interaction Modes for Enhancing Human-Large Language Model Collaboration

Core Concepts

This paper presents a taxonomy that categorizes the various interaction modes between humans and large language models (LLMs), aiming to empower users to tackle complex tasks by utilizing LLMs beyond the default conversational prompting paradigm.

Abstract

The paper presents a systematic review of existing literature in HCI venues published since 2021, which led to the identification of four key phases in human-LLM interaction flows: planning, facilitating, iterating, and testing. Additionally, the research introduces a detailed, structured taxonomy that encapsulates four primary interaction modes between humans and LLMs:

Standard Prompting:

Mode 1.1 Text-based Conversational Prompting
Mode 1.2 Text-based Conversational Prompting with Reasoning

User Interface (UI):

Mode 2.1 UI for Structured Prompts Input
Mode 2.2 UI for Varying Output
Mode 2.3 UI for Iteration of Interaction
Mode 2.4 UI for Testing of Interaction
Mode 2.5 UI for Reasoning

Context-based:

Mode 3.1 Explicit Context
Mode 3.2 Implicit Context

Agent Facilitator:

Mode 4.1 Team Process Facilitating
Mode 4.2 Capability-aware Task Delegation

The taxonomy provides a valuable tool for systematically understanding and analyzing the evolving landscape of human-LLM interaction and collaboration, guiding the design of human engagement with LLMs in increasingly complex and nuanced ways.

Stats

None.

Quotes

None.

Key Insights Distilled From

A Taxonomy for Human-LLM Interaction Modes

by Jie Gao,Simr... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2404.00405.pdf

Deeper Inquiries

How can this taxonomy be extended to include different types of tasks and design spaces beyond the current focus on writing and coding?

To extend the taxonomy to encompass different types of tasks and design spaces beyond writing and coding, we can consider the following approaches:

Task-specific Interaction Modes: Identify and categorize interaction modes tailored to specific tasks such as image generation, video editing, data analysis, or music composition. Each task may require unique prompts, interfaces, and collaboration methods with LLMs.

Domain-specific Taxonomy: Develop subcategories within the taxonomy that cater to diverse domains like healthcare, finance, education, or art. Each domain may have specific requirements and constraints that influence the interaction modes between humans and LLMs.

Multimodal Interaction: Include interaction modes that combine text-based prompts with other modalities like voice commands, gestures, or visual inputs. This extension can accommodate tasks that rely on multiple forms of input for LLMs to generate outputs effectively.

Collaborative Design Spaces: Explore interaction modes that facilitate collaborative design processes, where multiple users work together with LLMs in real-time. This extension can enhance teamwork, creativity, and decision-making in various design spaces beyond individual tasks.

By incorporating these extensions, the taxonomy can provide a comprehensive framework for designing human-LLM interactions across a wide range of tasks and design spaces, catering to the evolving applications of LLMs in diverse domains.

How can the taxonomy be applied to emerging applications of LLMs, such as image and video generation, to enhance the design of human-LLM interactions in those domains?

In the context of emerging applications like image and video generation, the taxonomy can be applied to enhance the design of human-LLM interactions in the following ways:

Task-specific Interaction Modes: Develop interaction modes tailored to image and video generation tasks, focusing on prompts that elicit specific visual outputs from LLMs. This can include structured prompts for image composition, style transfer, or video editing.

UI Design for Visual Outputs: Integrate UI elements that allow users to manipulate visual parameters such as color, texture, layout, and style in image and video generation processes. The taxonomy can include modes that emphasize visual feedback and control for users.

Iterative Testing Interfaces: Design UIs that enable users to test and refine multiple variations of image and video prompts, facilitating rapid prototyping and experimentation. Interaction modes for testing different visual outputs can enhance creativity and exploration in design processes.

Contextual Understanding: Incorporate interaction modes that leverage contextual information in image and video generation tasks. This can involve explicit context-based prompts or implicit context recognition to guide LLMs in producing relevant visual content.

By applying the taxonomy to these emerging applications, designers can create more effective and user-friendly interfaces for human-LLM interactions in image and video generation, fostering creativity and efficiency in visual content creation.

What are the potential ethical considerations and implications of the various interaction modes identified in the taxonomy, particularly in terms of transparency, accountability, and user agency?

Transparency: Ensuring transparency in interaction modes is crucial to build user trust. Designers should disclose how LLMs process user data, generate outputs, and make decisions based on prompts. Transparent prompts and interfaces can help users understand the system's behavior.

Accountability: Establishing accountability mechanisms is essential to address errors or biases in LLM outputs. Interaction modes should include features for error correction, feedback mechanisms, and clear attribution of responsibilities between users and LLMs.

User Agency: Upholding user agency involves empowering users to control the interaction process. Interaction modes should allow users to modify prompts, provide feedback on outputs, and make informed decisions about the generated content. Respecting user preferences and choices is key to preserving agency.

Bias Mitigation: Interaction modes should incorporate strategies to mitigate biases in LLM outputs, such as diverse training data, bias detection tools, and fairness assessments. Designers should proactively address bias issues to ensure fair and inclusive interactions.

Data Privacy: Interaction modes should prioritize data privacy by limiting data collection, securing user information, and obtaining explicit consent for data usage. Respecting user privacy rights is essential for ethical human-LLM interactions.

By considering these ethical considerations and implications in the design of interaction modes, designers can promote responsible and ethical use of LLMs, fostering a positive user experience and societal impact.