Kernekoncepter
Comcast has developed an "Ask Me Anything" (AMA) feature that leverages large language models (LLMs) to provide real-time responses to customer service agents, improving their efficiency and effectiveness in handling customer inquiries.
Resumé
Comcast, like many other companies, provides customer service through various communication channels. While digital automation capabilities have been replacing human customer representatives for many tasks, there are still many situations that require human-to-human interactions.
To assist agents in these complex scenarios, Comcast has introduced the "Ask Me Anything" (AMA) feature. AMA allows agents to ask questions to a large language model (LLM) on demand, as they are handling customer conversations. The LLM provides accurate responses in real-time, reducing the amount of context switching the agent needs.
The key aspects of the AMA system are:
Document Preprocessing: Comcast standardizes documents into plain text, chunks them into snippets, and assigns unique identifiers to enable efficient retrieval.
Retrieving Relevant Text Snippets: Comcast experimented with various retrieval models, including sparse (BM25) and dense (DPR, MPNet-base) approaches. They found that OpenAI's ada-002 embeddings performed best on their evaluation set.
Reranking Search Results: Comcast used synthetic data generation and fine-tuning to improve the relevancy of search results, leading to better answer quality.
Generating Answers from Snippets: Comcast follows the conventional Retrieval-Augmented Generation (RAG) approach, where the LLM is prompted to generate answers using the retrieved snippets. They also implemented a "Citation Rail" to provide references for the answers.
Offline Response Evaluation: Comcast evaluated the system's responses using human-annotated ground truth answers and metrics like Answer Quality, Citation Match Rate, and Recall@K.
In internal experiments, Comcast found that agents using AMA versus a traditional search experience spent approximately 10% fewer seconds per conversation containing a search, translating to millions of dollars of savings annually. Agents also provided positive feedback nearly 80% of the time, demonstrating the usefulness of the AMA feature.
Comcast further conducted an A/B test of the reranker module, observing a statistically significant decrease in the "No Answer Rate" and an increase in the "Positive Feedback Rate" for the treatment group using the reranker.
Overall, Comcast's AMA feature showcases how large language models can be effectively integrated into customer service workflows to enhance agent productivity and customer satisfaction.
Statistik
The average handle time for conversations containing a search improved by approximately 10% when agents used the AMA feature versus the traditional search option.
Citater
"AMA allows agents to ask questions to a large language model (LLM) on demand, as they are handling customer conversations—the LLM provides accurate responses in real-time, reducing the amount of context switching the agent needs."
"In our internal experiments, we find that agents using AMA versus a traditional search experience spend approximately 10% fewer seconds per conversation containing a search, translating to millions of dollars of savings annually."
"Agents that used the AMA feature provided positive feedback nearly 80% of the time, demonstrating its usefulness as an AI-assisted feature for customer care."