Automating Tabletop Exercises: Transitioning from Paper-based to Platform-based Delivery and Evaluation

Core Concepts

Tabletop exercises are an innovative teaching method for training teams in incident response and evaluating contingency plans, but they are not yet widely established in university education. This paper presents the development and deployment of a novel web-based learning environment, INJECT Exercise Platform (IXP), that substantially improves the prior practice of conducting tabletop exercises using pen and paper.

Abstract

This paper presents the evolution of a novel learning environment for delivering and evaluating tabletop exercises (TTXs) in computing education. The authors developed a web-based platform called INJECT Exercise Platform (IXP) to automate the repetitive tasks involved in hosting TTXs, which were previously conducted using pen and paper or simple online office applications.
The paper describes the key features of the proposed TTX format, including participant roles, exercise components (injects, tools, and milestones), and the exercise workflow. It then introduces the IXP platform, which allows designers to instantiate exercise definitions and instructors to deploy them for trainees.
The authors conducted three runs of a TTX on cybersecurity incident response using different versions of the IXP platform. The data and teaching experience from these runs are analyzed and compared. The automated data collection and analysis capabilities of IXP provided valuable insights into the trainees' behavior and learning that would be difficult to obtain in the traditional pen-and-paper format.
The results show that the IXP platform significantly reduces the workload and personnel requirements for TTX delivery, while also enabling detailed performance assessment and feedback for the trainees. The instructors' teaching experience also improved in terms of reliability and involvement of the trainees. The paper shares practical lessons learned from using the platform and provides open-source artifacts for the community.

Stats

The teams reached between 5 and 12 milestones out of 14 defined in the TTX, with an average of 10 (71%) in Run 2.
In Run 3, the teams reached between 4 and 11 milestones out of the 22 defined, with an average of 8 (57%).
A team used the available tools between 10 and 46 times throughout the TTX in Run 3, with 31 uses on average.
The teams engaged in 6 email threads on average during the TTX in Run 3.

Quotes

"At the beginning, we were quite lost. There is just so much difference between having a specified incident handling task and having to figure out everything by yourself."
"Instructors were briefed to assist by sending exercise emails to guide these teams. However, this approach proved challenging as instructors struggled to identify the right moments for intervention."

Key Insights Distilled From

From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises

by Vald... at arxiv.org 04-18-2024

https://arxiv.org/pdf/2404.10988.pdf

Deeper Inquiries

How can the platform's data analytics be further leveraged to provide real-time guidance and intervention for teams struggling during the tabletop exercise?

The platform's data analytics can be enhanced to offer real-time guidance and intervention by implementing machine learning algorithms to analyze team performance and behavior patterns during the exercise. By continuously monitoring the teams' interactions, tool usage, milestone achievements, and communication patterns, the platform can identify signs of struggle or inefficiency. For instance, if a team is taking longer than usual to reach a milestone or repeatedly using tools incorrectly, the system can flag these issues for instructors to intervene promptly.
Moreover, the platform can incorporate natural language processing to analyze the content of team communications and inject responses. By detecting keywords or sentiments indicative of confusion or uncertainty, the system can suggest tailored hints or guidance to steer the team in the right direction. This real-time feedback mechanism can help teams course-correct during the exercise, ensuring a more effective learning experience.

What are the potential limitations or drawbacks of automating tabletop exercises, and how can they be addressed to maintain the pedagogical value of these hands-on learning experiences?

While automating tabletop exercises offers numerous benefits, such as efficiency, scalability, and data analytics capabilities, there are potential limitations and drawbacks that need to be addressed to maintain the pedagogical value of these hands-on learning experiences.
One limitation is the risk of reducing the human interaction and spontaneity that are inherent in traditional tabletop exercises. To address this, the platform can incorporate features that promote collaboration and discussion among team members, such as virtual breakout rooms for private team discussions or real-time chat functionalities for instant communication.
Another drawback is the potential for technical issues or platform failures to disrupt the exercise flow. To mitigate this risk, robust testing and quality assurance processes should be implemented to ensure the platform's reliability. Additionally, providing clear instructions and technical support resources for both instructors and trainees can help address any issues that may arise during the exercise.
Furthermore, automating tabletop exercises may lead to a more structured and controlled environment, potentially limiting the creativity and adaptability of the exercise. To counter this, the platform can offer customizable scenarios, random events, or branching paths to introduce variability and challenge participants to think critically and creatively.

How can the tabletop exercise format and the IXP platform be adapted to support the development of other critical skills, such as leadership, communication, and decision-making, beyond just technical incident response capabilities?

To support the development of critical skills like leadership, communication, and decision-making, the tabletop exercise format and the IXP platform can be adapted in the following ways:

Role Assignment: Introduce diverse roles within the teams, such as team leader, communicator, and decision-maker, to encourage leadership development. Rotate these roles in each exercise to provide opportunities for different team members to practice leadership skills.

Scenario Complexity: Design scenarios that require effective communication and decision-making under pressure. Include elements like time constraints, conflicting priorities, and limited resources to simulate real-world challenges that test these skills.

Feedback Mechanisms: Implement feedback mechanisms within the platform that provide insights into team communication dynamics, leadership effectiveness, and decision-making processes. This feedback can help participants reflect on their performance and identify areas for improvement.

Debriefing Sessions: Conduct post-exercise debriefing sessions where instructors can facilitate discussions on leadership strategies, communication effectiveness, and decision-making processes observed during the exercise. Encourage participants to share their experiences and insights to enhance learning.

Scalability: Ensure the platform can accommodate larger teams or multiple teams working simultaneously to simulate complex organizational structures and dynamics. This scalability allows for more extensive collaboration and communication challenges to be addressed during the exercise.

By incorporating these adaptations, the tabletop exercise format and the IXP platform can serve as versatile tools for developing a wide range of critical skills beyond technical incident response, fostering holistic professional development in students.