Critical decision method

Critical decision method (CDM). 전문가가 실제로 겪었던 사건/사고에 대한 스토리텔링으로부터 전문성을 뽑아내는 체계적인 방법. 주로 Working minds 5장에서 인용.

History

Development of CDM:

We have developed the CDM to learn from specific incidents (Hoffman, Crandall, and Shadbolt 1998; Klein, Calderwood, and MacGregor 1989). Many CTA researchers use the CDM for conducting incident-based interviews (Blandford and Wong 2004; Ebright et al. 2003; Klein and Armstrong 2004; Militello and Crandall 1999; Omodei, Wearing, and McLennan 1998; Readinger, Roos, and Crandall 2004; Thordsen 1991; Wong, Sallis, and O’Hare 1997).

Procedure

Overview

The CDM is an intensive interview that often takes as long as two hours. In some cases such as weather forecasting where incident memories can be very rich, the CDM steps can even be broken up and conducted over several sessions. The CDM interview is conducted by two researchers. One interviewer acts as the primary facilitator, but also takes notes. The second interviewer is primarily responsible for taking a good set of notes and keeping track of the overall plan for interview…. The main portion of the interview is carried out by making several “sweeps” through an incident. Each sweep constitutes a pass through the incident and builds on the previous sweep(s). Each is focused on eliciting specific types of information.

Sweep 1. Selecting an Incident

The initial CDM step is focused on identifying candidate incidents and selecting an appropriate incident for deepening…. Generally, in this sweep you want to identify an incident that will contain cognitive components that go beyond background and routine procedural knowledge of the domain and that will enable you to learn about those components that characterize skilled performance and expertise.

CDM에 적합한 사건/사고 고르기:

당사자가 해당 사건에서 의사결정을 하는 위치에 있었는지 확인하기
“사건/사고에 대해 생각해오세요”라고 미리 알려주면 준비하는 과정에서 오히려 기억이 왜곡될 수 있음.
“일반적이지 않았던 사례”를 지나치게 강조하면 정말로 ‘이상한 이야기’만 듣고 끝나는 경우가 있음.
서두르거나 재촉하지 말기.

Sweep 2. Constructing a Timeline

The second sweep is aimed at getting a clear, refined, and verified overview of the incident structure, identifying key events and segments…. In diagramming the timeline, the critical points (sometimes called “decision points”) are when the practitioner experienced a major shift in his or her understanding of the situation or took some action that affected the events…. As the sequence and duration of events, actions, perceptions, thoughts, and decisions emerge, the interviewers and participant arrive at a shared view of the facts of the case from the participant’s perspective. Working through the incident in this way, interviewers are able to clear up inconsistencies, identify gaps, and fill in missing elements. Skilled interviewers are also able to begin tagging key segments and decision points to probe later on.

Sweep 3. Deepening

This sweep is the most challenging, but also the most fun part of the interview. This is where you have the opportunity to get inside the expert’s head and look at world through his or her eyes. From the interviewer’s perspective, the guiding question is: “What is the story behind this story? Based on the first two sweeps, I know what happened, who did what, and I know a bit about their role in the event. But what did they know, when did they know it, how did they know, and what did they do with what they knew?” … This sweep of the incident should yield a portrait of the participant’s cognitive experience, skills, and knowledge. At the end of sweep 3, you should have a detailed, specific, and fairly complete picture of each segment of the event and of the overall incident.

Probe questions:

Cues: What were you seeing, hearing, smelling, noticing etc.?
Information: What information did you use in marking this decision or judgment? How and where did you get this information, and from whom? What did you do with the information?
Analogs: Were you reminded of any previous experience? What about that previous experience seemed relevant for this case?
Standard operating procedures: Does this case fit a standard or typical scenario? Is it a type of event you were trained to deal with?
Goals and priorities: What were your specific goals and objectives at the time? What was most important to accomplish at this point in the incident?
Options: What other courses of action were considered or were available to you? How was this option chosen or others rejected? Was there a rule that you were following in choosing this option?
Experience: What specific training or experience was necessary or helpful in making this decision?
Assessment: Suppose you were asked to describe the situation to someone else at this point. How would you summarize the situation?
Mental models: Did you imagine the possible consequences of this actions? Did you create some sort of picture in your head? Did you imagine the events and how they would unfold?
Decision making: What let you know that this was the right thing to do at this point in the incident? How much time pressure was involved in making this decision? How long did it take to actually make this decision?
Guidance: Did you seek any guidance at this point in the incident? How did you know to trust the guidance you got?

언제 어떤 probe question을 던질 것인가?

If a critical point on the timeline involves making an observation, the probes about information and cues are useful.
If a critical point involves assessing or making sense of a situation or projecting a situation into the future, then probes about assessment and mental models are useful.
If a critical point on the timeline involves making a decision, questions should be about decisions, obviously, but also about goals and options.
At points where the story seems to refer to the participant’s knowledge, then probes about the basis of choice and about experience are useful.

Sweep 4. “What If” Queries

The final sweep of the CDM interview provides an opportunity to round out the interviewer’s insight into the participant’s experience, skill, and knowledge. Once again using the incident as a starting point, the interviewer poses various hypotheticals about the incident.

Probes:

Expert-novice contrasts: If a novice had been in charge at this particular point in the incident, what type of error might she or he have made and why? Would they have noticed what you noticed? Would they have known to do X?
Hypotheticals: If key feature of the situation had been different, what impact would it have had on your decision/assessment/actions/plan?
Experience: What training might have offered an advantage in this situation?
Aids: What knowledge, information, or tools/technologies could have helped?