Working minds

Preface

제목에 대해:

There are three senses to the title of this book, Working Minds. One is the notion that Cognitive task analysis is the study of cognition in real-world contexts and professional practice at work. A second is the sense that practitioners of CTA are themselves engaged in the work of studying the mind. The third is the sense of studying minds when they are engaged in successful accomplishment - when things “work.” Taken together, these three senses capture what this book is all about. —loc. 62

20년 역사:

We have been conducting applied studies (which today we call Cognitive task analysis) for about twenty years. … we have done CTA projects in more than a hundred distinct domains and have conducted over a thousand CTA interviews. —loc. 92

책의 목적:

In writing this book, we hope to demystify CTA by showing how it is done. … Also, at an emotional level, we want other researchers and practitioners to be able to enjoy and see value in this type of work. … We agreed from the start that the book would not be a survey of CTA methods, but instead would explain, in as much detail as we could provide, how to conduct a CTA study. … Another reason for writing this book is that we wanted to collaborate with each other. … Finally, another hope is that this is merely the first book of its kind. We do not pretend to have cast any “final words” in stone, or that everything we say here is exactly right.—loc. 109

Acknowledgments

Introduction

Chapter 1. Introduction

Henry Ford와 Nikola Tesla의 일화:

Tesla walked up to a wall of boilerplates, scanned them briefly, and then made an “X” in chalk on one of the plates. Examination of the boilerplate showed that it was indeed faulty. Ford was impressed, and told Tesla to send an invoice. The bill arrived, for $10,000. Ford, never known for his generosity, was astonished at the cost of writing an “X” on the boilerplate, and asked for a breakdown. Tesla sent another invoice, which read:

• Marking wall: $1
• Knowing where to mark: $9,999

This story speaks directly to the purposes and goals of this book in two respects. First, the story illustrates the “why” of Cognitive Task Analysis. What is it that Tesla knows, and how does he know it? What tells him what to do, with Henry Ford looking over his shoulder? Capturing that knowledge and reasoning is one of the things CTA can do. Second, the story illustrates “how” of CTA. CTA can be thought of as a set of tools in a toolkit. Like any tool, CTA can be employed well and wisely, or it can be employed poorly or inappropriately. What tool would you use if you wanted to understand how Tesla was able to grasp the nature of the problem so quickly?

—loc. 164

What CTA offers:

All CTA procedures have the general goal of helping researchers understand how cognition makes it possible for humans to get things done and then turning that understanding into aids—low or high tech—for helping people get things done better. —loc. 176

In all these cases, performance depends on what people know, what they perceive, what they believe, and how they think. —loc. 176

In writing this book, we hope to increase greatly the number of people with the skills and knowledge to conduct high-quality CTA. There are individuals and organizations with problems they cannot solve and opportunities they want to take advantage of. They need CTA tools and methods, and people who know how to apply them skillfully, across a range of problems and issues. —loc. 194

Unpacking CTA:

Cognitive: When the tasks that people are doing are complex, it is not enough to simply observe people’s actions and behaviors. It is also important to find out how they think and what they know, how they organize and structure information, and what they seek to understand better. This is a principal reason why the word “cognitive” begins the phrase CTA.

Task: … in complex cognitive systems, it is not always the literal action sequences that matter as much as the fact that practitioners are trying to get things done…. Therefore, we define task in this broader sense as the outcomes people are trying to achieve.

Analysis: We use the term “analysis” deliberately…. CTA methods provides procedures for systematic, scientific examination to support description and understanding. —loc. 194

Organization of the book:

Part I … provides detailed guidance for planning and carrying out CTA. It includes chapters on capturing knowledge and on capturing the way people reason. We rely on this distinction throughout the book: CTA investigates what people know and how they think.

Part II … provides a perspective on studying cognition in real-world settings and what an expanded view of cognition - a macrocognitive framework - offers. We describe some of the issues that surround CTA and what it means to study cognition in context. We end the section by exploring the challenges of rapidly changing technology.

Part III … describes key issues in applying CTA findings to several applications areas…. We also present a chapter on the role of CTA in the development of measures for evaluating cognitive work.

—loc. 275

Chapter 2. Overview of Cognitive Task Analysis Methods

Primary aspects of CTA:

The three primary aspects of CTA are knowledge elicitation, data analysis, and knowledge representation. Each of these aspects is critical to a successful CTA study. Many people equate CTA with the first aspect, eliciting the knowledge, because traditionally that has received the most attention. But if you don’t do a good job of analyzing your data, why bother collecting them? And if you don’t represent your findings so that others can understand them and why they matter, what have you accomplished? —loc. 288

Knowledge elicitation

Knowledge elicitation is the set of methods used to obtain information about what people know and how they know it: the judgments, strategies, knowledge, and skills that underlie performance. —loc. 288

Combination of CTA methods:

… in many CTA projects methods are used in combination. Using various tools and techniques in conjunction provides greater leverage and deeper insight. —loc. 576

Data analysis and knowledge representation:

The analysis phase of CTA is the process of structuring data, identifying findings, and discovering meaning. Knowledge representation includes the critical tasks of displaying data, presenting findings, and communicating meaning…. In much of the CTA literature, analysis and representation are inherently linked to knowledge elicitation and are not treated as separate processes at all…. However, many knowledge elicitation methods produce data that can be analyzed in many different ways and represented using a variety of formats….

• Capsulizing incidents
• Cataloguing cues and patterns
• Identifying themes
• Coding the data
• Describing cognitive sequences

Summary:

… there is not single right way to do CTA. Practitioners of CTA have a wide range of choices in the strategy to use in knowledge elicitation, data analysis, and knowledge representation. —loc. 719

Part I. Tools for Exploring Cognition in Context

Chapter 3. Preparation and Framing

시작하기:

… our advice is to resist the urge to jump into data collection. Experienced CTA researchers take the time to do initial groundwork…. In this chapter we suggest some key issues to think about and activities to carry out during the initial phase of a CTA project. —loc. 673

Framing the CTA project:

Framing the CTA project is the task of sharpening questions, focusing its goals, and identifying any constraints…. Let’s take a look at why framing might matter…. During the discussion, one of the nurses commented, “I’ve heard a good NICU nurse can tell 24 hours in advance of any lab test results that a baby is about to go bad” (Klein 1998). Other nurses in the room nodded…. Good nurses seem to have clear and accurate “intuitions” about infant illnesses. The key question became, “What is ‘nurses’ intuition?’ What does it look like, cognitively and behaviorally?” The study eventually centered on nurses’ perceptual judgments and assessment skills and how their expertise allows them to spot problems early. The frame for the project was about expert clinical assessment skills, but a study of critical care nurses could have been about many other issues. —loc. 693

CTA framing questions

Here are some suggestions:

• What issue or need do you plan to address?
• What sorts of people can tell you about this issue?
• What aspects of expertise or types of cognition do you need to know about?
• What type(s) of situation(s) will tell you the most about the issue you are exploring?

—loc. 700

Chapter 4. Using Concept Maps for Knowledge Elicitation and Representation

개요

In recent years, a technique called Concept mapping has been adopted by some CTA researchers as a method for both Knowledge elicitation and Knowledge representation. —loc. 925

Concept mapping

Background and research foundations

Concept mapping as CTA

What is concept map?

The concept mapping procedure

Team of group concept mapping

What makes for a good concept map in the CTA context?

Summary

CTA involves capturing what practitioners know about their domain: its concepts, principles, and events. We can think of no CTA process or project in which the CTA researchers did not have to elicit and then represent at least some domain knowledge. This chapter reviews the procedures and applications of concept mapping as a proven knowledge elicitation method for the efficient elicitation of practitioner knowledge. Concept maps involve labeled nodes and links, but concept maps differ in important ways from other types of diagrams that utilize combinations of graphical and textual elements to represent or express meanings. Concept mapping supports the practitioner’s effort to reach for crystal clarity about what he or she wishes to express. In concept-mapping knowledge elicitation, the researchers help the domain practitioner build up a representation of domain knowledge, in effect merging the activity of knowledge elicitation and the activity of knowledge representation. —loc. 1355

Chapter 5. Incident-Based CTA: Helping Practitioners “Tell Stories”

개요

One of the most powerful knowledge elicitation methods available to CTA practitioners s to probe actual incidents. People tell us about all kinds of details, challenges, subtle cues, background influences, and strategies that might never come to light in a general interview or a controlled simulation. Skilled decision makers have had many different experiences; that’s how they formed their knowledge and honed their skills. Their stories can be a doorway into that experience. —loc. 1285

What sorts of things can the CTA researcher find in stories?

• The cues and patterns that experts perceive
• The rules of thumb they have devised
• The kinds of decisions they have to make
• The features that make decisions tough
• The features that make cases typical
• The features of rare cases

The Critical decision method (CDM) Procedure

Boundaries and Limitations of the CDM:

The first condition that limits our ability to do a full CDM is when there simply are no real experts, or even highly skilled practitioners, to be found…. A second condition that can limit the usefulness of CDM is one in which participants are unable to generate useful incidents. Combat-like conditions, where people work under severely stressful conditions and handle very high workloads, can create a blur of events that are difficult to recall as discrete cases. —loc. 1631

Adaptations of CDM

CDM and Here-and-Now Incidents

CDM and Typical Incidents

… not all CTM methods that are incident-based rely on the study of critical incidents…. Sometimes CTA researchers want to understand how things usually or typically work. In other cases, concerns around memory issues may lead the researcher to go after very recent events, to make sure memory for details is fresh. —log. 1673

Variations on Use of a Timeline

Conducting CDM Over Multiple Sessions

The Knowledge Audit as Incident-Based CTA

The most thoroughly tested and validated adaptation of the CDM concept is the Knowledge Audit method (Hutton and Militello 1996; Hutton, Militello, and Miller 1997; Klein and Militello 2004; Militello and Hutton 1998). The Knowledge Audit examines the nature of the expertise needed to perform work skillfully. It structures an interview around a set of probes covering different aspects of expertise. —loc. 1696

Incident-Based CTA with Teams

Summary

In this chapter, we described one method for using incidents to extract cognitive elements - the Critical decision method. We described the procedures for conducting a CDM interview and offered an interviewer’s perspective on each of the CDM components. We examined the boundary conditions under which CDM is less likely to be effective, and we described some of the variations and adaptations that have developed to take advantage of the data collection opportunities that real, lived experience offers. —loc. 1727

Chapter 6. CTA Methods and Experiment-Like Tasks

개요:

Researchers have borrowed or adapted a number of methods from the psychology laboratory to examine cognitive work….

• In constrained processing (CP) methods, familiar tasks or routines are constrained in some way. The participant may be explicitly instructed to adopt a particular strategy, for example. Conversely, the participants may be confronted with a task that challenges their usual strategy.
• In limited information (LI) methods, participants are asked to solve problems given incomplete information.

—loc. 1748

Examples Example 1

In Hoffman’s (1987) study, experts were presented with aerial photos but were allowed only a two-minute inspection period. The experts then had to recall everything they could about the photos and provide their interpretation (e.g., “This region is an arid climate with shallow soils overlying tilted interbedded sandstone and limestone”). At first, the experts balked at the artificiality, but when encouraged to think of the task as a game rather than as a test or a challenge to their expertise, they found the task to be interesting. The task was based on one of the activities involved in the experts’ familiar task, but put a severe restriction on both time (constrained processing) and the amount of information available (limited information).

Results from this task revealed the extent to which the experts seem to achieve immediate perceptual understanding of terrain when viewing aerial photos. For instance, after inspecting the tropical imagery for two minutes one expert commented: “If you were to send troops there they would have to be protected from bacterial infections.” When asked how he knew that, the expert commented that he could tell from the ponds. The expert could see bacteria in a pond from forty thousand feet? No, but what the expert could see was flat interbedded limestone (in a homogeneous forest, the tree canopy informs about the terrain slope) in a tropical climate. Because the bedrock was flat-lying, the streams led into ponds having no major tributary or distributary for an outlet. Stagnant water in a tropical climate means leguminous water plants, implying that the waterways would be laden with bacteria. This all seems like a long inference chain in retrospect, but in the image inspection task, it seemed more a matter of immediate perception built upon a refined base of perceptual learning. —loc. 1791

Example 2

Think aloud problem solving

The TAPS Procedure

Questions about TAPS

Protocol analysis

Example 1: Coding for task procedures

Example 2: Coding for propositions for a model of knowledge

Example 3: Coding for leverage points

Coding verification

Effort considerations

Variations on the theme of experiment-like tasks

Experiment-like tasks may be useful, for example, in probing the specialized sub-domain knowledge or reasoning of experts (Hoffman 1987) or probing specific hypotheses about reasoning or strategies. They may be less useful if it seems impossible to compose a task that possesses ecological representativeness and validity for the participants who are domain practitioners, but also makes absolutely no sense to novices, apprentices, or journeymen….

We do not see this as an either-or choice. Cognitive Task Analysis conducted in a field setting can involve control and manipulation of variables, just as laboratory research can involve capturing the “real world.” However, the forms of control, selection, and manipulation of variables can be different in a CTA study than in a laboratory setting. Conversely, the ways in which the “real world” is captured in the laboratory result in significant differences from naturalistic studies. Fortunately, we see many ways to use experiment-like manipulations within CTA investigations, as described and illustrated in this chapter….

A combination of strengths would be to use what are perceived to be the more “natural” methods (interviews, observations, etc.) to construct a tentative macrocognitive model (of reasoning, cognition, knowledge, etc.), and then to use methods that are believed to involve more control, selection, and manipulation of variables to study the components of that model. —loc. 2020

Summary:

In this chapter we explored some intersections of laboratory methods and field research methods for conducting CTA. Researchers can tinker with aspects of a familiar task in a variety of ways, thereby eliciting experts’ strategies and reasoning, and do so in structured ways that provide empirical leverage. We also discussed think-aloud problem-solving and protocol methods and the use of analytic and representational formats that can reveal important aspects of cognitive process. —loc. 2020

Chapter 7. Analysis and Representation

앞단계에서 수집한 (주로) 비정형 정성 데이터를 분석하고 분석 결과를 정리하여 표현하는 다양한 방법들을 소개. 분석 및 표현을 제대로 못하면 앞단계에서 데이터를 아무리 많이/잘 수집해도 별 가치가 없음에도 불구하고 이 단계가 종종 간과된다고 지적.

정성데이터 분석 단계를 어떻게 하는지에 따라 프로젝트의 가치가 크게 좌우됨. 특히 전체 데이터를 여러번 검토하는 게 중요하다고. 한 번의 검토로 통찰을 충분히 뽑아내기는 불가능하다고(“it is simply not possible”) 단언. 또다른 중요한 요인은 분석가의 정성/정량 분석 역량.

Part II. Finding Cognition

Chapter 8. Thinking About Cognition

CTA를 잘하려면 인간의 인지 과정에 대해 잘 알아야 함. 특히 실제 상황 하에서의 인지(거시인지macrocognition)가 중요. NDM 학파가 실험실 연구보다 현장을 중시하는 이유.

몇가지 메모:

• 의도적 수련에 대한 연구(Ericsson, Krampe, and Tesch-Römer 1993)
• 전문가와 비전문가의 인지적 차이에 대한 연구(Klein and Militello 2004)
• 적응과 재계획replanning의 중요성. 항공 분야에서의 상당수 사고는 적시에 재계획을 하지 않았기 때문
• 재계획의 중요한 특징 중 하나는 대체로 목표에 대한 조정goal negotiation을 수반한다는 점. (애자일 방법론과 잘 연결되네요)

Chapter 9. Trends and Themes in the Development of Cognitive Task Analysis: The Rise of Modern Cognitive Psychology

9장의 주제는 CTA의 역사. 인지심리학, 인공지능 등 다양한 분야의 태생과 CTA가 어떻게 이어지는지 설명. 많은 이들이 CTA는 컴퓨터 관련 기술의 발전 및 컴퓨터로 인한 업무 환경의 급격한 변화로 인해 탄생한 것으로 여기지만 사실은 심리학과 더 관련이 깊음.

1950년대엔 AI(뉴럴넷 아님. 우리가 GOFAI라고 부르는 심볼릭 AI)와 인지심리학(촘스키 등의 인지혁명)이 발달. 그 밖에도 산업심리학의 작업분석, 인류학, HCI, 교육 설계 등 여러 분야가 CTA에 영향을 줌. (돈 노먼도 잠깐 나오는데 1980년대 HCI가 아니라 1960년대 인지심리에서 언급)

Chapter 10. Information Technology

IT가 삶의 모든 분야에 침투한 결과 다양한 인지적 도전과제가 생김. 인간이 기술과 어떻게 상호작용할지에 대해 많은 디자인 결정이 필요. 인간이 기술을 어떻게 사용할지, 기술의 역할은 무엇인지, 기능functions을 인간과 기술 사이에 어떻게 분배할지 등.

Envisioned world problem: IT 기술로 기존의 작업 방식을 개선하면, 그로 인해 IT 기술을 설계할 당시엔 생각하지 못했던 방향으로 작업 방식이 바뀌는 현상. 기존 방식을 관찰→IT를 도입→IT로 인해 업무 방식이 바뀜→새 업무 방식이 도입한 IT 기술이랑 잘 안 맞음. 이런 문제.

Part III. Putting CTA Findings to Use

Chapter 11. The Role of Cognitive Requirements in System Development

“These four features were coded into a training simulator at Brooks Air Force Base.”

“Brooks Air Force Base”는 어디선가 들어봤다 싶었는데, 피터 싱어의 동물 해방에서 언급된 잔인한 영장류 실험(PEP)을 했던 그곳. 원숭이들에게 고문을 가하며 모의비행기 조정법을 훈련키고 훈련이 끝나면 본격 실험(?)이 시작된다. 실험이란 원숭이들을 지속적인 방사능에 노출시키며 얼마나 오랜시간 모의기체의 평형을 유지하다가 죽는지 측정하는 것. 평형유지에 실패하면 전기충격이 가해지도록 훈련를 받아온 원숭이들은 죽어가면서도 평형을 유지하려고 애를 쓴다고.

(다시 본문 내용으로 돌아와서) 인지시스템공학(CSE; cognitive systems engineering)이라는 분야를 소개. 이 분야의 주요 방법 중 하나인 의사결정중심디자인(DCD; decision-centered design)이 자세히 설명한다고 함.

DCD에서는 의사결정요구사항표(DRT; decision requirements table)을 만드는데, 어떤 의사결정이 필요한지, 그게 왜 어려운지, 이에 대한 HCI 솔루션은 무엇인지를 정리.

5장에서 소개한 CDM의 결과로 어려운 의사결정의 목록, 왜 어려운지, 전문가들이 이런 상황에서 어떻게 의사결정을 내리는지 등의 정보가 수집되면, DCD에서는 이를 바탕으로 “기술을 활용하여 의사결정에 필요한 인지적 노력을 보조할 방법”을 찾는 식.

기존 디자인 방법과의 차이는 ‘과업목록과 데이터흐름’이 아니라 ‘어려운 사례’에서 시작한다는 점. 기존 방식의 디자인은 일상적이고 쉬운 과업을 자동화해줄 수 있지만 예외적 상황이나 어려운 상황에선 잘 작동하지 않음. 어설픈clumsy 자동화의 사례.

‘어려운 의사결정’은 기존 방식을 디지털로 전환한 후에도 대체로 유지되는 경향이 있다는 점에서 앞서 언급한 Envisioned world problem에 대한 좋은 해결책이기도 함.

Chapter 12. Cognitive Training

인지 기능에 대한 훈련(예: 올바른 멘탈모델 형성, 주요한 차이를 잘 지각하기, 주의력 관리하기)이 필요할 때 CTA를 활용하는 법. 특히 인지적으로 복잡한 과업이라면 교육 설계도 어려움. (Ten-steps to complex learning이 떠오르는 대목이네요)

CTA는 다음과 같은 도움을 줄 수 있음:

1. 인지적 훈련의 요구사항 분석하기
2. 훈련 시나리오 설계하기
3. 인지적 피드백 제공하기
4. 직무중훈련OJT 설계하기

Chapter 13. Understanding How Consumers Make Decisions: Using Cognitive Task Analysis for Market Research

마켓 리서치에 CTA를 활용하는 방법을 설명. 소비자의 구매 결정 과정, 사용 중인 제품을 바꾸기까지의 인지 과정, 제품에 대한 만족도를 어떻게 정의하는지, 제품을 신뢰한다는 것은 무슨 의미인지 등. 전통적 시장 분석은 소비자의 태도/믿음/선호를 지나치기 강조파며 인지적 분석을 간과.

Chapter 14. Cognitive Task Analysis for Measurement and Evaluation

CTA를 이용하여 인지적 숙련도를 측정하고 평가하는 방법을 다룸. 재미있는 점은 CTA에서 사용하는 방법들 자체를 평가하는 방법도 다룬다는 점. 저자들은 ‘측정’을 말할 때 기본적으로 ‘비교’를 염두에 둔다고. 개인 간 비교, 집단 간 비교, 어떤 표준(예: 알고리즘)과의 비교 등.

과정에 대한 평가(process evaluation)와 결과에 대한 평가(outcome evaluation)의 구분. 후자는 개선의 효과를 설명하고 전자는 그 이유를 설명. 과정 평가는 특히 훈련 과정에서 유익함. 한편, 정량화만으로는 놓칠 수 있는 부분들이 있으므로 정성 평가를 병행할 것을 권함.

Chapter 15. Future Directions for Cognitive Task Analysis

CTA의 미래. 지식, 전문성, 인지적 스킬을 ‘자원resource’으로 간주하면, CTA의 연구 주제를 자원 관리 관점에서 이해 가능. 광산업과 유사. CTA 방법들의 확장(자원을 더 잘 발굴), 응용분야의 확장(자원을 더 유용하게 활용), 관계자들의 전문성 향상을 돕기.

마지막 문단 요약:

• 독자들이 이 책을 통해 일부 CTA 방법들을 익혔기를 기대함.
• 우리가 가진 영업 비밀을 다 털어놓았고 숨은 비결 같은 건 없음.
• CTA를 잘 하기 위해 해야 할 남은 일은 오로지 열일과 연습 뿐임.
• 다른 모든 복잡한 스킬과 마찬가지.

끝.

Appendix: Guidance for Data Collection