History of Deceptive Patterns

I tried organizing the history of deceptive patterns and the related discussions. Deceptive patterns, or dark patterns, refer to “interfaces meticulously designed with the purpose of harming users’ autonomy, decision-making, and choice.” I’ve been interested in this field for a long time, so quite a few things come to mind.

Dark Patterns? Deceptive Patterns?

The term first proposed in 2010 was originally “dark patterns,” but following criticism that using “dark” with a negative meaning is not inclusive language, alternatives such as “deceptive patterns” and “damaging patterns” were later proposed. Harry Brignull, who first proposed the term dark pattern, recommends calling them deceptive patterns instead.

However, because the word “deceptive” is sometimes used in a special legal sense (for example, the U.S. Federal Trade Commission strictly distinguishes deception from manipulation), when there is a need to speak more precisely depending on the context, people sometimes write “deceptive and/or manipulative patterns.”

In this article, following Brignull’s suggestion, I will use “deceptive patterns.”

Definition

Deceptive patterns were first proposed by Harry Brignull in 2010 when he wrote Dark Patterns: dirty tricks designers use to make people do stuff.

Normally we think of bad design as the result of laziness, mistakes, or inexperience. We call these kinds of design patterns antipatterns. But there is another kind of bad design pattern as well. There are designs that are intentionally crafted with meticulous attention to detail, based on a deep understanding of human psychology, for the purpose of deceiving users into doing things they do not want. This is the dark side of design, and since there is not yet a term for this kind of pattern, I propose calling them dark patterns.

Normally we think of bad design as consisting of laziness, mistakes, or school-boy errors. We refer to these sorts of design patterns as Antipatterns. However, there’s another kind of bad design pattern, one that’s been crafted with great attention to detail, and a solid understanding of human psychology, to trick users into do things they wouldn’t otherwise have done. This is the dark side of design, and since these kind of design patterns don’t have a name, I’m proposing we start calling them Dark Patterns.

Design patterns are collections of reusable solutions that help with good design. The concept began with pattern language, proposed by architect Christopher Alexander, and was originally used in urban and architectural design before expanding into software design and interface design.

Antipatterns are the opposite of design patterns, referring to collections of bad solutions that recur frequently.

If antipatterns are the result of laziness, mistakes, or inexperience, deceptive patterns are distinguished from antipatterns by being intentional designs.

Spread

After Brignull’s first proposal in 2010, an official site was created collecting various examples, legal cases, and related writings, and in 2023 the book of the same name, Deceptive Patterns: Exposing the Tricks Tech Companies Use to Control You, was published, gathering the discussions that had accumulated up to that point.

History

Current discussions of deceptive patterns, including the patterns listed on the website, tend to focus mainly on the surface visible to users—that is, the UI. But unethical design also hides in the invisible back end, such as recommendation algorithms, and it also exists in subtler forms, such as social engineering methods that make use of other people connected through social networks.

Let’s look a bit more broadly at the related discussions and history.

1974. The dawn of human-computer interaction

In 1974, a small group called the Applied Information Processing Psychology Project (AIP) was formed at Xerox Palo Alto Research Center. The group’s achievements later became widely known through the 1983 book The Psychology of Human-Computer Interaction, which would come to be called the bible of the HCI field.

The group’s primary concern was devising ways for humans and computers to interact so that, in situations where a task to be processed had been given, that task could be completed as accurately and efficiently as possible. It is true that they had enormous interest in psychology, but topics such as human motivation, desire, emotion, and persuasion were outside the scope. That is because HCI researchers at the time were grounded in information processing theory, the perspective that regards the human brain as an information-processing machine.

… issues of motivation and personality are outside the scope. There is hardly any doubt that these topics ought to be included in an applied psychology, but it is unclear how the relevant existing knowledge should be well integrated.

… motivational and personality issues are not included. Again, there is hardly any doubt of the desirability of including them in an applied psychology, but it is unclear how to integrate the relevant existing knowledge of these topics.

—Chapter 1, The psychology of human-computer interaction

For this reason, there does not seem to have been any discussion related to deceptive patterns in the formative period of HCI.

HCI research that departed from traditional information processing theory did not seriously begin until the 1990s. For example, research on the aesthetic-usability effect was published in 1995, and Donald Norman’s Emotional Design was published in 2004.

1997. Captology

If previous design had been design that helps users do what they want to do well, then we can also think about design that induces and persuades users to do what the designer wants.

Discussion of this kind of design began in earnest at the CHI 1997 Captology SIG (Special Interest Group), convened at the suggestion of B. J. Fogg. This was a perspective that departed from the traditional information processing theory mentioned earlier.

Captology, as proposed by Fogg, is short for “Computer as a Persuasive Technology.”

An understanding of captology would not only enrich HCI theory about how humans interact with computers, but it would also lead to better design of interactive technologies, especially those that have the task of persuading users to change in beneficial ways.

An understanding of captology would not only enrich HCI theory about how humans interact with computers, but it would also lead to better design of interactive technologies, especially those that have the task of persuading users to change in beneficial ways.

According to this proposal, it seems Fogg expected captology to be used with good intentions.

The following year, in 1998, he published a paper titled Persuasive Computers: Perspectives and Research Directions. He analyzed computers as serving three functions—tool, media, and social actor—and named this the “Functional Triad.” Each of these three aspects has distinct persuasive affordances.

Persuasive affordances of the tool aspect: reducing barriers such as time/effort/cost, increasing self-efficacy, providing information for better decision-making, changing mental models
Persuasive affordances of the media aspect: providing direct learning/insight/visualization, helping understanding of causal relationships, motivating through experience and sensation
Persuasive affordances of the social actor aspect: establishing social norms, reminding people of social norms, providing social support or sanctions

Examples of the “good” uses of captology that Fogg expected were as follows.

Individual: educating people so they engage in more responsible sex (contraception, etc.)
Family: creating games that families can enjoy together so family members become more harmonious
Organization: developing remote work systems to increase employee satisfaction and help them stay at their jobs longer
Community: encouraging carpooling to ease traffic problems and reduce air pollution

Of course, it is not that Fogg naively thought only about the positive functions of captology.

… those who study persuasive technologies should have a sound understanding of the ethical implications of this field. A subfield on the ethics of persuasive computing is a worthy and important endeavor.

… those who study persuasive technologies should have a sound understanding of the ethical implications of this field. A subfield on the ethics of persuasive computing is a worthy and important endeavor.

He was already clearly aware of the potential for abuse, such as combining computers with slot machines to provide a more addictive gambling experience, or using computers to monitor employees’ web activity.

For reference, concerns about slot machines were not a “prediction” even by the standards of the time; they had already been realized long before. On the casino industry’s long effort to induce behavioral addiction, I strongly recommend Natasha Dow Schüll’s 2012 book Addiction by Design, which introduces the topic with incredible thoroughness. The gambling industry is the front line of deceptive patterns and addictive design.

In 1998, he founded the Stanford Persuasive Technology Lab and continued his research.

2002. “Using computers to change what we think and do”

In 2002, Fogg published Persuasive Technology: Using Computers to Change What We Think and Do. This book also makes very clear that he was distinctly aware of the possibility of unethical uses of captology.

Chapter 9, “The Ethics of Persuasive Technology,” deals with this issue intensively. Chapter 9 begins by listing hypothetical misuse cases, and all of the examples below became reality.

Forcibly collecting personal information incrementally through games played by children (corresponding to the deceptive pattern Forced Action)
Steering people toward buying a more expensive product than the one they originally intended to buy, while hiding necessary information (corresponding to deceptive patterns such as Nagging and Comparison Prevention)
Subtly hiding information and inducing people to make risky investments (corresponding to the deceptive pattern Trick Wording)

Professor Philip Zimbardo, who wrote the book’s endorsement, also repeatedly mentions concern about the possibility of misuse. (Fogg had been Zimbardo’s teaching assistant.)

(If we do not pay attention to captology,) we may be slow to respond to the problem of unethical actors using this technology for profit or political purposes. … If experts and ordinary citizens alike understand that computers can be used in ways that take away our power to decide matters affecting our health and well-being, then we will be able not only to use this technology for good, but also to remain vigilant against its evil side.

We would be slow to take necessary steps against unwanted persuasion by unethical agencies using this new technology for profit or political advantage. … If all of us, professionals and citizens alike, comprehend how computers can be used to persuade us to take control over the decisions affecting our health and well being, we can harness this power for the good it can impart, while sounding alarms against the dark side of this force of persuasive influence.

Unfortunately, all of those concerns also became reality.

(For reference, Professor Zimbardo’s Stanford prison experiment has not only long been criticized for the unethicality of its design, but it was also later revealed that Zimbardo himself actively intervened in and manipulated the experiment.)

2007. The Facebook platform and Mass Interpersonal Persuasion

In 2007, Facebook opened its platform to third-party developers. At the time, much like today’s app stores, users could register apps or games they had made on Facebook.

Immediately after Facebook opened the platform, Fogg gave a course on developing Facebook apps using captology, and the apps his students made achieved remarkable results, such as gaining 16 million users in just 10 weeks. Since Facebook had about 60 million users at the time, that meant they captured about 25% of all users. This was an early example showing the destructive power of persuasive technology when it met social media.

Fogg documented this case and published a paper in 2008 titled Mass Interpersonal Persuasion: An Early View of a New Phenomenon.

The six elements of MIP that he presented are still valid.

Persuasive experience: the existence of an experience created with the purpose of changing attitudes or behavior
Automated structure: the existence of (automated) digital technology that builds the persuasive experience
Social distribution: the persuasive experience will be shared/spread through acquaintance relationships
Rapid cycle: the persuasive experience can spread quickly through acquaintances
Huge social graph: the existence of a social graph that allows reaching millions of people through social relationships and structured interactions
Measured impact: the results of the persuasive experience can be observed by other users and creators

These principles began to be applied early on to various casual social games. This was also the period when companies started appearing that churned out games like factories—games whose core mechanics were identical, with only their outward appearance changed. Infamous(?) Zynga was also founded in 2007. The six mechanisms of MIP proposed by Fogg began to be applied not only to casual games but also to apps and services using gamification, and they are being actively used in a variety of fields including growth hacking.

At the time, Fogg rather naively predicted that MIP would lead to a de-centralization of persuasive power.

Over the past century, mass media has been the primary channel for persuasion. … They used mass media largely to achieve their own goals. Now, the landscape is changing. I believe the power to persuade will continue to become less centralized, thanks to MIP. For early evidence of decentralization, we can see how much impact ordinary individuals have had with blogs and online videos.

Over the past century, mass media has been the primary channel for persuasion. … They used mass media largely to achieve their own goals. Now, the landscape is changing. I believe the power to persuade will continue to become less centralized, thanks to MIP. For early evidence of decentralization, we can see how much impact ordinary individuals have had with blogs and online videos.

But this prediction was completely wrong. Looking back, this outlook rested on one mistaken assumption. The problem was the implicit assumption that social media platforms themselves are neutral spaces and that platform providers would not make use of MIP. In the 21st century, social media platforms are taking power away from 20th century mass media, but it is hard to say that things have become de-centralized.

For example, Meta, which owns Facebook, Instagram, and Threads, has been actively abusing centralized persuasive power for 20 years.

2007. Persuasion using mobile devices and the iPhone

In January 2007, Fogg published the book Mobile Persuasion, which proposes methods of just-in-time intervention using mobile devices. This book is the result of refining and expanding on material from Chapter 8 of his 2002 book (Persuasive Technology), “Increasing Persuasion through Mobility and Connectivity.”

Coincidentally, at about the same time (January 9, 2007), Steve Jobs unveiled the iPhone, and this became the turning point for smartphones to spread rapidly among the public.

As a result, the three elements Fogg had dreamed of were finally all in place.

Popularization of personal computers: persuasive technology using computers (Captology)
Social network platforms such as Facebook: Mass Interpersonal Persuasion using social networks
Popularization of smartphones: just-in-time intervention using mobile devices (Mobile Persuasion)

2009. The Fogg Behavior Model

In 2009, he published A Behavior Model for Persuasive Design, the paper that introduced the (now famous) Fogg behavior model (Fogg behavior model) “B=MAP.”

As Fogg’s research interests expanded from persuasive technology to behavior design in general, the name of his lab was later changed to the Behavior Design Lab.

After that, he published popular books such as Tiny Habits (2019) and has been giving public lectures. Of course, he also continues to hold paid lectures on the Fogg behavior model and captology aimed at Silicon Valley founders or aspiring founders.

2020. The Social Dilemma and the Center for Humane Technology

One of the students who took B. J. Fogg’s captology course, Tristan Harris, joined Google, but became disillusioned after seeing persuasive technology used unethically inside the company, left Google, and became a whistleblower. Together with Aza Raskin, Harris founded the Center for Humane Technology. The word “humane” was probably influenced by Jef Raskin’s The Humane Interface, Raskin being Aza Raskin’s father and the proposer of Apple’s Macintosh project. (The Korean translation was titled “Human-Centered Interface.”)

Aza Raskin is the designer who first devised the infinite scroll UI, and he has expressed regret, saying that his invention unintentionally led countless human beings into addiction. Unlike Raskin, Fogg has not, at least as far as I know, made statements expressing regret about research such as captology or MIP, and he is actively defending the past.

Harris and Raskin produced the documentaries The Social Dilemma and The AI Dilemma, arguing that giant technology companies are abusing technology for the purpose of maximizing profit and thereby harming the public. The two still run a podcast on the same topic, Your undivided attention.

2023. B. J. Fogg’s influence on Silicon Valley

Fogg’s research on persuasive technology and behavior design has had a tremendous influence on Silicon Valley, and if I name only some of the students he influenced, they would include the following.

Tristan Harris (cofounder of the Center for Humane Technology, former Google design ethicist)
Cristina Cordova (former business lead at Stripe, former platform & partnerships leader at Notion, current COO of Liner)
Mike Krieger (cofounder of Instagram)
Akshay Kothari (cofounder of Notion and current COO)
Dana Sittler (current product leader for Europe at Netflix, leader of Facebook’s product management team)
Dean Eckles (former Twitter consultant, former researcher on Facebook’s data science team)
Nina Kim Schultz (former engineering research lead and manager at Google)
Othman Laraki (cofounder of Color Genomics and current CEO, former VP of product management at Twitter, former advisor to Pinterest)

1984. Influence

Now let’s go back to the past.

Robert Cialdini’s 1984 book Influence is an important work that influenced Fogg’s captology, Nass’s CASA paradigm, which I’ll explain shortly, and Brignull’s deceptive patterns.

In this book, Cialdini presents six principles that can be used to exert influence on people and achieve what one wants.

reciprocity: if you first do someone a favor and then make a proposal, it becomes harder for them to refuse it.
consistency: people tend to behave consistently. If you elicit positive answers with light questions and then present what you really want, it becomes harder for them to give a negative answer.
social proof: people are influenced by other people’s decisions. This is especially true when ambiguity or uncertainty is high.
authority: people tend to respond automatically to authority or symbols of authority (such as academic degrees).
liking: people tend to accept proposals more readily from those they like—people who flatter them, are attractive, are funny, and so on.
scarcity: people tend to prefer what is scarce.

Widely used deceptive patterns make use of the concepts Cialdini described.

fake social proof: arbitrarily creating social proof
fake scarcity: arbitrarily creating scarcity
fake urgency: arbitrarily creating urgency (another form of scarcity)

Research on persuasion was, perhaps unsurprisingly, abused in diverse ways from the very beginning. A representative example would be hacker Kevin Mitnick’s The Art of Deception. In this book, Mitnick introduces methods of deceiving people and penetrating systems under the name social engineering, using persuasive techniques and various other methods.

1994. The CASA paradigm

Let’s return again to the 1990s.

Clifford Nass’s CASA paradigm was born in this period. CASA is short for Computer As a Social Actor (or it is sometimes used as short for “Computers Are Social Actors”). For reference, Nass was the doctoral advisor of B. J. Fogg, who proposed captology.

In his 1993 paper “Anthropomorphism, agency, and ethopoeia: Computers as social actors,” Nass argued that even when a computer interface provides only minimal “social cues,” users tend to anthropomorphize the computer, and that this can induce users to engage in social interaction with the computer.

The tendency of people to anthropomorphize computers had already been known for some time, but Nass’s research was significant in two respects.

First, before Nass, it had been assumed that inducing anthropomorphism would require fairly sophisticated and complex social cues, but Nass showed that even minimal cues can induce anthropomorphism.
Second, before Nass, the tendency toward anthropomorphism was regarded as something that appeared because of users’ ignorance or psychological error, but Nass showed that even users familiar with computers (that is, people who fully know that computers do not have a “self”) still anthropomorphize them.

After the 1993 anthropomorphism paper, Nass conducted several follow-up studies on the CASA paradigm together with Fogg. In particular, they showed that the concept of “reciprocity” proposed by social psychologist Robert Cialdini in Influence can also be induced in computer-human interaction. These studies became important foundations for Fogg’s later research, such as captology and MIP.

Philosopher Daniel Dennett had, more than ten years before Nass, already argued that humans often take an “intentional stance” or “intentional strategy” even toward certain artifacts such as a “thermostat,” and that this strategy is very effective (1981, True believers - The intentional strategy and why it works), but it is not clear whether Nass knew Dennett’s theory at the time.

2008. Nudge

It would be strange if only the IT or HCI fields were interested in ways of influencing human behavior. There are related discussions in behavioral economics as well, and 2008’s Nudge is a representative example. It introduces methods of guiding human behavior in a particular direction by designing the environment appropriately—the so-called libertarian paternalism.

If the nudges introduced in the first edition of 2008 were strategies for guiding human behavior in a “beneficial” direction, the 2021 “final” revised edition (apparently so titled to express the authors’ wish that they would never publish another revised edition) additionally introduces the concept of sludge. Sludge means “things that make it difficult for people to make wise choices.” The precise definition is below.

any aspect of choice architecture consisting of friction that makes it harder for people to obtain an outcome that will make them better off (by their own lights).

For example, if you want to be tested for COVID-19 but have to submit dozens of pages of paperwork, that is sludge. The authors say that some deceptive patterns (dark patterns) fit the definition of sludge. (That said, the sludge example above is closer to an antipattern than to a dark pattern.)

Some dark patterns count as sludge; they impose a lot of friction if people want to avoid certain charges.

For reference, the final revised edition of Nudge explains this on the basis of the original definition of deceptive patterns (“interfaces meticulously designed to make users take certain actions irrespective of their intentions for the purpose of making money”; 2010), but the definition in the deceptive patterns book published in 2023 (“interfaces meticulously designed with the purpose of harming users’ autonomy, decision-making, and choice”; 2023) has become somewhat more similar to the definition of sludge.