확률적 앵무새 논문의 오리진 스토리

확률적 앵무새 논문의 오리진 스토리. 패권: 누가 AI 전쟁의 승자가 될 것인가 12장 “Myth Busters”에서 발췌/인용.

Chapter 8. Everything is Awesome

Timnit Gebru:

December 2015, at the NeurIPS conference where Sam Altman and Elon Musk announced they were creating AI “for the benefit of humanity,” Gebru looked around at the thousands of other attendees and shuddered. Almost no one there looked like her. Gebru was in her early thirties and Black, and she’d had anything but a conventional upbringing with the support system that many of her peers had enjoyed. …

While it seemed like these systems could be the perfect neutral arbiter, they often were not. If the data they were trained on was biased, so was the system. and Gebru was painfully aware of bias. …

AI could make that worse. For a start, it was typically designed by people who hadn’t experienced racism, which was one reason why the data being used to train AI models also often failed to fairly represent people from minority groups and women.

COMPAS:

COMPAS used machine learning to give risk scores to defendants. The higher the score, the more likely they were to reoffend. The tool gave high scores to Black defendants far more than white defendants, but its predictions were often erroneous. COMPAS turned out to be twice as likely to be wrong about future criinal behavior by Black defendants as it was for Caucasian ones, according to a 2016 investigation by ProPublica, which looked at seven thousand risk scores given to people arrested in Florida and checked if they’d been charged with new offenses in the next two years. The tool was also more likely to misjudge white defendants who went on to commit other crimes as low-risk. America’s criminal justice system was already skewed against Black people, and that bias looked set to continue with the use of inscrutable AI tools. …

If the data showed a community was being overpoliced, the software would lead that community to continue being overpoliced, amplifying a pre-existing problem.

(전형적인 양성 되먹임 고리. 다만 소프트웨어 덕분에 더 빠르고 정확하고 효율적으로 차별을 실행한다. —ak)

일론 머스크에 대한 팀닛 게브루의 평가:

A white tech tycoon born and raised in South Africa during apartheid along with an all-white, all-male set of investors and researchers is trying to stop AI from ‘taking over the world’ and the only potential problem we see is that ‘all the researchers are working on deep learning?’

왜 실리콘 밸리의 부자들은 눈앞에 닥친 문제들은 무시하면서 먼 미래의 AI 종말론만 그토록 걱정할까?

She couldn’t help asking why some of Silicon Valley’s most powerful people were so worried about the possibility of AI doom when AI was already causing real harm to people today. There were two answers. The first was that few if any of the leaders of OpenAI and DeepMind had ever been, or ever would be, on the receiving end of racial or gender discrimination. The second was that it was, paradoxically, in their corporate interests to shout about the risks of an all-powerful superintelligence. It might not make perfect sense to warn people about the dangers of something you are trying to sell, but it was a brilliant marketing strategy. …

The strategy was also a clever way to avert the public’s attention from the thorny, more immediate problems that companies could take action on, requiring them to slow down their development and rein in the capabilities of their AI models.

1970년대 오일 업계의 전략과 유사:

This wouldn’t be the first time large companies had distracted the public while their businesses swelled. In the early 1970s, the plastic industry, backed by oil companies, began to promote the idea of recycling as a solution to the growing problem of plastic waste. …

Recycling is not a bad thing per se. But by promoting the practice, the industry could argue that plastics weren’t inherently bad so long as they were recycled properly, which shifted the perception of responsibility from producers to consumers. …

What the campaigns had achieved, though, was drawing public attention away from questioning the rapid expansion of plastic production and the toll that was having on the environment. …

In just the same way Big Oil redirected the world’s attention from their own significant environmental impact, AI’s leading builders could exploit the buzz around a future Terminator or Skynet to distract from the present-day problems that machine learning algorithms were causing. The burden of responsibility wasn’t on the creators or the industry to act now. It was an abstract problem to be dealt with later.

(빅테크가 장기주의에 빠져드는 이유. 한편, 스카이넷은 이미 우리 곁에 있고 다양한 방식으로 사람들을 죽이고 있다. 참고: 우발적 스카이넷 —ak)

Datasheets for Datasets 제안:

After joining Microsoft, she wrote up a set of rules called “Datasheets for Datasets”, which said that when training an AI model, programmers should create a datasheet that showed all the details about how it was created, what was in it, how it would be used, what its limitations might be, and any other ethical considerations. … If the model ended up being biased, it’d be a lot easier to figure out why.

구글로의 이직, Margaret Mitchell과의 만남:

After Google was in the news for the gorilla debacle, another computer scientist, named Margaret Mitchell, joined the search giant to try to prevent similar mistakes from happening. … Like Gebru, she was worried about the strange mistakes AI systems were making. …

In 2018, Mitchell sent Gebru an email asking to join her at Google.

But looking at the size of her new ethics team, it was clear where tech giants like Google prioritized their investment in AI: capabilities. …

When she warned managers at meetings about some of the potential problems their AI systems could introduce, she’d get emails from the Human Resources Department telling her to be more collaborative.

결국 해고된 두 사람:

One day when the pair were sitting in Gebru’s office at Building 41 of the Google campus, they were talking about an upsetting email that had come through from one of their managers, which reflected the discrimination they both felt at the company. Mitchell was on the verge of tears. Gebru took a different view.

“Don’t be depressed,” she told Mitchell. “Get angry.” …

Later, when both Mitchell and Gebru would be fired by Google, that same manager would publicly vouch for them both and then resign shortly after.

Gebru and Mitchell were about finally bring proper attention to their cause, even if it meant being kicked out in what would become a public scandal.

Chapter 12. Myth Busters

Inside Google, Gebru and Mitchell had become demoralized by signs that their bosses didn’t care about the risks of language models. At once point in late 2020, for instance, the pair heard about a key meeting between forty Google staff to discuss the future of large language models. A product manager led the discussion about ethics. Nobody had invited Gebru or Mitchell. …

They started throwing together ideas and called their project the stone soup paper …. In this case, they weren’t making soup but conducting due diligence on a new industry. Bender wrote the outline, while Gebru, Mitchell, one of Bender’s students, and three other from Google contributed all the text under her section headers. … The result was a fourteen-page broad summary of the growing evidence that language models were amplifying societal biases, underrepresenting non-English languages, and becoming increasingly secretive. …

Bender couldn’t stand the way GPT-3 and other large language models were dazzling their early users with what was, essentially, glorified auto-correct software. So she suggested putting “stochastic parrots” in the title to emphasize that the machines were simply parroting their training. …

Gebru and Mitchell quickly submitted the paper for review through Google’s internal process, through which the company checked its researchers weren’t leaking any sensitive material. The reviewer said it looked good, and their manager gave it all-clear. To make sure they ticked all the right boxes, Gebru and Mitchell also sent the paper to more than dozen other colleagues in and outside of Google, and they gave the company’s press relations team a heads-up. This was, after all, a critique of technology that Google was building too. …

Then something odd happened. A month after submitting the paper, Gebru, Mitchell, and their Google acoauthors were summoned to a meeting with Google executives. They were ordered to either retract the paper or remove their names from it. …

The executives said that after being further scrutinized by other anonymous reviewers, the paper hadn’t met the bar for publication. It was too negative about the problems of LLMs. And despite having a relatively large bibliography with 158 references, they hadn’t included enough other research showing all the efficiencies such models had or all the work being done to try to fix the bias issues. Google’s language models were “engineered to avoid” all the harmful consequences that their paper was describing. …

Gebru wrote a lengthy email to one of her superiors, trying to resolve the matter. Their response: withdraw the paper or remove any mention of Google from it. …

The following day, Gebru found an email in her inbox from her senior boss. Gebru hadn’t technically offered her resignation, but Google was accepting it anyway. …

A few months later, Google fired Mitchell too. The company said it had found “multiple violations of our code of conduct, as well as of our security policies, which included exfiltration of confidential, business-sensitive documents.” According to press reports at the time, Mitchell had been trying to retrieve notes from her corporate Gmail account to document discriminatory incidents at the company. …

The Stochastic Parrots paper hadn’t been all that earth-shattering in its findings. It was mainly an assemblage of other research work. But as word of the firings spread and the paper got leaked online, it took on a life of its own.