트랜스포머 아키텍쳐

트랜스포머 구조트랜스포머Transformer

거의 모든 LLM의 기반이 되는 신경망 아키텍쳐. 2017년 논문 Attention is all you need에서 처음으로 제안.

External links

poloclub.github.io/transformer-explainer/
bbycroft.net/llm
e2eml.school/transformers.html - One-hot encoding부터 시작하는 해설
nlp.seas.harvard.edu/annotated-transformer/ - 파이썬 구현과 해설

패권: 누가 AI 전쟁의 승자가 될 것인가 “트랜스포머 아키텍쳐의 오리진 스토리, GPT의 오리진 스토리, 확률적 앵무새 논문의 오리진 스토리, 샘 올트만의 어린 시절과 초반 경력, 데미스 허사비스의 어린 시절과 초반 경력”
MatFormer: Nested Transformer for Elastic Inference 이 글을 인용: “We present MatFormer, a novel Transformer architecture designed to provide elastic inference across diverse dep…”
GPT의 오리진 스토리 이 글을 인용: “Ilya Sutskever, OpenAI's star scientist, couldn't stop thinking about what transformer could do with language.”
BioGPT 이 글을 인용: “In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-…”
Tensor Logic: The Language of AI 이 글을 인용: “I show how to elegantly implement key forms of neural, symbolic and statistical AI in tensor logic, including t…”
Attention is Turing Complete 이 글을 인용: “We study the computational power of the Transformer, one of the most paradigmatic architectures exemplifying se…”
Survey of hallucination in natural language generation 이 글을 인용: “Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of seque…”
GPT-4 technical report 이 글을 인용: “GPT-4 is a Transformer-based model pre-trained to predict the next token in a document.”
Text-to-LoRA: Instant Transformer Adaption “T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass.”
Sparks of artificial general intelligence: Early experiments with GPT-4 이 글을 인용: “트랜스포머 아키텍쳐, 대규모 데이터, 자기지도학습 덕분.”
Microservice architecture “Microservices architecture is a design approach that structures an application as a collection of small, autono…”
사전 학습 모델 “Examples of successful large-scale pre-trained language models are Bidirectional Encoder Representations from T…”
Nested Learning: The Illusion of Deep Learning Architectures “Over the last decades, developing more powerful neural architectures and simultaneously designing optimization …”
Positional encoding 이 글을 인용: “RNN 같은 방식을 쓰지 않으면서도 토큰 순서에 대한 정보를 반영하기 위해 고안된 방법.”
ChatGPT의 짧은 역사 관련 절: Chapter 14. A Vague Sense of Doom
Vector Symbolic Architectures as a Computing Framework for Emerging Hardware “This article reviews recent progress in the development of the computing framework vector symbolic architecture…”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? “BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible bot…”

External links

See also