推荐系统和自然语言处理是大模型高发的两大应用领域，在大的特征下，它们又有各自的特点。下面以推荐系统的代表模型DLRM(Deep Learning Recommendation Model)和GPT-3(Generative Pre-Training)为例说明。

DLRM vs GPT-3

可以看到，推荐系统模型参数大部分在embeddings部分，而NLP模型参数大部分在FC(transformer)部分。可以认为推荐系统比NLP更稀疏(sparse)、且memory intensive得多。

Transformer block

基本结构

basic参数

• or : total number of transformer blocks

• or : number of units in each bottleneck layer, and number of units of each Q/K/V input

• or : number of heads of each transformer block

• or : input sequence length

derived参数

• : dimension of each attention head,

• : intermediate layer units of feed forward layer,

各参数在子图中的示意如下：

补充：Feed Forward子模块

典型模型基本参数

应用	模型
NLP	GPT-3	96	12288	96	2048
NLP	BERT_Base	12	768	12	128/512
NLP	BERT_Large	24	1024	16	128/512
RecSys	BST	1	128(max)	8	20

• BST: Behavior Sequence Transformer

References

1. The GPT-3 Architecture, on a Napkin

2. GPT-3 An Overview

3. Language Models are Few-Shot Learners

4. Improving Language Understanding by Generative Pre-Training

5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

6. Attention Is All You Need

7. BERT transformer block code

8. Deep Learning Recommendation Model for Personalization and Recommendation Systems

9. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba