tokenizer.encode（）_不知道起什么名字-CSDN博客_tokenizer.encode

tokenizer.encode("说你跑的挺远",add_special_tokenizerns=False)
相当于

seg=tokenizer.tokenize("说你跑的挺远")
encoder = tokenizer.convert_tokenizerns_to_ids(seg)
 

https://blog.csdn.net/Princeicon/article/details/108052302