์ฝ˜ํ…์ธ  ๋Œ€ํ‘œ ์ด๋ฏธ์ง€ - ๐Ÿงฎ ํŒŒ์ด์ฌ์œผ๋กœ ํ† ํฐ ๊ธธ์ด ์ž๋™ ๊ณ„์‚ฐ๊ธฐ ๊ฐœ๋ฐœํ•˜๊ธฐ: tiktoken๋ถ€ํ„ฐ ๋น„์šฉ ์ถ”์ •ยท์ฒญํฌ ๋ถ„ํ• ๊นŒ์ง€

๐Ÿงฎ ํŒŒ์ด์ฌ์œผ๋กœ ํ† ํฐ ๊ธธ์ด ์ž๋™ ๊ณ„์‚ฐ๊ธฐ ๊ฐœ๋ฐœํ•˜๊ธฐ: tiktoken๋ถ€ํ„ฐ ๋น„์šฉ ์ถ”์ •ยท์ฒญํฌ ๋ถ„ํ• ๊นŒ์ง€

"๊ธ€์ž ์ˆ˜ 3,000์ž๋ฉด ๋ช‡ ํ† ํฐ์ด์•ผ?" ์ด ์งˆ๋ฌธ์— ๋งค๋ฒˆ ๊ฐ์œผ๋กœ ๋Œ€๋‹ตํ•˜๊ณ  ์žˆ์—ˆ๋‹ค๋ฉด, ์˜ค๋Š˜ ๊ทธ ๊ฐ์„ ์ฝ”๋“œ๋กœ ๋ฐ”๊ฟ”๋ด…์‹œ๋‹ค. ํŒŒ์ด์ฌ ๋ช‡์‹ญ ์ค„๋กœ ์‹œ์ž‘ํ•ด์„œ ์‹ค์ „์—์„œ ์“ธ ์ˆ˜ ์žˆ๋Š” ํ† ํฐ ๊ณ„์‚ฐ๊ธฐ๊นŒ์ง€ ๊ฐ™์ด ๋งŒ๋“ค์–ด ๋ณผ๊ฒŒ์š”. ๐Ÿต

๋ฌธ์žฅ์ด ํ† ํฐ์ด ๋˜๋Š” ์ˆœ๊ฐ„ ์ž…๋ ฅ ๋ฌธ์žฅ "Hello ํ† ํฐ!" ๊ธ€์ž ์ˆ˜ 9์ž ํ† ํฌ๋‚˜์ด์ € ํ† ํฐ ID ๋ชฉ๋ก 9906 โ€ฆ 0 len() = ํ† ํฐ ์ˆ˜ ํ† ํฐ ์ˆ˜: 5 o200k_base ๋น„์šฉ ์ถ”์ • ํ•œ๋„ ๊ฒ€์‚ฌ ์ฒญํฌ ๋ถ„ํ• 

โ€ป ๊ทธ๋ฆผ ์† ํ† ํฐ ID๋Š” ์„ค๋ช…์šฉ ์˜ˆ์‹œ์˜ˆ์š”. ์‹ค์ œ ๊ฐ’์€ ์ธ์ฝ”๋”ฉ๋งˆ๋‹ค ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

1. ์™œ ํ•˜ํ•„ "ํ† ํฐ"์„ ์„ธ์•ผ ํ• ๊นŒ? ๐Ÿค”

LLM(๋Œ€๊ทœ๋ชจ ์–ธ์–ด ๋ชจ๋ธ)์€ ์šฐ๋ฆฌ๊ฐ€ ๋ณด๋‚ด๋Š” ๊ธ€์„ ๊ธ€์ž ๋‹จ์œ„๋กœ ์ฝ์ง€ ์•Š์•„์š”. ๋จผ์ € ํ† ํฌ๋‚˜์ด์ €๊ฐ€ ๋ฌธ์žฅ์„ ์ž˜๊ฒŒ ์ชผ๊ฐœ ์ •์ˆ˜ ID ๋ฐฐ์—ด๋กœ ๋ฐ”๊พธ๊ณ , ๋ชจ๋ธ์€ ๊ทธ ์ˆซ์ž ๋ฐฐ์—ด์„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ API ์š”๊ธˆ, ์ปจํ…์ŠคํŠธ ์œˆ๋„์šฐ(ํ•œ ๋ฒˆ์— ๋„ฃ์„ ์ˆ˜ ์žˆ๋Š” ๊ธธ์ด), ์ถœ๋ ฅ ๊ธธ์ด ์ œํ•œ์ด ์ „๋ถ€ ๊ธ€์ž ์ˆ˜๊ฐ€ ์•„๋‹ˆ๋ผ ํ† ํฐ ์ˆ˜ ๊ธฐ์ค€์œผ๋กœ ์ •ํ•ด์ ธ ์žˆ์–ด์š”.

๋ฌธ์ œ๋Š” "๊ธ€์ž ์ˆ˜ รท ๋ช‡" ๊ฐ™์€ ์–ด๋ฆผ์…ˆ์ด ์–ธ์–ด๋งˆ๋‹ค, ๋ชจ๋ธ๋งˆ๋‹ค, ์‹ฌ์ง€์–ด ๊ฐ™์€ ๋ชจ๋ธ์˜ ๋ฒ„์ „๋งˆ๋‹ค ๋‹ค๋ฅด๊ฒŒ ํ‹€๋ฆฐ๋‹ค๋Š” ์ ์ด์—์š”. ์˜์–ด ๋ฌธ์žฅ์€ ๋Œ€์ฒด๋กœ ๋‹จ์–ด ํ•˜๋‚˜๊ฐ€ ํ† ํฐ ํ•œ๋‘ ๊ฐœ๋กœ ๋๋‚˜์ง€๋งŒ, ํ•œ๊ตญ์–ด๋Š” ๊ฐ™์€ ์˜๋ฏธ๋ฅผ ํ‘œํ˜„ํ•  ๋•Œ ํ† ํฐ์ด ๋” ๋งŽ์ด ๋‚˜์˜ค๋Š” ๊ฒฝํ–ฅ์ด ์žˆ๊ณ , ์ฝ”๋“œยทURLยท์ด๋ชจ์ง€๋Š” ๋˜ ๋‹ค๋ฅธ ํŒจํ„ด์„ ๋ณด์ž…๋‹ˆ๋‹ค. ๊ฒฐ๊ตญ ์ •ํ™•ํ•œ ๊ฐ’์ด ํ•„์š”ํ•˜๋ฉด ํ•ด๋‹น ๋ชจ๋ธ์˜ ํ† ํฌ๋‚˜์ด์ €๋กœ ์ง์ ‘ ์„ธ๋Š” ๊ฒŒ ์œ ์ผํ•œ ์ •๋‹ต์ด์—์š”.

๐Ÿ’ธ ๋น„์šฉ ๊ด€๋ฆฌ๋Œ€๋ถ€๋ถ„์˜ ์ƒ์šฉ LLM API๋Š” ์ž…๋ ฅยท์ถœ๋ ฅ ํ† ํฐ ์ˆ˜์— ๋น„๋ก€ํ•ด ๊ณผ๊ธˆํ•ด์š”. ๋ฐฐ์น˜ ์ž‘์—… ์ „์— ํ† ํฐ์„ ์„ธ๋ฉด ์ฒญ๊ตฌ์„œ ์ถฉ๊ฒฉ์„ ๋ฏธ๋ฆฌ ๋ง‰์„ ์ˆ˜ ์žˆ์ฃ .
๐Ÿ“ ํ•œ๋„ ์ดˆ๊ณผ ๋ฐฉ์ง€์ปจํ…์ŠคํŠธ ์œˆ๋„์šฐ๋ฅผ ๋„˜๊ธฐ๋ฉด ์š”์ฒญ์ด ๊ฑฐ์ ˆ๋˜๊ฑฐ๋‚˜ ์•ž๋ถ€๋ถ„์ด ์ž˜๋ ค์š”. ๋ณด๋‚ด๊ธฐ ์ „์— ๊ฒ€์‚ฌํ•˜๋Š” ๊ฒŒ ํ›จ์”ฌ ์‹ธ๊ฒŒ ๋จนํž™๋‹ˆ๋‹ค.
โœ‚๏ธ RAG ์ฒญํฌ ์„ค๊ณ„๋ฌธ์„œ๋ฅผ ๊ฒ€์ƒ‰์šฉ ์กฐ๊ฐ์œผ๋กœ ๋‚˜๋ˆŒ ๋•Œ "500ํ† ํฐ ๋‹จ์œ„"์ฒ˜๋Ÿผ ํ† ํฐ ๊ธฐ์ค€์œผ๋กœ ์ž๋ฅด๋ฉด ์ž„๋ฒ ๋”ฉ ๋ชจ๋ธ ํ•œ๋„์— ์ •ํ™•ํžˆ ๋งž์ถœ ์ˆ˜ ์žˆ์–ด์š”.
ํ”ํ•œ ์˜คํ•ด: "1ํ† ํฐ = 1๋‹จ์–ด"์˜์–ด๊ถŒ ๋ฌธ์„œ์—์„œ ์ž์ฃผ ๋ณด์ด๋Š” "ํ† ํฐ 1๊ฐœ๋Š” ๋Œ€๋žต ์˜์–ด 4๊ธ€์ž, ๋‹จ์–ด์˜ 3/4 ์ •๋„" ๊ฐ™์€ ๋ง์€ ์˜์–ด ์ผ๋ฐ˜ ํ…์ŠคํŠธ์— ๋Œ€ํ•œ ๊ฒฝํ—˜์น™์ผ ๋ฟ์ด์—์š”. ํ•œ๊ตญ์–ด, ์ผ๋ณธ์–ด, ์†Œ์Šค ์ฝ”๋“œ์—๋Š” ๊ทธ๋Œ€๋กœ ์ ์šฉ๋˜์ง€ ์•Š์œผ๋‹ˆ ๊ฒฌ์ ์„œ๋‚˜ ํ•œ๋„ ๊ณ„์‚ฐ์— ์ด ์ˆซ์ž๋ฅผ ๋ฐ•์•„ ๋„ฃ์ง€ ๋งˆ์„ธ์š”.

2. ํ† ํฌ๋‚˜์ด์ €๋Š” ๋ฌธ์žฅ์„ ์–ด๋–ป๊ฒŒ ์ž๋ฅผ๊นŒ? (BPE ํ•œ ์ž… ๋ง›๋ณด๊ธฐ) ๐Ÿฐ

์š”์ฆ˜ GPT ๊ณ„์—ด์„ ๋น„๋กฏํ•œ ๋งŽ์€ ๋ชจ๋ธ์ด ์“ฐ๋Š” ๋ฐฉ์‹์€ BPE(Byte Pair Encoding) ๊ณ„์—ด์ด์—์š”. ์›๋ฆฌ๋Š” ์ƒ๊ฐ๋ณด๋‹ค ๋‹จ์ˆœํ•ฉ๋‹ˆ๋‹ค.

ํ…์ŠคํŠธ๋ฅผ ๊ฐ€์žฅ ์ž‘์€ ๋‹จ์œ„(๋ฌธ์ž ๋˜๋Š” ๋ฐ”์ดํŠธ)๋กœ ์ชผ๊ฐญ๋‹ˆ๋‹ค.
ํ•™์Šต ๋ฐ์ดํ„ฐ์—์„œ ๊ฐ€์žฅ ์ž์ฃผ ๋ถ™์–ด ๋‚˜์˜ค๋Š” ์Œ์„ ์ฐพ์•„ ํ•˜๋‚˜์˜ ์ƒˆ ๋‹จ์œ„๋กœ ํ•ฉ์นฉ๋‹ˆ๋‹ค. ์˜ˆ: "t"+"h" โ†’ "th"
์ด ๋ณ‘ํ•ฉ์„ ์ •ํ•ด์ง„ ์–ดํœ˜ ํฌ๊ธฐ์— ๋„๋‹ฌํ•  ๋•Œ๊นŒ์ง€ ์ˆ˜๋งŒ~์ˆ˜์‹ญ๋งŒ ๋ฒˆ ๋ฐ˜๋ณตํ•ด ๋ณ‘ํ•ฉ ๊ทœ์น™ํ‘œ์™€ ์–ดํœ˜ ์‚ฌ์ „์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
์ƒˆ ๋ฌธ์žฅ์ด ๋“ค์–ด์˜ค๋ฉด ์ด ๊ทœ์น™ํ‘œ๋ฅผ ์ˆœ์„œ๋Œ€๋กœ ์ ์šฉํ•ด ํ† ํฐ ID ๋ฐฐ์—ด๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.
BPE ๋ณ‘ํ•ฉ ๊ณผ์ • (๊ฐœ๋…๋„) 0๋‹จ๊ณ„ l o w e s t 6์กฐ๊ฐ 1๋‹จ๊ณ„ lo w e st 4์กฐ๊ฐ 2๋‹จ๊ณ„ low est 2์กฐ๊ฐ ์ž์ฃผ ๋ถ™์–ด ๋‹ค๋‹ˆ๋Š” ์Œ์ผ์ˆ˜๋ก ๋จผ์ € ํ•ฉ์ณ์ ธ์„œ ํ”ํ•œ ๋‹จ์–ด๋Š” ์ ์€ ํ† ํฐ์œผ๋กœ ํ‘œํ˜„๋ผ์š”

์—ฌ๊ธฐ์„œ ์ค‘์š”ํ•œ ํฌ์ธํŠธ! ๋ณ‘ํ•ฉ ๊ทœ์น™์€ ํ•™์Šต ๋ฐ์ดํ„ฐ์˜ ๋นˆ๋„๋กœ ๋งŒ๋“ค์–ด์ง€๊ธฐ ๋•Œ๋ฌธ์—, ํ•™์Šต ๋ฐ์ดํ„ฐ์— ๋งŽ์ด ๋“ฑ์žฅํ•œ ์–ธ์–ดยทํŒจํ„ด์ผ์ˆ˜๋ก ํฐ ๋ฉ์–ด๋ฆฌ๋กœ ๋ฌถ์—ฌ ํ† ํฐ์ด ์ ๊ฒŒ ๋‚˜์™€์š”. ์˜์–ด๊ฐ€ ์ƒ๋Œ€์ ์œผ๋กœ "ํ† ํฐ ํšจ์œจ"์ด ์ข‹์€ ์ด์œ ๊ฐ€ ์ด๊ฑฐ์˜ˆ์š”. ํ•œ๊ตญ์–ด๋Š” ํ•œ ์Œ์ ˆ์ด UTF-8๋กœ 3๋ฐ”์ดํŠธ๋ฅผ ์ฐจ์ง€ํ•˜๊ณ , ๋ฐ”์ดํŠธ ์ˆ˜์ค€ BPE์—์„œ๋Š” ์ž์ฃผ ์“ฐ์ด์ง€ ์•Š๋Š” ์Œ์ ˆ์ด ์—ฌ๋Ÿฌ ๋ฐ”์ดํŠธ ํ† ํฐ์œผ๋กœ ์ชผ๊ฐœ์งˆ ์ˆ˜ ์žˆ์–ด์„œ ํ† ํฐ ์ˆ˜๊ฐ€ ๋Š˜์–ด๋‚˜๊ธฐ ์‰ฝ์Šต๋‹ˆ๋‹ค.

๋ฐ”์ดํŠธ ์ˆ˜์ค€ BPE๋ž€?GPT-2 ์ดํ›„ OpenAI ๊ณ„์—ด ํ† ํฌ๋‚˜์ด์ €๋Š” ๋ฌธ์ž๊ฐ€ ์•„๋‹ˆ๋ผ UTF-8 ๋ฐ”์ดํŠธ๋ฅผ ๊ธฐ๋ณธ ๋‹จ์œ„๋กœ ์‚ผ์•„์š”. ๋•๋ถ„์— ์–ด๋–ค ์–ธ์–ดยท์ด๋ชจ์ง€๋“  "์•Œ ์ˆ˜ ์—†๋Š” ๋ฌธ์ž(UNK)" ์—†์ด ๋ฐ˜๋“œ์‹œ ์ธ์ฝ”๋”ฉ๋ฉ๋‹ˆ๋‹ค. ๋Œ€์‹  ํ† ํฐ ํ•˜๋‚˜๊ฐ€ ํ•œ ๊ธ€์ž์˜ ์ผ๋ถ€ ๋ฐ”์ดํŠธ๋งŒ ๋‹ด๋Š” ๊ฒฝ์šฐ๊ฐ€ ์ƒ๊ฒจ์„œ, ํ† ํฐ ํ•˜๋‚˜๋งŒ ๋”ฐ๋กœ ๋””์ฝ”๋”ฉํ•˜๋ฉด ๊นจ์ง„ ๋ฌธ์ž๊ฐ€ ๋‚˜์˜ฌ ์ˆ˜ ์žˆ์–ด์š”. ์ด๊ฑด ๊ณ„์‚ฐ๊ธฐ UI์—์„œ ํ† ํฐ์„ ์‹œ๊ฐํ™”ํ•  ๋•Œ ๊ผญ ๊ธฐ์–ตํ•ด์•ผ ํ•  ํ•จ์ •์ด์—์š”.

๋‹ค๋ฅธ ๋ฐฉ์‹๋„ ์žˆ์–ด์š”. Google ๊ณ„์—ด๊ณผ ์—ฌ๋Ÿฌ ์˜คํ”ˆ ๋ชจ๋ธ์—์„œ ์“ฐ์—ฌ ์˜จ SentencePiece(BPE ๋˜๋Š” Unigram ์•Œ๊ณ ๋ฆฌ์ฆ˜ ์ง€์›), BERT ๊ณ„์—ด์˜ WordPiece ๋“ฑ์ด์ฃ . ์•Œ๊ณ ๋ฆฌ์ฆ˜์ด ๋‹ฌ๋ผ๋„ ๊ณ„์‚ฐ๊ธฐ ์ž…์žฅ์—์„œ ํ•  ์ผ์€ ๋˜‘๊ฐ™์•„์š”. "ํ•ด๋‹น ๋ชจ๋ธ์˜ ํ† ํฌ๋‚˜์ด์ €๋ฅผ ๋ถˆ๋Ÿฌ์™€์„œ encodeํ•œ ๋’ค ๊ธธ์ด๋ฅผ ์žฐ๋‹ค." ๋!

3. ์–ด๋–ค ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋กœ ์…€๊นŒ? ๋„๊ตฌ ์ง€๋„ ๐Ÿ—บ๏ธ

"์–ด๋–ค ๋ชจ๋ธ์˜ ํ† ํฐ์„ ์„ธ๊ณ  ์‹ถ์€๊ฐ€"์— ๋”ฐ๋ผ ๋„๊ตฌ๊ฐ€ ๊ฐˆ๋ ค์š”. ํ‘œ๋กœ ์ •๋ฆฌํ•ด ๋ณผ๊ฒŒ์š”.

๋Œ€์ƒ ๋ชจ๋ธ์ถ”์ฒœ ๋ฐฉ๋ฒ•ํŠน์ง•
OpenAI GPT ๊ณ„์—ดtiktoken (pip install tiktoken)OpenAI๊ฐ€ ๊ณต๊ฐœํ•œ ๋น ๋ฅธ BPE ํ† ํฌ๋‚˜์ด์ €. ๋กœ์ปฌ์—์„œ ์˜คํ”„๋ผ์ธ ๊ณ„์‚ฐ ๊ฐ€๋Šฅ(์ตœ์ดˆ 1ํšŒ ์ธ์ฝ”๋”ฉ ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ)
Hugging Face ๊ณต๊ฐœ ๋ชจ๋ธ (Llama, Mistral, Qwen, Gemma ๋“ฑ)transformers์˜ AutoTokenizer๋ชจ๋ธ ์ €์žฅ์†Œ์˜ tokenizer ํŒŒ์ผ์„ ๊ทธ๋Œ€๋กœ ์‚ฌ์šฉ. ์ผ๋ถ€ ๋ชจ๋ธ์€ ์ ‘๊ทผ ์Šน์ธ ํ•„์š”
Anthropic Claude๊ณต์‹ SDK์˜ ํ† ํฐ ์นด์šดํŒ… API๋ฉ”์‹œ์ง€ยท์‹œ์Šคํ…œ ํ”„๋กฌํ”„ํŠธยท๋„๊ตฌ ์ •์˜๋ฅผ ํฌํ•จํ•œ ์ž…๋ ฅ ํ† ํฐ ์ˆ˜๋ฅผ ์„œ๋ฒ„๊ฐ€ ๊ณ„์‚ฐํ•ด ์คŒ
Google Gemini๊ณต์‹ SDK์˜ count_tokens์„œ๋ฒ„ ์ธก ๊ณ„์‚ฐ. ํ…์ŠคํŠธ ์™ธ ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ ์ž…๋ ฅ๋„ ์ง‘๊ณ„
์ฃผ์˜: ๋‚จ์˜ ํ† ํฌ๋‚˜์ด์ €๋กœ ์žฌ๋ฉด ์•ˆ ๋ผ์š”tiktoken์œผ๋กœ Claude๋‚˜ Llama ํ”„๋กฌํ”„ํŠธ๋ฅผ ์žฌ๋Š” ๊ฑด "์ค„์ž ๋Œ€์‹  30cm ์ž๋ฅผ ๋‹ค๋ฅธ ๋‹จ์œ„๋กœ ์ฝ๋Š” ๊ฒƒ"๊ณผ ๊ฐ™์•„์š”. ์–ดํœ˜ ์‚ฌ์ „์ด ๋‹ฌ๋ผ์„œ ๊ฒฐ๊ณผ๊ฐ€ ๊ฝค ์ฐจ์ด ๋‚  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทผ์‚ฌ์น˜๋กœ ์“ฐ๋ ค๋ฉด ๋ฐ˜๋“œ์‹œ "์ถ”์ •์น˜"๋ผ๊ณ  ํ‘œ์‹œํ•˜๊ณ  ์—ฌ์œ ๋ถ„์„ ๋‘์„ธ์š”.

4. ๊ฐ€์žฅ ์ž‘์€ ํ† ํฐ ๊ณ„์‚ฐ๊ธฐ: 10์ค„๋กœ ์‹œ์ž‘ํ•˜๊ธฐ โšก

๋จผ์ € ์„ค์น˜๋ถ€ํ„ฐ ํ•ด์š”.

pip install tiktoken

๊ทธ๋ฆฌ๊ณ  ๋Œ€๋ง์˜ ์ฒซ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # GPT-4o ๊ณ„์—ด์—์„œ ์“ฐ๋Š” ์ธ์ฝ”๋”ฉ
text = "์•ˆ๋…•ํ•˜์„ธ์š”! ํ† ํฐ ๊ณ„์‚ฐ๊ธฐ๋ฅผ ๋งŒ๋“ค์–ด ๋ด…์‹œ๋‹ค."
tokens = enc.encode(text)

print("ํ† ํฐ ์ˆ˜:", len(tokens))
print("ํ† ํฐ ID:", tokens)
print("๋ณต์›:", enc.decode(tokens))

์ด๊ฒŒ ์ „๋ถ€์˜ˆ์š”. encode()๊ฐ€ ์ •์ˆ˜ ๋ฆฌ์ŠคํŠธ๋ฅผ ๋Œ๋ ค์ฃผ๊ณ , ๊ทธ len()์ด ํ† ํฐ ์ˆ˜์˜ˆ์š”. decode()๋กœ ๋‹ค์‹œ ์›๋ฌธ์ด ๋ณต์›๋˜๋Š”์ง€๋„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์ฃ .

์ธ์ฝ”๋”ฉ ์ด๋ฆ„ vs ๋ชจ๋ธ ์ด๋ฆ„

tiktoken์€ ๋‘ ๊ฐ€์ง€ ๋ฐฉ์‹์œผ๋กœ ์ธ์ฝ”๋”๋ฅผ ๋ถˆ๋Ÿฌ์š”.

enc_a = tiktoken.get_encoding("cl100k_base")      # ์ธ์ฝ”๋”ฉ ์ด๋ฆ„์œผ๋กœ
enc_b = tiktoken.encoding_for_model("gpt-4o")     # ๋ชจ๋ธ ์ด๋ฆ„์œผ๋กœ
print(enc_b.name)   # o200k_base

๋Œ€ํ‘œ์ ์ธ ๋งคํ•‘์€ ์ด๋ž˜์š”. GPT-4ยทGPT-3.5-turbo ๋ฐ text-embedding-3 ๊ณ„์—ด ์ž„๋ฒ ๋”ฉ ๋ชจ๋ธ์€ cl100k_base, GPT-4o ๊ณ„์—ด์€ o200k_base, ๊ทธ๋ณด๋‹ค ์˜ค๋ž˜๋œ GPT-3 ๊ณ„์—ด์€ r50k_baseยทp50k_base๋ฅผ ์‚ฌ์šฉํ–ˆ์–ด์š”. ์ด๋ฆ„์˜ ์ˆซ์ž์ฒ˜๋Ÿผ cl100k_base๋Š” ์•ฝ 10๋งŒ ๊ฐœ, o200k_base๋Š” ์•ฝ 20๋งŒ ๊ฐœ ๊ทœ๋ชจ์˜ ์–ดํœ˜๋ฅผ ๊ฐ–๊ณ  ์žˆ๊ณ , ์–ดํœ˜๊ฐ€ ์ปค์ง€๋ฉด์„œ ์˜์–ด ์ด์™ธ ์–ธ์–ด์˜ ํ† ํฐ ํšจ์œจ์ด ๊ฐœ์„ ๋˜์—ˆ๋‹ค๊ณ  ์•Œ๋ ค์ ธ ์žˆ์–ด์š”.

ํŒ: ๋ชจ๋ฅด๋Š” ๋ชจ๋ธ๋ช… ๋Œ€์‘encoding_for_model()์€ tiktoken์ด ๋ชจ๋ฅด๋Š” ๋ชจ๋ธ๋ช…์„ ๋„ฃ์œผ๋ฉด KeyError๋ฅผ ๋˜์ ธ์š”. ์ƒˆ ๋ชจ๋ธ์ด ๋‚˜์˜ค๋ฉด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์—…๋ฐ์ดํŠธ ์ „๊นŒ์ง€ ์ด๋Ÿฐ ์ผ์ด ์ƒ๊ธฐ๋‹ˆ, ๊ณ„์‚ฐ๊ธฐ์—๋Š” ๊ธฐ๋ณธ ์ธ์ฝ”๋”ฉ์œผ๋กœ ๋–จ์–ด์ง€๋Š” ํด๋ฐฑ์„ ๊ผญ ๋„ฃ์–ด ๋‘์„ธ์š”. (์•„๋ž˜ ์‹ค์ „ ์ฝ”๋“œ์— ๋“ค์–ด ์žˆ์–ด์š”.)

ํ•จ์ • ํ•˜๋‚˜: ํŠน์ˆ˜ ํ† ํฐ ์—๋Ÿฌ

์‚ฌ์šฉ์ž๊ฐ€ ๋ถ™์—ฌ ๋„ฃ์€ ํ…์ŠคํŠธ์— <|endoftext|> ๊ฐ™์€ ๋ฌธ์ž์—ด์ด ๋“ค์–ด ์žˆ์œผ๋ฉด tiktoken์˜ encode()๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ValueError๋ฅผ ์ผ์œผ์ผœ์š”. ํŠน์ˆ˜ ํ† ํฐ์ด ์˜๋„์น˜ ์•Š๊ฒŒ ์ฃผ์ž…๋˜๋Š” ๊ฑธ ๋ง‰๋Š” ์•ˆ์ „์žฅ์น˜๊ฑฐ๋“ ์š”. ๊ณ„์‚ฐ๊ธฐ์—์„œ๋Š” ์ด๋ ‡๊ฒŒ ์ฒ˜๋ฆฌํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

# ํŠน์ˆ˜ ํ† ํฐ ๋ฌธ์ž์—ด๋„ ๊ทธ๋ƒฅ "์ผ๋ฐ˜ ํ…์ŠคํŠธ"๋กœ ์ทจ๊ธ‰ํ•ด์„œ ์…ˆ
n = len(enc.encode(text, disallowed_special=()))

# ๋˜๋Š” ํŠน์ˆ˜ ํ† ํฐ ์ฒ˜๋ฆฌ๋ฅผ ์•„์˜ˆ ํ•˜์ง€ ์•Š๋Š” ํ•จ์ˆ˜
n = len(enc.encode_ordinary(text))

๋กœ๊ทธ ํŒŒ์ผ์ด๋‚˜ ํฌ๋กค๋ง ๋ฐ์ดํ„ฐ์ฒ˜๋Ÿผ "๋ฌด์—‡์ด ๋“ค์–ด์˜ฌ์ง€ ๋ชจ๋ฅด๋Š”" ์ž…๋ ฅ์„ ๋‹ค๋ฃจ๋Š” ๊ณ„์‚ฐ๊ธฐ๋ผ๋ฉด ์ด ์ฒ˜๋ฆฌ๊ฐ€ ์‚ฌ์‹ค์ƒ ํ•„์ˆ˜์˜ˆ์š”. ์ด๊ฑฐ ์•ˆ ํ•˜๋ฉด ์ƒˆ๋ฒฝ์— ๋ฐฐ์น˜ ๋Œ๋ฆฌ๋‹ค ํ„ฐ์ง€๋Š” ๊ฒฝํ—˜์„ ํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๐Ÿ˜‡

5. ์ฑ„ํŒ… ๋ฉ”์‹œ์ง€๋Š” ๋ณธ๋ฌธ๋งŒ ์„ธ๋ฉด ์†ํ•ด ๋ณธ๋‹ค ๐Ÿ’ฌ

Chat Completions์ฒ˜๋Ÿผ messages ๋ฐฐ์—ด์„ ๋ณด๋‚ด๋Š” API๋Š” ๊ฐ ๋ฉ”์‹œ์ง€์— ์—ญํ• (role) ๊ตฌ๋ถ„์šฉ ํฌ๋งท ํ† ํฐ์ด ๋ถ™์–ด์š”. ๊ทธ๋ž˜์„œ ๋ฉ”์‹œ์ง€ ๋‚ด์šฉ๋งŒ ํ•ฉ์นœ ํ† ํฐ ์ˆ˜๋ณด๋‹ค ์‹ค์ œ ์ฒญ๊ตฌ ํ† ํฐ์ด ์กฐ๊ธˆ ๋” ๋งŽ์Šต๋‹ˆ๋‹ค. OpenAI Cookbook์˜ "How to count tokens with tiktoken" ์˜ˆ์ œ์—์„œ๋Š” gpt-3.5-turboยทgpt-4 ๊ณ„์—ด ๊ธฐ์ค€์œผ๋กœ ๋ฉ”์‹œ์ง€๋งˆ๋‹ค 3ํ† ํฐ, name ํ•„๋“œ๊ฐ€ ์žˆ์œผ๋ฉด 1ํ† ํฐ ์ถ”๊ฐ€, ๊ทธ๋ฆฌ๊ณ  ์‘๋‹ต ์‹œ์ž‘์šฉ์œผ๋กœ 3ํ† ํฐ์„ ๋”ํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ๊ณ„์‚ฐํ•ด์š”.

def count_chat_tokens(messages, model="gpt-4o"):
    """OpenAI Cookbook ๋ฐฉ์‹์˜ ๊ทผ์‚ฌ ๊ณ„์‚ฐ.
    ๋ชจ๋ธ์— ๋”ฐ๋ผ ์˜ค๋ฒ„ํ—ค๋“œ๊ฐ€ ๋‹ฌ๋ผ์งˆ ์ˆ˜ ์žˆ์œผ๋ฏ€๋กœ '์ถ”์ •์น˜'๋กœ ์‚ฌ์šฉ."""
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding("o200k_base")

    tokens_per_message = 3
    tokens_per_name = 1
    total = 0
    for msg in messages:
        total += tokens_per_message
        for key, value in msg.items():
            total += len(enc.encode(str(value), disallowed_special=()))
            if key == "name":
                total += tokens_per_name
    total += 3  # ์–ด์‹œ์Šคํ„ดํŠธ ์‘๋‹ต ํ”„๋ผ์ด๋ฐ
    return total

msgs = [
    {"role": "system", "content": "๋„ˆ๋Š” ์นœ์ ˆํ•œ ํŒŒ์ด์ฌ ํŠœํ„ฐ์•ผ."},
    {"role": "user", "content": "๋ฆฌ์ŠคํŠธ ์ปดํ”„๋ฆฌํ—จ์…˜ ์„ค๋ช…ํ•ด์ค˜"},
]
print(count_chat_tokens(msgs))
์ •ํ™•๋„์— ๋Œ€ํ•œ ์†”์งํ•œ ์ด์•ผ๊ธฐ์ด ์˜ค๋ฒ„ํ—ค๋“œ ์ˆซ์ž๋Š” Cookbook์—์„œ๋„ "๋ชจ๋ธ์ด ๋ฐ”๋€Œ๋ฉด ๋‹ฌ๋ผ์งˆ ์ˆ˜ ์žˆ๋‹ค"๊ณ  ๋ช…์‹œํ•œ ๊ทผ์‚ฌ์น˜์˜ˆ์š”. ๋„๊ตฌ ์ •์˜(function calling), ์ด๋ฏธ์ง€ ์ž…๋ ฅ, ๊ตฌ์กฐํ™” ์ถœ๋ ฅ ์Šคํ‚ค๋งˆ ๋“ฑ์€ ์ด ๊ณต์‹์— ํฌํ•จ๋˜์ง€ ์•Š์•„์š”. ์ตœ์ข… ์ง„์‹ค์€ API ์‘๋‹ต์˜ usage ํ•„๋“œ์ž…๋‹ˆ๋‹ค. ๊ณ„์‚ฐ๊ธฐ๋Š” "์‚ฌ์ „ ์ถ”์ •", usage๋Š” "์‚ฌํ›„ ์ •์‚ฐ"์ด๋ผ๊ณ  ์ƒ๊ฐํ•˜๋ฉด ๊น”๋”ํ•ด์š”.

6. ๋‹ค๋ฅธ ๋ชจ๋ธ ํŒจ๋ฐ€๋ฆฌ๋„ ์„ธ์–ด ๋ณด์ž ๐ŸŒ

Hugging Face ์˜คํ”ˆ ๋ชจ๋ธ

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
text = "์˜คํ”ˆ ๋ชจ๋ธ์˜ ํ† ํฐ ์ˆ˜๋ฅผ ์„ธ์–ด ๋ด…๋‹ˆ๋‹ค."

ids = tok.encode(text, add_special_tokens=False)
print(len(ids))

# ์ฑ„ํŒ… ํ…œํ”Œ๋ฆฟ๊นŒ์ง€ ํฌํ•จํ•œ '์‹ค์ œ ์ž…๋ ฅ' ๊ธธ์ด
messages = [{"role": "user", "content": text}]
chat_ids = tok.apply_chat_template(messages, add_generation_prompt=True)
print(len(chat_ids))

์—ฌ๊ธฐ์„œ ํ•ต์‹ฌ์€ ๋‘ ๊ฐ€์ง€์˜ˆ์š”. ์ฒซ์งธ, add_special_tokens ๊ธฐ๋ณธ๊ฐ’์— ๋”ฐ๋ผ BOS ๊ฐ™์€ ํŠน์ˆ˜ ํ† ํฐ์ด ๋ถ™์–ด ์ˆซ์ž๊ฐ€ 1~2๊ฐœ ๋‹ฌ๋ผ์งˆ ์ˆ˜ ์žˆ์–ด์š”. ๋‘˜์งธ, ์ฑ„ํŒ… ๋ชจ๋ธ์€ apply_chat_template์œผ๋กœ ๋งŒ๋“  ํฌ๋งท์ด ์‹ค์ œ ๋ชจ๋ธ ์ž…๋ ฅ์ด๋ฏ€๋กœ, ์ปจํ…์ŠคํŠธ ํ•œ๋„ ๊ฒ€์‚ฌ๋Š” ์ด ๊ฐ’์œผ๋กœ ํ•ด์•ผ ์ •ํ™•ํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋ธ ๊ฐ€์ค‘์น˜๋Š” ํ•„์š” ์—†๊ณ  ํ† ํฌ๋‚˜์ด์ € ํŒŒ์ผ๋งŒ ๋‚ด๋ ค๋ฐ›์œผ๋‹ˆ ์ƒ๊ฐ๋ณด๋‹ค ๊ฐ€๋ฒผ์›Œ์š”. (์ผ๋ถ€ ๋ชจ๋ธ์€ ํ—ˆ๋ธŒ์—์„œ ์ด์šฉ ์•ฝ๊ด€ ๋™์˜ ํ›„ ํ† ํฐ ์ธ์ฆ์ด ํ•„์š”ํ•ด์š”.)

Anthropic Claude

Claude๋Š” ๋กœ์ปฌ ํ† ํฌ๋‚˜์ด์ € ๋Œ€์‹  ๊ณต์‹ ํ† ํฐ ์นด์šดํŒ… API๋ฅผ ์ œ๊ณตํ•ด์š”. ๋ฉ”์‹œ์ง€, ์‹œ์Šคํ…œ ํ”„๋กฌํ”„ํŠธ, ๋„๊ตฌ ์ •์˜๊นŒ์ง€ ๋„ฃ์–ด ๋ณด๋‚ด๋ฉด ์ž…๋ ฅ ํ† ํฐ ์ˆ˜๋ฅผ ๋Œ๋ ค์ค๋‹ˆ๋‹ค.

import anthropic

client = anthropic.Anthropic()  # ํ™˜๊ฒฝ๋ณ€์ˆ˜ ANTHROPIC_API_KEY ์‚ฌ์šฉ
resp = client.messages.count_tokens(
    model="claude-sonnet-4-5",   # ์‚ฌ์šฉ ์ค‘์ธ ๋ชจ๋ธ ID๋กœ ๋ฐ”๊พธ์„ธ์š”
    system="๋„ˆ๋Š” ์ฝ”๋“œ ๋ฆฌ๋ทฐ์–ด์•ผ.",
    messages=[{"role": "user", "content": "์ด ํ•จ์ˆ˜ ๋ฆฌ๋ทฐํ•ด์ค˜: def f(x): return x*2"}],
)
print(resp.input_tokens)

Google Gemini

from google import genai

client = genai.Client()  # ํ™˜๊ฒฝ๋ณ€์ˆ˜ GEMINI_API_KEY ์‚ฌ์šฉ
resp = client.models.count_tokens(
    model="gemini-2.0-flash",   # ์‚ฌ์šฉ ์ค‘์ธ ๋ชจ๋ธ ID๋กœ ๋ฐ”๊พธ์„ธ์š”
    contents="ํ† ํฐ์„ ์„ธ์–ด ์ฃผ์„ธ์š”.",
)
print(resp.total_tokens)
๋กœ์ปฌ ๊ณ„์‚ฐ vs ์„œ๋ฒ„ ๊ณ„์‚ฐ์„œ๋ฒ„ ๊ณ„์‚ฐ ๋ฐฉ์‹์€ ์ •ํ™•ํ•˜์ง€๋งŒ ๋„คํŠธ์›Œํฌ ์™•๋ณต๊ณผ API ํ‚ค, ์š”์ฒญ ์ œํ•œ(rate limit)์ด ๋”ฐ๋ผ์™€์š”. ์ˆ˜์‹ญ๋งŒ ๊ฐœ ๋ฌธ์„œ๋ฅผ ํ›‘๋Š” ์ผ๊ด„ ์ž‘์—…์ด๋ผ๋ฉด "์ƒ˜ํ”Œ ๋ช‡๋ฐฑ ๊ฐœ๋Š” ์„œ๋ฒ„๋กœ ์ •ํ™•ํžˆ ์žฌ๊ณ , ๊ทธ ๋น„์œจ๋กœ ์ „์ฒด๋ฅผ ์ถ”์ •"ํ•˜๋Š” ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ์ „๋žต์ด ์‹ค์šฉ์ ์ž…๋‹ˆ๋‹ค. ๋ชจ๋ธ ID์™€ SDK ์‹œ๊ทธ๋‹ˆ์ฒ˜๋Š” ์—…๋ฐ์ดํŠธ๊ฐ€ ์žฆ์œผ๋‹ˆ ๊ณต์‹ ๋ฌธ์„œ์—์„œ ์ตœ์‹  ๊ฐ’์„ ํ™•์ธํ•˜์„ธ์š”.

7. ์„ค๊ณ„๋„ ๊ทธ๋ฆฌ๊ธฐ: ์“ธ๋ชจ ์žˆ๋Š” ๊ณ„์‚ฐ๊ธฐ์˜ ๊ตฌ์กฐ ๐Ÿ—๏ธ

์ด์ œ ์žฅ๋‚œ๊ฐ์„ ๋„๊ตฌ๋กœ ํ‚ค์›Œ ๋ด…์‹œ๋‹ค. ์‹ค์ œ๋กœ ์“ธ ๋งŒํ•œ ๊ณ„์‚ฐ๊ธฐ๋Š” ๋ณดํ†ต ์ด๋Ÿฐ ๊ธฐ๋Šฅ์ด ํ•„์š”ํ•ด์š”.

โ‘  ๋ฐฑ์—”๋“œ ์ถ”์ƒํ™”tiktoken / HF / ์„œ๋ฒ„ API๋ฅผ ๊ฐ™์€ ์ธํ„ฐํŽ˜์ด์Šค(count(text) โ†’ int)๋กœ ๊ฐ์‹ธ๊ธฐ
โ‘ก ์ž…๋ ฅ ๋‹ค์–‘ํ™”๋ฌธ์ž์—ด, ํŒŒ์ผ, ํด๋” ์ „์ฒด, ํ‘œ์ค€์ž…๋ ฅ(ํŒŒ์ดํ”„)
โ‘ข ๋ฆฌํฌํŠธํŒŒ์ผ๋ณ„ ํ† ํฐ ์ˆ˜, ํ•ฉ๊ณ„, ๊ธ€์ž๋‹น ํ† ํฐ ๋น„์œจ, ํ•œ๋„ ๋Œ€๋น„ %
โ‘ฃ ๋น„์šฉ ์ถ”์ •๋‹จ๊ฐ€๋Š” ์„ค์ • ํŒŒ์ผ๋กœ ๋ถ„๋ฆฌ (๊ฐ€๊ฒฉ์€ ์ˆ˜์‹œ๋กœ ๋ฐ”๋€Œ๋‹ˆ๊นŒ์š”!)
โ‘ค ์ฒญํฌ ๋ถ„ํ• ์ตœ๋Œ€ ํ† ํฐยท๊ฒน์นจ(overlap) ์ง€์ •ํ•ด RAG์šฉ ์กฐ๊ฐ ์ƒ์„ฑ
โ‘ฅ ์„ฑ๋Šฅ์ธ์ฝ”๋” ์บ์‹ฑ, ๋ฐฐ์น˜ ์ธ์ฝ”๋”ฉ, ๋Œ€์šฉ๋Ÿ‰ ํŒŒ์ผ ์ฒ˜๋ฆฌ
๊ณ„์‚ฐ๊ธฐ ๋‚ด๋ถ€ ํ๋ฆ„๋„ ํ…์ŠคํŠธยทํŒŒ์ผยทํด๋” CLI ์ธ์ž ํ•ด์„ ๋ชจ๋ธ โ†’ ๋ฐฑ์—”๋“œ ์„ ํƒ TokenCounter.count(text) tiktoken ยท HF ยท ์„œ๋ฒ„ API ํ† ํฐ ๋ฆฌํฌํŠธ ๋น„์šฉยทํ•œ๋„ ๊ฒฝ๊ณ  ์ฒญํฌ ํŒŒ์ผ ์ถœ๋ ฅ

8. ์‹ค์ „ ์ฝ”๋“œ: ๋ฐฑ์—”๋“œ ์ถ”์ƒํ™”๋ถ€ํ„ฐ ๐Ÿงฑ

ํ•˜๋‚˜์”ฉ ์Œ“์•„ ๋ณผ๊ฒŒ์š”. ๋จผ์ € "๋ฌด์—‡์œผ๋กœ ์„ธ๋“  ๊ฐ™์€ ๋ชจ์–‘"์œผ๋กœ ์“ฐ๊ธฐ ์œ„ํ•œ ์ถ”์ƒ ํด๋ž˜์Šค์˜ˆ์š”. ํŒŒ์ผ ์ด๋ฆ„์€ tokcalc.py๋กœ ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

from __future__ import annotations
from abc import ABC, abstractmethod
from functools import lru_cache
import tiktoken


class TokenCounter(ABC):
    name: str = "base"
    exact: bool = True   # ๋Œ€์ƒ ๋ชจ๋ธ ๊ธฐ์ค€์œผ๋กœ ์ •ํ™•ํ•œ ๊ฐ’์ธ๊ฐ€?

    @abstractmethod
    def count(self, text: str) -> int: ...

    def count_many(self, texts: list[str]) -> list[int]:
        return [self.count(t) for t in texts]


@lru_cache(maxsize=None)
def _get_encoding(name: str):
    # ์ธ์ฝ”๋” ๋กœ๋”ฉ์€ ๋น„์‹ธ๋ฏ€๋กœ ํ•œ ๋ฒˆ๋งŒ
    return tiktoken.get_encoding(name)


class TiktokenCounter(TokenCounter):
    def __init__(self, model: str | None = None, encoding: str | None = None,
                 fallback: str = "o200k_base"):
        if encoding:
            self.enc = _get_encoding(encoding)
        else:
            try:
                self.enc = tiktoken.encoding_for_model(model or "gpt-4o")
            except KeyError:
                self.enc = _get_encoding(fallback)
                self.exact = False      # ๋ชจ๋ฅด๋Š” ๋ชจ๋ธ โ†’ ์ถ”์ •์น˜
        self.name = f"tiktoken:{self.enc.name}"

    def count(self, text: str) -> int:
        return len(self.enc.encode(text, disallowed_special=()))

    def count_many(self, texts: list[str]) -> list[int]:
        # ๋‚ด๋ถ€์ ์œผ๋กœ ์Šค๋ ˆ๋“œ๋ฅผ ์จ์„œ ์—ฌ๋Ÿฌ ํ…์ŠคํŠธ๋ฅผ ํ•œ ๋ฒˆ์— ์ธ์ฝ”๋”ฉ
        batches = self.enc.encode_batch(texts, disallowed_special=())
        return [len(b) for b in batches]

    def encode(self, text: str) -> list[int]:
        return self.enc.encode(text, disallowed_special=())

    def decode(self, ids: list[int]) -> str:
        return self.enc.decode(ids)


class HFCounter(TokenCounter):
    def __init__(self, repo_id: str):
        from transformers import AutoTokenizer   # ํ•„์š”ํ•  ๋•Œ๋งŒ ์ž„ํฌํŠธ
        self.tok = AutoTokenizer.from_pretrained(repo_id)
        self.name = f"hf:{repo_id}"

    def count(self, text: str) -> int:
        return len(self.tok.encode(text, add_special_tokens=False))

    def encode(self, text: str) -> list[int]:
        return self.tok.encode(text, add_special_tokens=False)

    def decode(self, ids: list[int]) -> str:
        return self.tok.decode(ids)


def make_counter(model: str) -> TokenCounter:
    """'hf:์ €์žฅ์†ŒID' ํ˜•์‹์ด๋ฉด HF, ์•„๋‹ˆ๋ฉด tiktoken."""
    if model.startswith("hf:"):
        return HFCounter(model[3:])
    return TiktokenCounter(model=model)

ํฌ์ธํŠธ๋ฅผ ์งš์–ด ๋ณด๋ฉด์š”.

๐Ÿ‘‰ lru_cache๋กœ ์ธ์ฝ”๋” ์บ์‹ฑ: tiktoken ์ธ์ฝ”๋”๋Š” ์ฒ˜์Œ ๋กœ๋“œํ•  ๋•Œ ์–ดํœ˜ ํŒŒ์ผ์„ ์ฝ๊ณ (์ตœ์ดˆ์—” ๋‹ค์šด๋กœ๋“œ๊นŒ์ง€) ๋‚ด๋ถ€ ์ž๋ฃŒ๊ตฌ์กฐ๋ฅผ ๋งŒ๋“ค์–ด์š”. ํŒŒ์ผ 1,000๊ฐœ๋ฅผ ์…€ ๋•Œ๋งˆ๋‹ค ์ƒˆ๋กœ ๋งŒ๋“ค๋ฉด ์‹œ๊ฐ„ ๋‚ญ๋น„์ฃ .
๐Ÿ‘‰ exact ํ”Œ๋ž˜๊ทธ: ํด๋ฐฑ์œผ๋กœ ๋–จ์–ด์กŒ๋‹ค๋ฉด ์‚ฌ์šฉ์ž์—๊ฒŒ "์ด๊ฑด ์ถ”์ •์น˜์˜ˆ์š”"๋ผ๊ณ  ์ •์งํ•˜๊ฒŒ ์•Œ๋ ค ์ฃผ๊ธฐ ์œ„ํ•œ ์žฅ์น˜์˜ˆ์š”. ์ด๋Ÿฐ ๋””ํ…Œ์ผ์ด ๋„๊ตฌ์˜ ์‹ ๋ขฐ๋„๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
๐Ÿ‘‰ ์ง€์—ฐ ์ž„ํฌํŠธ: transformers๋Š” ๋ฌด๊ฑฐ์šด ํŒจํ‚ค์ง€๋ผ HF ๋ฐฑ์—”๋“œ๋ฅผ ์“ธ ๋•Œ๋งŒ ๋ถˆ๋Ÿฌ์™€์š”. CLI ์‹œ์ž‘ ์†๋„๊ฐ€ ํ™• ๋‹ฌ๋ผ์ง‘๋‹ˆ๋‹ค.

์˜คํ”„๋ผ์ธ ํ™˜๊ฒฝ์—์„œ tiktoken ์“ฐ๊ธฐtiktoken์€ ์ฒ˜์Œ ์ธ์ฝ”๋”ฉ์„ ๋ถˆ๋Ÿฌ์˜ฌ ๋•Œ ์ธํ„ฐ๋„ท์—์„œ ์–ดํœ˜ ํŒŒ์ผ์„ ๋ฐ›์•„ ๋กœ์ปฌ ์บ์‹œ์— ์ €์žฅํ•ด์š”. ์‚ฌ๋‚ด ํ์‡„๋ง ์„œ๋ฒ„๋ผ๋ฉด ์ธํ„ฐ๋„ท ๋˜๋Š” ๊ณณ์—์„œ ํ•œ ๋ฒˆ ์‹คํ–‰ํ•ด ์บ์‹œ๋ฅผ ๋งŒ๋“  ๋’ค, ๊ทธ ํด๋”๋ฅผ ์˜ฎ๊ธฐ๊ณ  ํ™˜๊ฒฝ๋ณ€์ˆ˜ TIKTOKEN_CACHE_DIR๋กœ ์œ„์น˜๋ฅผ ์ง€์ •ํ•˜๋Š” ๋ฐฉ์‹์ด ํ”ํžˆ ์“ฐ์—ฌ์š”.

9. ํŒŒ์ผยทํด๋” ์ผ๊ด„ ๊ณ„์‚ฐ๊ณผ ๋ฆฌํฌํŠธ ๐Ÿ“‚

์ด์ œ ์ง„์งœ "์ž๋™" ๊ณ„์‚ฐ๊ธฐ๋‹ค์šด ๋ถ€๋ถ„์ด์—์š”. ํด๋”๋ฅผ ํ†ต์งธ๋กœ ๋„ฃ์œผ๋ฉด ํŒŒ์ผ๋ณ„ ํ† ํฐ ์ˆ˜๋ฅผ ์ซ™ ๋ฝ‘์•„ ์ฃผ๋Š” ๊ธฐ๋Šฅ์ž…๋‹ˆ๋‹ค.

from dataclasses import dataclass
from pathlib import Path

TEXT_EXTS = {".txt", ".md", ".py", ".json", ".csv", ".html", ".js", ".ts", ".yaml", ".yml"}


@dataclass
class FileStat:
    path: str
    chars: int
    tokens: int

    @property
    def tokens_per_char(self) -> float:
        return self.tokens / self.chars if self.chars else 0.0


def read_text(path: Path) -> str | None:
    for encoding in ("utf-8", "cp949"):     # ๊ตญ๋‚ด ๋ ˆ๊ฑฐ์‹œ ํŒŒ์ผ ๋Œ€๋น„
        try:
            return path.read_text(encoding=encoding)
        except UnicodeDecodeError:
            continue
    return None   # ๋ฐ”์ด๋„ˆ๋ฆฌ ๋“ฑ์€ ๊ฑด๋„ˆ๋œ€


def iter_files(target: Path):
    if target.is_file():
        yield target
        return
    for p in sorted(target.rglob("*")):
        if p.is_file() and p.suffix.lower() in TEXT_EXTS:
            yield p


def scan(target: Path, counter: TokenCounter) -> list[FileStat]:
    paths, texts = [], []
    for p in iter_files(target):
        t = read_text(p)
        if t is not None:
            paths.append(p)
            texts.append(t)
    counts = counter.count_many(texts)
    return [FileStat(str(p), len(t), n) for p, t, n in zip(paths, texts, counts)]

cp949 ํด๋ฐฑ์€ ํ•œ๊ตญ ๊ฐœ๋ฐœ์ž๋ผ๋ฉด ๊ณต๊ฐํ•  ๋””ํ…Œ์ผ์ด์—์š”. ์˜ค๋ž˜๋œ ์œˆ๋„์šฐ ํ™˜๊ฒฝ์—์„œ ๋งŒ๋“  ๋ฉ”๋ชจ์žฅ ํŒŒ์ผ์€ UTF-8์ด ์•„๋‹Œ ๊ฒฝ์šฐ๊ฐ€ ๊ฝค ์žˆ๊ฑฐ๋“ ์š”. ๊ธ€์ž ์ˆ˜(chars)๋„ ๊ฐ™์ด ์ €์žฅํ•ด ๋‘๋ฉด "๊ธ€์ž๋‹น ํ† ํฐ ๋น„์œจ"์„ ๋ณผ ์ˆ˜ ์žˆ์–ด์„œ, ์šฐ๋ฆฌ ๋ฐ์ดํ„ฐ๊ฐ€ ์–ด๋–ค ํŠน์„ฑ์ธ์ง€ ๊ฐ ์žก๋Š” ๋ฐ ์•„์ฃผ ์ข‹์•„์š”.

ํŒ: ๋น„์œจ์€ "๋‚ด ๋ฐ์ดํ„ฐ"๋กœ ์ง์ ‘ ์žฌ์„ธ์š”์ธํ„ฐ๋„ท์—์„œ ๋– ๋„๋Š” "ํ•œ๊ตญ์–ด๋Š” ๋ช‡ ๋ฐฐ" ๊ฐ™์€ ์ˆซ์ž ๋Œ€์‹ , ๋‚ด ์„œ๋น„์Šค์˜ ์‹ค์ œ ํ”„๋กฌํ”„ํŠธ ์ƒ˜ํ”Œ ์ˆ˜๋ฐฑ ๊ฐœ๋กœ ๊ธ€์ž๋‹น ํ† ํฐ ๋น„์œจ์„ ์ธก์ •ํ•ด ๋‘์„ธ์š”. ๊ทธ ๋น„์œจ์ด ๊ณง ๋‚ด ์„œ๋น„์Šค ์ „์šฉ ๊ฒฌ์  ๊ณต์‹์ด ๋ฉ๋‹ˆ๋‹ค. ๋ฌธ์ฒด(๊ตฌ์–ด์ฒด/๋ฌธ์–ด์ฒด), ์˜์–ด ์„ž์ž„, ์ฝ”๋“œ ๋น„์ค‘์— ๋”ฐ๋ผ ๊ฝค ๋‹ฌ๋ผ์ ธ์š”.

10. ๋น„์šฉ ์ถ”์ •: ๋‹จ๊ฐ€๋Š” ์ฝ”๋“œ์— ๋ฐ•์ง€ ๋ง์ž ๐Ÿ’ฐ

์š”๊ธˆ์€ ๋ชจ๋ธ๋งˆ๋‹ค ๋‹ค๋ฅด๊ณ , ์ž…๋ ฅยท์ถœ๋ ฅ ๋‹จ๊ฐ€๊ฐ€ ๋”ฐ๋กœ์ด๋ฉฐ, ์บ์‹œ ํ• ์ธยท๋ฐฐ์น˜ ํ• ์ธ ๊ฐ™์€ ์˜ต์…˜๋„ ์žˆ๊ณ , ๋ฌด์—‡๋ณด๋‹ค ์ž์ฃผ ๋ฐ”๋€๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ๋‹จ๊ฐ€๋Š” ๋ณ„๋„ JSON ํŒŒ์ผ๋กœ ๋นผ๋Š” ๊ฒŒ ์ •์„์ด์—์š”.

# prices.json ์˜ˆ์‹œ โ€” ์ˆซ์ž๋Š” ํ˜•์‹ ์„ค๋ช…์šฉ ์ž๋ฆฌํ‘œ์‹œ์ž์ž…๋‹ˆ๋‹ค.
# ๋ฐ˜๋“œ์‹œ ๊ฐ ์‚ฌ ๊ณต์‹ ๊ฐ€๊ฒฉ ํŽ˜์ด์ง€์˜ ์ตœ์‹  ๊ฐ’์„ ๋„ฃ์œผ์„ธ์š”.
{
  "gpt-4o":      {"input_per_1m": 0.0, "output_per_1m": 0.0},
  "my-local-llm":{"input_per_1m": 0.0, "output_per_1m": 0.0}
}
import json

def estimate_cost(model: str, input_tokens: int, expected_output_tokens: int,
                  price_file: str = "prices.json") -> float | None:
    with open(price_file, encoding="utf-8") as f:
        prices = json.load(f)
    p = prices.get(model)
    if p is None:
        return None
    return (input_tokens * p["input_per_1m"]
            + expected_output_tokens * p["output_per_1m"]) / 1_000_000

์—ฌ๊ธฐ์„œ ๋งŽ์ด๋“ค ๋†“์น˜๋Š” ๊ฒŒ ์ถœ๋ ฅ ํ† ํฐ์ด์—์š”. ์ž…๋ ฅ์€ ๋ฏธ๋ฆฌ ์ •ํ™•ํžˆ ์…€ ์ˆ˜ ์žˆ์ง€๋งŒ ์ถœ๋ ฅ์€ ๋ชจ๋ธ์ด ์–ผ๋งˆ๋‚˜ ๊ธธ๊ฒŒ ๋‹ตํ• ์ง€ ๋ชจ๋ฅด์ž–์•„์š”. ๊ทธ๋ž˜์„œ ์‹ค๋ฌด์—์„œ๋Š” ๋ณดํ†ต ์„ธ ๊ฐ€์ง€ ๋ฐฉ๋ฒ• ์ค‘ ํ•˜๋‚˜๋ฅผ ์จ์š”.

์ƒํ•œ ๊ธฐ์ค€max_tokens ์„ค์ •๊ฐ’์„ ๊ทธ๋Œ€๋กœ ์ถœ๋ ฅ๋Ÿ‰์œผ๋กœ ๊ฐ€์ • โ†’ ์ตœ์•…์˜ ๋น„์šฉ
ํ†ต๊ณ„ ๊ธฐ์ค€๊ณผ๊ฑฐ usage ๋กœ๊ทธ์—์„œ ํ‰๊ท ยท์ƒ์œ„ ๋ฐฑ๋ถ„์œ„ ์ถœ๋ ฅ ๊ธธ์ด๋ฅผ ์‚ฌ์šฉ
๋น„์œจ ๊ธฐ์ค€์š”์•ฝ์ด๋ฉด ์ž…๋ ฅ์˜ ์ผ๋ถ€, ๋ฒˆ์—ญ์ด๋ฉด ์ž…๋ ฅ๊ณผ ๋น„์Šทํ•œ ๊ทœ๋ชจ ๋“ฑ ์ž‘์—… ์œ ํ˜•๋ณ„ ๊ฐ€์ •
์ถ”๋ก (reasoning) ๋ชจ๋ธ ์ฃผ์˜์ตœ๊ทผ์˜ ์ถ”๋ก ํ˜• ๋ชจ๋ธ์€ ํ™”๋ฉด์— ๋ณด์ด๋Š” ๋‹ต๋ณ€ ์™ธ์— ๋‚ด๋ถ€ ์ถ”๋ก  ๊ณผ์ •์—๋„ ํ† ํฐ์„ ์“ฐ๊ณ , ์ด ํ† ํฐ๋„ ์ถœ๋ ฅ ํ† ํฐ์œผ๋กœ ๊ณผ๊ธˆ๋˜๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์–ด์š”. ๋ˆˆ์— ๋ณด์ด๋Š” ๋‹ต์ด ์งง์•„๋„ ์ฒญ๊ตฌ ํ† ํฐ์€ ํด ์ˆ˜ ์žˆ์œผ๋‹ˆ, ์ด๋Ÿฐ ๋ชจ๋ธ์˜ ๋น„์šฉ ์ถ”์ •์€ ๊ผญ ์‹ค์ œ usage ๋กœ๊ทธ๋กœ ๋ณด์ •ํ•˜์„ธ์š”.

11. ํ† ํฐ ๊ธฐ์ค€ ์ฒญํฌ ๋ถ„ํ• ๊ธฐ โœ‚๏ธ

RAG(๊ฒ€์ƒ‰ ์ฆ๊ฐ• ์ƒ์„ฑ) ํŒŒ์ดํ”„๋ผ์ธ์„ ๋งŒ๋“ค ๋•Œ ๊ณ„์‚ฐ๊ธฐ์˜ ์ง„๊ฐ€๊ฐ€ ๋“œ๋Ÿฌ๋‚˜์š”. ๋ฌธ์„œ๋ฅผ "์ตœ๋Œ€ Nํ† ํฐ, ์•ž๋’ค Mํ† ํฐ ๊ฒน์นจ"์œผ๋กœ ์ž๋ฅด๋Š” ๊ธฐ๋Šฅ์ด์ฃ .

๊ฒน์นจ์ด ์žˆ๋Š” ์Šฌ๋ผ์ด๋”ฉ ์ฒญํฌ ๋ฌธ์„œ ์ „์ฒด ํ† ํฐ ์—ด ์ฒญํฌ 1 ์ฒญํฌ 2 ์ฒญํฌ 3 ๊ฒน์นจ ํ•œ ์นธ์”ฉ = ํฌ๊ธฐ โˆ’ ๊ฒน์นจ
def chunk_by_tokens(text: str, counter, max_tokens: int = 500,
                    overlap: int = 50) -> list[str]:
    if overlap >= max_tokens:
        raise ValueError("overlap์€ max_tokens๋ณด๋‹ค ์ž‘์•„์•ผ ํ•ฉ๋‹ˆ๋‹ค.")
    ids = counter.encode(text)
    step = max_tokens - overlap
    chunks = []
    for start in range(0, len(ids), step):
        piece = ids[start:start + max_tokens]
        chunks.append(counter.decode(piece))
        if start + max_tokens >= len(ids):
            break
    return chunks

๊ฐ„๋‹จํ•˜์ฃ ? ๊ทธ๋Ÿฐ๋ฐ ์ด ์ˆœ์ง„ํ•œ ๋ฒ„์ „์—๋Š” ํ•œ๊ตญ์–ด ์‚ฌ์šฉ์ž์—๊ฒŒ ์น˜๋ช…์ ์ธ ํ•จ์ •์ด ์žˆ์–ด์š”. ์•ž์„œ ๋งํ–ˆ๋“ฏ ๋ฐ”์ดํŠธ ์ˆ˜์ค€ BPE์—์„œ๋Š” ํ•œ๊ธ€ ํ•œ ๊ธ€์ž๊ฐ€ ์—ฌ๋Ÿฌ ํ† ํฐ์— ๊ฑธ์ณ ์žˆ์„ ์ˆ˜ ์žˆ์–ด์š”. ์ฒญํฌ ๊ฒฝ๊ณ„๊ฐ€ ํ•˜ํ•„ ๊ทธ ์ค‘๊ฐ„์— ๋–จ์–ด์ง€๋ฉด decode() ๊ฒฐ๊ณผ ๋์— ๊นจ์ง„ ๋ฌธ์ž(๏ฟฝ)๊ฐ€ ์ƒ๊น๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ์ฑ…: ๊ฒฝ๊ณ„๋ฅผ "์•ˆ์ „ํ•œ ๊ณณ"์œผ๋กœ ์˜ฎ๊ธฐ๊ธฐโ‘  ์ž˜๋ฆฐ ์กฐ๊ฐ์„ ๋””์ฝ”๋”ฉํ•ด ๋์— ๋Œ€์ฒด ๋ฌธ์ž(U+FFFD)๊ฐ€ ์žˆ์œผ๋ฉด ๊ฒฝ๊ณ„๋ฅผ ํ•œ๋‘ ํ† ํฐ ๋‹น๊ธฐ๊ฑฐ๋‚˜ ๋ฐ€์–ด ์กฐ์ •ํ•˜๊ธฐ
โ‘ก ๋” ์ข‹์€ ๋ฐฉ๋ฒ•: ๋จผ์ € ๋ฌธ๋‹จยท๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋‚˜๋ˆˆ ๋’ค, ํ† ํฐ ์ˆ˜๋ฅผ ์„ธ๋ฉด์„œ ํ•œ๋„๊นŒ์ง€ ์ด์–ด ๋ถ™์ด๊ธฐ. ์˜๋ฏธ ๋‹จ์œ„๊ฐ€ ๋ณด์กด๋ผ ๊ฒ€์ƒ‰ ํ’ˆ์งˆ๋„ ์ข‹์•„์ ธ์š”.
import re

def chunk_by_sentences(text: str, counter, max_tokens: int = 500) -> list[str]:
    # ๋งˆ์นจํ‘œยท๋ฌผ์Œํ‘œยท๋А๋‚Œํ‘œยท์ค„๋ฐ”๊ฟˆ ๊ธฐ์ค€์˜ ๋‹จ์ˆœ ๋ฌธ์žฅ ๋ถ„๋ฆฌ
    sentences = [s for s in re.split(r"(?<=[.!?ใ€‚])\s+|\n+", text) if s.strip()]
    chunks, buf, buf_tokens = [], [], 0
    for s in sentences:
        n = counter.count(s)
        if n > max_tokens:                       # ํ•œ ๋ฌธ์žฅ์ด ๋„ˆ๋ฌด ๊ธธ๋ฉด ํ† ํฐ ๋‹จ์œ„๋กœ ๊ฐ•์ œ ๋ถ„ํ• 
            if buf:
                chunks.append(" ".join(buf)); buf, buf_tokens = [], 0
            chunks.extend(chunk_by_tokens(s, counter, max_tokens, overlap=0))
            continue
        if buf_tokens + n > max_tokens:
            chunks.append(" ".join(buf)); buf, buf_tokens = [], 0
        buf.append(s); buf_tokens += n
    if buf:
        chunks.append(" ".join(buf))
    return chunks
"๋ถ€๋ถ„์˜ ํ•ฉ โ‰  ์ „์ฒด" ์ฃผ์˜๋ฌธ์žฅ๋ณ„ ํ† ํฐ ์ˆ˜๋ฅผ ๋”ํ•œ ๊ฐ’๊ณผ, ํ•ฉ์นœ ๋ฌธ์žฅ ์ „์ฒด๋ฅผ ํ•œ ๋ฒˆ์— ์„ผ ๊ฐ’์€ ์ •ํ™•ํžˆ ๊ฐ™์ง€ ์•Š์„ ์ˆ˜ ์žˆ์–ด์š”. ๊ณต๋ฐฑยท๊ตฌ๋‘์ ์ด ์•ž๋’ค ํ† ํฐ๊ณผ ๋ณ‘ํ•ฉ๋˜๋Š” ๋ฐฉ์‹์ด ๋‹ฌ๋ผ์ง€๊ธฐ ๋•Œ๋ฌธ์ด์—์š”. ํ•œ๋„๊ฐ€ ๋นก๋นกํ•˜๋‹ค๋ฉด ์™„์„ฑ๋œ ์ฒญํฌ๋ฅผ ๋งˆ์ง€๋ง‰์— ํ•œ ๋ฒˆ ๋” ์„ธ์„œ ๊ฒ€์ฆํ•˜์„ธ์š”. ์ฐจ์ด๋Š” ๋ณดํ†ต ์ž‘์ง€๋งŒ 0์€ ์•„๋‹™๋‹ˆ๋‹ค.

12. ๋ช…๋ น์ค„ ๋„๊ตฌ(CLI)๋กœ ๋ฌถ๊ธฐ ๐Ÿ–ฅ๏ธ

์ด์ œ ํ„ฐ๋ฏธ๋„์—์„œ

python tokcalc.py ./docs --model gpt-4o --limit 128000
์ฒ˜๋Ÿผ ์“ธ ์ˆ˜ ์žˆ๊ฒŒ ๋งŒ๋“ค์–ด ๋ด…์‹œ๋‹ค. ํ‘œ์ค€ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ argparse๋งŒ์œผ๋กœ ์ถฉ๋ถ„ํ•ด์š”.
import argparse
import sys

def main(argv=None):
    ap = argparse.ArgumentParser(description="ํŒŒ์ด์ฌ ํ† ํฐ ๊ธธ์ด ์ž๋™ ๊ณ„์‚ฐ๊ธฐ")
    ap.add_argument("target", nargs="?", help="ํŒŒ์ผ/ํด๋” ๊ฒฝ๋กœ (์ƒ๋žต ์‹œ ํ‘œ์ค€์ž…๋ ฅ)")
    ap.add_argument("--model", default="gpt-4o",
                    help="๋ชจ๋ธ๋ช… ๋˜๋Š” 'hf:์ €์žฅ์†ŒID'")
    ap.add_argument("--limit", type=int, default=None,
                    help="์ปจํ…์ŠคํŠธ ํ•œ๋„(ํ† ํฐ). ์ดˆ๊ณผ ํŒŒ์ผ ๊ฒฝ๊ณ ")
    ap.add_argument("--top", type=int, default=20, help="์ƒ์œ„ N๊ฐœ ํŒŒ์ผ๋งŒ ์ถœ๋ ฅ")
    ap.add_argument("--csv", help="๊ฒฐ๊ณผ๋ฅผ CSV๋กœ ์ €์žฅํ•  ๊ฒฝ๋กœ")
    args = ap.parse_args(argv)

    counter = make_counter(args.model)
    if not getattr(counter, "exact", True):
        print(f"โš ๏ธ  '{args.model}'์„ ๋ชจ๋ฆ…๋‹ˆ๋‹ค. {counter.name}๋กœ ์ถ”์ •ํ•ฉ๋‹ˆ๋‹ค.",
              file=sys.stderr)

    if args.target is None:                      # ํŒŒ์ดํ”„ ์ž…๋ ฅ: cat a.txt | python tokcalc.py
        text = sys.stdin.read()
        print(counter.count(text))
        return 0

    stats = scan(Path(args.target), counter)
    stats.sort(key=lambda s: s.tokens, reverse=True)
    total = sum(s.tokens for s in stats)

    print(f"๋ฐฑ์—”๋“œ: {counter.name} | ํŒŒ์ผ {len(stats)}๊ฐœ | ์ด {total:,} ํ† ํฐ\n")
    for s in stats[:args.top]:
        flag = ""
        if args.limit and s.tokens > args.limit:
            flag = "  ๐Ÿšจ ํ•œ๋„ ์ดˆ๊ณผ"
        print(f"{s.tokens:>10,}  {s.tokens_per_char:5.2f}/์ž  {s.path}{flag}")

    if args.csv:
        import csv
        with open(args.csv, "w", newline="", encoding="utf-8-sig") as f:
            w = csv.writer(f)
            w.writerow(["path", "chars", "tokens", "tokens_per_char"])
            for s in stats:
                w.writerow([s.path, s.chars, s.tokens, f"{s.tokens_per_char:.4f}"])
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

CSV ์ €์žฅ ์‹œ utf-8-sig๋ฅผ ์“ด ์ด์œ ๋Š” ์—‘์…€์—์„œ ํ•œ๊ธ€์ด ๊นจ์ง€์ง€ ์•Š๊ฒŒ ํ•˜๋ ค๋Š” ๊ฑฐ์˜ˆ์š”(BOM ํฌํ•จ). ์ด๋Ÿฐ ์‚ฌ์†Œํ•œ ๋ฐฐ๋ ค๊ฐ€ ๋™๋ฃŒ๋“คํ•œํ…Œ "์˜ค, ์ด๊ฑฐ ์ž˜ ๋งŒ๋“ค์—ˆ๋„ค" ์†Œ๋ฆฌ๋ฅผ ๋“ฃ๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค. ๐Ÿ˜Ž

ํŒ: ๊ฒฝ๊ณ ๋Š” stderr๋กœ์ถ”์ •์น˜ ๊ฒฝ๊ณ ๋ฅผ sys.stderr๋กœ ๋‚ด๋ณด๋‚ด๋ฉด, ๊ฒฐ๊ณผ๊ฐ’๋งŒ ๋‹ค๋ฅธ ๋ช…๋ น์–ด๋กœ ํŒŒ์ดํ”„ํ•  ๋•Œ ๊ฒฝ๊ณ  ๋ฌธ๊ตฌ๊ฐ€ ์„ž์ด์ง€ ์•Š์•„์š”. ์˜ˆ: cat prompt.txt | python tokcalc.py > count.txt

13. ํ™”๋ฉด์ด ์žˆ๋Š” ๊ณ„์‚ฐ๊ธฐ: Streamlit์œผ๋กœ 5๋ถ„ ์›น์•ฑ ๐ŸŒˆ

๋น„๊ฐœ๋ฐœ์ž ๋™๋ฃŒ(๊ธฐํš์ž, ๋งˆ์ผ€ํ„ฐ)๋„ ์“ฐ๊ฒŒ ํ•˜๋ ค๋ฉด ์›น UI๊ฐ€ ์ตœ๊ณ ์˜ˆ์š”. Streamlit์„ ์“ฐ๋ฉด ์ •๋ง ๊ธˆ๋ฐฉ ๋งŒ๋“ค์–ด์ง‘๋‹ˆ๋‹ค.

# app.py โ€” ์‹คํ–‰: streamlit run app.py
import streamlit as st
from tokcalc import make_counter

st.title("๐Ÿงฎ ํ† ํฐ ๊ธธ์ด ๊ณ„์‚ฐ๊ธฐ")
model = st.selectbox("๋ชจ๋ธ", ["gpt-4o", "gpt-4", "hf:Qwen/Qwen2.5-7B-Instruct"])
limit = st.number_input("์ปจํ…์ŠคํŠธ ํ•œ๋„(ํ† ํฐ)", value=128000, step=1000)
text = st.text_area("ํ…์ŠคํŠธ๋ฅผ ๋ถ™์—ฌ ๋„ฃ์œผ์„ธ์š”", height=250)

@st.cache_resource
def get_counter(m):
    return make_counter(m)

if text:
    counter = get_counter(model)
    ids = counter.encode(text)
    n = len(ids)
    col1, col2, col3 = st.columns(3)
    col1.metric("ํ† ํฐ ์ˆ˜", f"{n:,}")
    col2.metric("๊ธ€์ž ์ˆ˜", f"{len(text):,}")
    col3.metric("ํ•œ๋„ ์‚ฌ์šฉ๋ฅ ", f"{n / limit:.1%}")
    st.progress(min(n / limit, 1.0))

    with st.expander("ํ† ํฐ ์ชผ๊ฐœ์ง„ ๋ชจ์–‘ ๋ณด๊ธฐ (์•ž 200๊ฐœ)"):
        pieces = [counter.decode([i]) for i in ids[:200]]
        st.write(" | ".join(repr(p) for p in pieces))

st.cache_resource๋Š” ํ† ํฌ๋‚˜์ด์ € ๊ฐ™์€ ๋ฌด๊ฑฐ์šด ๊ฐ์ฒด๋ฅผ ์„ธ์…˜ ๊ฐ„์— ์žฌ์‚ฌ์šฉํ•˜๊ฒŒ ํ•ด ์ค˜์š”. ๋งˆ์ง€๋ง‰ "์ชผ๊ฐœ์ง„ ๋ชจ์–‘ ๋ณด๊ธฐ"์—์„œ ํ† ํฐ์„ ํ•˜๋‚˜์”ฉ ๋””์ฝ”๋”ฉํ•˜๋ฉด, ์•ž์—์„œ ๋งํ•œ ๋Œ€๋กœ ํ•œ๊ธ€ ๊ธ€์ž๊ฐ€ ๋ฐ”์ดํŠธ ์กฐ๊ฐ์œผ๋กœ ๋‚˜๋‰œ ๊ฒฝ์šฐ ๏ฟฝ์ฒ˜๋Ÿผ ๋ณด์ผ ์ˆ˜ ์žˆ์–ด์š”. ๋ฒ„๊ทธ๊ฐ€ ์•„๋‹ˆ๋ผ ๋ฐ”์ดํŠธ ์ˆ˜์ค€ BPE์˜ ์ •์ƒ ๋™์ž‘์ด๋‹ˆ ๋†€๋ผ์ง€ ๋งˆ์„ธ์š”! tiktoken์—๋Š” ํ† ํฐ๋ณ„ ์›๋ณธ ๋ฐ”์ดํŠธ๋ฅผ ๋Œ๋ ค์ฃผ๋Š” decode_single_token_bytes()๋„ ์žˆ์–ด์„œ, ๋ฐ”์ดํŠธ ๋‹จ์œ„๋กœ ์‹œ๊ฐํ™”ํ•˜๊ณ  ์‹ถ์„ ๋•Œ ์“ธ ์ˆ˜ ์žˆ์–ด์š”.

์ด๋ ‡๊ฒŒ ๋งŒ๋“  ๋„๊ตฌ๋Š” ํŒ€ ๋‚ด๋ถ€ ์œ„ํ‚ค์— ๋งํฌ๋งŒ ๊ฑธ์–ด ๋‘ฌ๋„ ๋ฐ˜์‘์ด ์ข‹์•„์š”. ํ”„๋กฌํ”„ํŠธ ์—”์ง€๋‹ˆ์–ด๋ง์ด๋‚˜ AI ์ž๋™ํ™” ์ž‘์—…์„ ์™ธ์ฃผ๋กœ ์ฃผ๊ณ ๋ฐ›๋Š” ์ผ์ด ๋Š˜๋ฉด์„œ, ์žฌ๋Šฅ๋„ท ๊ฐ™์€ ์žฌ๋Šฅ ๊ฑฐ๋ž˜ ํ”Œ๋žซํผ์—์„œ๋„ "ํ”„๋กฌํ”„ํŠธ ์ตœ์ ํ™”", "RAG ๊ตฌ์ถ•" ๊ฐ™์€ ์˜๋ขฐ๋ฅผ ํ•  ๋•Œ ํ† ํฐ ์˜ˆ์‚ฐ์„ ์ˆ˜์น˜๋กœ ์ œ์‹œํ•˜๋ฉด ๊ฒฌ์  ์†Œํ†ต์ด ํ›จ์”ฌ ๋ช…ํ™•ํ•ด์ง‘๋‹ˆ๋‹ค.

14. ์„ฑ๋Šฅ ์ด์•ผ๊ธฐ: ๋Œ€์šฉ๋Ÿ‰์„ ๋น ๋ฅด๊ฒŒ ๐Ÿš€

๋ช‡ ๊ฐœ ํŒŒ์ผ์ด๋ฉด ๋ญ๋“  ๋น ๋ฅด์ง€๋งŒ, ์ˆ˜ GB ๋กœ๊ทธ๋‚˜ ์ˆ˜์‹ญ๋งŒ ๊ฐœ ๋ฌธ์„œ๋ผ๋ฉด ์ด์•ผ๊ธฐ๊ฐ€ ๋‹ฌ๋ผ์ ธ์š”.

๊ธฐ๋ฒ•์„ค๋ช…์–ธ์ œ ์“ฐ๋‚˜
์ธ์ฝ”๋” ์บ์‹ฑget_encoding ๊ฒฐ๊ณผ ์žฌ์‚ฌ์šฉ (tiktoken ๋‚ด๋ถ€์—๋„ ์บ์‹œ๊ฐ€ ์žˆ์ง€๋งŒ ๋ช…์‹œ์ ์œผ๋กœ ๊ด€๋ฆฌํ•˜๋ฉด ๊น”๋”)ํ•ญ์ƒ
encode_batch์—ฌ๋Ÿฌ ํ…์ŠคํŠธ๋ฅผ ์Šค๋ ˆ๋“œ๋กœ ๋ณ‘๋ ฌ ์ธ์ฝ”๋”ฉ์งง์€ ํ…์ŠคํŠธ ๋‹ค์ˆ˜
HF Fast ํ† ํฌ๋‚˜์ด์ €Rust๋กœ ๊ตฌํ˜„๋œ tokenizers ๊ธฐ๋ฐ˜. ๋ฆฌ์ŠคํŠธ๋ฅผ ํ•œ ๋ฒˆ์— ๋„ฃ์œผ๋ฉด ๋ฐฐ์น˜ ์ฒ˜๋ฆฌHF ๋ชจ๋ธ ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ
๋ฉ€ํ‹ฐํ”„๋กœ์„ธ์‹ฑconcurrent.futures.ProcessPoolExecutor๋กœ ํŒŒ์ผ ๋‹จ์œ„ ๋ถ„์‚ฐํŒŒ์ผ ์ˆ˜๊ฐ€ ์•„์ฃผ ๋งŽ์„ ๋•Œ
์ค„ ๋‹จ์œ„ ์ŠคํŠธ๋ฆฌ๋ฐ๊ฑฐ๋Œ€ํ•œ ํŒŒ์ผ์„ ํ†ต์งธ๋กœ ๋ฉ”๋ชจ๋ฆฌ์— ์˜ฌ๋ฆฌ์ง€ ์•Š๊ณ  ์ค„ยท๋ธ”๋ก ๋‹จ์œ„๋กœ ์„ธ์„œ ํ•ฉ์‚ฐ๋ฉ”๋ชจ๋ฆฌ ์ œํ•œ ํ™˜๊ฒฝ (์•ฝ๊ฐ„์˜ ์˜ค์ฐจ ํ—ˆ์šฉ)
def count_large_file(path: Path, counter, block_lines: int = 2000) -> int:
    """๊ฑฐ๋Œ€ ํŒŒ์ผ์„ ๋ธ”๋ก ๋‹จ์œ„๋กœ ์„ธ์„œ ํ•ฉ์‚ฐ. ๋ธ”๋ก ๊ฒฝ๊ณ„์—์„œ ๋ฏธ์„ธํ•œ ์˜ค์ฐจ ๊ฐ€๋Šฅ."""
    total, buf = 0, []
    with path.open(encoding="utf-8", errors="replace") as f:
        for line in f:
            buf.append(line)
            if len(buf) >= block_lines:
                total += counter.count("".join(buf)); buf.clear()
    if buf:
        total += counter.count("".join(buf))
    return total
์™œ tiktoken์€ ๋น ๋ฅผ๊นŒ?tiktoken์˜ ํ•ต์‹ฌ ์ธ์ฝ”๋”ฉ ๋กœ์ง์€ Rust๋กœ ์ž‘์„ฑ๋˜์–ด ์žˆ๊ณ  ํŒŒ์ด์ฌ์€ ์–‡์€ ๋ž˜ํผ ์—ญํ• ์„ ํ•ด์š”. ๊ทธ๋ž˜์„œ ์ˆœ์ˆ˜ ํŒŒ์ด์ฌ BPE ๊ตฌํ˜„๋ณด๋‹ค ํ›จ์”ฌ ๋น ๋ฆ…๋‹ˆ๋‹ค. ๊ทธ๋ž˜๋„ ์ง„์งœ ๋ณ‘๋ชฉ์ด ์–ด๋””์ธ์ง€๋Š” cProfile์ด๋‚˜ time.perf_counter()๋กœ ์ง์ ‘ ์žฌ ๋ณด๋Š” ๊ฒŒ ์ •๋‹ต์ด์—์š”. ํŒŒ์ผ I/O๊ฐ€ ๋ณ‘๋ชฉ์ธ ๊ฒฝ์šฐ๋„ ๋งŽ๊ฑฐ๋“ ์š”.

15. ํ…Œ์ŠคํŠธ๋กœ ๊ณ„์‚ฐ๊ธฐ๋ฅผ ๋ฏฟ์„ ์ˆ˜ ์žˆ๊ฒŒ โœ…

์ˆซ์ž๋ฅผ ๋‚ด๋†“๋Š” ๋„๊ตฌ๋Š” ํ‹€๋ฆฌ๋ฉด ์‹ ๋ขฐ๋ฅผ ์žƒ์–ด์š”. pytest๋กœ ํ•ต์‹ฌ ์„ฑ์งˆ์„ ๊ฒ€์ฆํ•ด ๋‘ก์‹œ๋‹ค. ํŠน์ • ํ† ํฐ ์ˆ˜๋ฅผ ํ•˜๋“œ์ฝ”๋”ฉํ•˜๊ธฐ๋ณด๋‹ค "์„ฑ์งˆ(property)"์„ ๊ฒ€์‚ฌํ•˜๋Š” ๊ฒŒ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๋ฒ„์ „ ๋ณ€ํ™”์—๋„ ํŠผํŠผํ•ด์š”.

# test_tokcalc.py
import pytest
from tokcalc import TiktokenCounter, chunk_by_tokens, chunk_by_sentences

@pytest.fixture(scope="module")
def counter():
    return TiktokenCounter(encoding="o200k_base")

def test_empty_is_zero(counter):
    assert counter.count("") == 0

def test_roundtrip(counter):
    text = "ํ† ํฐ ๊ณ„์‚ฐ๊ธฐ ํ…Œ์ŠคํŠธ ๐Ÿงช with English & code: print('hi')"
    assert counter.decode(counter.encode(text)) == text

def test_special_token_text_does_not_crash(counter):
    assert counter.count("๋ณธ๋ฌธ <|endoftext|> ๋") > 0

def test_batch_equals_single(counter):
    texts = ["๊ฐ€๋‚˜๋‹ค", "hello world", "x = 1\n"]
    assert counter.count_many(texts) == [counter.count(t) for t in texts]

def test_chunks_respect_limit(counter):
    text = "๋ฌธ์žฅ์ž…๋‹ˆ๋‹ค. " * 500
    for c in chunk_by_sentences(text, counter, max_tokens=100):
        assert counter.count(c) <= 110   # ์žฌ๊ฒฐํ•ฉ ์˜ค์ฐจ ์—ฌ์œ 

def test_unknown_model_falls_back():
    c = TiktokenCounter(model="totally-unknown-model")
    assert c.exact is False

ํŠนํžˆ test_special_token_text_does_not_crash๋Š” ์•ž์—์„œ ๋งํ•œ ํŠน์ˆ˜ ํ† ํฐ ํ•จ์ •์ด ๋‹ค์‹œ ์ƒ๊ธฐ์ง€ ์•Š๊ฒŒ ๋ง‰์•„ ์ฃผ๋Š” "ํšŒ๊ท€ ๋ฐฉ์ง€ ํ…Œ์ŠคํŠธ"์˜ˆ์š”. ํ•œ ๋ฒˆ ๊ฒช์€ ๋ฒ„๊ทธ๋Š” ํ…Œ์ŠคํŠธ๋กœ ๋ฐ•์ œํ•ด ๋‘๋Š” ๊ฒŒ ๊ฐœ๋ฐœ์ž์˜ ์ƒ์กด ์ „๋žต์ž…๋‹ˆ๋‹ค. ๐Ÿงท

16. ํ•œ๊ตญ์–ด ํ† ํฐ์„ ์ค„์ด๋Š” ์‹ค์ „ ๊ฐ๊ฐ ๐Ÿ‡ฐ๐Ÿ‡ท

๊ณ„์‚ฐ๊ธฐ๋ฅผ ๋งŒ๋“ค์—ˆ์œผ๋ฉด ์ด์ œ ๊ทธ๊ฑธ๋กœ ์‹คํ—˜์„ ํ•ด ๋ณผ ์ฐจ๋ก€์˜ˆ์š”. ์ˆซ์ž๋ฅผ ์žฌ ๋ณด๋ฉด ์˜์™ธ์˜ ์‚ฌ์‹ค๋“ค์ด ๋ณด์ž…๋‹ˆ๋‹ค. ์•„๋ž˜๋Š” ๊ณ„์‚ฐ๊ธฐ๋กœ ์ง์ ‘ ๋น„๊ตํ•ด ๋ณด๊ธธ ๊ถŒํ•˜๋Š” ์‹คํ—˜ ๋ชฉ๋ก์ด์—์š”.

๐Ÿ”ฌ ์ธ์ฝ”๋”ฉ ๋น„๊ต๊ฐ™์€ ํ•œ๊ตญ์–ด ๋ฌธ์„œ๋ฅผ cl100k_base์™€ o200k_base๋กœ ๊ฐ๊ฐ ์„ธ์–ด ๋ณด์„ธ์š”. ์–ดํœ˜๊ฐ€ ํฐ ์ชฝ์—์„œ ํ† ํฐ์ด ์ค„์–ด๋“œ๋Š” ๊ฒฝํ–ฅ์„ ์ง์ ‘ ํ™•์ธํ•  ์ˆ˜ ์žˆ์–ด์š”.
๐Ÿ”ฌ ๊ณต๋ฐฑยท์„œ์‹๋“ค์—ฌ์“ฐ๊ธฐ, ์—ฐ์† ๊ณต๋ฐฑ, ์žฅ์‹์šฉ ๊ตฌ๋ถ„์„ (=====)์ด ์ƒ๊ฐ๋ณด๋‹ค ํ† ํฐ์„ ๋จน์–ด์š”. JSON์„ minifyํ•˜๋ฉด ์ค„์–ด๋“œ๋Š”์ง€๋„ ์žฌ ๋ณด์„ธ์š”.
๐Ÿ”ฌ ์‹œ์Šคํ…œ ํ”„๋กฌํ”„ํŠธ๋งค ์š”์ฒญ๋งˆ๋‹ค ๋ฐ˜๋ณต๋˜๋Š” ์‹œ์Šคํ…œ ํ”„๋กฌํ”„ํŠธ๋Š” "ํ† ํฐ ร— ์š”์ฒญ ์ˆ˜"๋กœ ๋น„์šฉ์ด ์ปค์ ธ์š”. ๊ฐ€์žฅ ๋จผ์ € ๋‹ค์ด์–ดํŠธํ•  ๋Œ€์ƒ!
๐Ÿ”ฌ ์ˆซ์žยท๋‚ ์งœ๊ธด ์ˆซ์ž, ํƒ€์ž„์Šคํƒฌํ”„, UUID๋Š” ์—ฌ๋Ÿฌ ํ† ํฐ์œผ๋กœ ์ชผ๊ฐœ์ง€๋Š” ๊ฒฝ์šฐ๊ฐ€ ๋งŽ์•„์š”. ๋กœ๊ทธ๋ฅผ ํ”„๋กฌํ”„ํŠธ์— ๋„ฃ๊ธฐ ์ „์— ํ•„์š”ํ•œ ํ•„๋“œ๋งŒ ๋‚จ๊ธฐ์„ธ์š”.
์ธก์ •ํ•˜์ง€ ์•Š์œผ๋ฉด ๊ฐœ์„ ํ•  ์ˆ˜ ์—†๋‹ค. ํ† ํฐ๋„ ๋งˆ์ฐฌ๊ฐ€์ง€๋‹ค. ๊ฐ์œผ๋กœ ํ”„๋กฌํ”„ํŠธ๋ฅผ ๊นŽ์ง€ ๋ง๊ณ , ๊ณ„์‚ฐ๊ธฐ๋กœ ์žฌ๊ณ  ๊นŽ์ž.
๊ทธ๋ ‡๋‹ค๊ณ  ๋ฌด์ž‘์ • ์˜์–ด๋กœ ๋ฐ”๊พธ์ง„ ๋งˆ์„ธ์š”"์˜์–ด๋กœ ์“ฐ๋ฉด ํ† ํฐ์ด ์ค„์–ด๋“œ๋‹ˆ ํ”„๋กฌํ”„ํŠธ๋ฅผ ์ „๋ถ€ ์˜์–ด๋กœ!"๋Š” ๋‹จ์ˆœํ•˜๊ฒŒ ๋ณผ ๋ฌธ์ œ๊ฐ€ ์•„๋‹ˆ์—์š”. ์ถœ๋ ฅ์ด ํ•œ๊ตญ์–ด์—ฌ์•ผ ํ•˜๋Š” ์„œ๋น„์Šค๋ผ๋ฉด ๋ฒˆ์—ญ ํ’ˆ์งˆ, ์œ ์ง€๋ณด์ˆ˜์„ฑ, ํŒ€์›์˜ ๊ฐ€๋…์„ฑ๊นŒ์ง€ ๋”ฐ์ ธ์•ผ ํ•ด์š”. ํ† ํฐ ์ ˆ๊ฐ ํšจ๊ณผ๋Š” ๋‚ด ๋ฐ์ดํ„ฐ๋กœ ๊ณ„์‚ฐ๊ธฐ๋กœ ์žฌ๊ณ  ํ’ˆ์งˆ ํ‰๊ฐ€์™€ ํ•จ๊ป˜ ํŒ๋‹จํ•˜์„ธ์š”.

17. ํ”„๋กœ์ ํŠธ ๋งˆ๋ฌด๋ฆฌ: ๋ฐฐํฌ ํ˜•ํƒœ ๊ณ ๋ฅด๊ธฐ ๐Ÿ“ฆ

๋‹จ์ผ ์Šคํฌ๋ฆฝํŠธ: tokcalc.py ํ•˜๋‚˜๋ฅผ ํŒ€ ์ €์žฅ์†Œ์— ๋‘๊ณ  ๊ณต์œ . ๊ฐ€์žฅ ๋น ๋ฅด๊ณ  ๋‹จ์ˆœํ•ด์š”.
ํŒจํ‚ค์ง€ํ™”: pyproject.toml์— [project.scripts] ํ•ญ๋ชฉ์œผ๋กœ tokcalc = "tokcalc:main"์„ ๋“ฑ๋กํ•˜๋ฉด pip install ํ›„ ์–ด๋””์„œ๋“  tokcalc ๋ช…๋ น์œผ๋กœ ์‹คํ–‰๋ผ์š”.
์‚ฌ์ „ ๊ฒ€์‚ฌ ํ›…: pre-commit์ด๋‚˜ CI์—์„œ ํ”„๋กฌํ”„ํŠธ ํ…œํ”Œ๋ฆฟ ํŒŒ์ผ์ด ํ•œ๋„๋ฅผ ๋„˜์œผ๋ฉด ์‹คํŒจํ•˜๋„๋ก ๊ฑธ์–ด ๋‘๊ธฐ. ํ”„๋กฌํ”„ํŠธ๊ฐ€ ์Šฌ๊ธˆ์Šฌ๊ธˆ ๋น„๋Œ€ํ•ด์ง€๋Š” ๊ฑธ ๋ง‰์•„์ค˜์š”.
์„œ๋น„์Šค ๋ฏธ๋“ค์›จ์–ด: LLM ํ˜ธ์ถœ ์ง์ „์— ํ† ํฐ์„ ์„ธ์„œ ํ•œ๋„ ์ดˆ๊ณผ ์‹œ ํžˆ์Šคํ† ๋ฆฌ๋ฅผ ์ž˜๋ผ๋‚ด๊ฑฐ๋‚˜ ์š”์•ฝ ๋‹จ๊ณ„๋กœ ๋ณด๋‚ด๋Š” ๊ฐ€๋“œ๋กœ ํ™œ์šฉ.

๋งˆ์ง€๋ง‰ "๋ฏธ๋“ค์›จ์–ด" ํŒจํ„ด์€ ์ฑ—๋ด‡์—์„œ ํŠนํžˆ ์œ ์šฉํ•ด์š”. ๋Œ€ํ™”๊ฐ€ ๊ธธ์–ด์ง€๋ฉด ์˜ค๋ž˜๋œ ๋ฉ”์‹œ์ง€๋ถ€ํ„ฐ ๋นผ๋ฉด์„œ ํ•œ๋„ ์•ˆ์— ๋งž์ถ”๋Š” ๊ฑฐ์ฃ .

def trim_history(messages, model="gpt-4o", budget=8000, keep_system=True):
    """์˜ค๋ž˜๋œ ๋Œ€ํ™”๋ถ€ํ„ฐ ์ œ๊ฑฐํ•ด ํ† ํฐ ์˜ˆ์‚ฐ ์•ˆ์— ๋งž์ถ˜๋‹ค."""
    system = [m for m in messages if m["role"] == "system"] if keep_system else []
    rest = [m for m in messages if m["role"] != "system"]
    while rest and count_chat_tokens(system + rest, model) > budget:
        rest.pop(0)          # ๊ฐ€์žฅ ์˜ค๋ž˜๋œ ๋ฉ”์‹œ์ง€ ์ œ๊ฑฐ
    return system + rest
ํ•ต์‹ฌ ์ •๋ฆฌ๐Ÿ”น ํ† ํฐ ์ˆ˜๋Š” ๋Œ€์ƒ ๋ชจ๋ธ์˜ ํ† ํฌ๋‚˜์ด์ €๋กœ ์„ธ์•ผ ์ •ํ™•ํ•˜๋‹ค (OpenAI โ†’ tiktoken, ์˜คํ”ˆ ๋ชจ๋ธ โ†’ AutoTokenizer, ClaudeยทGemini โ†’ ๊ณต์‹ ์นด์šดํŒ… API)
๐Ÿ”น tiktoken์€ disallowed_special=() ๋˜๋Š” encode_ordinary๋กœ ํŠน์ˆ˜ ํ† ํฐ ์—๋Ÿฌ๋ฅผ ์˜ˆ๋ฐฉ
๐Ÿ”น ์ฑ„ํŒ… API๋Š” ๋ฉ”์‹œ์ง€ ํฌ๋งท ์˜ค๋ฒ„ํ—ค๋“œ๊ฐ€ ๋ถ™์œผ๋ฏ€๋กœ ๋ณธ๋ฌธ ํ•ฉ๊ณ„๋ณด๋‹ค ํฌ๋‹ค. ์ตœ์ข… ๊ธฐ์ค€์€ API์˜ usage
๐Ÿ”น ๋‹จ๊ฐ€๋Š” ์„ค์ • ํŒŒ์ผ๋กœ ๋ถ„๋ฆฌ, ์ถœ๋ ฅ ํ† ํฐ์€ ์ƒํ•œยทํ†ต๊ณ„ยท๋น„์œจ ์ค‘ ํ•˜๋‚˜๋กœ ์ถ”์ •
๐Ÿ”น ์ฒญํฌ ๋ถ„ํ• ์€ ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ํ•˜๊ณ , ์™„์„ฑ๋œ ์ฒญํฌ๋ฅผ ๋‹ค์‹œ ์„ธ์–ด ๊ฒ€์ฆ
๐Ÿ”น ์ˆซ์ž๋ฅผ ๋‚ด๋Š” ๋„๊ตฌ์—” ํ…Œ์ŠคํŠธ์™€ "์ถ”์ •์น˜" ํ‘œ์‹œ๋กœ ์‹ ๋ขฐ๋ฅผ ๋”ํ•œ๋‹ค

์—ฌ๊ธฐ๊นŒ์ง€ ๋”ฐ๋ผ์™”๋‹ค๋ฉด ์ด์ œ ์—ฌ๋Ÿฌ๋ถ„ ์†์—๋Š” "๊ธ€์ž ์ˆ˜ ๋Œ€์ถฉ ๊ณฑํ•˜๊ธฐ"๊ฐ€ ์•„๋‹ˆ๋ผ, ๋ชจ๋ธ๋ณ„๋กœ ์ •ํ™•ํžˆ ์„ธ๊ณ , ๋น„์šฉ์„ ์˜ˆ์ธกํ•˜๊ณ , ํ•œ๋„๋ฅผ ์ง€ํ‚ค๊ณ , ๋ฌธ์„œ๋ฅผ ์•Œ๋งž๊ฒŒ ์ž๋ฅด๋Š” ์ง„์งœ ๋„๊ตฌ๊ฐ€ ์žˆ์–ด์š”. ์ฒ˜์Œ์—” 10์ค„์งœ๋ฆฌ ์Šคํฌ๋ฆฝํŠธ์˜€๋Š”๋ฐ ๋ง์ด์ฃ ! ๋‹ค์Œ ํ”„๋กœ์ ํŠธ์—์„œ ํ”„๋กฌํ”„ํŠธ ๊ธธ์ด ๋•Œ๋ฌธ์— ๊ณ ๋ฏผ๋  ๋•Œ, ๊ฐ ๋ง๊ณ  ๊ณ„์‚ฐ๊ธฐ๋ฅผ ๋จผ์ € ์ผœ๋Š” ์Šต๊ด€์„ ๋“ค์—ฌ ๋ณด์„ธ์š”. ์ƒ๊ฐ๋ณด๋‹ค ๋งŽ์€ ๋ˆ๊ณผ ์‹œ๊ฐ„์ด ์ ˆ์•ฝ๋ฉ๋‹ˆ๋‹ค. ๐ŸŒฑ

#tiktoken#BPE#ํ† ํฐ๊ณ„์‚ฐ#RAG์ฒญํฌ#LLM๋น„์šฉ#ํŒŒ์ด์ฌCLI

์ฐธ๊ณ  ์ž๋ฃŒ

ยท openai/tiktoken (GitHub) โ€” https://github.com/openai/tiktoken
ยท OpenAI Cookbook, "How to count tokens with tiktoken" โ€” https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
ยท Hugging Face Transformers ๋ฌธ์„œ, Tokenizer / Chat templates โ€” https://huggingface.co/docs/transformers
ยท Anthropic ๋ฌธ์„œ, Token counting โ€” https://docs.anthropic.com/en/docs/build-with-claude/token-counting
ยท Google AI for Developers, Understand and count tokens โ€” https://ai.google.dev/gemini-api/docs/tokens
ยท Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" (BPE) โ€” https://arxiv.org/abs/1508.07909

๋Œ“๊ธ€ ์ž‘์„ฑ

์ด ๊ธ€์— ๋Œ€ํ•œ ์—ฌ๋Ÿฌ๋ถ„์˜ ์ƒ๊ฐ์„ ๋“ค๋ ค์ฃผ์„ธ์š”

๋Œ“๊ธ€ 0
๋Œ“๊ธ€์„ ๋ถˆ๋Ÿฌ์˜ค๋Š” ์ค‘์ž…๋‹ˆ๋‹ค.

์•„์ง ๋Œ“๊ธ€์ด ์—†์Šต๋‹ˆ๋‹ค.