์ฝ˜ํ…์ธ  ๋Œ€ํ‘œ ์ด๋ฏธ์ง€ - ๐Ÿš€ Sentence Transformers๋กœ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑํ•˜๊ธฐ: ํ…์ŠคํŠธ๋ฅผ ์ˆซ์ž๋กœ ๋ฐ”๊พธ๋Š” ๋งˆ๋ฒ• ๊ฐ™์€ ์—ฌํ–‰

๐Ÿš€ Sentence Transformers๋กœ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑํ•˜๊ธฐ: ํ…์ŠคํŠธ๋ฅผ ์ˆซ์ž๋กœ ๋ฐ”๊พธ๋Š” ๋งˆ๋ฒ• ๊ฐ™์€ ์—ฌํ–‰

์ž์—ฐ์–ด๋ฅผ ์ปดํ“จํ„ฐ๊ฐ€ ์ดํ•ดํ•˜๋Š” ๋ฒกํ„ฐ๋กœ ๋ณ€ํ™˜ํ•˜๋Š” ์‹ค์ „ ๊ฐ€์ด๋“œ

A Text [0.23, 0.87, 0.45...] โœ“ Vector Sentence Transformers Pipeline Input Embedding Output

๐ŸŽฏ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ์ด ๋ญ๊ธธ๋ž˜ ์ด๋ ‡๊ฒŒ ๋‚œ๋ฆฌ์•ผ?

์•ผ, ์นœ๊ตฌ! ํ˜น์‹œ ์ปดํ“จํ„ฐํ•œํ…Œ "์‚ฌ๋ž‘ํ•ด"๋ผ๋Š” ๋ง์„ ์–ด๋–ป๊ฒŒ ์ดํ•ด์‹œํ‚ฌ ์ˆ˜ ์žˆ์„๊นŒ ์ƒ๊ฐํ•ด๋ณธ ์  ์žˆ์–ด? ๐Ÿค”
์ปดํ“จํ„ฐ๋Š” ์šฐ๋ฆฌ์ฒ˜๋Ÿผ ๊ฐ์ •์„ ๋А๋ผ์ง€ ๋ชปํ•˜์ž–์•„. ๊ทธ๋ƒฅ 0๊ณผ 1๋งŒ ์•Œ์•„๋“ฃ๋Š” ๊ธฐ๊ณ„๋‹ˆ๊นŒ ๋ง์ด์•ผ.

๊ทธ๋Ÿฐ๋ฐ ์—ฌ๊ธฐ์„œ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ(Vector Embedding)์ด๋ผ๋Š” ๊ฒŒ ๋“ฑ์žฅํ•ด!
์ด๊ฑด ๋งˆ์น˜ ๋ฒˆ์—ญ๊ธฐ ๊ฐ™์€ ๊ฑด๋ฐ, ์šฐ๋ฆฌ๊ฐ€ ์“ฐ๋Š” ๋ง์„ ์ปดํ“จํ„ฐ๊ฐ€ ์ดํ•ดํ•  ์ˆ˜ ์žˆ๋Š” ์ˆซ์ž ๋ฐฐ์—ด๋กœ ๋ฐ”๊ฟ”์ฃผ๋Š” ๊ฑฐ์•ผ.
"์‚ฌ๋ž‘ํ•ด"๋ฅผ [0.8, 0.9, 0.2, 0.7, ...] ์ด๋Ÿฐ ์‹์œผ๋กœ ๋ณ€ํ™˜ํ•˜๋Š” ๊ฑฐ์ง€.

๊ทธ๋ฆฌ๊ณ  ์ด ์ž‘์—…์„ ์—„์ฒญ ์ž˜ํ•˜๋Š” ๋„๊ตฌ๊ฐ€ ๋ฐ”๋กœ Sentence Transformers์•ผ! ๐ŸŽ‰

๐Ÿ’ก ์ž ๊น๋งŒ! ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ์„ ์™œ ๋ฐฐ์›Œ์•ผ ํ•˜๋Š”๋ฐ? โ€ข ๊ฒ€์ƒ‰ ์—”์ง„ ๋งŒ๋“ค๊ธฐ: ์‚ฌ์šฉ์ž๊ฐ€ ์ž…๋ ฅํ•œ ์งˆ๋ฌธ๊ณผ ๊ฐ€์žฅ ๋น„์Šทํ•œ ๋‹ต๋ณ€ ์ฐพ๊ธฐ
โ€ข ์ถ”์ฒœ ์‹œ์Šคํ…œ: ๋น„์Šทํ•œ ์ƒํ’ˆ์ด๋‚˜ ์ฝ˜ํ…์ธ  ์ถ”์ฒœํ•˜๊ธฐ
โ€ข ์ฑ—๋ด‡ ๊ฐœ๋ฐœ: ์‚ฌ์šฉ์ž ์˜๋„ ํŒŒ์•…ํ•˜๊ณ  ์ ์ ˆํ•œ ๋‹ต๋ณ€ ์ƒ์„ฑํ•˜๊ธฐ
โ€ข ๋ฌธ์„œ ๋ถ„๋ฅ˜: ๋‰ด์Šค ๊ธฐ์‚ฌ๋ฅผ ์ž๋™์œผ๋กœ ์นดํ…Œ๊ณ ๋ฆฌ๋ณ„๋กœ ๋ถ„๋ฅ˜ํ•˜๊ธฐ
โ€ข ๊ฐ์„ฑ ๋ถ„์„: ๋ฆฌ๋ทฐ๊ฐ€ ๊ธ์ •์ ์ธ์ง€ ๋ถ€์ •์ ์ธ์ง€ ํŒ๋‹จํ•˜๊ธฐ

์š”์ฆ˜ AI ์‹œ๋Œ€์— ์ด๋Ÿฐ ๊ธฐ์ˆ  ์—†์ด๋Š” ์ œ๋Œ€๋กœ ๋œ ์„œ๋น„์Šค๋ฅผ ๋งŒ๋“ค๊ธฐ ์–ด๋ ค์›Œ.
์žฌ๋Šฅ๋„ท ๊ฐ™์€ ํ”Œ๋žซํผ์—์„œ๋„ ์‚ฌ์šฉ์ž๊ฐ€ ์›ํ•˜๋Š” ์žฌ๋Šฅ์„ ์ •ํ™•ํ•˜๊ฒŒ ๋งค์นญํ•˜๋ ค๋ฉด ์ด๋Ÿฐ ๊ธฐ์ˆ ์ด ํ•„์ˆ˜์ ์ด๊ฑฐ๋“ ! ๐ŸŽฏ

๐Ÿง  Sentence Transformers๊ฐ€ ๋ญ”์ง€ ์ œ๋Œ€๋กœ ์•Œ์•„๋ณด์ž

Sentence Transformers๋Š” 2019๋…„์— ๋‚˜์˜จ Python ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ธ๋ฐ,
๋ฌธ์žฅ์ด๋‚˜ ๋‹จ๋ฝ์„ ์˜๋ฏธ๋ฅผ ๋‹ด์€ ๋ฒกํ„ฐ๋กœ ๋ณ€ํ™˜ํ•ด์ฃผ๋Š” ๋„๊ตฌ์•ผ.

๊ธฐ์กด์˜ BERT ๊ฐ™์€ ๋ชจ๋ธ๋“ค์€ ๋ฌธ์žฅ ๊ฐ„ ์œ ์‚ฌ๋„๋ฅผ ๊ณ„์‚ฐํ•˜๋ ค๋ฉด ๋ชจ๋“  ์กฐํ•ฉ์„ ์ผ์ผ์ด ๋น„๊ตํ•ด์•ผ ํ–ˆ์–ด.
์˜ˆ๋ฅผ ๋“ค์–ด 10,000๊ฐœ ๋ฌธ์žฅ์˜ ์œ ์‚ฌ๋„๋ฅผ ๊ณ„์‚ฐํ•˜๋ ค๋ฉด ์•ฝ 5์ฒœ๋งŒ ๋ฒˆ์˜ ๋น„๊ต๊ฐ€ ํ•„์š”ํ–ˆ์ง€! ๐Ÿ˜ฑ

ํ•˜์ง€๋งŒ Sentence Transformers๋Š” ๊ฐ ๋ฌธ์žฅ์„ ๋จผ์ € ๋ฒกํ„ฐ๋กœ ๋ณ€ํ™˜ํ•œ ๋‹ค์Œ,
์ฝ”์‚ฌ์ธ ์œ ์‚ฌ๋„ ๊ฐ™์€ ๊ฐ„๋‹จํ•œ ๊ณ„์‚ฐ์œผ๋กœ ๋น„๊ตํ•  ์ˆ˜ ์žˆ์–ด์„œ ์†๋„๊ฐ€ ์—„์ฒญ ๋นจ๋ผ!

๐Ÿ“Š ์„ฑ๋Šฅ ๋น„๊ต

๋ฐฉ๋ฒ• 10,000๊ฐœ ๋ฌธ์žฅ ๋น„๊ต ์‹œ๊ฐ„ ํ™•์žฅ์„ฑ
BERT Cross-Encoder ์•ฝ 65์‹œ๊ฐ„ โŒ ๋‚ฎ์Œ
Sentence Transformers ์•ฝ 5์ดˆ โœ… ๋†’์Œ

์ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์˜ ํ•ต์‹ฌ์€ Siamese Network ๊ตฌ์กฐ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค๋Š” ๊ฑฐ์•ผ.
์Œ๋‘ฅ์ด ๋„คํŠธ์›Œํฌ๋ผ๋Š” ๋œป์ธ๋ฐ, ๊ฐ™์€ ๊ฐ€์ค‘์น˜๋ฅผ ๊ณต์œ ํ•˜๋Š” ๋‘ ๊ฐœ์˜ ์‹ ๊ฒฝ๋ง์ด ๋ฌธ์žฅ์„ ์ฒ˜๋ฆฌํ•ด์„œ
์˜๋ฏธ์ ์œผ๋กœ ๋น„์Šทํ•œ ๋ฌธ์žฅ๋“ค์€ ๋ฒกํ„ฐ ๊ณต๊ฐ„์—์„œ ๊ฐ€๊นŒ์ด ์œ„์น˜ํ•˜๋„๋ก ํ•™์Šต๋ผ! ๐ŸŽฏ

๐Ÿ› ๏ธ ์„ค์น˜๋ถ€ํ„ฐ ์‹œ์ž‘ํ•ด๋ณด์ž! ํ™˜๊ฒฝ ์„ธํŒ…ํ•˜๊ธฐ

์ž, ์ด์ œ ๋ณธ๊ฒฉ์ ์œผ๋กœ ์‹œ์ž‘ํ•ด๋ณผ๊นŒ? ๋จผ์ € ํ•„์š”ํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋“ค์„ ์„ค์น˜ํ•ด์•ผ ํ•ด.
Python 3.6 ์ด์ƒ ๋ฒ„์ „์ด ํ•„์š”ํ•˜๊ณ , ๊ฐ€๋Šฅํ•˜๋ฉด ๊ฐ€์ƒํ™˜๊ฒฝ์„ ๋งŒ๋“ค์–ด์„œ ์ž‘์—…ํ•˜๋Š” ๊ฑธ ์ถ”์ฒœํ•ด! ๐Ÿ

โœจ ๊ฐ€์ƒํ™˜๊ฒฝ ๋งŒ๋“ค๊ธฐ (์„ ํƒ์‚ฌํ•ญ์ด์ง€๋งŒ ๊ฐ•๋ ฅ ์ถ”์ฒœ!) ๊ฐ€์ƒํ™˜๊ฒฝ์„ ์‚ฌ์šฉํ•˜๋ฉด ํ”„๋กœ์ ํŠธ๋ณ„๋กœ ๋…๋ฆฝ์ ์ธ ํŒจํ‚ค์ง€ ํ™˜๊ฒฝ์„ ์œ ์ง€ํ•  ์ˆ˜ ์žˆ์–ด์„œ
๋‹ค๋ฅธ ํ”„๋กœ์ ํŠธ์™€ ์ถฉ๋Œ์ด ์—†๊ณ  ๊ด€๋ฆฌ๊ฐ€ ํŽธํ•ด์ ธ!
# ๊ฐ€์ƒํ™˜๊ฒฝ ์ƒ์„ฑ
python -m venv sentence_env

# ๊ฐ€์ƒํ™˜๊ฒฝ ํ™œ์„ฑํ™” (Windows)
sentence_env\Scripts\activate

# ๊ฐ€์ƒํ™˜๊ฒฝ ํ™œ์„ฑํ™” (Mac/Linux)
source sentence_env/bin/activate

์ด์ œ Sentence Transformers๋ฅผ ์„ค์น˜ํ•ด๋ณด์ž! ๐Ÿš€

# ๊ธฐ๋ณธ ์„ค์น˜
pip install sentence-transformers

# ๋˜๋Š” ์ตœ์‹  ๋ฒ„์ „์œผ๋กœ ์„ค์น˜
pip install -U sentence-transformers

# ์ถ”๊ฐ€๋กœ ์œ ์šฉํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋“ค
pip install numpy pandas scikit-learn
โš ๏ธ ์ฃผ์˜์‚ฌํ•ญ ์ฒ˜์Œ ๋ชจ๋ธ์„ ๋กœ๋“œํ•  ๋•Œ ์ธํ„ฐ๋„ท์—์„œ ๋ชจ๋ธ ํŒŒ์ผ์„ ๋‹ค์šด๋กœ๋“œํ•˜๊ธฐ ๋•Œ๋ฌธ์—
์ธํ„ฐ๋„ท ์—ฐ๊ฒฐ์ด ํ•„์š”ํ•˜๊ณ , ๋ชจ๋ธ ํฌ๊ธฐ์— ๋”ฐ๋ผ ์ˆ˜๋ฐฑ MB์—์„œ ์ˆ˜ GB๊นŒ์ง€ ๋‹ค์šด๋กœ๋“œ๋  ์ˆ˜ ์žˆ์–ด!
์‹œ๊ฐ„์ด ์ข€ ๊ฑธ๋ฆด ์ˆ˜ ์žˆ์œผ๋‹ˆ ์ปคํ”ผ ํ•œ ์ž” ํ•˜๋ฉด์„œ ๊ธฐ๋‹ค๋ ค~ โ˜•

์„ค์น˜๊ฐ€ ์™„๋ฃŒ๋˜๋ฉด ์ œ๋Œ€๋กœ ์„ค์น˜๋๋Š”์ง€ ํ™•์ธํ•ด๋ณด์ž!

import sentence_transformers
print(sentence_transformers.__version__)

# ๊ฐ„๋‹จํ•œ ํ…Œ์ŠคํŠธ
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
print("๋ชจ๋ธ ๋กœ๋“œ ์„ฑ๊ณต! ๐ŸŽ‰")

์—๋Ÿฌ ์—†์ด ๋ฒ„์ „์ด ์ถœ๋ ฅ๋˜๊ณ  ๋ชจ๋ธ์ด ๋กœ๋“œ๋˜๋ฉด ์„ฑ๊ณต์ด์•ผ! ๐Ÿ‘

๐ŸŽจ ์ฒซ ๋ฒˆ์งธ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑํ•˜๊ธฐ: Hello World!

์ž, ์ด์ œ ์ง„์งœ ์žฌ๋ฏธ์žˆ๋Š” ๋ถ€๋ถ„์ด์•ผ! ์ฒซ ๋ฒˆ์งธ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ์„ ๋งŒ๋“ค์–ด๋ณด์ž! ๐ŸŽŠ
์ฝ”๋“œ๋Š” ์ƒ๊ฐ๋ณด๋‹ค ์—„์ฒญ ๊ฐ„๋‹จํ•ด. ๋†€๋ž„ ์ค€๋น„ ๋์–ด?

from sentence_transformers import SentenceTransformer

# ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋ฌธ์žฅ ์ค€๋น„
sentences = [
    "๋‚˜๋Š” ํŒŒ์ด์ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ์„ ์ข‹์•„ํ•ด",
    "Python programming is my favorite",
    "์˜ค๋Š˜ ๋‚ ์”จ๊ฐ€ ์ •๋ง ์ข‹๋„ค์š”"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ (์ด๊ฒŒ ์ „๋ถ€์•ผ!)
embeddings = model.encode(sentences)

# ๊ฒฐ๊ณผ ํ™•์ธ
print(f"์ž„๋ฒ ๋”ฉ shape: {embeddings.shape}")
print(f"์ฒซ ๋ฒˆ์งธ ๋ฌธ์žฅ์˜ ๋ฒกํ„ฐ (์ผ๋ถ€): {embeddings[0][:5]}")

์ถœ๋ ฅ ๊ฒฐ๊ณผ๋Š” ์ด๋Ÿฐ ์‹์œผ๋กœ ๋‚˜์˜ฌ ๊ฑฐ์•ผ:


์ž„๋ฒ ๋”ฉ shape: (3, 384)

์ฒซ ๋ฒˆ์งธ ๋ฌธ์žฅ์˜ ๋ฒกํ„ฐ (์ผ๋ถ€): [ 0.0234 -0.1234  0.5678 -0.2345  0.8901]

์—ฌ๊ธฐ์„œ (3, 384)๋Š” 3๊ฐœ์˜ ๋ฌธ์žฅ์ด ๊ฐ๊ฐ 384์ฐจ์›์˜ ๋ฒกํ„ฐ๋กœ ๋ณ€ํ™˜๋๋‹ค๋Š” ๋œป์ด์•ผ!
384๊ฐœ์˜ ์ˆซ์ž๋กœ ๋ฌธ์žฅ์˜ ์˜๋ฏธ๋ฅผ ํ‘œํ˜„ํ•˜๋Š” ๊ฑฐ์ง€. ์‹ ๊ธฐํ•˜์ง€? ๐Ÿ˜ฎ

์ด ๋ฒกํ„ฐ๋“ค์€ ๋‹จ์ˆœํ•œ ์ˆซ์ž ๋ฐฐ์—ด์ด ์•„๋‹ˆ๋ผ ์˜๋ฏธ๋ฅผ ๋‹ด๊ณ  ์žˆ์–ด.
๋น„์Šทํ•œ ์˜๋ฏธ์˜ ๋ฌธ์žฅ๋“ค์€ ๋ฒกํ„ฐ ๊ณต๊ฐ„์—์„œ ๊ฐ€๊นŒ์ด ์œ„์น˜ํ•˜๊ฒŒ ๋ผ!

๐Ÿ’ก Pro Tip: ๋ชจ๋ธ ์„ ํƒํ•˜๊ธฐ all-MiniLM-L6-v2๋Š” ๊ฐ€๋ณ๊ณ  ๋น ๋ฅธ ๋ชจ๋ธ์ด์•ผ.
โ€ข ํฌ๊ธฐ: ์•ฝ 80MB
โ€ข ์†๋„: ๋งค์šฐ ๋น ๋ฆ„
โ€ข ์„ฑ๋Šฅ: ๋Œ€๋ถ€๋ถ„์˜ ์ž‘์—…์— ์ถฉ๋ถ„
โ€ข ์ถ”์ฒœ ์šฉ๋„: ํ”„๋กœํ† ํƒ€์ž…, ์‹ค์‹œ๊ฐ„ ์„œ๋น„์Šค

๋” ๋†’์€ ์„ฑ๋Šฅ์ด ํ•„์š”ํ•˜๋ฉด all-mpnet-base-v2๋ฅผ ์‚ฌ์šฉํ•ด๋ด!
โ€ข ํฌ๊ธฐ: ์•ฝ 420MB
โ€ข ์†๋„: ์ค‘๊ฐ„
โ€ข ์„ฑ๋Šฅ: ๋” ์ •ํ™•ํ•จ
โ€ข ์ถ”์ฒœ ์šฉ๋„: ๋†’์€ ์ •ํ™•๋„๊ฐ€ ํ•„์š”ํ•œ ํ”„๋กœ์ ํŠธ

๐Ÿ” ์œ ์‚ฌ๋„ ๊ณ„์‚ฐํ•˜๊ธฐ: ๋ฌธ์žฅ๋“ค์ด ์–ผ๋งˆ๋‚˜ ๋น„์Šทํ• ๊นŒ?

์ž„๋ฒ ๋”ฉ์„ ๋งŒ๋“ค์—ˆ์œผ๋ฉด ์ด์ œ ๋ฌธ์žฅ๋“ค์ด ์–ผ๋งˆ๋‚˜ ๋น„์Šทํ•œ์ง€ ๊ณ„์‚ฐํ•ด๋ณผ ์ฐจ๋ก€์•ผ!
์ด๊ฒŒ ๋ฐ”๋กœ ์ž„๋ฒ ๋”ฉ์˜ ์ง„์งœ ํŒŒ์›Œ๋ฅผ ๋А๋‚„ ์ˆ˜ ์žˆ๋Š” ์ˆœ๊ฐ„์ด์ง€. ๐Ÿ”ฅ

from sentence_transformers import SentenceTransformer, util
import torch

# ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋น„๊ตํ•  ๋ฌธ์žฅ๋“ค
sentences = [
    "๊ณ ์–‘์ด๊ฐ€ ์†ŒํŒŒ์—์„œ ์ž๊ณ  ์žˆ์–ด",
    "์†ŒํŒŒ ์œ„์—์„œ ๊ณ ์–‘์ด๊ฐ€ ์ž ์„ ์ž๋„ค",
    "๊ฐ•์•„์ง€๊ฐ€ ๊ณต์›์—์„œ ๋›ฐ์–ด๋†€๊ณ  ์žˆ์–ด",
    "์˜ค๋Š˜ ์ €๋… ๋ฉ”๋‰ด๋Š” ๋ญ๊ฐ€ ์ข‹์„๊นŒ?"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(sentences, convert_to_tensor=True)

# ์ฝ”์‚ฌ์ธ ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
cosine_scores = util.cos_sim(embeddings, embeddings)

# ๊ฒฐ๊ณผ ์ถœ๋ ฅ
print("๋ฌธ์žฅ ๊ฐ„ ์œ ์‚ฌ๋„ ๋งคํŠธ๋ฆญ์Šค:")
print(cosine_scores)

# ํŠน์ • ๋ฌธ์žฅ ์Œ์˜ ์œ ์‚ฌ๋„ ํ™•์ธ
print(f"\n๋ฌธ์žฅ 0๊ณผ 1์˜ ์œ ์‚ฌ๋„: {cosine_scores[0][1]:.4f}")
print(f"๋ฌธ์žฅ 0๊ณผ 2์˜ ์œ ์‚ฌ๋„: {cosine_scores[0][2]:.4f}")
print(f"๋ฌธ์žฅ 0๊ณผ 3์˜ ์œ ์‚ฌ๋„: {cosine_scores[0][3]:.4f}")

๊ฒฐ๊ณผ๋ฅผ ๋ณด๋ฉด ์ด๋ ‡๊ฒŒ ๋‚˜์˜ฌ ๊ฑฐ์•ผ:

๐Ÿ“Š ์œ ์‚ฌ๋„ ๊ฒฐ๊ณผ ๋ถ„์„ โ€ข ๋ฌธ์žฅ 0๊ณผ 1์˜ ์œ ์‚ฌ๋„: 0.8756 โ†’ ๊ฑฐ์˜ ๊ฐ™์€ ์˜๋ฏธ! ๐ŸŽฏ
โ€ข ๋ฌธ์žฅ 0๊ณผ 2์˜ ์œ ์‚ฌ๋„: 0.4523 โ†’ ๋‘˜ ๋‹ค ๋™๋ฌผ ๊ด€๋ จ์ด๋ผ ์•ฝ๊ฐ„ ๋น„์Šท
โ€ข ๋ฌธ์žฅ 0๊ณผ 3์˜ ์œ ์‚ฌ๋„: 0.1234 โ†’ ์™„์ „ํžˆ ๋‹ค๋ฅธ ์ฃผ์ œ

์œ ์‚ฌ๋„๋Š” -1์—์„œ 1 ์‚ฌ์ด์˜ ๊ฐ’์ด๊ณ , 1์— ๊ฐ€๊นŒ์šธ์ˆ˜๋ก ๋น„์Šทํ•œ ์˜๋ฏธ์•ผ!
๋ณดํ†ต 0.7 ์ด์ƒ์ด๋ฉด ๋งค์šฐ ์œ ์‚ฌํ•˜๋‹ค๊ณ  ๋ณผ ์ˆ˜ ์žˆ์–ด.

์ด์ œ ์ข€ ๋” ์‹ค์šฉ์ ์ธ ์˜ˆ์ œ๋ฅผ ๋งŒ๋“ค์–ด๋ณผ๊นŒ? ์งˆ๋ฌธ-๋‹ต๋ณ€ ๋งค์นญ ์‹œ์Šคํ…œ์„ ๋งŒ๋“ค์–ด๋ณด์ž! ๐ŸŽฏ

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

# FAQ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค
faq_questions = [
    "๋ฐฐ์†ก์€ ์–ผ๋งˆ๋‚˜ ๊ฑธ๋ฆฌ๋‚˜์š”?",
    "ํ™˜๋ถˆ ์ •์ฑ…์ด ์–ด๋–ป๊ฒŒ ๋˜๋‚˜์š”?",
    "ํšŒ์›๊ฐ€์ž…์€ ์–ด๋–ป๊ฒŒ ํ•˜๋‚˜์š”?",
    "๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”",
    "์ƒํ’ˆ ๊ตํ™˜์€ ๊ฐ€๋Šฅํ•œ๊ฐ€์š”?"
]

faq_answers = [
    "์ผ๋ฐ˜์ ์œผ๋กœ 2-3์ผ ์ •๋„ ์†Œ์š”๋ฉ๋‹ˆ๋‹ค.",
    "๊ตฌ๋งค ํ›„ 7์ผ ์ด๋‚ด ํ™˜๋ถˆ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.",
    "ํ™ˆํŽ˜์ด์ง€ ์ƒ๋‹จ์˜ ํšŒ์›๊ฐ€์ž… ๋ฒ„ํŠผ์„ ํด๋ฆญํ•˜์„ธ์š”.",
    "๋กœ๊ทธ์ธ ํŽ˜์ด์ง€์—์„œ ๋น„๋ฐ€๋ฒˆํ˜ธ ์ฐพ๊ธฐ๋ฅผ ์ด์šฉํ•˜์„ธ์š”.",
    "๊ตฌ๋งค ํ›„ 14์ผ ์ด๋‚ด ๊ตํ™˜ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค."
]

# FAQ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ (ํ•œ ๋ฒˆ๋งŒ ํ•˜๋ฉด ๋ผ!)
faq_embeddings = model.encode(faq_questions, convert_to_tensor=True)

# ์‚ฌ์šฉ์ž ์งˆ๋ฌธ
user_query = "๋ฌผ๊ฑด์ด ์–ธ์ œ ๋„์ฐฉํ•˜๋‚˜์š”?"

# ์‚ฌ์šฉ์ž ์งˆ๋ฌธ ์ž„๋ฒ ๋”ฉ
query_embedding = model.encode(user_query, convert_to_tensor=True)

# ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
similarities = util.cos_sim(query_embedding, faq_embeddings)[0]

# ๊ฐ€์žฅ ์œ ์‚ฌํ•œ ์งˆ๋ฌธ ์ฐพ๊ธฐ
best_match_idx = similarities.argmax()
best_score = similarities[best_match_idx]

print(f"์‚ฌ์šฉ์ž ์งˆ๋ฌธ: {user_query}")
print(f"๋งค์นญ๋œ FAQ: {faq_questions[best_match_idx]}")
print(f"์œ ์‚ฌ๋„ ์ ์ˆ˜: {best_score:.4f}")
print(f"๋‹ต๋ณ€: {faq_answers[best_match_idx]}")

์ด ์ฝ”๋“œ๋Š” ์‚ฌ์šฉ์ž๊ฐ€ "๋ฌผ๊ฑด์ด ์–ธ์ œ ๋„์ฐฉํ•˜๋‚˜์š”?"๋ผ๊ณ  ๋ฌผ์–ด๋ณด๋ฉด
์ž๋™์œผ๋กœ "๋ฐฐ์†ก์€ ์–ผ๋งˆ๋‚˜ ๊ฑธ๋ฆฌ๋‚˜์š”?"์™€ ๋งค์นญ์‹œ์ผœ์„œ ์ ์ ˆํ•œ ๋‹ต๋ณ€์„ ์ฐพ์•„์ค˜! ๐Ÿค–

์‹ค์ œ๋กœ ์žฌ๋Šฅ๋„ท ๊ฐ™์€ ํ”Œ๋žซํผ์—์„œ ์‚ฌ์šฉ์ž ๋ฌธ์˜๋ฅผ ์ž๋™์œผ๋กœ ์ฒ˜๋ฆฌํ•˜๋Š” ๋ฐ ์ด๋Ÿฐ ๊ธฐ์ˆ ์„ ํ™œ์šฉํ•  ์ˆ˜ ์žˆ์–ด!

๐ŸŒ ๋‹ค๊ตญ์–ด ์ง€์›: ํ•œ๊ตญ์–ด๋„ ๋ฌธ์ œ์—†์–ด!

์—ฌ๊ธฐ์„œ ์ค‘์š”ํ•œ ํฌ์ธํŠธ! Sentence Transformers๋Š” ๋‹ค๊ตญ์–ด๋ฅผ ์ง€์›ํ•˜๋Š” ๋ชจ๋ธ๋“ค๋„ ์žˆ์–ด! ๐ŸŒ
ํ•œ๊ตญ์–ด ๋ฌธ์žฅ๋„ ์™„๋ฒฝํ•˜๊ฒŒ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฑฐ์ง€.

from sentence_transformers import SentenceTransformer

# ๋‹ค๊ตญ์–ด ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')

# ์—ฌ๋Ÿฌ ์–ธ์–ด์˜ ๋ฌธ์žฅ๋“ค
sentences = [
    "์•ˆ๋…•ํ•˜์„ธ์š”, ๋ฐ˜๊ฐ‘์Šต๋‹ˆ๋‹ค",
    "Hello, nice to meet you",
    "ใ“ใ‚“ใซใกใฏใ€ใฏใ˜ใ‚ใพใ—ใฆ",
    "Hola, mucho gusto",
    "Bonjour, enchantรฉ"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(sentences)

# ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
from sentence_transformers import util
similarities = util.cos_sim(embeddings, embeddings)

print("๋‹ค๊ตญ์–ด ๋ฌธ์žฅ ๊ฐ„ ์œ ์‚ฌ๋„:")
for i in range(len(sentences)):
    for j in range(i+1, len(sentences)):
        print(f"{sentences[i][:20]}... โ†” {sentences[j][:20]}...")
        print(f"์œ ์‚ฌ๋„: {similarities[i][j]:.4f}\n")

๋†€๋ž๊ฒŒ๋„ ์„œ๋กœ ๋‹ค๋ฅธ ์–ธ์–ด๋กœ ๋œ ๊ฐ™์€ ์˜๋ฏธ์˜ ๋ฌธ์žฅ๋“ค์ด ๋†’์€ ์œ ์‚ฌ๋„๋ฅผ ๋ณด์—ฌ์ค˜!
์ด๊ฑด ๋ชจ๋ธ์ด ์–ธ์–ด๋ฅผ ๋„˜์–ด์„œ ์˜๋ฏธ๋ฅผ ์ดํ•ดํ•œ๋‹ค๋Š” ์ฆ๊ฑฐ์•ผ. ๐ŸŽฏ

๐ŸŒ ์ถ”์ฒœ ๋‹ค๊ตญ์–ด ๋ชจ๋ธ๋“ค

๋ชจ๋ธ๋ช… ์ง€์› ์–ธ์–ด ํŠน์ง•
paraphrase-multilingual-MiniLM-L12-v2 50๊ฐœ ์ด์ƒ ๋น ๋ฅด๊ณ  ๊ฐ€๋ฒผ์›€
paraphrase-multilingual-mpnet-base-v2 50๊ฐœ ์ด์ƒ ๋†’์€ ์„ฑ๋Šฅ
distiluse-base-multilingual-cased-v2 15๊ฐœ ์ฃผ์š” ์–ธ์–ด ๊ท ํ˜•์žกํžŒ ์„ฑ๋Šฅ

ํ•œ๊ตญ์–ด ์ฒ˜๋ฆฌ๊ฐ€ ์ค‘์š”ํ•œ ํ”„๋กœ์ ํŠธ๋ผ๋ฉด paraphrase-multilingual-mpnet-base-v2๋ฅผ ์ถ”์ฒœํ•ด!
์กฐ๊ธˆ ๋ฌด๊ฒ์ง€๋งŒ ํ•œ๊ตญ์–ด ์„ฑ๋Šฅ์ด ์ •๋ง ์ข‹๊ฑฐ๋“ . ๐Ÿ‘

โšก ์„ฑ๋Šฅ ์ตœ์ ํ™”: ๋” ๋น ๋ฅด๊ฒŒ, ๋” ํšจ์œจ์ ์œผ๋กœ!

์‹ค์ œ ์„œ๋น„์Šค์— ์ ์šฉํ•˜๋ ค๋ฉด ์„ฑ๋Šฅ ์ตœ์ ํ™”๊ฐ€ ํ•„์ˆ˜์•ผ!
์ˆ˜์ฒœ, ์ˆ˜๋งŒ ๊ฐœ์˜ ๋ฌธ์žฅ์„ ์ฒ˜๋ฆฌํ•ด์•ผ ํ•  ๋•Œ ์–ด๋–ป๊ฒŒ ํ•ด์•ผ ํ• ๊นŒ? ๐Ÿš€

1๏ธโƒฃ ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ ํ™œ์šฉํ•˜๊ธฐ

๋ฌธ์žฅ์„ ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌํ•˜๋Š” ๊ฒƒ๋ณด๋‹ค ์—ฌ๋Ÿฌ ๊ฐœ๋ฅผ ํ•œ ๋ฒˆ์— ์ฒ˜๋ฆฌํ•˜๋Š” ๊ฒŒ ํ›จ์”ฌ ๋นจ๋ผ!

import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# ํ…Œ์ŠคํŠธ์šฉ ๋ฌธ์žฅ 1000๊ฐœ ์ƒ์„ฑ
sentences = [f"์ด๊ฒƒ์€ ํ…Œ์ŠคํŠธ ๋ฌธ์žฅ ๋ฒˆํ˜ธ {i}์ž…๋‹ˆ๋‹ค" for i in range(1000)]

# ๋ฐฉ๋ฒ• 1: ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌ (๋А๋ฆผ)
start = time.time()
embeddings_slow = []
for sentence in sentences:
    emb = model.encode(sentence)
    embeddings_slow.append(emb)
time_slow = time.time() - start

# ๋ฐฉ๋ฒ• 2: ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ (๋น ๋ฆ„)
start = time.time()
embeddings_fast = model.encode(sentences, batch_size=32)
time_fast = time.time() - start

print(f"ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌ: {time_slow:.2f}์ดˆ")
print(f"๋ฐฐ์น˜ ์ฒ˜๋ฆฌ: {time_fast:.2f}์ดˆ")
print(f"์†๋„ ํ–ฅ์ƒ: {time_slow/time_fast:.1f}๋ฐฐ ๋นจ๋ผ์ง! ๐Ÿš€")
๐Ÿ’ก ๋ฐฐ์น˜ ํฌ๊ธฐ ์„ ํƒ ๊ฐ€์ด๋“œ โ€ข GPU ์‚ฌ์šฉ ์‹œ: 32~128 (GPU ๋ฉ”๋ชจ๋ฆฌ์— ๋”ฐ๋ผ ์กฐ์ ˆ)
โ€ข CPU ์‚ฌ์šฉ ์‹œ: 8~32
โ€ข ๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ๋ถ€์กฑํ•˜๋ฉด ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ ์ค„์—ฌ๋ด!
โ€ข ๋„ˆ๋ฌด ํฌ๋ฉด ์˜คํžˆ๋ ค ๋А๋ ค์งˆ ์ˆ˜ ์žˆ์–ด

2๏ธโƒฃ GPU ํ™œ์šฉํ•˜๊ธฐ

GPU๊ฐ€ ์žˆ๋‹ค๋ฉด ๋ฐ˜๋“œ์‹œ ํ™œ์šฉํ•ด์•ผ ํ•ด! ์†๋„๊ฐ€ 10๋ฐฐ ์ด์ƒ ๋นจ๋ผ์งˆ ์ˆ˜ ์žˆ๊ฑฐ๋“ . ๐Ÿ’ช

import torch
from sentence_transformers import SentenceTransformer

# GPU ์‚ฌ์šฉ ๊ฐ€๋Šฅ ์—ฌ๋ถ€ ํ™•์ธ
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"์‚ฌ์šฉ ์ค‘์ธ ๋””๋ฐ”์ด์Šค: {device}")

# ๋ชจ๋ธ์„ GPU๋กœ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2', device=device)

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ (์ž๋™์œผ๋กœ GPU ์‚ฌ์šฉ)
sentences = ["๋ฌธ์žฅ 1", "๋ฌธ์žฅ 2", "๋ฌธ์žฅ 3"]
embeddings = model.encode(sentences, convert_to_tensor=True)

print(f"์ž„๋ฒ ๋”ฉ์ด GPU์— ์žˆ๋‚˜์š”? {embeddings.is_cuda}")

3๏ธโƒฃ ์ž„๋ฒ ๋”ฉ ์บ์‹ฑํ•˜๊ธฐ

๊ฐ™์€ ๋ฌธ์žฅ์„ ์—ฌ๋Ÿฌ ๋ฒˆ ์ฒ˜๋ฆฌํ•  ํ•„์š” ์—†์–ด! ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•œ ์ž„๋ฒ ๋”ฉ์€ ์ €์žฅํ•ด๋‘๊ณ  ์žฌ์‚ฌ์šฉํ•˜์ž. ๐Ÿ’พ

import pickle
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋ฌธ์žฅ๊ณผ ์ž„๋ฒ ๋”ฉ์„ ๋”•์…”๋„ˆ๋ฆฌ๋กœ ์ €์žฅ
embedding_cache = {}

def get_embedding(sentence):
    """์บ์‹œ๋ฅผ ํ™œ์šฉํ•œ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ"""
    if sentence not in embedding_cache:
        embedding_cache[sentence] = model.encode(sentence)
    return embedding_cache[sentence]

# ์‚ฌ์šฉ ์˜ˆ์‹œ
sentences = ["์•ˆ๋…•ํ•˜์„ธ์š”", "๋ฐ˜๊ฐ‘์Šต๋‹ˆ๋‹ค", "์•ˆ๋…•ํ•˜์„ธ์š”"]  # "์•ˆ๋…•ํ•˜์„ธ์š”"๊ฐ€ ์ค‘๋ณต

for sent in sentences:
    emb = get_embedding(sent)
    print(f"'{sent}' ์ฒ˜๋ฆฌ ์™„๋ฃŒ")

# ์บ์‹œ ์ €์žฅ
with open('embedding_cache.pkl', 'wb') as f:
    pickle.dump(embedding_cache, f)

# ๋‚˜์ค‘์— ์บ์‹œ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ
with open('embedding_cache.pkl', 'rb') as f:
    loaded_cache = pickle.load(f)
โš ๏ธ ๋ฉ”๋ชจ๋ฆฌ ๊ด€๋ฆฌ ์ฃผ์˜! ์ž„๋ฒ ๋”ฉ์„ ๋„ˆ๋ฌด ๋งŽ์ด ์บ์‹ฑํ•˜๋ฉด ๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ๋ถ€์กฑํ•ด์งˆ ์ˆ˜ ์žˆ์–ด.
โ€ข 384์ฐจ์› ์ž„๋ฒ ๋”ฉ 1๊ฐœ โ‰ˆ 1.5KB
โ€ข 100๋งŒ ๊ฐœ ์ž„๋ฒ ๋”ฉ โ‰ˆ 1.5GB

LRU(Least Recently Used) ์บ์‹œ๋ฅผ ์‚ฌ์šฉํ•˜๊ฑฐ๋‚˜,
๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์— ์ €์žฅํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๊ณ ๋ คํ•ด๋ด!

4๏ธโƒฃ ๋ชจ๋ธ ์–‘์žํ™”(Quantization)

๋ชจ๋ธ ํฌ๊ธฐ๋ฅผ ์ค„์ด๊ณ  ์†๋„๋ฅผ ๋†’์ด๋Š” ๊ณ ๊ธ‰ ๊ธฐ๋ฒ•์ด์•ผ! ๐ŸŽฏ

from sentence_transformers import SentenceTransformer
import torch

# ์ผ๋ฐ˜ ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋™์  ์–‘์žํ™” ์ ์šฉ
model = torch.quantization.quantize_dynamic(
    model, 
    {torch.nn.Linear}, 
    dtype=torch.qint8
)

# ์–‘์žํ™”๋œ ๋ชจ๋ธ๋กœ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
sentences = ["ํ…Œ์ŠคํŠธ ๋ฌธ์žฅ์ž…๋‹ˆ๋‹ค"]
embeddings = model.encode(sentences)

print("์–‘์žํ™” ์™„๋ฃŒ! ๋ชจ๋ธ์ด ๋” ๊ฐ€๋ฒผ์›Œ์กŒ์–ด์š” ๐ŸŽ‰")

์–‘์žํ™”๋ฅผ ํ•˜๋ฉด ๋ชจ๋ธ ํฌ๊ธฐ๊ฐ€ ์•ฝ 4๋ฐฐ ์ค„์–ด๋“ค๊ณ , CPU์—์„œ ์ถ”๋ก  ์†๋„๊ฐ€ 2-3๋ฐฐ ๋นจ๋ผ์ ธ!
์ •ํ™•๋„๋Š” ์•ฝ๊ฐ„ ๋–จ์–ด์งˆ ์ˆ˜ ์žˆ์ง€๋งŒ ๋Œ€๋ถ€๋ถ„์˜ ๊ฒฝ์šฐ ๋ฌด์‹œํ•  ์ˆ˜ ์žˆ๋Š” ์ˆ˜์ค€์ด์•ผ. ๐Ÿ‘

๐ŸŽฏ ์‹ค์ „ ํ”„๋กœ์ ํŠธ: ์˜๋ฏธ ๊ธฐ๋ฐ˜ ๊ฒ€์ƒ‰ ์—”์ง„ ๋งŒ๋“ค๊ธฐ

์ด์ œ ๋ฐฐ์šด ๊ฑธ ์ข…ํ•ฉํ•ด์„œ ์‹ค์ œ๋กœ ๋™์ž‘ํ•˜๋Š” ๊ฒ€์ƒ‰ ์—”์ง„์„ ๋งŒ๋“ค์–ด๋ณด์ž!
ํ‚ค์›Œ๋“œ ๋งค์นญ์ด ์•„๋‹ˆ๋ผ ์˜๋ฏธ๋ฅผ ์ดํ•ดํ•˜๋Š” ๊ฒ€์ƒ‰์ด์•ผ. ๐Ÿ”

from sentence_transformers import SentenceTransformer, util
import numpy as np

class SemanticSearchEngine:
    def __init__(self, model_name='all-MiniLM-L6-v2'):
        """์˜๋ฏธ ๊ธฐ๋ฐ˜ ๊ฒ€์ƒ‰ ์—”์ง„ ์ดˆ๊ธฐํ™”"""
        self.model = SentenceTransformer(model_name)
        self.documents = []
        self.embeddings = None
        
    def add_documents(self, documents):
        """๋ฌธ์„œ ์ถ”๊ฐ€ ๋ฐ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ"""
        self.documents.extend(documents)
        print(f"๐Ÿ“š {len(documents)}๊ฐœ ๋ฌธ์„œ ์ถ”๊ฐ€ ์ค‘...")
        
        # ์ƒˆ๋กœ์šด ๋ฌธ์„œ๋“ค์˜ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
        new_embeddings = self.model.encode(
            documents, 
            convert_to_tensor=True,
            show_progress_bar=True
        )
        
        # ๊ธฐ์กด ์ž„๋ฒ ๋”ฉ๊ณผ ํ•ฉ์น˜๊ธฐ
        if self.embeddings is None:
            self.embeddings = new_embeddings
        else:
            self.embeddings = torch.cat([self.embeddings, new_embeddings])
            
        print(f"โœ… ์ด {len(self.documents)}๊ฐœ ๋ฌธ์„œ ์ธ๋ฑ์‹ฑ ์™„๋ฃŒ!")
        
    def search(self, query, top_k=5):
        """์˜๋ฏธ ๊ธฐ๋ฐ˜ ๊ฒ€์ƒ‰ ์ˆ˜ํ–‰"""
        if not self.documents:
            return []
            
        # ์ฟผ๋ฆฌ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
        query_embedding = self.model.encode(query, convert_to_tensor=True)
        
        # ์ฝ”์‚ฌ์ธ ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
        cos_scores = util.cos_sim(query_embedding, self.embeddings)[0]
        
        # ์ƒ์œ„ k๊ฐœ ๊ฒฐ๊ณผ ์ถ”์ถœ
        top_results = torch.topk(cos_scores, k=min(top_k, len(self.documents)))
        
        results = []
        for score, idx in zip(top_results[0], top_results[1]):
            results.append({
                'document': self.documents[idx],
                'score': float(score),
                'rank': len(results) + 1
            })
            
        return results

# ์‚ฌ์šฉ ์˜ˆ์‹œ
search_engine = SemanticSearchEngine()

# ์ƒ˜ํ”Œ ๋ฌธ์„œ๋“ค
documents = [
    "Python์€ ๋ฐฐ์šฐ๊ธฐ ์‰ฝ๊ณ  ๊ฐ•๋ ฅํ•œ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด์ž…๋‹ˆ๋‹ค",
    "๋จธ์‹ ๋Ÿฌ๋‹์„ ์œ„ํ•œ ์ตœ๊ณ ์˜ ์–ธ์–ด๋Š” Python์ž…๋‹ˆ๋‹ค",
    "JavaScript๋Š” ์›น ๊ฐœ๋ฐœ์— ํ•„์ˆ˜์ ์ธ ์–ธ์–ด์ž…๋‹ˆ๋‹ค",
    "Java๋Š” ์—”ํ„ฐํ”„๋ผ์ด์ฆˆ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์— ๋งŽ์ด ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค",
    "C++์€ ๊ฒŒ์ž„ ๊ฐœ๋ฐœ๊ณผ ์‹œ์Šคํ…œ ํ”„๋กœ๊ทธ๋ž˜๋ฐ์— ์ ํ•ฉํ•ฉ๋‹ˆ๋‹ค",
    "์ธ๊ณต์ง€๋Šฅ ๊ฐœ๋ฐœ์ž๊ฐ€ ๋˜๋ ค๋ฉด Python์„ ๋ฐฐ์›Œ์•ผ ํ•ฉ๋‹ˆ๋‹ค",
    "React๋Š” ํ˜„๋Œ€์ ์ธ ์›น UI๋ฅผ ๋งŒ๋“œ๋Š” JavaScript ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ž…๋‹ˆ๋‹ค",
    "๋”ฅ๋Ÿฌ๋‹ ํ”„๋ ˆ์ž„์›Œํฌ๋กœ๋Š” TensorFlow์™€ PyTorch๊ฐ€ ์œ ๋ช…ํ•ฉ๋‹ˆ๋‹ค"
]

# ๋ฌธ์„œ ์ถ”๊ฐ€
search_engine.add_documents(documents)

# ๊ฒ€์ƒ‰ ์ˆ˜ํ–‰
query = "AI ๊ฐœ๋ฐœ์„ ๋ฐฐ์šฐ๊ณ  ์‹ถ์–ด์š”"
results = search_engine.search(query, top_k=3)

print(f"\n๐Ÿ” ๊ฒ€์ƒ‰์–ด: '{query}'")
print("=" * 60)
for result in results:
    print(f"\n{result['rank']}์œ„ (์œ ์‚ฌ๋„: {result['score']:.4f})")
    print(f"๐Ÿ“„ {result['document']}")

์ด ์ฝ”๋“œ๋ฅผ ์‹คํ–‰ํ•˜๋ฉด ์ด๋Ÿฐ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ฌ ๊ฑฐ์•ผ:

๐Ÿ” ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ ๊ฒ€์ƒ‰์–ด: 'AI ๊ฐœ๋ฐœ์„ ๋ฐฐ์šฐ๊ณ  ์‹ถ์–ด์š”'

1์œ„ (์œ ์‚ฌ๋„: 0.7234)
๐Ÿ“„ ์ธ๊ณต์ง€๋Šฅ ๊ฐœ๋ฐœ์ž๊ฐ€ ๋˜๋ ค๋ฉด Python์„ ๋ฐฐ์›Œ์•ผ ํ•ฉ๋‹ˆ๋‹ค

2์œ„ (์œ ์‚ฌ๋„: 0.6891)
๐Ÿ“„ ๋จธ์‹ ๋Ÿฌ๋‹์„ ์œ„ํ•œ ์ตœ๊ณ ์˜ ์–ธ์–ด๋Š” Python์ž…๋‹ˆ๋‹ค

3์œ„ (์œ ์‚ฌ๋„: 0.6543)
๐Ÿ“„ ๋”ฅ๋Ÿฌ๋‹ ํ”„๋ ˆ์ž„์›Œํฌ๋กœ๋Š” TensorFlow์™€ PyTorch๊ฐ€ ์œ ๋ช…ํ•ฉ๋‹ˆ๋‹ค

๋ณด๋‹ค์‹œํ”ผ "AI"๋‚˜ "๊ฐœ๋ฐœ"์ด๋ผ๋Š” ๋‹จ์–ด๊ฐ€ ์ •ํ™•ํžˆ ํฌํ•จ๋˜์ง€ ์•Š์€ ๋ฌธ์„œ๋„ ์ฐพ์•„๋‚ด!
์ด๊ฒŒ ๋ฐ”๋กœ ์˜๋ฏธ ๊ธฐ๋ฐ˜ ๊ฒ€์ƒ‰์˜ ํŒŒ์›Œ์•ผ. ๐ŸŽฏ

๐Ÿ”ง ๊ณ ๊ธ‰ ๊ธฐ๋Šฅ: ํŒŒ์ธํŠœ๋‹์œผ๋กœ ์„ฑ๋Šฅ ๋Œ์–ด์˜ฌ๋ฆฌ๊ธฐ

๊ธฐ๋ณธ ๋ชจ๋ธ๋„ ์ข‹์ง€๋งŒ, ํŠน์ • ๋„๋ฉ”์ธ์— ๋งž๊ฒŒ ํŒŒ์ธํŠœ๋‹ํ•˜๋ฉด ์„ฑ๋Šฅ์ด ํ›จ์”ฌ ์ข‹์•„์ ธ!
์˜ˆ๋ฅผ ๋“ค์–ด ์˜๋ฃŒ, ๋ฒ•๋ฅ , ๊ธˆ์œต ๊ฐ™์€ ์ „๋ฌธ ๋ถ„์•ผ์—์„œ๋Š” ํŒŒ์ธํŠœ๋‹์ด ํ•„์ˆ˜์•ผ. ๐ŸŽ“

ํŒŒ์ธํŠœ๋‹ ๋ฐ์ดํ„ฐ ์ค€๋น„ํ•˜๊ธฐ

ํŒŒ์ธํŠœ๋‹์„ ์œ„ํ•ด์„œ๋Š” ํ•™์Šต ๋ฐ์ดํ„ฐ๊ฐ€ ํ•„์š”ํ•ด. ๋ณดํ†ต ์„ธ ๊ฐ€์ง€ ํ˜•ํƒœ๋กœ ์ค€๋น„ํ•  ์ˆ˜ ์žˆ์–ด:

๐Ÿ“Š ํ•™์Šต ๋ฐ์ดํ„ฐ ํ˜•ํƒœ 1. Positive Pairs: ๋น„์Šทํ•œ ์˜๋ฏธ์˜ ๋ฌธ์žฅ ์Œ
์˜ˆ) ("๊ณ ์–‘์ด๊ฐ€ ์ž๊ณ  ์žˆ์–ด", "๊ณ ์–‘์ด๊ฐ€ ์ž ์„ ์ž๋„ค")

2. Triplets: (๊ธฐ์ค€ ๋ฌธ์žฅ, ์œ ์‚ฌ ๋ฌธ์žฅ, ๋น„์œ ์‚ฌ ๋ฌธ์žฅ)
์˜ˆ) ("๊ฐ•์•„์ง€", "๊ฐœ", "๊ณ ์–‘์ด")

3. Labeled Data: ์œ ์‚ฌ๋„ ์ ์ˆ˜๊ฐ€ ๋งค๊ฒจ์ง„ ๋ฌธ์žฅ ์Œ
์˜ˆ) ("๋ฌธ์žฅA", "๋ฌธ์žฅB", 0.85)
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

# ํ•™์Šต ๋ฐ์ดํ„ฐ ์ค€๋น„ (Positive Pairs ์˜ˆ์‹œ)
train_examples = [
    InputExample(texts=['๊ณ ์–‘์ด๊ฐ€ ์ž๊ณ  ์žˆ์–ด', '๊ณ ์–‘์ด๊ฐ€ ์ž ์„ ์ž๋„ค'], label=1.0),
    InputExample(texts=['๊ฐ•์•„์ง€๊ฐ€ ๋›ฐ์–ด๋†€์•„', '๊ฐœ๊ฐ€ ๋‹ฌ๋ฆฌ๊ณ  ์žˆ์–ด'], label=1.0),
    InputExample(texts=['๋‚ ์”จ๊ฐ€ ์ข‹์•„', '๋น„๊ฐ€ ์˜ค๊ณ  ์žˆ์–ด'], label=0.3),
    InputExample(texts=['ํŒŒ์ด์ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ', 'Python ์ฝ”๋”ฉ'], label=0.95),
    # ... ๋” ๋งŽ์€ ๋ฐ์ดํ„ฐ
]

# ๋ฐ์ดํ„ฐ๋กœ๋” ์ƒ์„ฑ
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)

# ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2')

# ์†์‹ค ํ•จ์ˆ˜ ์ •์˜
train_loss = losses.CosineSimilarityLoss(model)

# ํŒŒ์ธํŠœ๋‹ ์‹œ์ž‘
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=4,
    warmup_steps=100,
    output_path='./my_finetuned_model'
)

print("๐ŸŽ‰ ํŒŒ์ธํŠœ๋‹ ์™„๋ฃŒ!")
๐Ÿ’ก ํŒŒ์ธํŠœ๋‹ ํŒ โ€ข ์ตœ์†Œ 1,000๊ฐœ ์ด์ƒ์˜ ํ•™์Šต ์ƒ˜ํ”Œ ์ค€๋น„
โ€ข ๋„๋ฉ”์ธ ํŠนํ™” ๋ฐ์ดํ„ฐ์ผ์ˆ˜๋ก ํšจ๊ณผ๊ฐ€ ์ข‹์•„
โ€ข Epoch๋Š” 3-5 ์ •๋„๊ฐ€ ์ ๋‹น (๋„ˆ๋ฌด ๋งŽ์œผ๋ฉด ๊ณผ์ ํ•ฉ)
โ€ข Learning rate๋Š” 2e-5 ์ •๋„๋กœ ์‹œ์ž‘
โ€ข ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ๋กœ ์„ฑ๋Šฅ์„ ๋ชจ๋‹ˆํ„ฐ๋งํ•ด์•ผ ํ•ด

ํ‰๊ฐ€ ๋ฐ ์„ฑ๋Šฅ ์ธก์ •

from sentence_transformers import evaluation

# ํ‰๊ฐ€ ๋ฐ์ดํ„ฐ ์ค€๋น„
sentences1 = ['๊ณ ์–‘์ด๊ฐ€ ์ž๊ณ  ์žˆ์–ด', '๊ฐ•์•„์ง€๊ฐ€ ๋›ฐ์–ด๋†€์•„']
sentences2 = ['๊ณ ์–‘์ด๊ฐ€ ์ž ์„ ์ž๋„ค', '๊ฐœ๊ฐ€ ๋‹ฌ๋ฆฌ๊ณ  ์žˆ์–ด']
scores = [0.9, 0.85]  # ์‹ค์ œ ์œ ์‚ฌ๋„ ์ ์ˆ˜

# ํ‰๊ฐ€์ž ์ƒ์„ฑ
evaluator = evaluation.EmbeddingSimilarityEvaluator(
    sentences1, 
    sentences2, 
    scores
)

# ๋ชจ๋ธ ํ‰๊ฐ€
model = SentenceTransformer('./my_finetuned_model')
score = evaluator(model)

print(f"๋ชจ๋ธ ์„ฑ๋Šฅ ์ ์ˆ˜: {score:.4f}")

๐ŸŒŸ ์‹ค์ „ ํ™œ์šฉ ์‚ฌ๋ก€: ๋‹ค์–‘ํ•œ ์‘์šฉ ๋ถ„์•ผ

Sentence Transformers๋ฅผ ์‹ค์ œ๋กœ ์–ด๋–ป๊ฒŒ ํ™œ์šฉํ•  ์ˆ˜ ์žˆ์„๊นŒ?
์—ฌ๋Ÿฌ ๊ฐ€์ง€ ์žฌ๋ฏธ์žˆ๋Š” ์‚ฌ๋ก€๋“ค์„ ์‚ดํŽด๋ณด์ž! ๐Ÿš€

1๏ธโƒฃ ์ค‘๋ณต ๋ฌธ์„œ ํƒ์ง€

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋ฌธ์„œ ๋ฆฌ์ŠคํŠธ
documents = [
    "์ธ๊ณต์ง€๋Šฅ์€ ๋ฏธ๋ž˜์˜ ํ•ต์‹ฌ ๊ธฐ์ˆ ์ž…๋‹ˆ๋‹ค",
    "AI๋Š” ์•ž์œผ๋กœ ๊ฐ€์žฅ ์ค‘์š”ํ•œ ๊ธฐ์ˆ ์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค",
    "์˜ค๋Š˜ ๋‚ ์”จ๊ฐ€ ์ •๋ง ์ข‹๋„ค์š”",
    "์ธ๊ณต์ง€๋Šฅ ๊ธฐ์ˆ ์€ ๋ฏธ๋ž˜์— ๋งค์šฐ ์ค‘์š”ํ•ฉ๋‹ˆ๋‹ค",
    "์ ์‹ฌ ๋ฉ”๋‰ด๋กœ ๋ญ ๋จน์„๊นŒ์š”?"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(documents, convert_to_tensor=True)

# ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
similarities = util.cos_sim(embeddings, embeddings)

# ์ค‘๋ณต ๋ฌธ์„œ ์ฐพ๊ธฐ (์œ ์‚ฌ๋„ 0.8 ์ด์ƒ)
print("๐Ÿ” ์ค‘๋ณต ๊ฐ€๋Šฅ์„ฑ์ด ๋†’์€ ๋ฌธ์„œ ์Œ:")
for i in range(len(documents)):
    for j in range(i+1, len(documents)):
        if similarities[i][j] > 0.8:
            print(f"\n๋ฌธ์„œ {i+1}: {documents[i]}")
            print(f"๋ฌธ์„œ {j+1}: {documents[j]}")
            print(f"์œ ์‚ฌ๋„: {similarities[i][j]:.4f} โš ๏ธ")

2๏ธโƒฃ ๋ฌธ์„œ ํด๋Ÿฌ์Šคํ„ฐ๋ง

from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋‹ค์–‘ํ•œ ์ฃผ์ œ์˜ ๋ฌธ์„œ๋“ค
documents = [
    "Python ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๋ฐฐ์šฐ๊ธฐ",
    "๋จธ์‹ ๋Ÿฌ๋‹ ์ž…๋ฌธ ๊ฐ€์ด๋“œ",
    "๋ง›์žˆ๋Š” ํŒŒ์Šคํƒ€ ๋ ˆ์‹œํ”ผ",
    "JavaScript ๊ธฐ์ดˆ ๊ฐ•์ขŒ",
    "๋”ฅ๋Ÿฌ๋‹์œผ๋กœ ์ด๋ฏธ์ง€ ๋ถ„๋ฅ˜ํ•˜๊ธฐ",
    "์ง‘์—์„œ ๋งŒ๋“œ๋Š” ํ”ผ์ž",
    "React ์ปดํฌ๋„ŒํŠธ ์„ค๊ณ„",
    "์ž์—ฐ์–ด ์ฒ˜๋ฆฌ ๊ธฐ์ดˆ",
    "์ดˆ์ฝœ๋ฆฟ ์ผ€์ดํฌ ๋งŒ๋“ค๊ธฐ"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(documents)

# K-Means ํด๋Ÿฌ์Šคํ„ฐ๋ง (3๊ฐœ ๊ทธ๋ฃน)
num_clusters = 3
clustering_model = KMeans(n_clusters=num_clusters, random_state=42)
clustering_model.fit(embeddings)
cluster_assignment = clustering_model.labels_

# ๊ฒฐ๊ณผ ์ถœ๋ ฅ
clustered_sentences = [[] for _ in range(num_clusters)]
for sentence_id, cluster_id in enumerate(cluster_assignment):
    clustered_sentences[cluster_id].append(documents[sentence_id])

print("๐Ÿ“Š ํด๋Ÿฌ์Šคํ„ฐ๋ง ๊ฒฐ๊ณผ:")
for i, cluster in enumerate(clustered_sentences):
    print(f"\nํด๋Ÿฌ์Šคํ„ฐ {i+1}:")
    for doc in cluster:
        print(f"  โ€ข {doc}")

๊ฒฐ๊ณผ๋ฅผ ๋ณด๋ฉด ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๊ด€๋ จ, ์š”๋ฆฌ ๊ด€๋ จ, AI ๊ด€๋ จ ๋ฌธ์„œ๋“ค์ด ์ž๋™์œผ๋กœ ๊ทธ๋ฃนํ™”๋ผ! ๐ŸŽฏ

3๏ธโƒฃ ์ถ”์ฒœ ์‹œ์Šคํ…œ

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

# ์ƒํ’ˆ ์„ค๋ช…
products = [
    "๋ฌด์„  ๋ธ”๋ฃจํˆฌ์Šค ์ด์–ดํฐ - ๊ณ ์Œ์งˆ ์‚ฌ์šด๋“œ",
    "๋…ธ์ด์ฆˆ ์บ”์Šฌ๋ง ํ—ค๋“œํฐ - ํ”„๋ฆฌ๋ฏธ์—„ ์Œ์งˆ",
    "์Šค๋งˆํŠธ์›Œ์น˜ - ๊ฑด๊ฐ• ๊ด€๋ฆฌ ๊ธฐ๋Šฅ",
    "ํ”ผํŠธ๋‹ˆ์Šค ํŠธ๋ž˜์ปค - ์šด๋™ ๊ธฐ๋ก",
    "ํœด๋Œ€์šฉ ์Šคํ”ผ์ปค - ๋ฐฉ์ˆ˜ ๊ธฐ๋Šฅ",
    "๊ฒŒ์ด๋ฐ ํ—ค๋“œ์…‹ - 7.1 ์„œ๋ผ์šด๋“œ",
    "๋Ÿฌ๋‹ํ™” - ์ฟ ์…”๋‹ ์šฐ์ˆ˜",
    "์š”๊ฐ€ ๋งคํŠธ - ๋ฏธ๋„๋Ÿผ ๋ฐฉ์ง€"
]

# ์‚ฌ์šฉ์ž๊ฐ€ ๊ด€์‹ฌ ์žˆ๋Š” ์ƒํ’ˆ
user_interest = "์Œ์•… ๋“ฃ๊ธฐ ์ข‹์€ ์ด์–ดํฐ"

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
product_embeddings = model.encode(products, convert_to_tensor=True)
user_embedding = model.encode(user_interest, convert_to_tensor=True)

# ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
similarities = util.cos_sim(user_embedding, product_embeddings)[0]

# ์ƒ์œ„ 3๊ฐœ ์ถ”์ฒœ
top_k = 3
top_results = similarities.topk(k=top_k)

print(f"๐ŸŽฏ '{user_interest}'์— ๋Œ€ํ•œ ์ถ”์ฒœ ์ƒํ’ˆ:")
for score, idx in zip(top_results[0], top_results[1]):
    print(f"\n์ถ”์ฒœ๋„: {score:.4f}")
    print(f"์ƒํ’ˆ: {products[idx]}")

4๏ธโƒฃ ๊ฐ์„ฑ ๋ถ„์„ ๊ฐ•ํ™”

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๊ฐ์„ฑ ๊ธฐ์ค€ ๋ฌธ์žฅ๋“ค
positive_reference = "์ •๋ง ์ข‹์•„์š”! ์ตœ๊ณ ์˜ˆ์š”! ๋งŒ์กฑ์Šค๋Ÿฝ์Šต๋‹ˆ๋‹ค!"
negative_reference = "๋ณ„๋กœ์˜ˆ์š”. ์‹ค๋งํ–ˆ์–ด์š”. ์ตœ์•…์ž…๋‹ˆ๋‹ค."
neutral_reference = "๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”. ๋ณดํ†ต์ž…๋‹ˆ๋‹ค. ํ‰๋ฒ”ํ•ด์š”."

# ๋ถ„์„ํ•  ๋ฆฌ๋ทฐ๋“ค
reviews = [
    "์ด ์ œํ’ˆ ์ง„์งœ ๋Œ€๋ฐ•์ด์—์š”! ๊ฐ•๋ ฅ ์ถ”์ฒœํ•ฉ๋‹ˆ๋‹ค!",
    "๋ฐฐ์†ก์ด ๋„ˆ๋ฌด ๋Šฆ์–ด์„œ ์งœ์ฆ๋‚˜๋„ค์š”",
    "๊ฐ€๊ฒฉ ๋Œ€๋น„ ๊ดœ์ฐฎ์€ ๊ฒƒ ๊ฐ™์•„์š”",
    "ํ’ˆ์งˆ์ด ํ˜•ํŽธ์—†์–ด์š”. ํ™˜๋ถˆํ•˜๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค",
    "๋”ฑ ๊ธฐ๋Œ€ํ•œ ๋งŒํผ์ด๋„ค์š”"
]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
ref_embeddings = model.encode([positive_reference, negative_reference, neutral_reference])
review_embeddings = model.encode(reviews)

# ๊ฐ ๋ฆฌ๋ทฐ์˜ ๊ฐ์„ฑ ๋ถ„์„
print("๐Ÿ“Š ๋ฆฌ๋ทฐ ๊ฐ์„ฑ ๋ถ„์„ ๊ฒฐ๊ณผ:\n")
for i, review in enumerate(reviews):
    similarities = util.cos_sim(review_embeddings[i], ref_embeddings)[0]
    
    sentiment = ["๊ธ์ • ๐Ÿ˜Š", "๋ถ€์ • ๐Ÿ˜ž", "์ค‘๋ฆฝ ๐Ÿ˜"]
    max_idx = similarities.argmax()
    
    print(f"๋ฆฌ๋ทฐ: {review}")
    print(f"๊ฐ์„ฑ: {sentiment[max_idx]} (์‹ ๋ขฐ๋„: {similarities[max_idx]:.4f})")
    print()

๐Ÿ›ก๏ธ ์ฃผ์˜์‚ฌํ•ญ๊ณผ ํ•œ๊ณ„์  ์•Œ์•„๋‘๊ธฐ

Sentence Transformers๊ฐ€ ์•„๋ฌด๋ฆฌ ์ข‹์•„๋„ ์™„๋ฒฝํ•˜์ง„ ์•Š์•„.
์‹ค์ œ ํ”„๋กœ์ ํŠธ์— ์ ์šฉํ•˜๊ธฐ ์ „์— ์ด๋Ÿฐ ์ ๋“ค์„ ๊ผญ ์•Œ์•„๋‘ฌ์•ผ ํ•ด! โš ๏ธ

1๏ธโƒฃ ๋ฌธ๋งฅ ๊ธธ์ด ์ œํ•œ

๋Œ€๋ถ€๋ถ„์˜ ๋ชจ๋ธ์€ ์ตœ๋Œ€ 512 ํ† ํฐ๊นŒ์ง€๋งŒ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์–ด.
๊ธด ๋ฌธ์„œ๋Š” ์ž˜๋ผ์„œ ์ฒ˜๋ฆฌํ•˜๊ฑฐ๋‚˜ ์š”์•ฝ์ด ํ•„์š”ํ•ด!

# ๊ธด ํ…์ŠคํŠธ ์ฒ˜๋ฆฌ ์˜ˆ์‹œ
def process_long_text(text, model, max_length=512):
    # ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋ถ„ํ• 
    sentences = text.split('. ')
    
    # ์ฒญํฌ๋กœ ๋‚˜๋ˆ„๊ธฐ
    chunks = []
    current_chunk = ""
    
    for sentence in sentences:
        if len(current_chunk) + len(sentence) < max_length:
            current_chunk += sentence + ". "
        else:
            chunks.append(current_chunk)
            current_chunk = sentence + ". "
    
    if current_chunk:
        chunks.append(current_chunk)
    
    # ๊ฐ ์ฒญํฌ์˜ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ ํ›„ ํ‰๊ท 
    embeddings = model.encode(chunks)
    return embeddings.mean(axis=0)
2๏ธโƒฃ ๋„๋ฉ”์ธ ํŠนํ™” ์„ฑ๋Šฅ ์ผ๋ฐ˜ ๋ชจ๋ธ์€ ์ „๋ฌธ ์šฉ์–ด๋‚˜ ํŠน์ˆ˜ํ•œ ๋„๋ฉ”์ธ์—์„œ ์„ฑ๋Šฅ์ด ๋–จ์–ด์งˆ ์ˆ˜ ์žˆ์–ด.
์˜๋ฃŒ, ๋ฒ•๋ฅ , ๊ธˆ์œต ๋“ฑ ์ „๋ฌธ ๋ถ„์•ผ์—์„œ๋Š” ํŒŒ์ธํŠœ๋‹์ด ํ•„์ˆ˜!
3๏ธโƒฃ ์–ธ์–ด๋ณ„ ์„ฑ๋Šฅ ์ฐจ์ด ์˜์–ด ๋ฐ์ดํ„ฐ๋กœ ์ฃผ๋กœ ํ•™์Šต๋˜์–ด์„œ ํ•œ๊ตญ์–ด ์„ฑ๋Šฅ์ด ์˜์–ด๋ณด๋‹ค ๋‚ฎ์„ ์ˆ˜ ์žˆ์–ด.
๋‹ค๊ตญ์–ด ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๊ฑฐ๋‚˜ ํ•œ๊ตญ์–ด ๋ฐ์ดํ„ฐ๋กœ ํŒŒ์ธํŠœ๋‹ํ•˜๋Š” ๊ฒŒ ์ข‹์•„!
4๏ธโƒฃ ๊ณ„์‚ฐ ๋น„์šฉ ์‹ค์‹œ๊ฐ„์œผ๋กœ ์ˆ˜๋ฐฑ๋งŒ ๊ฐœ์˜ ๋ฌธ์„œ๋ฅผ ๊ฒ€์ƒ‰ํ•˜๋ ค๋ฉด ์ธํ”„๋ผ ๋น„์šฉ์ด ๋งŒ๋งŒ์น˜ ์•Š์•„.
FAISS ๊ฐ™์€ ๋ฒกํ„ฐ ๊ฒ€์ƒ‰ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜๋Š” ๊ฑธ ์ถ”์ฒœํ•ด!
# FAISS๋ฅผ ํ™œ์šฉํ•œ ํšจ์œจ์ ์ธ ๊ฒ€์ƒ‰
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๋Œ€๋Ÿ‰์˜ ๋ฌธ์„œ (์˜ˆ: 100๋งŒ ๊ฐœ)
documents = ["๋ฌธ์„œ 1", "๋ฌธ์„œ 2", ...]  # ์‹ค์ œ๋กœ๋Š” ์—„์ฒญ ๋งŽ์Œ

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(documents, show_progress_bar=True)
embeddings = np.array(embeddings).astype('float32')

# FAISS ์ธ๋ฑ์Šค ์ƒ์„ฑ
dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(embeddings)

# ๋น ๋ฅธ ๊ฒ€์ƒ‰
query = "๊ฒ€์ƒ‰ํ•  ๋‚ด์šฉ"
query_embedding = model.encode([query]).astype('float32')

# ์ƒ์œ„ 10๊ฐœ ๊ฒฐ๊ณผ ๊ฒ€์ƒ‰ (๋งค์šฐ ๋น ๋ฆ„!)
k = 10
distances, indices = index.search(query_embedding, k)

print("๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ:")
for i, idx in enumerate(indices[0]):
    print(f"{i+1}. {documents[idx]} (๊ฑฐ๋ฆฌ: {distances[0][i]:.4f})")

๐Ÿ’ผ ํ”„๋กœ๋•์…˜ ํ™˜๊ฒฝ์— ๋ฐฐํฌํ•˜๊ธฐ

๊ฐœ๋ฐœ ํ™˜๊ฒฝ์—์„œ ์ž˜ ๋™์ž‘ํ•œ๋‹ค๊ณ  ๋์ด ์•„๋‹ˆ์•ผ!
์‹ค์ œ ์„œ๋น„์Šค์— ๋ฐฐํฌํ•˜๋ ค๋ฉด ์ด๋Ÿฐ ๊ฒƒ๋“ค์„ ๊ณ ๋ คํ•ด์•ผ ํ•ด. ๐Ÿš€

Flask API ์„œ๋ฒ„ ๋งŒ๋“ค๊ธฐ

from flask import Flask, request, jsonify
from sentence_transformers import SentenceTransformer, util
import torch

app = Flask(__name__)

# ๋ชจ๋ธ ๋กœ๋“œ (์„œ๋ฒ„ ์‹œ์ž‘ ์‹œ ํ•œ ๋ฒˆ๋งŒ)
print("๋ชจ๋ธ ๋กœ๋”ฉ ์ค‘...")
model = SentenceTransformer('all-MiniLM-L6-v2')
print("๋ชจ๋ธ ๋กœ๋“œ ์™„๋ฃŒ!")

# ๋ฌธ์„œ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค (์‹ค์ œ๋กœ๋Š” DB์—์„œ ๊ฐ€์ ธ์˜ด)
documents = [
    "Python ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๊ฐ•์ขŒ",
    "๋จธ์‹ ๋Ÿฌ๋‹ ๊ธฐ์ดˆ",
    "์›น ๊ฐœ๋ฐœ ์ž…๋ฌธ",
    # ... ๋” ๋งŽ์€ ๋ฌธ์„œ
]

# ๋ฏธ๋ฆฌ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
document_embeddings = model.encode(documents, convert_to_tensor=True)

@app.route('/search', methods=['POST'])
def search():
    try:
        # ์š”์ฒญ ๋ฐ์ดํ„ฐ ๋ฐ›๊ธฐ
        data = request.get_json()
        query = data.get('query', '')
        top_k = data.get('top_k', 5)
        
        if not query:
            return jsonify({'error': '๊ฒ€์ƒ‰์–ด๋ฅผ ์ž…๋ ฅํ•ด์ฃผ์„ธ์š”'}), 400
        
        # ์ฟผ๋ฆฌ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
        query_embedding = model.encode(query, convert_to_tensor=True)
        
        # ์œ ์‚ฌ๋„ ๊ณ„์‚ฐ
        similarities = util.cos_sim(query_embedding, document_embeddings)[0]
        
        # ์ƒ์œ„ ๊ฒฐ๊ณผ ์ถ”์ถœ
        top_results = torch.topk(similarities, k=min(top_k, len(documents)))
        
        # ๊ฒฐ๊ณผ ํฌ๋งทํŒ…
        results = []
        for score, idx in zip(top_results[0], top_results[1]):
            results.append({
                'document': documents[idx],
                'score': float(score)
            })
        
        return jsonify({
            'query': query,
            'results': results,
            'total': len(results)
        })
        
    except Exception as e:
        return jsonify({'error': str(e)}), 500

@app.route('/embed', methods=['POST'])
def embed():
    try:
        data = request.get_json()
        texts = data.get('texts', [])
        
        if not texts:
            return jsonify({'error': 'ํ…์ŠคํŠธ๋ฅผ ์ž…๋ ฅํ•ด์ฃผ์„ธ์š”'}), 400
        
        # ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
        embeddings = model.encode(texts)
        
        return jsonify({
            'embeddings': embeddings.tolist(),
            'dimension': embeddings.shape[1]
        })
        
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=False)

Docker๋กœ ์ปจํ…Œ์ด๋„ˆํ™”ํ•˜๊ธฐ

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

# ํ•„์š”ํ•œ ํŒจํ‚ค์ง€ ์„ค์น˜
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ฝ”๋“œ ๋ณต์‚ฌ
COPY . .

# ๋ชจ๋ธ ๋ฏธ๋ฆฌ ๋‹ค์šด๋กœ๋“œ (์„ ํƒ์‚ฌํ•ญ)
RUN python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"

# ํฌํŠธ ๋…ธ์ถœ
EXPOSE 5000

# ์„œ๋ฒ„ ์‹คํ–‰
CMD ["python", "app.py"]
# requirements.txt
flask==2.3.0
sentence-transformers==2.2.2
torch==2.0.0
numpy==1.24.0
# Docker ์ด๋ฏธ์ง€ ๋นŒ๋“œ ๋ฐ ์‹คํ–‰
docker build -t sentence-transformer-api .
docker run -p 5000:5000 sentence-transformer-api
๐Ÿš€ ์„ฑ๋Šฅ ์ตœ์ ํ™” ํŒ โ€ข ๋ชจ๋ธ ์บ์‹ฑ: ์„œ๋ฒ„ ์‹œ์ž‘ ์‹œ ๋ชจ๋ธ์„ ํ•œ ๋ฒˆ๋งŒ ๋กœ๋“œ
โ€ข ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ: ์—ฌ๋Ÿฌ ์š”์ฒญ์„ ๋ชจ์•„์„œ ํ•œ ๋ฒˆ์— ์ฒ˜๋ฆฌ
โ€ข ๋น„๋™๊ธฐ ์ฒ˜๋ฆฌ: FastAPI + async/await ์‚ฌ์šฉ
โ€ข ๋กœ๋“œ ๋ฐธ๋Ÿฐ์‹ฑ: ์—ฌ๋Ÿฌ ์„œ๋ฒ„ ์ธ์Šคํ„ด์Šค ์šด์˜
โ€ข ์บ์‹ฑ ๋ ˆ์ด์–ด: Redis๋กœ ์ž์ฃผ ๊ฒ€์ƒ‰๋˜๋Š” ๊ฒฐ๊ณผ ์บ์‹ฑ

๐ŸŽ“ ๋” ๊นŠ์ด ํŒŒ๊ณ ๋“ค๊ธฐ: ๊ณ ๊ธ‰ ์ฃผ์ œ๋“ค

๊ธฐ๋ณธ์„ ๋งˆ์Šคํ„ฐํ–ˆ๋‹ค๋ฉด ์ด์ œ ๋” ๊นŠ์€ ๋‚ด์šฉ์„ ์•Œ์•„๋ณผ ์ฐจ๋ก€์•ผ! ๐Ÿค“

Cross-Encoder vs Bi-Encoder

Sentence Transformers๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ Bi-Encoder ๋ฐฉ์‹์ด์•ผ.
ํ•˜์ง€๋งŒ ๋” ์ •ํ™•ํ•œ ๊ฒฐ๊ณผ๊ฐ€ ํ•„์š”ํ•˜๋ฉด Cross-Encoder๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์–ด!

๐Ÿ“Š Bi-Encoder vs Cross-Encoder ๋น„๊ต

ํŠน์ง• Bi-Encoder Cross-Encoder
์†๋„ โšก ๋งค์šฐ ๋น ๋ฆ„ ๐Ÿข ๋А๋ฆผ
์ •ํ™•๋„ ๐Ÿ‘ ์ข‹์Œ ๐ŸŽฏ ๋งค์šฐ ์ข‹์Œ
ํ™•์žฅ์„ฑ โœ… ์šฐ์ˆ˜ โŒ ์ œํ•œ์ 
์‚ฌ์šฉ ์‚ฌ๋ก€ ๋Œ€๊ทœ๋ชจ ๊ฒ€์ƒ‰, 1์ฐจ ํ•„ํ„ฐ๋ง ์ตœ์ข… ์ˆœ์œ„ ๊ฒฐ์ •, ์žฌ์ •๋ ฌ
from sentence_transformers import SentenceTransformer, CrossEncoder, util

# Bi-Encoder๋กœ ํ›„๋ณด ์ถ”์ถœ
bi_encoder = SentenceTransformer('all-MiniLM-L6-v2')
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

query = "ํŒŒ์ด์ฌ์œผ๋กœ ์›น ํฌ๋กค๋งํ•˜๋Š” ๋ฐฉ๋ฒ•"
documents = [
    "Python BeautifulSoup์„ ์ด์šฉํ•œ ์›น ์Šคํฌ๋ž˜ํ•‘",
    "JavaScript๋กœ ์›น์‚ฌ์ดํŠธ ๋งŒ๋“ค๊ธฐ",
    "Selenium์„ ํ™œ์šฉํ•œ ์ž๋™ํ™”",
    "Java ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๊ธฐ์ดˆ",
    "์›น ํฌ๋กค๋ง ๋ฒ•์  ์ด์Šˆ"
]

# 1๋‹จ๊ณ„: Bi-Encoder๋กœ ๋น ๋ฅด๊ฒŒ ์ƒ์œ„ ํ›„๋ณด ์ถ”์ถœ
query_emb = bi_encoder.encode(query, convert_to_tensor=True)
doc_embs = bi_encoder.encode(documents, convert_to_tensor=True)
similarities = util.cos_sim(query_emb, doc_embs)[0]

# ์ƒ์œ„ 3๊ฐœ ์ถ”์ถœ
top_k = 3
top_results = torch.topk(similarities, k=top_k)
candidates = [documents[idx] for idx in top_results[1]]

print("1๋‹จ๊ณ„ - Bi-Encoder ๊ฒฐ๊ณผ:")
for i, doc in enumerate(candidates):
    print(f"{i+1}. {doc}")

# 2๋‹จ๊ณ„: Cross-Encoder๋กœ ์ •ํ™•ํ•˜๊ฒŒ ์žฌ์ •๋ ฌ
pairs = [[query, doc] for doc in candidates]
cross_scores = cross_encoder.predict(pairs)

# ์ ์ˆ˜์ˆœ์œผ๋กœ ์ •๋ ฌ
ranked_results = sorted(zip(candidates, cross_scores), key=lambda x: x[1], reverse=True)

print("\n2๋‹จ๊ณ„ - Cross-Encoder ์žฌ์ •๋ ฌ ๊ฒฐ๊ณผ:")
for i, (doc, score) in enumerate(ranked_results):
    print(f"{i+1}. {doc} (์ ์ˆ˜: {score:.4f})")

๋‹ค์ค‘ ๋ฒกํ„ฐ ํ‘œํ˜„ (Multi-Vector Representations)

๋ฌธ์„œ๊ฐ€ ๊ธธ ๋•Œ๋Š” ์—ฌ๋Ÿฌ ๊ฐœ์˜ ๋ฒกํ„ฐ๋กœ ํ‘œํ˜„ํ•˜๋Š” ๊ฒŒ ๋” ํšจ๊ณผ์ ์ผ ์ˆ˜ ์žˆ์–ด!

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# ๊ธด ๋ฌธ์„œ๋ฅผ ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋ถ„ํ• 
long_document = """
์ธ๊ณต์ง€๋Šฅ์€ ํ˜„๋Œ€ ์‚ฌํšŒ์˜ ํ•ต์‹ฌ ๊ธฐ์ˆ ์ž…๋‹ˆ๋‹ค.
๋จธ์‹ ๋Ÿฌ๋‹๊ณผ ๋”ฅ๋Ÿฌ๋‹์„ ํ†ตํ•ด ๋‹ค์–‘ํ•œ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
์ž์—ฐ์–ด ์ฒ˜๋ฆฌ, ์ปดํ“จํ„ฐ ๋น„์ „, ์Œ์„ฑ ์ธ์‹ ๋“ฑ ์—ฌ๋Ÿฌ ๋ถ„์•ผ์—์„œ ํ™œ์šฉ๋ฉ๋‹ˆ๋‹ค.
์•ž์œผ๋กœ AI ๊ธฐ์ˆ ์€ ๋”์šฑ ๋ฐœ์ „ํ•  ๊ฒƒ์œผ๋กœ ์˜ˆ์ƒ๋ฉ๋‹ˆ๋‹ค.
"""

sentences = [s.strip() for s in long_document.split('.') if s.strip()]

# ๊ฐ ๋ฌธ์žฅ์˜ ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
sentence_embeddings = model.encode(sentences)

# ๋ฐฉ๋ฒ• 1: ํ‰๊ท  ํ’€๋ง
avg_embedding = np.mean(sentence_embeddings, axis=0)

# ๋ฐฉ๋ฒ• 2: ์ตœ๋Œ€ ํ’€๋ง
max_embedding = np.max(sentence_embeddings, axis=0)

# ๋ฐฉ๋ฒ• 3: ๊ฐ€์ค‘ ํ‰๊ท  (์•ž ๋ฌธ์žฅ์— ๋” ๋†’์€ ๊ฐ€์ค‘์น˜)
weights = np.array([1.0, 0.8, 0.6, 0.4])
weighted_embedding = np.average(sentence_embeddings, axis=0, weights=weights)

print(f"๋ฌธ์žฅ ์ˆ˜: {len(sentences)}")
print(f"ํ‰๊ท  ์ž„๋ฒ ๋”ฉ shape: {avg_embedding.shape}")
print(f"์ตœ๋Œ€ ์ž„๋ฒ ๋”ฉ shape: {max_embedding.shape}")
print(f"๊ฐ€์ค‘ ํ‰๊ท  ์ž„๋ฒ ๋”ฉ shape: {weighted_embedding.shape}")

์ž„๋ฒ ๋”ฉ ์ฐจ์› ์ถ•์†Œ

384์ฐจ์›์ด ๋„ˆ๋ฌด ํฌ๋‹ค๋ฉด ์ฐจ์›์„ ์ค„์—ฌ์„œ ์ €์žฅ ๊ณต๊ฐ„๊ณผ ๊ณ„์‚ฐ ๋น„์šฉ์„ ์ ˆ์•ฝํ•  ์ˆ˜ ์žˆ์–ด!

from sentence_transformers import SentenceTransformer
from sklearn.decomposition import PCA
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# ์ƒ˜ํ”Œ ๋ฌธ์žฅ๋“ค
sentences = [f"์ƒ˜ํ”Œ ๋ฌธ์žฅ {i}" for i in range(1000)]

# ์ž„๋ฒ ๋”ฉ ์ƒ์„ฑ
embeddings = model.encode(sentences)
print(f"์›๋ณธ ์ฐจ์›: {embeddings.shape}")

# PCA๋กœ ์ฐจ์› ์ถ•์†Œ (384 -> 128)
pca = PCA(n_components=128)
reduced_embeddings = pca.fit_transform(embeddings)
print(f"์ถ•์†Œ ์ฐจ์›: {reduced_embeddings.shape}")

# ์ •๋ณด ์†์‹ค ํ™•์ธ
explained_variance = sum(pca.explained_variance_ratio_)
print(f"๋ณด์กด๋œ ์ •๋ณด: {explained_variance:.2%}")

# ํฌ๊ธฐ ๋น„๊ต
original_size = embeddings.nbytes / 1024 / 1024  # MB
reduced_size = reduced_embeddings.nbytes / 1024 / 1024  # MB
print(f"\n์›๋ณธ ํฌ๊ธฐ: {original_size:.2f} MB")
print(f"์ถ•์†Œ ํฌ๊ธฐ: {reduced_size:.2f} MB")
print(f"์ ˆ์•ฝ๋ฅ : {(1 - reduced_size/original_size)*100:.1f}%")

๐Ÿ› ํŠธ๋Ÿฌ๋ธ”์ŠˆํŒ…: ์ž์ฃผ ๋งŒ๋‚˜๋Š” ๋ฌธ์ œ๋“ค

๊ฐœ๋ฐœํ•˜๋‹ค ๋ณด๋ฉด ์ด๋Ÿฐ ๋ฌธ์ œ๋“ค์„ ์ž์ฃผ ๋งŒ๋‚˜๊ฒŒ ๋ผ. ๋ฏธ๋ฆฌ ์•Œ์•„๋‘๋ฉด ์‹œ๊ฐ„์„ ๋งŽ์ด ์ ˆ์•ฝํ•  ์ˆ˜ ์žˆ์–ด! ๐Ÿ’ก

โŒ ๋ฌธ์ œ 1: CUDA out of memory GPU ๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ๋ถ€์กฑํ•  ๋•Œ ๋ฐœ์ƒํ•ด.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•:
โ€ข ๋ฐฐ์น˜ ํฌ๊ธฐ ์ค„์ด๊ธฐ: batch_size=16 โ†’ batch_size=8
โ€ข CPU ์‚ฌ์šฉ: device='cpu'
โ€ข ๋” ์ž‘์€ ๋ชจ๋ธ ์‚ฌ์šฉ: all-MiniLM-L6-v2
โ€ข ๊ทธ๋ž˜๋””์–ธํŠธ ๋ˆ„์  ์‚ฌ์šฉ
โŒ ๋ฌธ์ œ 2: ํ•œ๊ตญ์–ด ์„ฑ๋Šฅ์ด ๋‚ฎ์Œ ์˜์–ด ์ค‘์‹ฌ ๋ชจ๋ธ์ด๋ผ ํ•œ๊ตญ์–ด ์ฒ˜๋ฆฌ๊ฐ€ ์•ฝํ•  ์ˆ˜ ์žˆ์–ด.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•:
โ€ข ๋‹ค๊ตญ์–ด ๋ชจ๋ธ ์‚ฌ์šฉ: paraphrase-multilingual-mpnet-base-v2
โ€ข ํ•œ๊ตญ์–ด ๋ฐ์ดํ„ฐ๋กœ ํŒŒ์ธํŠœ๋‹
โ€ข KoBERT ๊ธฐ๋ฐ˜ ๋ชจ๋ธ ์‚ฌ์šฉ
โ€ข ์ „์ฒ˜๋ฆฌ ๊ฐ•ํ™” (๋„์–ด์“ฐ๊ธฐ, ์ •๊ทœํ™”)
โŒ ๋ฌธ์ œ 3: ์ถ”๋ก  ์†๋„๊ฐ€ ๋„ˆ๋ฌด ๋А๋ฆผ ์‹ค์‹œ๊ฐ„ ์„œ๋น„์Šค์—์„œ ์‘๋‹ต์ด ๋А๋ ค.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•:
โ€ข ๋ฐฐ์น˜ ์ฒ˜๋ฆฌ ํ™œ์šฉ
โ€ข ๋ชจ๋ธ ์–‘์žํ™”
โ€ข ONNX๋กœ ๋ณ€ํ™˜
โ€ข ์ž„๋ฒ ๋”ฉ ์บ์‹ฑ
โ€ข GPU ์‚ฌ์šฉ
โŒ ๋ฌธ์ œ 4: ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์ด ๋„ˆ๋ฌด ํผ ์ˆ˜๋ฐฑ๋งŒ ๊ฐœ์˜ ์ž„๋ฒ ๋”ฉ์„ ์ €์žฅํ•˜๋ฉด ๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ๋ถ€์กฑํ•ด.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•:
โ€ข ์ฐจ์› ์ถ•์†Œ (PCA)
โ€ข ๋ฒกํ„ฐ DB ์‚ฌ์šฉ (Pinecone, Weaviate)
โ€ข FAISS ์ธ๋ฑ์Šค ํ™œ์šฉ
โ€ข ๋””์Šคํฌ ๊ธฐ๋ฐ˜ ์ €์žฅ
# ONNX๋กœ ๋ชจ๋ธ ์ตœ์ ํ™” ์˜ˆ์‹œ
from sentence_transformers import SentenceTransformer
import torch

# ๋ชจ๋ธ ๋กœ๋“œ
model = SentenceTransformer('all-MiniLM-L6-v2')

# ONNX๋กœ ๋ณ€ํ™˜
model.save('my_model')

# ๋ณ€ํ™˜๋œ ๋ชจ๋ธ ์‚ฌ์šฉ
# (์‹ค์ œ๋กœ๋Š” onnxruntime ํ•„์š”)
print("๋ชจ๋ธ ์ตœ์ ํ™” ์™„๋ฃŒ!")

๐ŸŽฏ ์‹ค์ „ ํ”„๋กœ์ ํŠธ ์•„์ด๋””์–ด

๋ฐฐ์šด ๋‚ด์šฉ์„ ํ™œ์šฉํ•ด์„œ ์ด๋Ÿฐ ํ”„๋กœ์ ํŠธ๋“ค์„ ๋งŒ๋“ค์–ด๋ณผ ์ˆ˜ ์žˆ์–ด! ๐Ÿš€
ํฌํŠธํด๋ฆฌ์˜ค์—๋„ ์ข‹๊ณ , ์‹ค์ œ ์„œ๋น„์Šค๋กœ๋„ ๋ฐœ์ „์‹œํ‚ฌ ์ˆ˜ ์žˆ์ง€.

๐Ÿ’ผ ํ”„๋กœ์ ํŠธ 1: ์Šค๋งˆํŠธ ๋ฌธ์„œ ๊ฒ€์ƒ‰ ์‹œ์Šคํ…œ ํšŒ์‚ฌ ๋‚ด๋ถ€ ๋ฌธ์„œ, ๋งค๋‰ด์–ผ, FAQ๋ฅผ ์˜๋ฏธ ๊ธฐ๋ฐ˜์œผ๋กœ ๊ฒ€์ƒ‰

ํ•ต์‹ฌ ๊ธฐ๋Šฅ:
โ€ข ์ž์—ฐ์–ด ์งˆ๋ฌธ์œผ๋กœ ๊ด€๋ จ ๋ฌธ์„œ ์ฐพ๊ธฐ
โ€ข ์œ ์‚ฌ ๋ฌธ์„œ ์ถ”์ฒœ
โ€ข ์ž๋™ ํƒœ๊น… ๋ฐ ๋ถ„๋ฅ˜
โ€ข ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ ํ•˜์ด๋ผ์ดํŒ…

๊ธฐ์ˆ  ์Šคํƒ: Sentence Transformers + FAISS + Flask + React
๐Ÿ›’ ํ”„๋กœ์ ํŠธ 2: AI ์ƒํ’ˆ ์ถ”์ฒœ ์—”์ง„ ์‚ฌ์šฉ์ž ์ทจํ–ฅ๊ณผ ์ƒํ’ˆ ์„ค๋ช…์„ ๋ถ„์„ํ•ด์„œ ๋งž์ถค ์ถ”์ฒœ

ํ•ต์‹ฌ ๊ธฐ๋Šฅ:
โ€ข ์ƒํ’ˆ ์„ค๋ช… ๊ธฐ๋ฐ˜ ์œ ์‚ฌ ์ƒํ’ˆ ์ฐพ๊ธฐ
โ€ข ์‚ฌ์šฉ์ž ๋ฆฌ๋ทฐ ๊ฐ์„ฑ ๋ถ„์„
โ€ข ๊ฐœ์ธํ™”๋œ ์ถ”์ฒœ
โ€ข ํŠธ๋ Œ๋“œ ๋ถ„์„

๊ธฐ์ˆ  ์Šคํƒ: Sentence Transformers + PostgreSQL + Django
๐Ÿ’ฌ ํ”„๋กœ์ ํŠธ 3: ์Šค๋งˆํŠธ ์ฑ—๋ด‡ ๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ์ดํ•ดํ•˜๊ณ  ์ ์ ˆํ•œ ๋‹ต๋ณ€ ์ œ๊ณต

ํ•ต์‹ฌ ๊ธฐ๋Šฅ:
โ€ข ์˜๋„ ํŒŒ์•… ๋ฐ ์—”ํ‹ฐํ‹ฐ ์ถ”์ถœ
โ€ข FAQ ์ž๋™ ๋งค์นญ
โ€ข ๋Œ€ํ™” ์ปจํ…์ŠคํŠธ ์œ ์ง€
โ€ข ๊ฐ์ • ์ธ์‹

๊ธฐ์ˆ  ์Šคํƒ: Sentence Transformers + Rasa + FastAPI
๐Ÿ“ฐ ํ”„๋กœ์ ํŠธ 4: ๋‰ด์Šค ํ๋ ˆ์ด์…˜ ์„œ๋น„์Šค ์‚ฌ์šฉ์ž ๊ด€์‹ฌ์‚ฌ์— ๋งž๋Š” ๋‰ด์Šค ์ž๋™ ์ถ”์ฒœ

ํ•ต์‹ฌ ๊ธฐ๋Šฅ:
โ€ข ๋‰ด์Šค ์ž๋™ ๋ถ„๋ฅ˜
โ€ข ์ค‘๋ณต ๊ธฐ์‚ฌ ์ œ๊ฑฐ
โ€ข ๊ฐœ์ธํ™” ํ”ผ๋“œ
โ€ข ํŠธ๋ Œ๋”ฉ ํ† ํ”ฝ ๋ถ„์„

๊ธฐ์ˆ  ์Šคํƒ: Sentence Transformers + Scrapy + MongoDB
๐ŸŽ“ ํ”„๋กœ์ ํŠธ 5: ํ•™์Šต ์ž๋ฃŒ ์ถ”์ฒœ ์‹œ์Šคํ…œ ํ•™์ƒ์˜ ์ˆ˜์ค€๊ณผ ๊ด€์‹ฌ์‚ฌ์— ๋งž๋Š” ํ•™์Šต ์ž๋ฃŒ ์ถ”์ฒœ

ํ•ต์‹ฌ ๊ธฐ๋Šฅ:
โ€ข ๋‚œ์ด๋„ ์ž๋™ ๋ถ„๋ฅ˜
โ€ข ์œ ์‚ฌ ๊ฐ•์˜/์ž๋ฃŒ ์ถ”์ฒœ
โ€ข ํ•™์Šต ๊ฒฝ๋กœ ์ œ์•ˆ
โ€ข ์งˆ๋ฌธ-๋‹ต๋ณ€ ๋งค์นญ

๊ธฐ์ˆ  ์Šคํƒ: Sentence Transformers + Neo4j + Vue.js

๐Ÿ“š ๋” ๊ณต๋ถ€ํ•˜๊ณ  ์‹ถ๋‹ค๋ฉด?

Sentence Transformers๋ฅผ ๋งˆ์Šคํ„ฐํ•˜๋Š” ์—ฌ์ •์€ ์—ฌ๊ธฐ์„œ ๋์ด ์•„๋‹ˆ์•ผ!
๊ณ„์† ๋ฐœ์ „ํ•˜๊ณ  ์žˆ๋Š” ๋ถ„์•ผ๋‹ˆ๊นŒ ๊พธ์ค€ํžˆ ๊ณต๋ถ€ํ•ด์•ผ ํ•ด. ๐Ÿ“–

๐ŸŒŸ ์ถ”์ฒœ ํ•™์Šต ์ž๋ฃŒ ๊ณต์‹ ๋ฌธ์„œ ๋ฐ ํŠœํ† ๋ฆฌ์–ผ:
โ€ข Sentence Transformers ๊ณต์‹ ๋ฌธ์„œ
โ€ข Hugging Face ๋ชจ๋ธ ํ—ˆ๋ธŒ
โ€ข SBERT.net ํŠœํ† ๋ฆฌ์–ผ

๋…ผ๋ฌธ:
โ€ข "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks"
โ€ข "Making Monolingual Sentence Embeddings Multilingual"
โ€ข "Augmented SBERT"

์ปค๋ฎค๋‹ˆํ‹ฐ:
โ€ข Hugging Face ํฌ๋Ÿผ
โ€ข Reddit r/MachineLearning
โ€ข Stack Overflow
โ€ข GitHub Issues

์‹ค์Šต ํ”Œ๋žซํผ:
โ€ข Kaggle ๋Œ€ํšŒ ๋ฐ ๋…ธํŠธ๋ถ
โ€ข Google Colab
โ€ข Paperspace Gradient

๊ทธ๋ฆฌ๊ณ  ์‹ค์ œ๋กœ ํ”„๋กœ์ ํŠธ๋ฅผ ๋งŒ๋“ค์–ด๋ณด๋Š” ๊ฒŒ ๊ฐ€์žฅ ์ค‘์š”ํ•ด!
์ด๋ก ๋งŒ ๊ณต๋ถ€ํ•˜์ง€ ๋ง๊ณ  ์ง์ ‘ ์ฝ”๋“œ๋ฅผ ์งœ๊ณ , ์—๋Ÿฌ๋ฅผ ๋งŒ๋‚˜๊ณ , ํ•ด๊ฒฐํ•˜๋ฉด์„œ ๋ฐฐ์›Œ์•ผ ์ง„์งœ ์‹ค๋ ฅ์ด ๋Š˜์–ด. ๐Ÿ’ช

๐ŸŽฌ ๋งˆ๋ฌด๋ฆฌํ•˜๋ฉฐ

์ž, ์—ฌ๊ธฐ๊นŒ์ง€ Sentence Transformers๋กœ ๋ฒกํ„ฐ ์ž„๋ฒ ๋”ฉ์„ ์ƒ์„ฑํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋ดค์–ด! ๐ŸŽ‰

์ฒ˜์Œ์—๋Š” ๋ณต์žกํ•ด ๋ณด์˜€์„ ์ˆ˜๋„ ์žˆ์ง€๋งŒ, ์‹ค์ œ๋กœ ์‚ฌ์šฉํ•ด๋ณด๋ฉด ์ƒ๊ฐ๋ณด๋‹ค ๊ฐ„๋‹จํ•˜์ง€?
๋ช‡ ์ค„์˜ ์ฝ”๋“œ๋งŒ์œผ๋กœ ๊ฐ•๋ ฅํ•œ ์ž์—ฐ์–ด ์ฒ˜๋ฆฌ ๊ธฐ๋Šฅ์„ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฒŒ ์ •๋ง ๋†€๋ผ์›Œ! ๐Ÿ˜ฎ

์ด ๊ธฐ์ˆ ์€ ์ง€๊ธˆ๋„ ๊ณ„์† ๋ฐœ์ „ํ•˜๊ณ  ์žˆ์–ด.
์ƒˆ๋กœ์šด ๋ชจ๋ธ๋“ค์ด ๊ณ„์† ๋‚˜์˜ค๊ณ , ์„ฑ๋Šฅ๋„ ์ ์  ์ข‹์•„์ง€๊ณ  ์žˆ์ง€.
๊ทธ๋Ÿฌ๋‹ˆ๊นŒ ๊ธฐ๋ณธ์„ ํƒ„ํƒ„ํžˆ ๋‹ค์ ธ๋‘๋ฉด ์•ž์œผ๋กœ ๋‚˜์˜ฌ ์ƒˆ๋กœ์šด ๊ธฐ์ˆ ๋“ค๋„ ์‰ฝ๊ฒŒ ๋ฐฐ์šธ ์ˆ˜ ์žˆ์„ ๊ฑฐ์•ผ! ๐Ÿš€

์žฌ๋Šฅ๋„ท์—์„œ๋„ ์ด๋Ÿฐ AI ๊ธฐ์ˆ ์„ ํ™œ์šฉํ•œ ๋‹ค์–‘ํ•œ ํ”„๋กœ์ ํŠธ๋“ค์„ ๋งŒ๋‚˜๋ณผ ์ˆ˜ ์žˆ์–ด.
ํ˜น์‹œ ํ”„๋กœ์ ํŠธ๋ฅผ ์ง„ํ–‰ํ•˜๋‹ค๊ฐ€ ๋ง‰ํžˆ๋Š” ๋ถ€๋ถ„์ด ์žˆ๋‹ค๋ฉด,
์ „๋ฌธ๊ฐ€๋“ค์˜ ๋„์›€์„ ๋ฐ›๋Š” ๊ฒƒ๋„ ์ข‹์€ ๋ฐฉ๋ฒ•์ด์•ผ! ๐Ÿ’ก

๊ธฐ์–ตํ•ด! ์™„๋ฒฝํ•œ ์ฝ”๋“œ๋ฅผ ์ฒ˜์Œ๋ถ€ํ„ฐ ์งœ๋Š” ์‚ฌ๋žŒ์€ ์—†์–ด.
์—๋Ÿฌ๋ฅผ ๋งŒ๋‚˜๊ณ , ๋””๋ฒ„๊น…ํ•˜๊ณ , ๊ฐœ์„ ํ•˜๋Š” ๊ณผ์ •์ด ๋ฐ”๋กœ ์„ฑ์žฅํ•˜๋Š” ๊ณผ์ •์ด์•ผ.
ํฌ๊ธฐํ•˜์ง€ ๋ง๊ณ  ๊ณ„์† ๋„์ „ํ•ด๋ด! ๐Ÿ”ฅ

๊ทธ๋Ÿผ ์ด์ œ ์ง์ ‘ ์ฝ”๋“œ๋ฅผ ์งœ๋ณด๋ฉด์„œ ์‹ค์Šตํ•ด๋ณด์ž!
์งˆ๋ฌธ์ด ์žˆ๊ฑฐ๋‚˜ ๋ง‰ํžˆ๋Š” ๋ถ€๋ถ„์ด ์žˆ์œผ๋ฉด ์–ธ์ œ๋“  ์ปค๋ฎค๋‹ˆํ‹ฐ์— ๋ฌผ์–ด๋ด.
์šฐ๋ฆฌ ๋ชจ๋‘ ํ•จ๊ป˜ ๋ฐฐ์šฐ๊ณ  ์„ฑ์žฅํ•˜๋Š” ๊ฑฐ๋‹ˆ๊นŒ! ๐Ÿค

ํ–‰์šด์„ ๋นŒ์–ด! ํ™”์ดํŒ…! ๐Ÿ’ชโœจ

๐ŸŽ“ Happy Coding! Sentence Transformers Master
๋Œ“๊ธ€ ์ž‘์„ฑ

์ด ๊ธ€์— ๋Œ€ํ•œ ์—ฌ๋Ÿฌ๋ถ„์˜ ์ƒ๊ฐ์„ ๋“ค๋ ค์ฃผ์„ธ์š”

๋Œ“๊ธ€ 0
๋Œ“๊ธ€์„ ๋ถˆ๋Ÿฌ์˜ค๋Š” ์ค‘์ž…๋‹ˆ๋‹ค.

์•„์ง ๋Œ“๊ธ€์ด ์—†์Šต๋‹ˆ๋‹ค.