Day 1 What I learn From Making Rag by using go instead of Python

🛑 what is Overlap ?
⭕️ what is With vs. Without👇?
Question: Show me how overlap looks with and without it.
Concept:

When we cut text into chunks, we risk cutting a sentence in half. If Chunk 1 ends with "The capital of Ethiopia is" and Chunk 2 starts with "Addis Ababa.", the AI might not understand Chunk 1 fully. Overlap repeats some words at the end of Chunk 1 in the beginning of Chunk 2.

Example Text:

"The quick brown fox jumps over the lazy dog. The cat sat on the mat."

Let's say Size = 5 words, Overlap = 2 words.

WITHOUT Overlap (Step = Size = 5):
Chunk 1: "The quick brown fox jumps"
Chunk 2: "over the lazy dog. The"
Chunk 3: "cat sat on the mat."
Problem: Chunk 1 cuts off "jumps". Chunk 2 starts with "over". The context "jumps over" is split.

WITH Overlap (Size = 5, Overlap = 2, Step = 3):

Step = Size - Overlap = 5 - 2 = 3.

Chunk 1 (Words 0-4): "The quick brown fox jumps"
Chunk 2 (Words 3-7): "fox jumps over the lazy"
Notice: "fox jumps" is repeated. Now Chunk 2 has the context of what happened before "over".
Chunk 3 (Words 6-10): "the lazy dog. The cat"
Chunk 4 (Words 9-13): "The cat sat on the mat."
Reason: Overlap ensures that no important context is lost at the boundary of two chunks.