Split documents into passages small enough to match a question, big enough to make sense.
- LoadRead PDFs, pages and docs as text.
- SplitCut by headings or about 500 tokens.
- OverlapRepeat a little text between chunks.
- MetadataKeep title, section and page with each chunk.
Explain 1/5
Source: LangChain: Text splitters