← All projectsAdvanced · About 12–16 hours
Databricks document pipeline with vector search
Ingest documents into Delta tables, chunk and embed them in a scheduled job, and serve retrieval.
Good for: Data engineer, AI engineer, Data scientist
What you will have at the endA lakehouse pipeline (bronze, silver, gold) that keeps an index of document chunks fresh and answers questions over it.
- Delta Lake
- PySpark
- Databricks Jobs
- Unity Catalog
- Vector search
- Incremental processing
New to this? Do the toolkit setup first, then add these one at a time.
- 1
Workspace, notebooks, jobs and catalog.
# sign up for Databricks Free Edition
$ brew tap databricks/tap && brew install databricks
$ databricks auth login --host <your-workspace-url>
- 2
- 3
How to show it to an employer
1Job run screenshot and lineage graph
2README with a medallion diagram and the incremental logic
3Cost and timing table
Stretch goals
Data quality checks that fail the job
Cost per 1,000 documents in the README
A small RAG app on top of the index
Questions this prepares you for
How do you make embedding incremental?
Why bronze, silver and gold?
How do you govern who can read the chunks?
Practise them in a mock interview Next projectAzure OpenAI assistant over company data→