RAG Training Ground
Mines historical hosting conversations through the lakehouse into a vector library, teaching the agent to answer guests in my own voice.
Teaching a machine to sound like me
If you want an AI to replace you, you first have to let it stalk you.
The first draft of my co-host sounded like every other chatbot on earth — "Greetings, esteemed traveler!" — and guests can smell that from a mile away. I didn't want a polite robot. I wanted something that answered the way I answer: my real early-check-in policy, how I actually handle a noise complaint, the exact way I reassure someone that yes, the Wi-Fi router is plugged in.
That voice doesn't come from a prompt. It comes from data — every conversation I'd ever had with a guest. The catch: that history was trapped in two very different places, my Gmail inbox and Airbnb's host dashboard.
Two ways to liberate a conversation
The civilized path — Gmail. Google actually ships a documented API, so this one stayed clean. A small OAuth2 script authenticates once, caches a token.json, and quietly pulls every email thread tied to my listings — no browser, no scraping.
The trench-warfare path — Airbnb. Airbnb guards its API like launch codes and buries host–guest chat behind a fortified React portal, so the polite route was off the table. Back came Playwright. The scraper drives a headless browser, but the real trick is a saved session: I clear 2FA and the captcha once, persist the session to airbnb_state.json, and from then on the browser injects the cookie, walks the inbox, and parses the DOM — even reading the little middle-dot (·) separators in each bubble to tell who said what: guest, host, or the automated system.
These extractors are heavy and authenticated. Running them on every pipeline pass is wasted compute and a fast way to get a host account flagged as a bot. So they sit behind a single qa_extractor toggle in the orchestrator, flipped to True only when enough new conversations have piled up to be worth mining.
# Toggle this to True when there are enough new conversations to mine
qa_extractor = False
if qa_extractor:
run_pipeline_step([venv_python, "src/extractors/gmail_connect.py"],
"TRAINING EXTRACTION (Gmail API)", run_dir=project_root)
run_pipeline_step([venv_python, "src/extractors/inbox_scrapper.py"],
"TRAINING EXTRACTION (Airbnb Inbox)", run_dir=project_root)
The output is thousands of clean, chronological JSON dialogue trees. But raw JSON is the last thing you want to hand a language model — it's a swamp of React artifacts, HTML boilerplate, and inconsistent timestamps. Smart agents need clean data, so the conversations run through the same Medallion architecture I use for market data — except this time we're refining human dialogue, not prices.
Bronze → Silver → Gold, for conversations
Bronze is the quarantine zone: an append-only Delta table where the wildly different Gmail and Playwright JSON structures land untouched. No cleaning, no judgment — just an immutable, version-controlled record of exactly what was scraped.
Silver is where PySpark does the heavy lifting. It shreds the nested arrays, strips the HTML, standardizes timestamps, and tags every line with a role — guest vs. host. It also deduplicates, because Airbnb loves to deliver the same message as both an email and an in-app notification. What leaves Silver is a clean, chronological transcript of every interaction.
Gold turns transcripts into teaching material. It drops the noise — the endless "Thanks!" / "You're welcome!" threads — isolates the high-value question/answer pairs, chunks the text, and pulls out my actual host policies, formatted into structured prompt/completion pairs.
run_pipeline_step([venv_python, "src/etl/silver_qa.py"], "SILVER LAYER Q&A (Clean)", run_dir=project_root)
run_pipeline_step([venv_python, "src/etl/gold_qa.py"], "GOLD LAYER Q&A (RAG dataset)", run_dir=project_root)
The Gold table stops being a dataset and becomes a contextual library. Every chunk gets embedded into a vector store, and at runtime the co-host retrieves the closest past conversations to whatever a guest just asked — then answers in my voice instead of a generic one. That retrieval step is the entire point of the next project.
What it produces
A searchable library of my own hosting history — every real answer I've ever given, cleaned, chunked, embedded, and ready to retrieve. It's the difference between an assistant that sounds like a brand and one that sounds like the actual host. Next stop: wiring it into a live, tool-using agent that handles guests on its own.