I’ve read a lot of LLM chats, not of BayLeaf users, but of users of a precursor system that I ran in 2024. There’s a lot you can learn about how they think by seeing how they write to these machine ghosts. Since BayLeaf started operating, however, I’ve barely read any. This post explores a way to let me get a sense of what people are doing with the BayLeaf Chat service without, for their safety and my own, me directly eyeballing raw conversation data.
Chat is the only part of BayLeaf that records the content of your messages, and even those are deleted automatically after a while. Of your conversations, my privacy statement says “They are accessible only to you and the system administrator.” That’s me, the system administrator. If I’m going to keep operating BayLeaf, I want some assurance that my system isn’t responding in ways that are too cringe. I want to know what my system is talking about, at the level of broad categories, and, for sensitive categories, I want to use my privileged access to spot check a few items. Eventually, I’ll make this last level of access impossible, even for myself.
A few years ago, the folks over at Anthropic shared the design of Clio: “a system for privacy-preserving insights into real-world AI use.” The core idea is to do the first few levels of analysis of raw customer data automatically, only letting human analysts get involved after a few layers of summarization and aggregation. In literary studies, we might call this a distant reading technique (in contrast to close readings where the multiple meanings of specific words and phrases might be scrutinized). Clio works through four phases: extracting facets (extracting rough summaries and metadata), semantic clustering (grouping related conversation by theme or general topic), cluster description (giving clusters a descriptive title and summary without bubbling up sensitive data), and building hierarchies (assembling clusters of clusters until a digestible bigger picture emerges). I ran something similar.
The first phase of my pipeline was conversation data extraction. This was a little tricky because conversations on BayLeaf Chat are structured as trees rather than linear lists of messages and that the storage format has changed over the last few months. The only interesting move I made in the extraction phase was to mask out my own usage data. I already know what I do with BayLeaf, so my analysis was focused on others.
Next came the compaction phase. Even though users don’t spend a lot of time typing messages into BayLeaf Chat, conversations can get pretty large when they upload or paste large documents, elicit long replies from the agent, or when the agent does several rounds of interaction with third party services (e.g. web search) before formulating the user-facing response. My strategy was to mid-truncate user messages and agent replies (keeping the first and last chunks of a message and throwing away anything left in the middle) and to discard any reasoning blocks or tool call results. This discarded about 89% of the total text volume keeping what I assumed to be the interesting parts.
For facet extraction, I ran each of the compacted conversation streams through a mid-sized language model (DeepSeek V4 Flash 0731, the currently recommended model on the BayLeaf API). I asked the model to produce a topic (2–6 word phrase), user message language (e.g. “en” or “zh”), and summary (1–2 sentence de-identified gist of the user’s intent). This analysis was pretty cheap. I spent about $0.08 in total for facet extraction (compared to a total of about $3.15 on a coding agent based on the same model to execute the rest of this project and help edit the blog post).
Once I had the facets extracted for each conversation, I needed a way to group related conversations together. It’s not completely unreasonable to simply ask another LLM to name the common patterns, but I opted for something more mechanical and scalable: classical unsupervised machine learning. I used OpenAI’s text-embedding-3-small model to get a vector for each conversation’s extracted topic, then I threw those through HDBSCAN. HDBSCAN works by estimating how densely clustered each conversation is in the vector space, growing clusters out of regions of stable density and keeping the ones that persist across density thresholds. Conversations too spread out to belong to any stable region are left over, and I treated that leftover as its own long-tail bucket rather than forcing it into a cluster. I think that, in Anthropic’s approach, human analysts got involved in some of these groupings, but it was fully automatic in my design. I don’t think my design is better, just a faster path to a first look at the big picture.
Once I had the conversations clustered, I wanted to be able to describe the result. Taking the top twelve or so clusters by size, I again asked a mid-sized language model to come up with a de-personalized summary. This time, the describer model only got access to the per-conversation summaries (not the super-short topic or the longer, compacted message history). For each cluster, the describer gave me a 2–5 word title naming the theme of the cluster and 1–2 sentence summary of what the items in the cluster had in common.
So, what did I find? Here are the model-generated titles and summaries, in order of decreasing cluster size. (Does Substack not support tables?? Okay, the format here is going to be “Title: Summary”.)
Rust GUI development & tutorial building: One sustained project: egui/eframe/GPUI node editors, mdBook tutorial sites, prompt-crafting for AI coding
Model identity & privacy inquiries: ”What model are you?”, “who’s the provider?”, “what’s your data retention?”
Football simulation development: Python NFL-sim: scheduling, drafts, GM logic, play-by-play
Academic recommendation letters: Rec letters, grade appeals, GSI support letters, program-status decisions
AI in learning & game-assistance research: Literature searches on AI tutoring, self-directed learning, player-assistance
Reflective academic writing: Essays, self-evaluations, fellowship statements (ethnic studies, education)
Hilly-commute vehicles: E-bikes/scooters for steep campus terrain
Marine biology coursework: Right whales, energetics, lecture review
Linux setup & troubleshooting: Drivers, displays, VPN
Campus mascot humor: UCSC jokes (banana slugs, hills)
Agentic coding model selection: OpenChamber vs Cline, BayLeaf/OpenRouter config
Faculty burnout & boundaries: Educators in distress seeking crisis support, boundaries, exit strategies
Physics homework: E&M, circuits
…plus ~12 smaller clusters: Phaser, LaTeX, Canvas LMS, salamanders, SSH, AI-detection, etc.
Long tail: One-off personal queries: recipes, dating, health, flights, taxes, jokes
As someone with a very personal interest in the topic of “Faculty burnout & boundaries”, I did drill down to the raw data for a few of these. I can report that the agent is responding well enough, the facet extraction and cluster summaries are accurate enough, and, ugh, it’s a good thing that I’ve found a way to maintain a lot of distance from the raw data over the recent two years. I’m glad my system could provide someone support, but I’m also glad I didn’t get sucked into trying to provide that support directly myself.
Even though this analysis is basically fully automated, I’m not sure that I want to rig it up to run regularly. I’m writing this blog post as a starting point for in-person conversations about how much a community should be allowed to know about what others in that community are doing in chats that feel private. It is one thing for Anthropic to use Clio on a tiny fraction of data from their international user base. It’s another to run it for just my single-campus community.