Today, I’m blocking access to closed-weight models in the BayLeaf API.

Why make such a big deal about whether you can download the model weights? I don’t think I want to try to convince you on this specific point today. Some of you might already be sold on it on the basis of relational reciprocity (giving back to the Web that made these models possible) or free-market economics (access to model weights enables an ecosystem where many different implementations of AI can thrive). Others might just read it as me trying to argue against the generic architectural pattern of software as a service, which (and you’re so smart) I seem to use all over the BayLeaf project, actually.

I’m making a big deal of it because I think some big deals ought to be made, with promises, compromises, and with robust and inspectable implementations. Let’s draw some lines, propose not to cross them, and see what living with that decision feels like (at least for a while). At this specific moment, I don’t think it is super important precisely where those lines get drawn.

Here are a few packages I’m imagining:

• No closed-weight models (the one I’m implementing): Just don’t serve inference for models for which we can’t find an associated public weight repository (e.g. a HuggingFace link). This might lock us out of future proprietary models that make more efficient use of energy (proxied by inference cost in dollars) or human attention (proxied by number of agent steps to complete a task). However, because whether the models are downloadable isn’t intrinsically coupled to efficiency and other issues, the risk of potential, future, temporary regret seems low. Seems reasonable. Let’s do it. Done, even if it reverses one of my earlier deliberate choices to preserve access for research and comparison.

• No data or money for inference to problematic providers (or whoever is suitably OpenAI/Google/Anthropic-like to attract your ire): I like the hit-em-where-it-hurts spirit of this one, but it gets tricky to implement. Can data flow back to OpenAI for the use of their open-weight GPT-OSS models? Can we use GPT-OSS from a non-OpenAI provider? Can money flow to Amazon, who pays Anthropic behind the scenes? What about how Anthropic pays SpaceXAI behind the scenes for Colossus? Can we still use Google Drive or even Google Search if we’re boycotting Gemini? If we effectively reward Google for good behavior by paying them for inference of open-weight (or other less-problematic category of) models through their Model-as-a-Service (MaaS) system, are we still feeding the beast? This confusion is what kept drawing me back to just blocking closed-weight models and seeing what it is like to live with that choice for a while.

• Model–harness disaggregation: Microsoft once famously got in big trouble for bundling a browser too tightly with an operating system, but it was fine for Google to do the same a decade later, at least for a while. There’s currently tension around how strongly Google pushes to be the default search engine in beyond-Google browsers. Something analogous is happening with mobile operating systems and their associated (or importantly not-associated) app stores. In the generative AI space, I can foresee problematic power relations when models and harnesses are too tightly integrated. Currently, you can use non-OpenAI models inside of OpenAI’s Codex harness, and you can use Claude models in harnesses beyond Claude Code. But you can’t leverage your Claude subscription when you leave an Anthropic-provided harness. And the situation between Cursor (the harness), Cursor Composer (the model), and being acquired by SpaceXAI (who makes their own models and harnesses with different degrees of mandatory coupling) is rapidly changing as I write this. Rather than a wait-and-see, sure-hope-we-don’t-get-locked-in strategy, we could just commit to exercising the discipline of never running a harness that comes from the same organization that provides the model. BayLeaf encourages but does not enforce this usage pattern. Trying to live by this one myself has gotten me to mostly give up on Anthropic models, because I can’t access them in a cost-efficient way without leaving my agentic home in OpenCode and other open-source harnesses that I’ve contributed to. When model–harness antitrust action comes, I’ll be ready, but it doesn’t feel like a big issue for others.

• Capped context windows: This one is trying to merge some kind of self-care practice with labor action by way of an inference server parameter tweak, but there is some logic to it. Context windows limit how far back in a conversation (or other agent activity trace) a model can reach when trying to decide how to respond. What happens if we cap context windows on supported inference endpoints at, say, sixteen thousand tokens? (This is maybe comparable to twenty-five pages of text.) This limits how deep of a relationship you can get into with your robot buddy. It limits how complex of a job your robot buddy can do after they let you go. It limits how lost you can get in a conversation before the session hits a hard stop. It also puts a nice low cap on your maximum per-request spend, far below the cost associated with those long-horizon agentic traces that many model-trainers like to brag about. It’s not a cap on the agent’s intelligence, maybe just on the scope of the application of that intelligence. Someone needs to test this to see if the effects on self-care or labor displacement point in the right direction, but it would be super cheap to implement.

Backing up to the title of this post, I’m blocking access to closed-weight models only a little bit because I care about reciprocity and healthy markets. I mostly want to get people thinking about specific, actionable demands and to have municipal-scale institutions adopt them (even if only temporarily) so we can all gain experience with what’s feasible and effective.

(Because my agents have the data at hand: “Models without published weights accounted for about 33% of BayLeaf’s OpenRouter inference spending in August. Claude Sonnet 5 alone accounted for 23% of total spending, and Claude Opus 4.8 accounted for another 7%. The remaining 3.5% went mostly to Grok, with smaller amounts spent on Gemini and GPT models.”)

One lasting effect of that history is that, for a certain population, “military-grade encryption” has a very precise connotation: It’s the actual encryption you can’t have, while the encryption you can have isn’t actually that good. It’s unsafe to let others have it. Or it’s too expensive to let you have it. The only way to stop a bad guy with a gun is for the other guys with guns to be able to read your email, or something like that. There is no alternative.

To rapidly bring this back to generative AI in higher education, I’m going to talk about who can see the data processed by LLMs. It might seem that if a language model needs to read your text to understand it, and a remote provider runs the model, then the provider needs to be able to read your text. I can require that it be encrypted on the way there and that the provider encrypt it when stored (or simply not store it at all). But surely the provider needs to be able to read it while the model is working? Nope! The model needs to to read it, but the model provider doesn’t.

It’s time for some military-grade encryption, of a sort.

Suppose you are the United States Department of War, and you need a good coding agent for, you know, war stuff. Do you just make a POST request to https://api.openai.com/v1 like any other customer? Probably not. I can’t find the precise details of how it all works. I got lost among FedRAMP, DISA, ESI, IL5 and IL6, why the Pentagon cleared AWS, Google, Microsoft, OpenAI, and others but not Anthropic, blah. Let’s work from basic words on public webpages, not acronyms and mysterious dead-end landing pages.

Our sister campus UC Riverside was proud to announce a partnership with Google to address “technical red tape and rigorous security and technical oversight that occasionally forced the university to decline critical funding.” The solution was a “secure enclave” with a “secure network boundary that keeps sensitive information such as Controlled Unclassified Information (CUI) protected.”

Here I hit an delightful name collision (or perhaps an aspirational declaration of intent). UCR’s Secure Enclave seems to be a tightly controlled, monitored Google Cloud environment built with Stellar Engine (an open-source Terraform framework for deploying infrastructure that maps onto federal security requirements). The enclosure is mostly administrative and architectural: identity controls, isolated networks, managed encryption keys, restricted ingress and egress, extensive logging, and a substantial pile of compliance documentation. Stellar Engine can also provision Confidential VMs, but UCR’s public documentation doesn’t say whether it uses them. More importantly, it describes authorized UCR staff managing the environment and its decryption keys. This kind of secure enclave controls which operators may see the data. It doesn’t necessarily mean operator access cryptographically impossible. Zero operator access (ZOA) is the specific property I’m itching for.

Meanwhile, Google sells something closer to another meaning of secure enclave. Confidential Space runs a workload inside a hardware-backed trusted execution environment (TEE). Remote attestation lets someone check which code is running before releasing secrets to it, while memory encryption and a hardened operating system keep the workload hidden even from the people operating the surrounding infrastructure (awesome!). Skipping some amazing computer science and engineering detail (including cryptography), this is how a cloud provider can offer compute without retaining the ordinary technical ability to peek at what is being computed. It is a bit more expensive and complicated than the typical way of running cloud services, but it is an actual commercially available alternative. Many organizations want this level of privacy, not just those with guns.

Google isn’t the only player in the cryptographically sealed enclave game. Other cloud providers offer comparable systems, including AWS with its Nitro Enclaves. Specifically in the world of confidential LLM inference, ConfidentialInference.net tracks available providers and their model catalogs, tagged by security architecture. When a provider combines remote attestation with end-to-end encryption to the attested workload, you are getting, to my knowledge, at least the cryptographic component of military-grade encryption for LLMs. (I’m sure there’s a “does the guy guarding the door have a gun?” aspect that I’m leaving out here.)

Can we have this at UC Santa Cruz, to protect our student data, our research data, and our late-night, stressed-out rants about housing (un)afforability? Yes, and it doesn’t even need to be difficult for users to set up.

When you are having that sensitive chat, my recently-added agent skill will nudge you away from BayLeaf Chat and into a desktop app running on your own computer, such as OpenCode or OpenChamber. I give you one-line setup commands like opencode auth login https://api.bayleaf.dev to hook those desktop apps with cloud inference services. Until last night, this configured only standard LLM inference services: encrypted transport to a provider that contractually promises not snarf your data. But now a similar command also configures our sealed LLM inference service. Your device verifies the enclave’s attestation, then encrypts your conversation to a key bound to that enclave. Neither the provider’s ordinary infrastructure nor the BayLeaf operator (that’s me) is capable of getting at the plaintext. Per-token prices are higher and the model catalog is a bit sparse, but it works.

Skipping back, you don’t use one of these confidential inference services simply by making a POST request to https://api.bayleaf.dev/v1. That would use encrypted transport to deliver plaintext to the ordinary BayLeaf API route. Instead, you need extra code on your computer to verify the attestation and pack up your data in a form that can be unpacked only inside the secure enclave, nowhere in between. Most agent harnesses don’t support this, and I didn’t want to make users install a separate desktop sidecar app that they would need to launch before their agent started working. So, for OpenCode, I created and published the opencode-tinfoil plugin for integrating Tinfoil-compatible confidential LLM inference providers, including the one available through the BayLeaf API. Once you have it set up, you’ll see a new batch of models in the picker:

This isn’t IL7, but it is better than you can get in any Education package from a major AI provider.

We don’t spend big on BayLeaf. We don’t build much, and we don’t buy much either. We don’t train models, don’t build inference servers, don’t assemble physical hardware, don’t develop general-purpose chat interfaces, and don’t build coding-agent harnesses. Okay, we did make one coding-agent harness, Lathe, but we spun that out into a BayLeaf-independent project immediately after it was useful enough to be worth sharing. We do, however, contribute patches upstream to Open WebUI, OpenCode, and OpenChamber. We do use LLM inference through OpenRouter and Tinfoil, web context from Tavily, and sandboxes from Daytona. We do have a custom service running for the BayLeaf API, but did we make the right build versus buy decision there? A recent, uh, incident got me thinking.

Earlier today I realized a problem with the BayLeaf API that might have been caught if there were more than one person looking over the implementation. Users of the BayLeaf API (at least the part of it that wraps OpenRouter) are subject to a daily spending limit. However, users could effectively bypass that limit by repeatedly revoking and re-provisioning their BayLeaf API key. Users need to be able to rotate their keys in response to personal leaks, so it’s a legitimate usage pattern. My earlier implementation of the BayLeaf API specifically accounted for this exact potential abuse pattern, but I didn’t closely track the implementation for regressions. Later work to add Sealed inference accidentally changed the key lifecycle: re-provisioning a BayLeaf key caused the next request to mint a fresh back-end key, resetting the provider-side spending limit. There’s a more precise writeup of the incident on GitHub. What matters is that I made a mistake, it cost me some money (perhaps $300), and it got me thinking.

Of course the buggiest parts of BayLeaf are in the parts we built ourselves. But why did we need to build the BayLeaf API in the first place? Other campuses are deploying the free and open-source API Gateway LiteLLM, even the NSF-funded National Research Platform LLM Service. Should I have just used LiteLLM? This could mean less building, more reuse, and more opportunities to have local frictions turn into upstream contributions with global impact. I may yet decide to adopt LiteLLM, but it would currently run counter to a strategy to reduce total system complexity and institutional cost or environmental impacts.

Any institutional adopter of a gateway product, whether self-hosted like LiteLLM or managed like PortKey, is going to need to glue the LLM inference service to institutional identity and policy: people need a way to log in with their existing accounts. The trouble is that people need more than just LLM inference services to make widely productive use of generative AI technology. The BayLeaf API abstracts over the web fetch and web services provided by Tavily and the isolated code execution sandboxes provided by Daytona. We could have decided to build these versus buy them on a per-usage basis, but we’d still need them. The LiteLLM developers know this, and they have special-case support for web search integration and code interpreters. However, these are slight generalizations of the instantiations of those ideas present in ChatGPT, not the weird and wild versions we have. So, let’s presume any campus-scale adopter is going to build just enough custom glue to solve at least authentication and moderate extensibility (rather than getting lost in LiteLLM’s plugin system). For BayLeaf, that ease of extensibility bought us quick integration of E2EE inference providers, smoothed desktop harness onboarding, a place to document how to use coding agents with Canvas, etc.

It’s hard to speculate whether this kind of incident would have been just as likely if I had been using LiteLLM (or PortKey) instead of OpenRouter for the normal LLM inference path. But maybe?

My cool-headed editing agent, who didn’t just find $300 on the floor, wants to put it this way: “The local seam cannot be purchased away. Therefore, reduce its scope, reuse everything on either side of it, and recycle its inevitable failures into shared institutional memory.” Fine. In this case, recycling means turning the incident into automated lifecycle assertions and this public reflection. Zooming out, what else can we reduce, reuse, or recycle through some different combination of building, buying, or simply b-not engaging? I’ll be on the lookout.

Ah, because this whole incident started with me looking for where the biggest flow of BayLeaf’s spending was going: Stop using Claude Sonnet 5, UCSC folks, or just everyone. According to the DeepSWE benchmark, Sonnet 5 is everywhere dominated by other models, including open-weight models like yesterday’s (mid-sized and vision-capable) GLM-5.3-Flash. You could use this model to generate fewer output tokens over fewer total agent steps to get your thing done while, under reasonable assumptions, using less energy and spending less (of my) money to do it. I like giving people room to explore and learn for themselves, but I’d also like to build (or maybe even buy?) better tooling to gather evidence about irresponsible model choices. Right now I have personal vibes, stale reviews, and scattered benchmarks that aren’t aligned with campus interests. Deciding that a model choice is conclusively irresponsible requires political and institutional analysis. Does anyone want to try to sell us tools for that?

You know, when AI crashes, we’ll have GPUs at ten cents on the dollar. We’ll have an army of skilled workers who know how to make those GPUs do really interesting and cool things. We’ll have these open-source models that have barely been touched and optimized. [...] I think we’ll have open-source models. I think they’ll do really interesting things. I think we’ll have a lot of cheap hardware, and I think we’ll have a lot of people who know how to use it.

Doctorow calls this the productive residue of the bubble. I wanna know more about that productive residue. I want to jump ahead to the sustainable part. But, augh, I don’t want to erase the crash itself. The pensions, jobs, public services, and household savings exposed to the bubble belong mostly to people who did not create it. Doctorow suggests that if the crash is followed by austerity, as recent financial crises have been, its costs will be imposed downward while many of the people who inflated the bubble leave with their wealth intact.

So the question isn’t how to make the crash arrive faster. It is how to start building the post-bubble settlement before it arrives. It’s not X, it’s Y, of course.

One way to read the current stage of LLM development is as unnecessary production. A few-days-old preprint applies the European Union’s waste hierarchy to language models: prevention, reuse, recycling, recovery, and disposal. A model can become waste when enormous resources go into training it, it is used briefly, and then it is displaced by another model trained from scratch. Fine-tuning, model merging, and other adaptations can extend a model’s useful life. And sometimes the least wasteful choice is not to use an LLM at all.

After the speculative demand for constant replacement dries up, models don’t need to vanish. We can keep using, adapting, combining, and eventually retiring them. That cultural and technical capacity to maintain the parts of this system that we need can live in public, in individuals and organizations that don’t think of themselves as representing an AI industry.

Interactive planning has a technique called Idealized Design. You plan for what should come next by assuming that the current system failed last night. “Gentlemen, the telephone system of the United States was destroyed last night” is the opening premise in a fun story about Bell Labs redesigning the telephone system. Continuing the design exercise, two constraints:

First, technological feasibility. This means we cannot use any but currently available knowledge. No science fiction. We can’t replace the phone with mental telepathy. The second constraint, the system we design must be operationally viable. [...] The system must be able to function and survive in the current environment. For example, it will have to obey current laws and regulations.

Let’s run with it. Gentlepersons, the centralized generative AI service system was destroyed last night. ChatGPT, Claude, Gemini, and their commercial API endpoints are gone. Neoclouds, wrapper apps, and a good slice of the hardware design and manufacturing industry was destroyed too, why not? The rest of the environment remains: snapshots of open-weight models, already manufactured hardware, public research, open-source software, and people who know how to work with all of it. This is cleaner than Doctorow’s crash, but it’s a design exercise for asking what we could build from the residue without quietly assuming that the existing service providers will return.

If you want LLM services, you might run them for yourself or your organization on what is now cheap hardware. Small-ish models are already pretty good. We could get several years of useful work from models whose knowledge stops in 2026, especially when skill documents and retrieval tools can supply current context and people can continue adapting the models at the level of parameters or architectural adapters. Or we might pool hardware through universities, libraries, cooperatives, or an actually-open successor to today’s inference marketplaces. Access would probably be slower and less uniform. It would also be less dependent on a few firms subsidizing usage while trying to capture a market.

You would still need a harness: a place to keep conversations and files, connect models to other systems, and decide what context they can act on. We already have a healthy range of free, open-source harnesses. Even if the commercial support behind like tools like Open WebUI, OpenCode, and OpenChamber is lost in the crash, their snapshotted sources can be inspected, forked, and maintained or even cloned and rebooted without the permission their current industry backers.

Now onto human knowledge and cultural practice. How do millions of people (is it more than a billion now? whatever) move from services that hides nearly every technical decision to systems they can understand and control? We should probably start to build the offramps now, document them, and give people routine opportunities to practice using them.

I recently added a skill to the Basic agent on BayLeaf Chat, and I called it offramp. This one gives the agent a few key tips to help someone stop using BayLeaf Chat. It explicitly says that using BayLeaf Chat to get help leaving BayLeaf Chat is legitimate.

The skill is configured to trigger under two conditions. First, if the user is explicitly asking how to grow beyond BayLeaf Chat, sure, help them grow. Second, if the user is discussing a particularly sensitive subject for which they might benefit from enhanced privacy protections, they might need this advice too. Are you asking for financial advice, personal health advice, or career advice laced with talk of self-harm? I think LLMs can be useful in some of those scenarios, but not here. Not on the record, not in a database some professor can browse when they are curious. Let’s take this conversation onto your personal device so that you own the terms and scope of data retention.

The first level of offramp is to move the harness onto the user’s own device. The skill currently suggests OpenChamber, a desktop interface for the OpenCode agent harness, connected to the BayLeaf API for inference. Conversation history then stays on the device and is no longer available to the BayLeaf Chat administrator (that’s me). It’s a meaningful step, but it still leaves you partially dependent on BayLeaf’s inference service.

Offramps, the blog post title is plural for a reason. A user can take further offramps from BayLeaf inference to an individual zero-data-retention service, maybe ordinary pay-per-month subscription like OpenCode Go or high-security pay-per-token inference service like Tinfoil. Or maybe they could dabble in disconnecting from cloud services altogether by model running on hardware they control. It’ll cost them some in time and/or money, but the end of the ramp leads to a productive life even in a world where BayLeaf is destroyed alongside ChatGPT in our idealized design exercise.

The offramp skill is part of BayLeaf’s counterfoil research strategy. In Illich’s sense, counterfoil research is inquiry conducted against the grain of institutional self-interest: research that questions the necessity of an institution’s expansion rather than continually finding reasons for it to grow. For BayLeaf, that means operating a useful campus service that helps us develop to a level where we don’t need BayLeaf any more.

In the moment when someone turning to a robot buddy is stressed, exposed, or frightened, that is when I want them to see the offramp. Withholding it at precisely that moment is a kind of invitation to dependence. I don’t want people to practice comfortable dependence on something shaped almost like ChatGPT. I want them to become comfortable carrying their own practices, context, and judgment from one system to another, including into whatever we can build from the residue.

Some part of me needs to clarify up front that the tokens processed by LLMs aren’t literally words, just word-like. I think you should accept this, not just so we can get on with the blog post, but because even literal English words are only a bit like words in other languages anyway. A linguist might step in to distinguish lexemes from morphemes from graphemes, but let’s just say that sometimes the natural chunks of text don’t align with apparent word boundaries. You can decompose “precaution” into “pre-” and “caution” or acknowledge that “put up” (as in “to put up with”) isn’t just putting something in an upward direction.

For now, let’s assume we have enough of a shared sense of words to talk about word counts. I can talk about how I read a twelve-thousand-word research paper or how I need to write a new paper’s abstract in no more than 250 words. Already, counts of words read versus written are distinct. Further, reading twelve thousand words of unfamiliar text is quite a different task from reading twelve thousand words of familiar text, and maybe the text is familiar because I just wrote it myself or at recently reviewed it for another purpose. These intuitive distinctions between kinds of word counts will come in handy in just a moment.

To get an LLM hosted on a commercial inference service to generate text for you, you need to pay. So what does a token cost? Let’s look up the pricing for a currently leading open-weight LLM, Kimi K3, on the OpenRouter inference marketplace. As I write this, there’s a headline price of $13 per million tokens. So, that’s it, right? Wait, this page is covered with slightly different prices, all measured in the same dollars-per-million-token units, and the numbers are changing over time. Some of these prices seem to distinguish “input” tokens from “output” tokens and even from “cached read” input tokens. Let’s map it out using the idea of tokens as words.

$13/M is the current output-token price for the cheapest provider of K3. This is like saying there are many writers for hire, all writing in the same style, but maybe they write at different speeds and are more or less busy with work from other clients. The tables show that different K3 providers have different recent throughput rates (some have recently been generating about 10 tokens per second, whereas others have been running around 70 tokens per second). At the same time, the tables show different uptime percentages. This tells us how often the different providers are turning customers away because they are unavailable or overloaded. These writers-for-hire are moody! It’s not always the case that the more expensive providers are the fastest or most reliable ones, so it is justifiable that OpenRouter takes a cut to route requests among providers according to preferences for price, throughput, latency, and availability.

Okay, we’ve figured out how to map the idea of written words onto output tokens, and we’ve accepted that prices can vary because there are many different writing teams out there. But what about reading? We haven’t even explained half the prices on that page.

The next headline number on the OpenRouter K3 page is the “in” or “input token” cost, currently $2.60/M. This tells us something about the cost for models to process input text before deciding how they are going to generate the outputs. But what even are the inputs when you are using a chatbot or coding agent?

Imagine you are ten turns deep into a conversation with a chatbot. You’ve just sent your latest question, and you want the harnessed model (an agent) to reply. To generate the reply, your harness sends your ten previous user messages, the ten previous agent replies, and your most recent user message. This is the text the model needs to read before writing a reply. Even though reading is generally easier than writing (and this is reflected in the prices we saw, $2.60/M in versus $13/M out), keeping a conversation going requires a lot of reading. In particular, it requires a lot of rereading text that should be familiar, assuming our request is routed to the same provider endpoint that handled the earlier responses. Familiar reading has a different market price from unfamiliar reading, and for K3 that’s currently about $0.30/M (varying by provider, of course).

In transformer-architecture LLMs, this notion of familiarity with text has a technical interpretation. In most LLMs, the in-context meaning of each word (sorry, token) is influenced by all of the words that came before it in the text. Consider “the cat sat on the mat.” Specifically, consider being partway through reading it: “the cat sat on the _.” The second “the” isn’t just an isolated definite article; it’s the definite article of the as-yet-unspecified place where the cat is sitting. This contextualized, thickened representation is part of why the model is more likely to say “the cat sat on the mat” than “the cat sat on the cat,” even if “the cat” was more common than “the mat” in the model’s training data. It requires compute to build up these thickened representations, so we’d like to save them to avoid recomputing them for each new request that shares an identical input text prefix. At the same time, these thickened representations take extra storage space to keep around, so providers won’t keep them around forever. Providers give you a discount on reading text that is familiar in the technical sense that its previously computed representation remains available through prompt caching.

At this point, we have three different dollars-per-million-token prices for a specific model from a specific provider at a specific point in time in a competitive marketplace. We can trace these to the distinct efforts needed to write versus read, or to read familiar versus unfamiliar text, all measured in word counts. But we’re not done interpreting that OpenRouter page! If you click the “Pricing” tab, you get yet another price: “Weighted Avg Input Price,” currently $0.79/M (somewhere between the cached and uncached input prices). What’s this?

People use different models for different purposes, so the typical ratio of cached to uncached inputs (of requests containing familiar text to those containing unfamiliar text) can vary. Further, providers with different amounts of cache storage or different cache-eviction (forgetting? defamiliarizing?) strategies can influence this balance. “Weighted Avg Input Price” is trying to abstract over a bunch of details to give you one simple input price. The “Weighted Avg Output Price” (currently $14.84/M for K3) is trying to do the same on the output side. Even though a lot of interesting technical and economic detail is getting erased here, these prices are starting to become useful.

Suppose I want to ask my assistant to generate a 250-word summary of a 12,000-word research paper. I’ve got input and output word counts. We can apply the rough rule of thumb that an English word maps to about 1.3 LLM tokens. From there, we can plug in these weighted average input and output prices to find that our summarization request might cost, uh, about 1.7 cents. (The true cost is likely to be higher because this paper is likely unfamiliar and uncached, and newer models will likely write additional internal reasoning text to themselves before writing the final summary. Nevertheless, it’ll be cheap.)

When a new model comes out or an existing provider adds support for an old model, we sometimes want to summarize the usage cost of the model with a single number, without reference to a specific task. Artificial Analysis calculates its blended prices using a “7:2:1” ratio: assume that requests involve 70% cached inputs, 20% uncached inputs, and 10% outputs, by token. If we’re considering blended cost ratios, the units of text cancel out: it doesn’t matter that tokens aren’t exactly words so long as they are typically proportional. Because the blended cost of inputs is cheaper than outputs by about the same ratio that inputs are more common than outputs, we sometimes just read the raw dollars-per-million output token market price as the single summary figure for the model as a whole.

Even though tokens aren’t money, there is a justifiable chain of abstractions and approximations that lets us say “GLM-5.2 costs $3.60/M” (this is the model powering the Basic agent on BayLeaf Chat) and “Opus-5 costs $25/M” (this is the model I often use for maintenance of BayLeaf’s infrastructure). When a new model comes out that scores perhaps “82% as smart as Opus-5” by the ratio of their Artificial Analysis Intelligence Index scores, at a blended cost of $0.18/M, hmm, maybe we should switch to that one. (DeepSeek V4 Flash 0731 is the one I’m thinking about today, and it is already marked as the recommended model on the BayLeaf API.)

Even if there’s an analytically useful way to treat LLM tokens as if they were money (or simply cost money), this immediately erases the rich distinction between reading and writing and the importance of familiarity when reading. If you want to control the US-dollar cost of your LLM usage, simply asking your assistant to be concise in replies isn’t going to matter much. Getting your assistant to read less, or to read in ways that put much more of the text into the familiar category, matters much more.

Perhaps I should write another post specifically about these responsible cost-control techniques. The key concept you’d need to learn is how to identify the longest identical prefix shared from one request to the next. But before getting into techniques, we should ask why users need to recover these distinctions from provider accounting categories in the first place.

Most people using LLMs access them via pay-per-month subscription services that offer them a quantitatively unspecified amount of service. Some are a bit more transparent, promising that your subscription covers a certain number of requests within rate-limiting windows of a certain duration. If you are only thinking in terms of atomic requests (each of which involves some specific amount of familiar and unfamiliar reading before some hidden reasoning and visible writing), nothing is nudging you to learn even about token counts, let alone the distinctions between cached and uncached inputs.

Until the last month, I had gone about two years without monthly subscriptions to LLM inference services. I probably paid a lot more as a result. Flat-rate individual subscriptions can provide far more inference than the same payment would buy at API rates, while enterprise services increasingly expose metered consumption. Whatever the providers’ motives for that subsidy, I think paying per token taught me things I couldn’t have learned as easily as a subscription customer.

On BayLeaf Chat, under each agent reply, there’s a little info button you can click to get some usage and cost data about that reply. The display is a little too technical, but what’s important is that it has these unpacked token counts. The cached_tokens figure is less than the prompt_tokens figure, but only by a bit (because most of the text in the conversation is recycled from one reply to the next), and the total_tokens figure is only a little higher than that (because the replies are typically small compared with the inputs that prompted them).

One of the things you can learn from examining these usage cards is that the actual size (token or word count) of a conversation is often not easy to predict from the words you see on the screen. When you ask the agent something that causes it to look things up on the web or interact with other external systems, the agent needs to write some additional words to do those searches and read a whole lot more additional words to make sense of the search results before it writes back to you. This isn’t an argument for us to turn off web search to optimize costs. This is an argument for people to demand visibility into token counts so they can understand what’s actually driving the system’s individual and socialized costs.

The everyday interfaces of ChatGPT, Claude, and Gemini generally hide token counts from users as if they were a distracting technical detail mostly only relevant to system integrators. But if you can allow the slight fudge that tokens are words, every user already has some intuition for interpreting word counts and making responsible choices on that basis. The “token” terminology makes you think about accounting, whereas “word” sets you up to think about reading versus writing, how meaning thickens in context, how familiarity degrades when things fall out of working memory, etc. It’s easy to look at a chatbot’s apparently lengthy reply and wonder if the output was worth the cost. I want you to think about all of the words that got read before writing that reply, think about which human wrote them, and how we can plug in more relevant context sources while encoding them compactly.

Tokens are words, not money. Hopefully I’ve thickened your representation for “tokens” in a generative AI context so that you’re ready to make choices differently in the future.

Chat is the only part of BayLeaf that records the content of your messages, and even those are deleted automatically after a while. Of your conversations, my privacy statement says “They are accessible only to you and the system administrator.” That’s me, the system administrator. If I’m going to keep operating BayLeaf, I want some assurance that my system isn’t responding in ways that are too cringe. I want to know what my system is talking about, at the level of broad categories, and, for sensitive categories, I want to use my privileged access to spot check a few items. Eventually, I’ll make this last level of access impossible, even for myself.

A few years ago, the folks over at Anthropic shared the design of Clio: “a system for privacy-preserving insights into real-world AI use.” The core idea is to do the first few levels of analysis of raw customer data automatically, only letting human analysts get involved after a few layers of summarization and aggregation. In literary studies, we might call this a distant reading technique (in contrast to close readings where the multiple meanings of specific words and phrases might be scrutinized). Clio works through four phases: extracting facets (extracting rough summaries and metadata), semantic clustering (grouping related conversation by theme or general topic), cluster description (giving clusters a descriptive title and summary without bubbling up sensitive data), and building hierarchies (assembling clusters of clusters until a digestible bigger picture emerges). I ran something similar.

The first phase of my pipeline was conversation data extraction. This was a little tricky because conversations on BayLeaf Chat are structured as trees rather than linear lists of messages and that the storage format has changed over the last few months. The only interesting move I made in the extraction phase was to mask out my own usage data. I already know what I do with BayLeaf, so my analysis was focused on others.

Next came the compaction phase. Even though users don’t spend a lot of time typing messages into BayLeaf Chat, conversations can get pretty large when they upload or paste large documents, elicit long replies from the agent, or when the agent does several rounds of interaction with third party services (e.g. web search) before formulating the user-facing response. My strategy was to mid-truncate user messages and agent replies (keeping the first and last chunks of a message and throwing away anything left in the middle) and to discard any reasoning blocks or tool call results. This discarded about 89% of the total text volume keeping what I assumed to be the interesting parts.

For facet extraction, I ran each of the compacted conversation streams through a mid-sized language model (DeepSeek V4 Flash 0731, the currently recommended model on the BayLeaf API). I asked the model to produce a topic (2–6 word phrase), user message language (e.g. “en” or “zh”), and summary (1–2 sentence de-identified gist of the user’s intent). This analysis was pretty cheap. I spent about $0.08 in total for facet extraction (compared to a total of about $3.15 on a coding agent based on the same model to execute the rest of this project and help edit the blog post).

Once I had the facets extracted for each conversation, I needed a way to group related conversations together. It’s not completely unreasonable to simply ask another LLM to name the common patterns, but I opted for something more mechanical and scalable: classical unsupervised machine learning. I used OpenAI’s text-embedding-3-small model to get a vector for each conversation’s extracted topic, then I threw those through HDBSCAN. HDBSCAN works by estimating how densely clustered each conversation is in the vector space, growing clusters out of regions of stable density and keeping the ones that persist across density thresholds. Conversations too spread out to belong to any stable region are left over, and I treated that leftover as its own long-tail bucket rather than forcing it into a cluster. I think that, in Anthropic’s approach, human analysts got involved in some of these groupings, but it was fully automatic in my design. I don’t think my design is better, just a faster path to a first look at the big picture.

Once I had the conversations clustered, I wanted to be able to describe the result. Taking the top twelve or so clusters by size, I again asked a mid-sized language model to come up with a de-personalized summary. This time, the describer model only got access to the per-conversation summaries (not the super-short topic or the longer, compacted message history). For each cluster, the describer gave me a 2–5 word title naming the theme of the cluster and 1–2 sentence summary of what the items in the cluster had in common.

So, what did I find? Here are the model-generated titles and summaries, in order of decreasing cluster size. (Does Substack not support tables?? Okay, the format here is going to be “Title: Summary”.)

• Rust GUI development & tutorial building: One sustained project: egui/eframe/GPUI node editors, mdBook tutorial sites, prompt-crafting for AI coding

• Model identity & privacy inquiries: ”What model are you?”, “who’s the provider?”, “what’s your data retention?”

• Football simulation development: Python NFL-sim: scheduling, drafts, GM logic, play-by-play

• Academic recommendation letters: Rec letters, grade appeals, GSI support letters, program-status decisions

• AI in learning & game-assistance research: Literature searches on AI tutoring, self-directed learning, player-assistance

• Reflective academic writing: Essays, self-evaluations, fellowship statements (ethnic studies, education)

• Hilly-commute vehicles: E-bikes/scooters for steep campus terrain

• Marine biology coursework: Right whales, energetics, lecture review

• Linux setup & troubleshooting: Drivers, displays, VPN

• Campus mascot humor: UCSC jokes (banana slugs, hills)

• Agentic coding model selection: OpenChamber vs Cline, BayLeaf/OpenRouter config

• Faculty burnout & boundaries: Educators in distress seeking crisis support, boundaries, exit strategies

• Physics homework: E&M, circuits

• …plus ~12 smaller clusters: Phaser, LaTeX, Canvas LMS, salamanders, SSH, AI-detection, etc.

• Long tail: One-off personal queries: recipes, dating, health, flights, taxes, jokes

As someone with a very personal interest in the topic of “Faculty burnout & boundaries”, I did drill down to the raw data for a few of these. I can report that the agent is responding well enough, the facet extraction and cluster summaries are accurate enough, and, ugh, it’s a good thing that I’ve found a way to maintain a lot of distance from the raw data over the recent two years. I’m glad my system could provide someone support, but I’m also glad I didn’t get sucked into trying to provide that support directly myself.

Even though this analysis is basically fully automated, I’m not sure that I want to rig it up to run regularly. I’m writing this blog post as a starting point for in-person conversations about how much a community should be allowed to know about what others in that community are doing in chats that feel private. It is one thing for Anthropic to use Clio on a tiny fraction of data from their international user base. It’s another to run it for just my single-campus community.

I haven’t received any NSLs (yet), so please don’t interpret this post as a telegraphed announcement of disclosure. I’m just thinking things through.

For narrative purposes, let’s say that you, dear reader, are one of my faculty colleagues. Even more, you are actively using BayLeaf Chat to plan and execute terrorist operations. You are the bad guy, and you need to be stopped. Or maybe you are a just doing Middle East politics research, an international student, or someone organizing a union drive, etc. And I, Adam (the BayLeaf operator), get this letter, and the letter is about you. The FBI or whoever needs me to share what I know about you, and they don’t let me tell the public or even you about how I’m doing it. I can talk to my own layer about it, but that’s about it.

The letter can’t ask for everything about you. Per usual interpretations of 18 U.S. Code § 2709 - Counterintelligence access to telephone toll and transactional records, the letter can only ask me for transactional metadata, not the content of your messages. So, I might be compelled to reveal that someone with your campus email address sent a message or got an agent’s response at a specific date and time, and maybe I’d also reveal the conversation identifier that links several messages into a specific flow. An NSL isn’t going to require me to share the payload of these messages. That’d take a search warrant, but we’re talking about NSLs today. You can learn a bit more about “electronic communication transactional records” (ECTRs) in this DOJ FAQ (hosted by CDT).

I think ECTRs are about where, when, and with whom an individual is communicating. Is a chat with an agent even two-party communication? Is the name of the model-generated title of your conversation data or metadata? Are tool calls and responses protected? How about the length of the messages (token counts)? I don’t know. My understanding is that the boundaries between data and metadata are murky even for traditional communication between two human parties. In this world where you are plotting your attack in a computer-mediated conversation, apparently, with yourself, there’s even less precedent I can find.

Let’s assume the letter is just asking for message timestamps (analogous to phone call initiation records). How much of those could BayLeaf Chat leak? Per our current data retention policy (and automated cleanup machinery), data and metadata for your old chats are faithfully wiped 90 days after you last interact with them (on a per-conversation basis). [✨ Editing agent’s note: Two honest caveats I found while auditing this: attachments are not deleted along with their conversation, they become orphans and get swept on a later run, so a file outlives its chat by up to about 48 hours; and deleted rows persist in the managed database’s backups until those age out on the hosting provider’s schedule.] Well, if your account is in a special records hold group, your records are not cleaned up automatically. However, the Help agent on BayLeaf Chat can transparently tell you if you are in one of those groups. Currently, I’m the only user in such a group because that’s part of how I test the retention exception mechanism.

In my own audit of the data retention machinery for this post, I realized that a user’s custom instructions (system prompts) and explicitly written memories are currently retained forever. Given that these are where you might have the system remember a clue like “Attack in September”, I need to think about how these are retained. Previously, I had been considering them like innocuous profile data (like your display name), but they plausibly contain the precursors to sensitive messages that you’d like to not have haunting you years after writing them. Ah, and if I’m considering them to be subscriber metadata (akin to a billing address in a traditional phone system) rather than message content, are they available to snarf via NSLs? Dunno.

I was relieved to learn that “§ 2709 does not require communication service providers to create records which they do not maintain in the ordinary course of business” (OLC, 32 Op. O.L.C. 145 (2008)). This means I can’t, at least via NSLs, be compelled to start recording additional information. The same opinion is blunt about the other side of the deal: if you do keep a record, even scattered across aggregate data, you have to go extract it and hand it over. If BayLeaf were operated by a big company with tons of internal logs and metrics already being recorded, there might be a rich picture of you being painted by previously-recorded metadata. Here at bus-factor-one BayLeaf, however, there is much less metadata about you than a commercial provider would hold, and I don’t plan on logging more any time soon. That said, I shouldn’t pretend the pile is empty: your account record and group memberships (which can indirectly encode course enrollment) are currently kept indefinitely, the API database keeps a per-backend record of the last date you used each one, revoked API key rows are never purged so an email address is permanent once I’ve seen it, and a handful of log lines in the API carry your email address for a few days.

Speaking of logging, it is important that BayLeaf Chat isn’t the only service available from BayLeaf. Using the BayLeaf API, you can setup chat and coding agent apps on your own personal devices. When you use those, your personal device has the only copy of the message content. So, even if I were served with a search warrant (secret or otherwise), I would have nothing stored to hand over. If you are concerned about privacy of your chats, please stop using hosted chat services altogether, even BayLeaf’s. This is the same move of giving up using Google Docs for sensitive document editing. Keep those documents on your personal devices, please, for your own sake.

Back on NSLs, it doesn’t seem that realistic that I, the BayLeaf operator, would actually be the recipient of the NSL. To the degree that BayLeaf is associated with UC Santa Cruz (which is, to say, partial), NSLs are likely going to be delivered directly to UC administration or to one of my subprocessors (like DigitalOcean or Cloudflare) better equipped to respond in a timely manner. [✨ Editing agent’s note: In fact I am not authorized to accept service on behalf of the Regents at all: UC’s own guidance tells employees to redirect anyone attempting service to the Office of the General Counsel in Oakland.] When your messages come through BayLeaf services, your email address generally travels along with them. So, if someone wants your metadata and has the legal authorization to get it, they can get it without going through me. (Even more reason to practice ways of using Generative AI without a centralized service in the first place!)

If BayLeaf’s records about you were cleanly interpretable as my own records in my role as faculty, they might be additionally protected under the UC Electronic Communications Policy. However, to the degree that I’m operating BayLeaf as a shadow IT system, I’m unclear on how others in power might interpret the policy. It sure would be simpler if the university actually owned this whole thing.

So, if you really are doing something nefarious (or something benign that someone else decides is nefarious), how are you supposed to keep it under wraps? Using your own chat or coding agent harness on your own device will keep your data out of BayLeaf’s databases, but a compromised back-end service provider could leak your data. In the last few weeks, I’ve added a Sealed Inference service facet of the BayLeaf API based on Tinfoil. If you get your local agent to use the Sealed service, the content of your requests is protected from both myself and the back-end providers on a level that you simply can’t get from OpenAI/Anthropic/Google, at least in their Education packages. What Sealed does not hide is the metadata this whole post is about: the fact that you made a request, when you made it, how big it was, roughly how many tokens it used, and even which model you picked. Should this inference mode be the default on BayLeaf? Maybe. The threat model behind that choice goes beyond NSLs, but the associated paranoia isn’t entirely unjustified.

When I saw this slide over a decade ago, I thought to myself “If I’m ever operating a service like this, I’m not going to design it to have that one convenient point for the full take.” But here I am, running BayLeaf Chat in a shape where there is such a bottleneck. At least I’m running an open-source chat harness with transparent operations. I tell myself that the Chat service’s design is justified because it offers people an incremental offramp from Google-shaped services. Time will tell how many decide to follow that offramp.

I’ve a few sources of historical data to work with. Each gives a partial view, but there’s pretty strong consensus in what they say.

First, I can look at the last 90 days of message traffic metadata on BayLeaf Chat. I can’t look further back than 90 days because, per our data retention policy, that’s literally all we store. So, this’ll tell us about the tail end of spring quarter and some quieter and less structured usage in the summer quarter.

Next, I have six months of logs from the OpenCode coding agent harness on my laptop and more than two years of activity on a friends-and-family instance of Open WebUI. Together, these represent something close to what we might expect for the BayLeaf API and BayLeaf Chat services over a longer period of time, but biased toward my personal usage patterns.

Before I launch into what I found, think about what you might expect the data to show. If UC Santa Cruz were a stereotypical business organization, you might expect steady weekday usage during business hours, and maybe even a clear lunchtime notch. Maybe Monday mornings would show users accessing the system to get oriented for the week, and Friday afternoons would show a gentle ramp down as people left early. Now, consider that we’re at a university, with folks staying up late to prepare for exams, balancing social lives, perhaps specifically *not* accessing the services while they are in scheduled lectures. What’s that usage going to look like?

According to the data, weekends aren’t a big thing. Whatever effects you might think would push weekday and weekend patterns apart seem to mostly cancel out. Usage on Saturdays and Sundays falls between the highest and lowest weekday usage in all three datasets. The location and ratio of peaks and troughs vary a bit by dataset and by whether you aggregate by session count, message count within a session, token count within a message, and so on. My personal data shows I’m almost twice as active on Thursdays as I am on Fridays, but the campus data suggests this is just a personal quirk. Someone else with a different schedule might have a consistently different profile.

Time of day is where things get a bit more interesting. Across all three datasets, there’s a clear on-off pattern, with activity starting around 8am and turning off around midnight. In the off period, we’re not fully off. Activity is minimal but nonzero (even in my personal datasets) at 4am. Meanwhile, 4pm is a daytime peak. My personal data shows distinct notches at 5pm, for commuting home, and 7pm, for cooking and eating dinner. However, these notches only appear when you look at user message counts. For requests at the API level (which include agents following up on tool call results, etc.), they go away. I often leave my agents cooking up some code while I’m cooking my family dinner.

Overall, there’s a wake-sleep pattern that comes from most of the UC Santa Cruz campus community being in the same place at the same time. We’re maybe 20 times as active in wake mode as in sleep mode. But is this pattern exploitable?

By analogy to electrical power generation, it can be cheaper and more efficient to keep some kinds of generators running continuously around the clock. To handle demand spikes, you might then bring peaker plants online at critical moments. This isn’t even generally true across the full range of generation technologies, but let’s see how far the analogy gets us.

If we ran BayLeaf using on-campus GPUs (we don’t), maybe we could turn off those servers at midnight and wake them up at 8am. You’d hope that, if we had 20 GPUs, we’d simply turn off 19 for sleep mode so that the service remained available to night owls. However, the situation isn’t so simple. For a given size of model, there’s a minimum hardware configuration needed to generate tokens at a reasonable rate. We outsource inference to third-party services because, even at our 4pm peak usage, we can’t fully utilize a single dedicated inference server.

If our wake-sleep patterns were 12 hours on, 12 hours off, perhaps we could coordinate with a sister campus in (*checks globe*) Iran, Kazakhstan, or western Russia. They’d be active when we weren’t. With a 16-hour active period, however, we’d have overlap. Maybe we could band together with other West Coast universities (perhaps in some sort of system of universities in California) and reach enough aggregate demand that diurnal resource scaling starts to make sense. Until then, it seems prudent to pool our usage with the much larger national market and let providers with much more comprehensive data decide how many servers to keep active at various times of day.

I realized I’ve gotten to the end of this data-oriented post without sharing any concrete data. So, uh, here’s a snapshot of my personal coding agent data, which turns out to be fairly representative of campus-wide usage as well:

Lest we settle for AI pig iron, I don’t think we should run inference for BayLeaf services on campus. However, there’s a lot to be learned from an earnest attempt.

A Laptop in my Office

It’s commonly assumed that you need (multiple) GPUs to run LLMs. This is untrue, but it gives us somewhere to start. Let’s pick a location on campus, stick a machine with a GPU there, load an LLM on it, and measure how much power it takes to generate tokens of text from some model at some speed.

My faculty office in the Engineering 2 building has university-provided electrical power, cooling, and network access. The computing hardware for my personal data center will be my personal laptop, an M3 Max MacBook Pro. Because this computer has one large block of unified memory (64GB), it will let us try running some models that would otherwise require multiple desktop-class GPU devices (e.g. NVIDIA RTX 4090s) or a server-class GPU (e.g. an NVIDIA A100).

I’ll start with two of the strongest open-weight LLMs for their total size at this point in the summer of 2026: Qwen3.6-27B, the “dense” one, and Qwen3.6-35B-A3B, the “sparse” one. I’ll use the 8-bit quantization of each model, so you can easily estimate the on-disk storage size and in-memory size by the parameter count: a 35 billion parameter model is going to take up about 35 gigabytes. With 64 gigabytes of memory in this machine, I’ll have lots of room left over to store KV-cache for whichever model I’m using. (The cache is an important data structure for making LLM inference efficient, but it requires almost as much memory as the model itself for storage in typical agentic usage scenarios.)

You don’t just double-click the models and run them like apps; you need to run an inference server that will load and interpret them in response to user queries. I’ll use the oMLX inference server. While you might use vLLM for production-grade multi-user inference services, oMLX is a nice match for my laptop.

This setup with those two Qwen3.6 models in oMLX isn’t something new for me. I’ve been running this setup for a few months, and I’m a monetary and technical contributor to the oMLX open-source project.

When I fire off some benchmarking runs, I find the dense model generates about 12 tokens per second (tps) and the sparse model runs at 78 tps. The sparse model is faster because it only activates a fraction of the model for each output token (35B-A3B means only 3 billion out of 35 billion parameters are active at each token, a density factor of about 8.5%; 27B means all 27 billion parameters of the 27 billion parameter model are active each time, 100% density). According to published quality benchmarks (and my vague, subjective sense), the dense model is smarter than the sparse one, but the quality difference isn’t big enough to justify the throughput sacrifice. So, the sparse model is what I’d realistically use in production (in some bizarre world where my laptop were the inference substrate).

During inference tests, I can feel my laptop physically heating up. I measured it drawing 120 Watts of power. As I write this blog post (on the waterfront steps of Copenhagen Opera House), my laptop is idling at 16W. But most of that is from having the screen brightness cranked up so I can read text outside. If I were writing this post in my hotel room, I’d probably be using more like 8W, or even 4W if I kept the screen extra dim. (This trip to Copenhagen for an academic conference is going to cost me more than a year’s operating costs of BayLeaf, by the way.)

The numbers I reported above were for processing a single request at a time. This makes sense for personal use, but for a service, we might reasonably be asked to generate responses for several requests concurrently. When I re-run my tests with multiple concurrent requests, the per-request token generation rate goes down, but the total token generation rate goes up. For a server-class GPU, you might configure the server to work on 32 distinct requests at a time. For my high-memory, weak-compute laptop, handling even a second request concurrently makes the first request’s response trickle back uncomfortably slowly.

Okay, I ran an on-campus inference service, and it totally worked. I’m done, right? No: it is all of this beyond-the-GPU stuff that matters a lot.

Suppose I wanted to wire my laptop’s oMLX instance up as one of the back-ends for BayLeaf. My laptop isn’t always in my office (it’s in Copenhagen right now!), so the BayLeaf user-facing services need to find the inference hardware as it comes around the world. This isn’t a major problem, as I can logically wire things up with my Tailscale VPN. But what if I need to close the lid of my laptop? What if the times of day when BayLeaf users want inference services collide with when I want to use my computer’s GPU and RAM for something else?

Even if it can technically run LLM inference services, a laptop in my office is clearly not a satisfying replacement for your typical data center.

(Rabbit hole: Qwen3.6-35B-A3B is an excellent model for its size, but it’s also not smart enough to handle many of the things I’d like to do with BayLeaf. It’s often the case that bigger models are better. It’s also often the case that newer models, of a given size, are getting better every few months. It seems likely that “35B-A3B” is a good shape for MoE models, even if the Qwen3.6 instantiation of that shape isn’t fully satisfying.)

A Campus Data Center

Just a one-minute walk from my office in the Engineering 2 building takes you to the Communications building on campus. This building serves multiple kinds of communications: the Film and Digital Media department has offices, studios, and editing bays in there, it’s got a satellite dish on the roof that might have something to do with radio communications, and, most important for us, it houses a machine room with racks of servers and data networking.

I went down to the basement of the Communications building, following the high-pitched whine of server fans, and found a locked door. Through the window, I could see the machines with their glowing lights. It’s a data center with industrial grade power and thermal utilities. If I somehow got my hands on a server-class GPU, I could run it from there, right? That’d clearly be the appropriate location to run on-campus compute infrastructure. But, I don’t have a key to the door.

Or maybe I do have a way in. It turns out that several of the machines in the Communications Building Data Center are already rigged up to participate in the Nautilus Hyperconverged GPU Cluster of the National Research Platform (NRP). For unrelated reasons, I already have admin privileges for a resource namespace on this cluster. So, a few Kubernetes commands later, I had access to a machine that identified itself as k8s-gpu-6.ucsc.edu with an NVIDIA RTX 5000 Ada Generation GPU with 32 GB VRAM.

I got the two Qwen3.6 models running on that GPU server, this time using vLLM rather than oMLX, and I collected some unremarkable throughput measurements. I say unremarkable because many others have gathered the same data. For a single request, the server-class GPU runs somewhat faster than the laptop-class GPU. For multiple requests, the server-class GPU maintains per-request performance much better. But this is common knowledge that doesn’t require an experiment. What mattered more is that, as an individual faculty member, I was able to figure out how to put this all together from already-purchased, already-running resources.

The problem is that these resources aren’t arranged to be able to support a service like BayLeaf. There’s no problem with running LLMs on NRP: it’s an officially supported service. The bigger issue is that this research platform is allocated for research use, and the LLM service in particular is not operated with any of the privacy and security guarantees we need. Even if I wanted to use UCSC’s existing NRP nodes to run my own LLM service (under my own privacy terms), I shouldn’t. GPU nodes on NRP are supposed to be allocated only temporarily, for jobs that can keep the GPU at 40% utilization or above at all times. Even with whole-campus adoption, I don’t think BayLeaf has enough around-the-clock demand to justify keeping a single dedicated GPU reserved for BayLeaf inference services.

I ran another experiment, this time using CPU-only inference on a machine that identified itself as fiona8.ucsc.edu. The same two models are, of course, capable of running inside vLLM. GPUs were never technically required. The dense model generates at 4 tps and the sparse model 7 tps. Natural human speech rate is somewhere between these figures, so we shouldn’t rule out CPU-only inference on generation speed alone.

The problem is reading, not writing, in two senses. First, we humans read the output of the model, we don’t speak it. Text that trickles out at speech rate doesn’t afford skimming or even reading particularly comfortably. Second, models don’t just generate output, they process inputs that contextualize the future output. If you ask your friend to summarize a 10-page research paper, you don’t mind it when they take an hour to read the paper first. But if your LLM-based assistant takes an hour to read the paper first before typing out the summary, you’ll be frustrated. (There’s an emerging technical literature on “disaggregated inference” where input processing and output generation are separated, run by different software systems on different hardware. But let’s not get lost in that rabbit hole.) If we’re too focused on generating tokens, we’ll miss the constraints that make one system design or another feasible for use in BayLeaf.

Think of it like power generation. Just because I’m able to rent a diesel generator, haul it up to campus, and generate electricity, doesn’t mean it is a good idea. For all sorts of reasons (efficiency, environmental, and otherwise), we centralize power generation at scales larger than a single university campus.

Say, where does the power for the campus data center, or my office for that matter, come from anyway?

A Campus Power Plant

Just another one-minute walk from the Communications building on campus takes you to the campus cogeneration plant. Cogeneration, here, refers to simultaneous generation of electrical power and useful heat. Useful heat, in the form of near-boiling water, can be piped between buildings to provide much more efficient heating (or even cooling, through the use of absorption chillers) than electricity provides. If the Communications building houses our data center, then the Cogen facility houses our power plant. Or at least it did, sort of, partially.

According to campus sources, about 64% of campus electricity comes from our cogen plant. About 6% comes from our various photovoltaic arrays (mostly the canopy in the East Remote Parking Lot). The remaining 30% is made up by on-demand imports from PG&E. PG&E is also the supplier for the natural gas (methane) burned in the “Solar Turbines”-branded Mercury 50 PG Generator Set in the cogen plant.

Numbers aside, it is fair to say most of our electrical power is generated locally. However, even that local generation capacity is not entirely situated here. We didn’t frack the methane we burn or smelt the iron used to forge the turbine parts here. There are important reasons, some strategic and some historical, for us to continue to burn the methane here rather than having PG&E burn it for us off-campus.

It might be fun to stash a computer in the control room of the cogen plant, to have it generate tokens using hyper-locally produced energy and (at least by proximity) be part of heat recovery and reuse systems. But it would just be a stunt. The amount of energy used by AI services at a campus scale is just so tiny compared to other kinds of campus energy usage. On the individual scale, dimming your laptop screen, or, even better, remembering to close the exterior door to the lecture hall when the heating or cooling systems are working, is going to matter a bit more than where your AI tokens are generated.

Decommissioning and Decarbonization

In most of my years as a student and later faculty on the UC Santa Cruz campus, I saw the metallic lettering of the “F. LOUIS FACKLER COGENERATION PLANT” sign attached to a large beige building. When I went to go capture a photograph of the building for this blog post, I noticed the sign has recently moved to a smaller, brown building.

There’s a whole story that could be told about Fackler, the man, and the plant(s) named after him. However, I’ll cut it short to say that even the new building, which houses the Mercury 50, is planned to be decommissioned and not replaced. The future of energy at UC Santa Cruz involves ramping down, rather than up, carbon-emitting energy generation.

On a much shorter time scale, the Communications Building Data Center is also being decommissioned. NRP servers currently operating there will need to move to off-campus data centers. For reasons mostly to do with power and water access, this means using a slice of space in an existing data center in Quincy, Washington, not even in California.

As I imagine an on-campus inference service for BayLeaf, it involves commissioning new infrastructure, re-carbonizing the campus in ways that directly contradict other well-reasoned plans.

If there is a role for university-only inference services, I think the appropriate scale of it is at something larger than a single campus (as it is for power generation). It seems appropriate that UC San Diego’s San Diego Supercomputer Center (SDSC) is being used to experimentally run LLMs for multiple UC campuses. It offers inference for UCSD’s TritonGPT and UCB’s BearGPT. Should it run a hypothetical SlugGPT for UCSC? I’m not sure. All of California’s higher education institutions are located in one timezone and have similar academic calendars, so our load is peaky, and we haven’t yet invented some kind of “AI battery” that would allow us to store excess inference capability in off-hours to spend down during peak hours.

We want our LLM-based agents to respond quickly, but that might mean responding within half a second, not within a few milliseconds. With that kind of latency tolerance, the inference service could be almost anywhere else on the Internet-connected Earth without us noticing. We should probably have base-load inference services follow the sun as it shines on different parts of the world rather than push too hard for regional (e.g. campus-level or even state-level) self-sufficiency.

Metallurgical Literacy

I opened this post with an oblique reference to backyard furnaces (土法炼钢). During China’s Great Leap Forward (1958–1962), the term referred to efforts to radically decentralize steel production. To cut a long story short and avoid torturing a weak analogy, what came out of those furnaces was mostly brittle pig iron rather than usable steel, and production was eventually re-centralized.

I hope you can see the analogy between backyard furnaces and me generating tokens with my laptop in my faculty office. It seems like it sort of works for me, if you excuse some response quality issues, but there’s no way I can usefully compete with cloud inference services when you look at the bigger picture of system reliability, energy efficiency, or alignment with campus decarbonization goals.

So why did Mao Zedong and I even go down this path in the first place? To speculate (as I’m an engineer rather than a historian or political scientist), I think it’s about developing people, perspectives, and credibility of exit from other plans. I want people to understand that the path from methane to chatbots (or from photovoltaics to research automation assistance) is traceable at the human scale, that many of the important processes are close analogs to things that are already happening on campus within a few minutes’ walking distance from one another.

When there’s something we don’t like about how Generative AI currently works in society, we need to be able to build and operate alternatives so we can be precise about the definition of the problem and the feasibility of the solutions. If we’re not able to make alternatives tangible, we’ll be too cautious to try new things and cling too tightly to the first things that seem to even partially work.

Should a university procure every student a Netflix account and every employee a LinkedIn Career plan? Probably not, for so, so many reasons. It makes a lot more sense to offer everyone basic Internet access when they are on campus via convenient Wi-Fi. Individual users or smaller-than-a-whole-campus groups can opt into additional services on their own. Oh, and it would be nice for the university to provide some kind of hosting services so that specific members of the campus community have some capability of offering their services to others in the community.

There’s a similar picture we need to draw for Generative AI. This post is going to try to leverage your well-established familiarity with the Internet and the Web (even if you are mostly a casual user of it) to help you pull apart some ideas that are too easily collapsed in the current public discourse around Generative AI.

Let’s focus on retail Generative AI subscription services associated with brands like ChatGPT, Claude, and Gemini. Under problematic privacy terms and tight usage limits, you can usually access some version of these services for free. Or, if you decide to pay, you can get some kind of Plus or Pro subscription, perhaps $5 or $20 per month, offering more. For some folk, even that’s not enough, and there are Max or Ultra subscriptions, maybe $100 or $200 per month, that give you even more smarts or more messages per day. The privacy terms, usage limits, and per-user prices of the subscriptions available for universities and other large organizations are a little different than those for individual users, but we’ll ignore those distinctions for today.

Should the campus procure everyone a ChatGPT Plus account (or some close, institutionally-negotiated equivalent)? You can’t say it would be unrealistic: we’ve seen this done for individual UC campuses like UCLA and for the entire California State University system. But, augh, I can’t help but think these purchases were made without a clear understanding of alternatives. Was a campus-wifi-like design even considered?

What does a raw Generative AI service look like? It would be something that gives you access to the smarts (the text extruder, the spicy autocomplete engine, the stochastic parrot, etc.) without a brand-name chat interface. It might be a pay-as-you-go service like OpenRouter or a flat-fee subscription service like OpenCode Go. Typically, these are available under excellent privacy terms (usually zero data retention, ZDR) and with generous usage limits. Unless you are a software developer directly integrating your custom software with these services, you can’t make much use of these services by yourself. It’d be like trying to use the Web without a browser! (Your programmer friend, a fan of curl, asserts that this is a possibility, but they are suspiciously reluctant to show you how to do it yourself.) We often call this level of Generative AI service the API: the application programming interface. Specifically, we might call it an inference API providing access to pre-trained models (to distinguish it from APIs capable of training or fine-tuning new models).

So, what’s like the browser for Generative AI? The two places you’ll most often see Generative AI come to life (in the sense of making spicy autocomplete into something useful) are chat interfaces and agentic coding tools. In chat interfaces, your opening query and subsequent responses are valuable context that steer the underlying model to generate text that is relevant to your interests that day. In agentic coding tools, this context is typically thickened to include all of the code and documentation files for a project, and the generated text can be executed as commands in that project environment to have impacts on the real world (e.g. sending a message or deploying a new version of a web service). In 2026, we might generalize both of these with the term “harness.” We say a model can become an agent when it is connected to a harness and supplied with context.

Can you get a chat interface or a coding interface à la carte, without any bundled subscription smarts? Sure, Open WebUI offers a browser-based chat interface that feels a lot like the one you’d see at ChatGPT.com or Claude.ai. Importantly, you can plug in various different inference API services. Open WebUI is software that you can run on your own computer or have someone in your community run on a cloud service on your behalf. OpenCode is a terminal-based coding agent harness that feels like Claude Code, and OpenChamber is a desktop app that feels like Claude Cowork. But, again, they can be connected to any back-end API service. Harnesses are typically free to run directly on your own computer, or they can be run for your community on a cloud hosting service at very low aggregate cost. Setting up Open WebUI for your friends and family is only about as hard as setting up a self-hosted blog package like WordPress or Ghost. If the harness only needs to serve you as an individual, there’s often a convenient desktop app (or even mobile app!) package of that same software available.

Running your own harness (or having someone run it for you) doesn’t mean you need to fully disengage from big-name Generative AI providers and their flagship models. You can use a specific model like Claude Opus 5 inside of OpenCode or chat with the same model that powers ChatGPT from inside of Open WebUI. You can switch models or even back-end providers mid-session to explore alternatives or optimize your costs. You don’t need to export or import your data when you switch providers because your data is stored near you and your harness. Alternatively, you can fully disengage from the big-name providers if you like (perhaps for cost, privacy, ethics, politics, or other reasons).

You can use the Google Chrome browser with your Verizon home internet to watch a documentary on Netflix. I can use the Firefox browser with my Xfinity home internet to watch a documentary on YouTube. Such flexibility, very variety. Now, consider the idea of a university purchasing Google AI Pro for Education (like they’ve done at UC Berkeley). Students, faculty, and staff can only use one family of harnesses (Google-branded surfaces like Google Docs, Gemini Notebook, etc.) to access one family of Generative AI models (Gemini) with a cloud-only context storage model. In this design, there’s no way for me to have an off-the-record chat or try out a different harness or model because the API access layer has been erased. It’d be like an Internet service that only offered Facebook access, as if Facebook simply were the Internet.

BayLeaf is trying to make the alternative, the inference-APIs-and-agent-harnesses design, tangible. Students, faculty, and staff can access a wide variety of Generative AI models, all under zero-data-retention terms, via the BayLeaf API. We have a generous default usage limit (currently $5/day for most users) with the ability to make exceptions. We also run harnesses for chat and coding (using Open WebUI for chat and the Lathe toolkit to add sandboxed code execution abilities). We even have a smoother onramp for using BayLeaf in local-to-your-computer harnesses like OpenCode and OpenChamber. We don’t require your chat data to be stored on our servers (if you use your own harness), and we’ve even figured out how to offer you the smarts (the spicy autocomplete engine) without either us or even our back-end providers being able to access your data, even in flight, using E2EE (end-to-end encryption).

At this point in the blog post, I want you to be able to read retail Generative AI subscriptions (and their institutional equivalents) as if they were Facebook-only Internet access plans. I want you to be able to identify the harness as the thing that makes the underlying generative models useful. I want you to feel a creeping discomfort with the idea that California’s public universities are ceding control and making additional payments to access services with strictly less capability than API-level services from those same providers.

BayLeaf is my fairly hard-line take on what’s appropriate for an institution like UC Santa Cruz. However, I want to applaud the hundred-flowers approach taken by our sister campus UC Berkeley: https://ai.berkeley.edu/tools-training/licensed-ai-tools They offer harnessed and API-level access to models from various providers, some channels centrally funded while others based on recharge, some self-service and others by approval only. Beyond the big-name bundles like ChatGPT Edu and Google AI Pro, they have in-house offerings like their BearGPT and Campus AI Sandbox (roughly analogous to BayLeaf Chat and BayLeaf API). From this breadth of offerings, you can infer that a single big-name bundled-subscription service was not by itself sufficient to meet the needs of the campus community. You could also infer that the investment to explore in-house and open-source alternatives was affordable.

Before I close, I should say that inference API services don’t necessarily have to be things you access over the Internet. The hardware and software implementing the smarts of the API could also be run on your own computer. The responses from laptop-sized models might be a little disappointing, the text trickling out slowly as your laptop gets uncomfortably warm, but it is possible. Generative AI isn’t inherently an Internet technology. It’s just much more convenient (and energy-, time-, and cost-efficient) to treat it as one. We, a big-ass public university that already runs other IT infrastructure, could run the models ourselves, but I don’t think we should go that far. I can analyze the thermodynamics of on-campus inference in another post. For today, just hang on to this idea of separating inference APIs from harnesses. Keep control of your data (your context!), keep your options open, and, when possible, buy your spicy autocomplete in bulk.

I like the way that open-weight models are available for unauthenticated download from the web in the same way that much of the presumed-public data used to train them was. That’s a great expression of reciprocity. Am I giving that up or diluting my focus by allowing people to access closed-weight models on my platform? I’m still not sure.

This morning, I pushed a change to the BayLeaf API that effectively delists closed-weight models. If you craft a request with the correct model slug for a closed-weight model you know of (perhaps you found it listed on OpenRouter), the request will still go through (assuming there are ZDR providers for it available), but I’m not making it too easy.

What do I gain by taking this step, or the next step beyond delisting to actually block access to closed-weight models? I gain some ideological purity points, but who is counting? I gain a talking point about how I operate an open-weight-only LLM service, and that feels good personally. I’d hope, a few months after delisting, that I can tell stories of how people didn’t knock on my door asking for access to those unlisted models. It’d show sufficiency and that access to plenty-good-enough doesn’t require dancing on the frontier.

What do I lose by making those brand-name models harder or impossible to access? Writing this in July 2026, just a few days after the Open Weights and American AI Leadership open letter and the launch and full binary release of Kimi K3, I’m not clear I’m giving up much at all either in the present or in the long term. There’s an increasing thin sliver of capability that is only accessible via specific closed-weight models, and it would be easy for me to justify that accepting (excepting?) this sliver is a fair price to pay for being able to declare ourselves satisfied within the open-weight sphere.

To be fair, it is reasonable to want access to the full variety of models to run comparative experiments. This is something I’m a little interested in myself as an AI nerd, but the vast majority of BayLeaf’s audience, the UC Santa Cruz campus community, doesn’t need this, and it is available at reasonable costs outside the walls of the BayLeaf playground. Maybe I’ll take that next step in the near future.

If my choice is between needing to tell someone that, sorry, we don’t have ClaudeGPT-9000 or getting to tell someone that there’s no way using BayLeaf can contribute data or money to providers who don’t play the Web reciprocity game… uh, the choice is getting clearer by the second.

You should know that the vast majority of BayLeaf’s development and maintenance was done and continues to be done with agents based on closed-weight models, but that fraction is steadily ticking down. And that’s not just because I’m trying out monstrously-large open-weight models like Kimi K3. The fraction done with laptop-scale (and even phone-scale) open-weight models is ticking up from below. It’s just computers, man. It doesn’t have to be empires.

Edit on September 2, 2026: I decided to fully block closed-weight models.

The leaves of the California bay have a particular symbolism for me. They are abundant, abundant to such a degree that they are hard to notice. It takes someone calling them out, maybe remarking on the scent, or telling a story of how the campus mascot banana slug’s coloring is a form of natural camouflage that only works because of the contrast of the occasional yellowed leaf against the more common green and brown... Someone makes them visible, and then you see them everywhere. They are useful too (yes, really), but again it also takes someone stepping you through it to make it believable. Did you know you can make a kind of chocolatey treat from the roasted nuts of the bay tree? Reader, we’ll get to that part later.

Abundance

A pile of leaves doesn’t immediately seem to evoke anything you’d attribute to a generative AI service platform, so give me a moment to build the connections.

BayLeaf (the platform) didn’t arrive with a splashy product launch, fully formed, and shaped by a product team to be consumed by a paying (or captive for the harvesting) audience. Most people in the campus community will encounter it as some thing that was quietly already existing. It should feel fit to the environment because it grew here by incremental evolution of something that was already working for me. You don’t have to use it, and it doesn’t demand much of you when you decide to try it.

Any plant, but certainly something as complicated as a tree, integrates with a larger environment. The bay trees on campus don’t need watering or pesticide treatments. They just sort of continue because they found a niche, and they occupy it abundantly. BayLeaf, as a pile of software, can’t be immediately transported into a new organization, or even another higher education campus, without local adaptation. You need someone to plug it into the authentication system used to identify campus community users, you need someone to register domains, create cloud service accounts, paste API keys (and be responsible for the results of pasting those keys), all in a way that is hopefully compatible with existing campus IT services and policies. But once you’ve got those, it doesn’t take much to keep it going. You don’t need full-time staff, you don’t need corporate gifts or even funding from state or federal initiatives. There’s some level of labor and funding flowing through it, but those fit within my budget of personal activities to stay on top of the happenings of my field, constructed to center on artificial intelligence, interactive media, and the combination of these in higher-education context.

BayLeaf is this thing that appears all around you, once someone points it out, as an already-existing thing that you might make use of if you got the guidance or summoned the curiosity to figure it out on your own. Maybe abundance isn’t quite the right term for a bus-factor-one pseudo-hobby project, but it’s what I’m reaching for with the name.

Usefulness

Have you ever paid for a jar of dried bay leaves at the grocery store only to have it sit unused on the shelf for years because so few recipes seemed to call for them? Stop doing that. The bay leaves you get off a tree on campus aren’t quite the same as the ones you buy from the store, and there are some ways that they are actually better. Don’t wait for them to be mentioned by name in a recipe. Put them in places where they might fit, and see what happens. Bay leaves are moderately useful, and your use of them will expand if you take it upon yourself to talk to others about how they are using them.

You don’t just, like, stuff them into your face like other things you’d buy from the same grocery store. It really matters what context (wink) you place them in. You can learn some recipes. You can invent new ones. You aren’t switching to a bay-only diet, but maybe you are making it a little bit of your identity to try using the leaves in new ways.

BayLeaf, the AI thing, occupies a category of software that some people believe to be categorically useless or even harmful to use. I disagree, but at the same time I see how, if your only exposure to a bay leaves was just directly biting one at some point back in 2023, you’d find them pretty disgusting. I can write much more in the future about what makes most commercial generative AI service offerings unpalatable to me, but I want to stay focused on usefulness here.

Bay trees, in addition to leaves, grow a fruit that is a bit like an avocado, a powerfully bitter avocado with a thin and quick-rotting meat. Useless, right? The pit, the peppernut as it is sometimes called, is this interesting thing that you can collect and roast. When you crack open the shell to access and then grind up the roasted nut-meat, you get this curious oily substance that is a lot like chocolate. It is brown and roasty, sure, but it melts in your hand and freezes solid at room temperature. It tastes like coffee, lightly-burnt popcorn, and, yes, chocolate. Basically, it’s like an alt-nutella that you can make from squirrel-discarded lumps you can just pick up off the ground. It’s a thing (and probably has been a thing around here for the last ten thousand years).

Acorns are a thing too, and they are even more abundant (in masting years) and quite a bit more useful too. But, I didn’t name my service after acorns, so I’ll scale back to the more subtle and let-me-introduce-you nature of the California bay.

I haven’t said anything about LLMs, agents, skills, sandboxes, or security reviews, labor relations, or even anarchist philosophy. We’ll get to those aspects of BayLeaf in another post. For today, let’s say that BayLeaf does offer something useful to those who are patient and curious to build skill and experience using it. It’s not the revolutionary future of anything. It’s the already-present, a pretty normal and, in the big picture, moderately useful thing you could learn about. Beyond individual use, there’s another level that we, the higher education community, might find useful too, but we can take our time on that. There’s no rush.

Thanks for your attention. This has been a test of the BayLeaf Blog publishing system. (First post.)