Earlier, I rejected the idea of on-campus inference services. Even though the energy and hardware resources are available, there are good reasons not to use them. But does this mean I need to accept that a campus-specific Generative AI platform must use third-party inference services? I’ve got a deep feeling that it doesn’t, but I don’t have a clear vision of why. In this post, I’m trying to figure out what “first-party inference” would mean for the BayLeaf counterplatform.
We have campus IT services that aren’t physically sited on campus. Surely we do.
Like, how does our main website at https://ucsc.edu get hosted? It looks like ucsc.edu redirects through Fly.io to an instance of WordPress hosted on Cloudflare. Is this something we’re, in some sense, hosting ourselves? WordPress is open-source and self-hostable. No, we’re contracting with CampusPress, and those folks use Cloudflare. If we had decided to run WordPress ourselves atop virtual machines from AWS, I’d be willing to call that first-party because I care about the application layer, data ownership, and authorability more than I do who pumps the electrons into the physical hardware, at least for the purposes of this blog post.
We run our own DNS services, we have our own ASN (AS5739), and we announce our networks via BGP through CENIC, which connects us to Internet2. We can hold our physical ground on the Internet, but these aren’t things that most faculty, students, and staff interact with directly. What applications do we meaningfully control, ones where we haven’t ceded autonomy to a party outside of the university, at least at the relevant level of discourse?
Let me try again. How does the campus directory work? At the DNS level, campusdirectory.ucsc.edu is a CNAME for campusdirectory.prd.idm.aws.ucsc.edu, and that resolves to endpoints in AWS us-west-2 (Oregon). UCSC moved a lot of physical IT infrastructure to AWS, an Infrastructure as a Service (IaaS) provider, around 2018. From the looks of it, the campus directory is plausibly campus-specific software written by us and running on our preferred place to (virtually) site our services. I’m happy to call the campus directory a first-party service.
What about MyUCSC, our academic information system (AIS)? This seems to be an instance of PeopleSoft Campus Solutions running on hardware we manage in that same AWS data center in Oregon. If PeopleSoft were just a heap of text we pulled from a public repository without any special negotiated license, I’d call this another first-party service. However, it seems to be proprietary software we’ve purchased so that we can run it ourselves. If we want to make campus-specific changes to our instance, it isn’t clear that we can just go do that without getting permission, support, or some other kind of buy-in from the vendor.
Oh, maybe the UC Santa Cruz Library? They run the open-source Drupal content management system (CMS) themselves, via AWS. Even better, the library publishes a heap of locally-developed open-source software including plugins for the Omeka digital collections and online exhibit service, which they also run on AWS. This is how they operate the Grateful Dead Archive Online.
Taking this all back to first-party (LLM) inference, I think we could do it. We could run locally developed software (or locally configured open-source software) on IaaS-style infrastructure, and it would feel, more or less, like we’re running the AI services ourselves. In the constellation of these others, it wouldn’t even be out of the ordinary.
I ran a little experiment last night to see what running a first-party inference service atop IaaS would be like. It wasn’t targeting something that would cover the entire campus demand for inference services, just something that could sustainably show capability, to make a credible exit from BayLeaf’s current use of OpenRouter plausible. I spun up some GPU sandboxes on Daytona (Daytona is an IaaS provider that we already use to power the Code Sandbox feature in BayLeaf Chat), installed the open-source vLLM serving engine (used by many other inference providers on OpenRouter), and pulled down the weights of a 27-billion-parameter language model. I guess I should call this one a small language model, SLM, because it is also one that fits on my laptop.
Skipping a bunch of details and rounding the numbers, here’s the finding: For about $1 per month, I can maintain the capacity to spin up, on 2 minutes’ notice, a first-party inference service that costs $1 per hour while active. It hits maximum efficiency handling about four user requests in parallel (with others waiting in a queue), and it supports long-context caching well enough for entry-level agentic use patterns.
This isn’t as cheap and convenient as the always-on inference available from commodity markets, but it would be something we own, that we author. We could host weird, unpopular, campus-specific models. We could offer inference for doing, say, automatic transcription of sensitive audio and video data, without needing to trust a third-party’s zero-data-retention (ZDR) pinky promise. We could, with the code pulled from the BayLeaf software repository, just clearly not store that data on disk or send it anywhere else.
Yes, of course, you need to trust the IaaS. You need to trust someone if you aren’t going to melt the sand into a GPU yourself. What matters is whether the third party you need to trust is a data-sucking edtech monster or whether they are an ultra-generic containerized compute provider that you could, with high credibility, bring on-prem on short notice.