Note · Infrastructure

Self-hosting a 26B model behind my portfolio chatbot

How the "Ask my site" widget actually works: a static site, a Cloudflare Worker, and a quantized 26B model running on hardware I own.

The chat widget on this site, the one that answers questions about my projects, doesn't call a hosted API from a large provider. It calls a language model I run myself, on hardware sitting in my own home, reached through a tunnel. This note is the architecture and the trade-offs, without the parts that would just be an invitation to poke at my infrastructure.

The architecture, in one pass

The site itself is static: HTML, CSS and vanilla JavaScript, served by a Cloudflare Worker with an assets binding. Almost every request never touches the Worker's code at all; it's routed straight to static files. The one exception is a single API route, a POST endpoint, which is the entire dynamic surface of the site. That route proxies chat requests to an OpenAI-compatible endpoint served from a machine at home, reached through a tunnel. The model behind that endpoint is a quantized 26B model: small enough to run on hardware I own, capable enough to hold a decent conversation about my own work.

The bot's knowledge is deliberately narrow. Every request is grounded in the site's own public project catalogue, the same document that answer engines and AI crawlers read. That means the assistant can only discuss what's already public on the site. It can't leak anything, because there's nothing private in its context to leak.

A few design decisions worth explaining

The API token for the model never leaves the server. It's read inside the Worker from a server-side secret and attached to the outbound request; the browser only ever talks to the same-origin chat route and never sees a credential.

There's no in-Worker rate limiter, and that's deliberate rather than an oversight. Cloudflare Workers run across many short-lived isolates that share no memory, so an in-process counter for "requests per IP" doesn't survive between requests. It resets constantly and protects nothing. Instead the route enforces a same-origin check, rejecting anything whose Origin or Referer isn't this site, plus hard caps on request size and message count. Real rate limiting belongs at the edge, as a WAF rule in front of the Worker, and that's where I put it.

Replies stream, but not over EventSource. EventSource is a GET-only API, and this is a POST endpoint carrying a chat history, so the Worker instead reads the upstream model's stream and re-emits it as newline-delimited JSON, one small object per chunk. The client reads that stream and renders text as it arrives: the same user-facing effect as EventSource, without fighting its GET-only constraint.

And when the tunnel is down, which does happen, the widget doesn't fail silently or throw a raw error. It falls back to a plain message pointing at my email. A visitor should never wonder whether the thing is broken; they should just get a way to reach me anyway.

The honest trade-off

A model running on a personal machine behind a tunnel is a single point of failure. If my home connection drops or the machine reboots, the chatbot goes quiet until something notices. For a portfolio site, that's a fine trade: the cost of downtime is a slightly less impressive demo for whoever happens to be visiting at that moment. For a client's production system, the same architecture would be the wrong call entirely, because downtime there costs someone else's business, not just my own polish.

Knowing which situation you're in, whether a single point of failure is an acceptable cost or a liability you're quietly building into someone else's system, is the actual engineering skill here. The architecture itself is simple. The judgment about where it's appropriate took longer to get right.

There's one piece of monitoring on top of all this: a cron job runs every 15 minutes, pings the tunnel, and posts to Discord only when the state changes, up to down or back to up. I don't need a dashboard for a two-route API. I need to know the moment it stops answering, and nothing more than that.

Try it: the ✳ button on any page of this site is that exact pipeline.