The Prompt Is Not a Security Boundary

Most AI features that touch sensitive data protect it the same way: with a paragraph. Somewhere in the system prompt there's an instruction like never reveal personal information, and that instruction is the privacy architecture. It usually works, in the sense that a locked door usually works when nobody pulls the handle. But an instruction is a request, and a security boundary that depends on the other party honoring a request isn't a boundary. It's etiquette.

This post is about a system built on the opposite assumption. TeachMetrics is a self-hosted analytics platform for Teachable school owners that I'm developing. We use Teachable for cRisk Academy. TeachMetrics syncs a school's full history into the operator's database. That is full of student PII. Names, emails, per-student progress. We have a feature called "Ask-AI" that lets the operator ask questions in plain English: which courses had the most enrollments this quarter? How's my school doing? Answering those questions means involving a third-party language model, and the product is bring-your-own-provider: anything speaking the OpenAI-compatible protocol, from OpenAI and OpenRouter to a local Ollama running on the same machine.

That last detail is what forces the interesting design. If the operator can point the feature at any provider, the design cannot assume the provider is trustworthy. So the architecture starts from a deliberately uncharitable premise: treat the AI model as an untrusted SQL author. Assume the provider is hostile. Assume the model will one day attempt to extract the full database via `SELECT * FROM users`. Then build the system to protect against this, because the guarantee never depended on the model behaving in the first place.

The prompt is UX. The boundary is code.

A semantic layer is a privacy layer

The foundation is what the model is allowed to see, and it's less than you'd guess.

TeachMetrics exposes exactly 24 read-only MySQL views to its Artificial Intelligence paths, all named `ai_*`, all defined in migrations. Each view is an aggregate: one row per course, per month, per country, per price band, per coupon, per cohort. The columns are counts, sums, rates, and labels, with self-documenting naming conventions. Here is some of what the views contain: course names, revenue sums, completion rates, calendar periods. What they structurally cannot contain: a student's name, email, IP address, or any per-student row. Not because a rule filters those out at query time, but because no such column exists in any view. There is nothing student-shaped to leak.

Here's the part I find genuinely elegant: this layer wasn't built as a privacy control. It was built for accuracy. Text-to-SQL models are notoriously bad at guessing the right joins across a real schema, so the vetted joins live inside the views, and the model only ever writes simple queries against pre-aggregated, correctly-computed tables. The same construction that makes the numbers correct is the construction that makes PII unreachable. Accuracy and privacy came from the same design move.

One implementation detail worth stealing if you're here for that: the catalog of these views is a single source of truth twice over. The same array that renders the data dictionary in the model's prompt is the allow-list the query executor enforces. Adding a view means one migration plus one catalog entry, and the prompt and the enforcement can never drift apart, because they are literally the same data structure. Anyone who has watched documentation and validation diverge over time will recognize what that's worth.

Six layers, each assuming the ones above it failed

The model writes SQL; the SQL is treated exactly like user input. What follows is defense in depth in a plain PHP application on shared hosting: no vector database, no agent framework, no cloud dependency. Six layers, each designed on the assumption that everything above it has already been defeated.

Layer 1: static validation. The generated SQL is rejected unless it is a single statement, starts with `SELECT` or `WITH`, contains no banned keywords (writes, DDL, file access, schema probing, timing and resource-abuse primitives), and every `FROM` and `JOIN` target is on the `ai_*` allow-list. Two details here are the war stories. First, SQL comments are banned outright: MySQL treats `/* */` as whitespace, so `FROM/**/users` would sail past a naive `FROM <name>` regex while executing against the real `users` table. The model has no legitimate need for comments, so rather than parse them, the validator refuses them entirely. Second, schema-qualified escapes like `db.users` don't work either. If a qualified name resolves to something that isn't on the allow-list, and the query is rejected.

Layer 2: a read-only envelope. The validated query still runs inside a `READ ONLY` transaction with a five-second statement timeout and a 200-row cap on what returns. Any write transaction attempt that somehow survived Layer 1 dies at the database.

Layer 3: a grant sandbox, where hosting allows it. On webhosts that permit a second MySQL user, the AI path connects as a SELECT-only user granted on exactly the `ai_*` views and nothing else. Enforcement moves into the database's own privilege system, the layer beneath the application entirely. There's even a "test sandbox" button in settings so the operator can verify the user genuinely cannot read base tables, rather than taking the claim on faith.

Layer 4: error hygiene. Database error messages can contain column names and SQL fragments, which makes them an exfiltration channel: probe with deliberately broken queries, read the schema out of the exceptions. So raw errors are never surfaced. A failed query produces a generic "could not be completed," to the LLM. Errors are output that LLMs are trained to resolve and work through; treat them like output.

Layer 5: outbound transport. The AI provider endpoint is operator-configured in TeachMetrics, which makes it a server-side request forgery surface. The endpoint guard blocks cloud-metadata addresses over both IPv4 and IPv6, rejects non-canonical numeric hosts that decode to internal IPs, and pins the actual connection to the vetted IP with redirects disabled, closing the classic DNS-rebinding window where an attacker's nameserver answers the safety check with a public address and the real connection with a private one. Localhost, notably, is allowed. A local Ollama AI is the maximum-privacy configuration, and the guard is built to permit exactly that.

Layer 6: transparency. Every Ask-AI answer ships with a collapsible "What was sent to the AI" disclosure showing the exact payload that left the server. The privacy claim is inspectable per-answer, not just asserted in a policy document or technical manual.

It's fair to ask whether publishing this map hands attackers a list of things to probe. It does, and that's acceptable for the oldest reason in the field: Kerckhoffs's principle. A design that only holds while its mechanism stays secret doesn't hold. The TeachMetrics design was built assuming the attacker knows everything, which is exactly why the read-only transaction and the grant sandbox sit beneath the validator: pattern-matching layers get bypassed eventually, so the layers that don't pattern-match are there to backstop them. Publishing the design invites scrutiny on our terms. What you won't find here, deliberately, are the literal validation patterns. The categories are the architecture; the patterns are just today's implementation.

Prompt injection, cut down to size

Prompt injection is usually framed as the unsolved problem of the LLM era, and as a way of making models reliably ignore adversarial instructions, it largely is. But the framing smuggles in an assumption: that a successful injection matters. It only matters in proportion to what the injected instructions can reach.

Walk it through here. Suppose an operator pastes a question containing a maximally successful injection, and the model obeys it completely. What can the attacker's instructions actually do? Produce a SELECT statement. That statement must pass the allow-list validator, execute read-only against aggregate views containing no student data, under a user that may hold no other grants, returning at most 200 rows of counts and sums. The blast radius of total prompt-injection success is: a different aggregate query ran.

That's the payoff of putting the boundary in code. You don't have to solve prompt injection. You have to make it not worth solving.

Saying precisely what the guarantee is

Trust systems live or die on the precision of their claims, so here is this one stated carefully.

The claim is not "no data is ever sent to the AI provider." Data is sent: the operator's question, a schema dictionary, and aggregate rows labeled with course names, enrollments, revenue, and coupon codes. The claim is that no individual student record ever leaves the operator's server, and that this holds even against a hostile provider, because it's enforced in code the AI LLM provider never touches. (For operators who want the stronger property, it's available: point the feature at a local LLM model and nothing leaves the server at all.)

Nor is the claim "impossible to misuse." TeachMetrics is source-available, and a self-hoster who edits the views, the executor, or the snapshot builder can absolutely leak whatever they expose; the public policy documents place responsibility for modified installations on the modifier, and they name "data views and query safeguards" specifically rather than prompts alone, because the prompts were never where the guarantee lived. Notably, a modified prompt still can't leak anything: the executor doesn't care what the prompt said. That asymmetry is the whole thesis in one sentence.

The pattern travels

None of this required exotic infrastructure. TeachMetrics a PHP application, a set of SQL views, a validator that fits in a few dozen lines, a read-only transaction, and some careful curl options, deployable on shared hosting. What it required instead was a decision made early and held consistently: the model's output is untrusted input, full stop, and every layer downstream of the model is built accordingly.

The industry keeps trying to make models behave by asking them nicely, then patching the asks when they fail. The more durable posture is older than any of this: don't put the guard where the persuasion happens. Put it where the persuasion can't reach.

The prompt is not a security boundary. Build one out of code, and the prompt can go back to doing its actual job, which is being helpful.

---

The architecture described here ships on TeachMetrics); The data views, the validator, and the read-only transaction do the guaranteeing; the prompt just does the talking.