An AI feedback interviewer, rebuilt after a security review showed the first version was an open LLM proxy.

Survey Guardian

The feedback widget after one message. The reply says the product only offers daily or weekly schedules and asks what the respondent needs hourly reports for. A counter reads "1 of ~5".
RoleSolo — product, security review, engineering
Timeline2026
StackNext.js, TypeScript, Supabase, Claude, Tailwind

The problem

Feedback forms get you "it's confusing" and stop there. The useful part is why. Someone who writes "where's the Slack integration?" about a product that has none doesn't need a thank-you. They need a short correction and a question about what they were trying to do.

I had two Lovable-generated prototypes of that idea from March 2026. One asked a single "why" follow-up after each message. The other, a multi-tenant version, checked what respondents said against a description of the product. Both demoed fine. Then I read the code. Both AI endpoints were public: no login, CORS open to every origin, and they took the message history from the caller, role included. Anyone who found the URL could use my AI credits as a free LLM. The multi-tenant one also took the tenant and conversation IDs from the request body and wrote with the service role key, so it trusted the caller to say whose data to touch.

What I built

Solo, end to end: product decisions, security review, schema, backend, UI and an eval. I rebuilt it as one Next.js app with Claude Code, and wrote the rules that must not regress into the repo.

Dark mode. A green check and "Thanks for your feedback" above a short summary of what the respondent said.
The end of a conversation, in dark mode. The summary is written by the model, and the respondent sees it too.
The owner dashboard listing one survey, Acme Exports, with colored counts by kind: bug 2, feature request 2, confusion 4, praise 1.
The owner's view, on demo data for a fictional product. The counts come from a database view that applies the owner's row-level security.
A survey's conversations, each with a summary or "No summary" while it is still active, a kind badge on every respondent message, and an open transcript. The top one shows a respondent asking for an hourly schedule and the interviewer correcting it.
Per-message classification on demo data. The top conversation is a live run against the model. The others are seeded.
  • One door for respondents. Every respondent write goes through a single route handler, after validation. The public database key can't read or write anything: row-level security is owner-only and the grants are revoked as well. A script asks the live project what the public key can reach. Against the hosted test project, the answer is "permission denied" everywhere.
  • No client-sent history. The request body is a strict schema with no role field and no message list. The server loads the conversation from the database, and respondent text only ever lands in user turns, so a caller can't smuggle in a system prompt.
  • The turn cap is one SQL statement. A single function checks the conversation token, the survey, the status and the cap, and claims the turn atomically. Against a local Postgres, twenty simultaneous requests against a cap of three produce exactly three winners.
  • A monthly cap per owner. Respondents are anonymous and every conversation spends my AI credits, so each owner gets 200 new conversations a month, roughly a dollar. The count and the insert run under one lock. To check the lock matters, I ran the same test on a copy without it: twenty simultaneous starts let 13 through where 5 fit. With the lock it was exactly 5, every round.
  • Tokens are stored hashed. A respondent's 32-byte token is returned once and kept only as a SHA-256 hash.
  • It fails closed. In production, a missing rate-limit store or a missing bot-check secret means the endpoint refuses instead of running unprotected.
  • One LLM call per turn (the prototype made two). The call returns the reply, a classification (bug, feature request, confusion, praise, other) and a note. When the respondent expects something the product description says doesn't exist, the reply corrects them first.
  • Closed signups. Owners sign in by magic link and are invited by hand. The page is framed only on the survey route, under a per-request nonce CSP.
  • Tests that match the hosted setup. An offline schema test runs the migrations in an in-memory Postgres, in two grant modes, and probes the policies, grants and the turn function. I checked that each new check fails without its fix. The database's own security advisor then found two things my tests hadn't: a function created by the platform's "automatic RLS" option was callable by the public through the API, and the API roles held table privileges the app never uses. Both are fixed in migrations, with tests.
  • An eval. 30 labeled messages through the real code: 29 correct, about 808 input and 60 output tokens per turn, around a tenth of a cent per turn at Haiku 4.5's list price ($1 and $5 per million tokens). Small, written by me, not a held-out benchmark. The one miss: "the link says it expired" came back as other, not bug.
  • A widget. A chat bubble that loads its iframe on first click, with light and dark following the visitor. It copies the interaction of the original prototype.

Where it stands

It isn't deployed. There's no URL, no outside user and no real respondent yet. What exists: the rebuilt app, a hosted Supabase test project that passes the public-key audit (as of 2026-10-27) and has had its database-level advisor findings fixed, and a full local run with a live AI conversation and the owner dashboard.

Not tested yet: rate limiting and the bot check end to end, and a hosted deploy. Next is a deploy and an invite-only beta, starting with a widget on kirkclarke.com.

I'm not launching it as a SaaS. AI follow-up questions already ship in the big survey tools, so the concept isn't mine to claim. I'd open it up only if three outside teams ask.

What this taught me

An AI-built prototype that demos well can ship an open LLM proxy, and the demo never shows it. The question to ask is what a stranger can do with the endpoint.

My first schema test passed under permissive defaults while the real project handed extra privileges to the API roles. I only saw that by asking the live database. A test has to run in the same mode as production.

My first concurrency test ran in a one-connection in-memory database, where requests queue up instead of racing, so it couldn't have caught a race. What settled it was real parallel sessions, plus a copy of the code without the fix to show the test can fail.

The eval turned "seems to classify well" into 29 of 30 and showed the exact message it missed.

Got a hard problem? I’d like to hear about it.