Your company’s most sensitive data is being pasted into a chat box, and nobody signed off on it. Customer records, unreleased code, contract language, and incident postmortems with real hostnames in them all go out over an API to a vendor you have a ToS with and not much else.
I’m not claiming Anthropic or OpenAI are out to steal anything. Their business terms generally say they won’t train on your data, and I’d take them at their word. But “we promise” is a policy, not an architecture. Your data still crosses your perimeter, still sits in someone else’s logs for some retention window, and is still subject to someone else’s subpoenas, breaches, and contract changes. If your legal team can’t point to where the data lives, you don’t have a data governance story. You have a vibe.
What Actually Leaves Your Building
Nobody thinks of an LLM call as a data transfer, so nobody audits it like one. Here’s what’s in those prompts right now:
- Source code, including the proprietary parts, the ones with hardcoded config, and the ones with a TODO that says “fix auth before launch.”
- Customer support transcripts with names, emails, and account details, sent in for summarization.
- Stack traces and logs with internal hostnames, IPs, and the occasional token someone forgot to scrub.
- Contracts, pricing sheets, and board-deck drafts, because someone wanted a tighter summary.
- Database schemas and sample rows, because “just paste the table so it understands the structure.”
Multiply that by every developer, analyst, and PM with a browser tab open, and you have a continuous, unmonitored export of your company’s internal knowledge. It’s happening whether or not you’ve ever approved it. The only open question is whether you have a say in where it goes.
Why We Keep Doing This
Convenience. An API key takes five minutes. A GPU cluster sounds like a quarter-long project, so everyone picks the five-minute option and calls it a “pilot.” That pilot is now in production and handles customer PII.
Hype. Nobody gets fired for buying the famous model. “We use the frontier API” is a safe sentence to say in a board meeting, even when the task is classifying support tickets.
Resume-driven procurement. “Integrated GPT-whatever across the org” looks great on a promo packet. “Kept the data boring and in-house” doesn’t.
Fear of the hard part. Self-hosting used to mean racking GPUs and babysitting CUDA drivers. A lot of teams are still making 2023 decisions based on 2023 fear.
Nobody owns the question. Security assumes the platform team approved it. The platform team assumes security did. Meanwhile, shadow AI usage spreads because the sanctioned option is “wait for the committee.”
The Cost Reality Check
Your bill scales with enthusiasm. Once developers discover they can shove a whole repo into a prompt, token spend stops tracking value and starts tracking laziness. Agentic tools make it worse, because a single “fix this bug” request can fan out into dozens of model calls that nobody on the team ever sees. You’re renting a meter, and someone else sets the rate.
Your product depends on a vendor’s roadmap. Models get deprecated, behavior shifts between versions, rate limits change, and prices move. Your evals passed last month and nobody knows why they’re failing now. You don’t own the thing you built on.
Compliance is a recurring cost. Every customer security questionnaire asks where the data goes. “To a third party, and here’s their DPA” delays or kills deals in healthcare, finance, and defense. Every delayed deal is a cost that never shows up on the AI line item.
Cognitive cost. Someone has to keep track of which teams use which provider, under which agreement, with which retention settings. That’s real overhead, and it grows with every new tool that quietly ships with an LLM integration.
The Honest Objections
Let’s deal with the objections up front, because pretending they don’t exist is how self-hosting posts lose credibility.
“The closed frontier models are smarter.” On the hardest reasoning tasks, sometimes they are. But most internal workloads (summarization, extraction, classification, code assist, internal Q&A) don’t need the best model on the planet. They need a competent one that’s always available and doesn’t leak. Don’t use a surgeon to take your temperature.
“We don’t have the expertise.” You’ll need someone who understands inference serving, and you’ll want to budget for that. But you already run Kubernetes, databases, and queues. This is another stateful-ish service with unusual hardware needs, not a new discipline.
“GPUs are expensive.” Idle GPUs are expensive. A fleet running at 8% utilization is a space heater with a procurement budget. This is the objection that kills most self-hosting projects, and it’s the one with an actual fix.
The Boring Alternative: Open Models, Your Cluster, Autoscaled GPUs
Kimchi for the model layer. Kimchi is a governed AI coding platform built around open-source models. Its enforcement layer applies spending, data, deployment, and model-routing policies before requests reach a model, and teams can set hard budget limits by user, team, API key, or model. That kills the runaway-token-bill problem at the source. It exposes production-ready open-source models like Kimi K2.6 and MiniMax M2.7 through an OpenAI-compatible API, so your existing tooling keeps working and you mostly change a base URL. It can even route Claude Code through an Anthropic-compatible endpoint, so developers keep the workflow they already like.
Deploy it in your own environment for the sensitive stuff. Here’s the distinction that matters. Kimchi’s serverless API runs models on Kimchi’s own GPU infrastructure, which is fine for low-stakes work but is still a third party. For stricter requirements, you can deploy on Kubernetes in AWS, GCP, Azure, or on-premises, which keeps model data within your own environment. Use serverless for the low-stakes stuff and self-deploy for anything that touches customer data or proprietary code. Pick the deployment mode per workload, not per company.
Cast AI for GPUs on demand. This fixes the utilization problem. Cast AI’s autoscaler provisions GPU instances on demand, scales them down when they’re no longer needed, and uses spot instances to cut costs further. It also supports GPU time-slicing, so multiple workloads can share a single GPU. It handles the NVIDIA driver installation and configuration too, which is the unglamorous part that eats weekends. You write a standard Kubernetes Deployment and Cast AI handles the optimization underneath, and if spot capacity dries up, its fallback feature temporarily runs the workload on on-demand instances until spot is available again. It runs in your own cloud account, so your data and your GPUs stay in your tenancy, and your bill follows actual load instead of hoped-for load.
Doing the Math Honestly
Don’t take my word for the economics. Run the numbers on your own workload:
- Measure your current spend. Pull a month of API usage and break it down by team and use case. You’ll probably find that a handful of workloads account for most of the tokens.
- Estimate GPU hours for the same workloads. Be honest about peak versus average load. Bursty, business-hours traffic is where autoscaling pays for itself.
- Add the human cost. Include on-call, upgrades, and the engineer who understands the serving stack. Leaving this out is how self-hosting looks cheap on a spreadsheet and expensive in real life.
- Compare at your real utilization, not the best case. A cheap GPU hour you can’t fill is still a wasted hour.
For steady, high-volume, internal workloads, self-hosting tends to win. For tiny, spiky, or frontier-dependent ones, the API may stay cheaper. The point is to make that call deliberately instead of by default.
A Sane Rollout
- Pick the one workload that touches your most sensitive data and move it first.
- Deploy Kimchi on Kubernetes in your own cloud account, behind its OpenAI-compatible endpoint.
- Let Cast AI autoscale the GPU nodes, and set Kimchi’s hard budget limits so a runaway loop can’t surprise you.
- Build a small eval set from your real tasks and compare self-hosted against the API honestly. Rerun it whenever you change models.
- Keep API access only for tasks where the eval shows a real gap, with a written reason and an owner.
- Lock down egress so the sanctioned path is the easy path and the unsanctioned one gets noticed.
Mistakes to Avoid
- Buying hardware first. You don’t know your utilization yet. Rent until you do.
- Skipping evals. “It feels about as good” is how you ship a regression to production.
- Self-hosting everything on principle. Use the API where it’s genuinely better and the data isn’t sensitive. Ideology is not an architecture.
- Forgetting the humans. If the self-hosted option is slower or clunkier than the browser tab, people will keep using the browser tab.
- No budget guardrails. An agent in a retry loop will happily burn a month’s budget over a weekend. Hard limits are not optional.
The Bottom Line
Own your inference the way you own your database. Nobody sane ships production Postgres to a vendor’s multi-tenant black box and hopes for the best, and your prompts are at least as sensitive as your tables. Keep the data in your account, scale the compute on demand, and send things outside only when you’ve decided it’s worth it.
It’s not exciting, and it won’t get you a conference keynote. It will get you a security questionnaire you can answer in one sentence, and a bill that makes sense.
