Beyond Tokenomics: A Framework for Monitoring AI Costs at Enterprise Scale

Uber’s CTO recently disclosed that the company burned through its entire 2026 AI coding budget in four months. By March, 84% of Uber’s engineers had adopted Claude Code, and roughly 70% of newly committed code at the company was AI-authored — impressive adoption, and an entirely predictable way to blow through a budget that was sized for a different usage pattern. One unnamed enterprise reportedly ran up a $500 million bill in a single month on Claude usage after nobody had set per-employee usage caps. Nvidia’s Bryan Catanzaro has said plainly that for many of his own workflows, “the cost of compute…now far exceeds what the company spends on the employees using it.” Jensen Huang’s own recommendation — a $250,000 annual token budget per $500,000 engineer — is itself an admission that AI inference is now a line item comparable to headcount, not a rounding error under it. (Forbes)

None of this is a fringe problem. TechRadar reported on a KPMG survey in which 29% of senior leaders across 20 countries said they were struggling to understand the rise in their own operating costs as they scaled AI, and close to half of organizations surveyed said they’d had to rephase AI deployments after costs outweighed the expected value. An MIT study cited in the same reporting found AI automation is currently economically justified in only about 23% of roles it’s being pointed at. Companies are laying off workers to fund tools that, in a lot of these cases, cost more than the workers they replaced.

I don’t think this is because AI is inherently unaffordable. I think it’s because most organizations are monitoring AI spend the same crude way you’d monitor a phone bill — total tokens times a per-million rate — and that kind of tokenomics genuinely cannot answer the questions a CEO ends up asking after the invoice arrives: which team, which workflow, and was this normal. Below is a small, personal-scale experiment in the kind of monitoring that actually would answer those questions, plus what I think the enterprise version of it needs to look like.

Why Tokenomics Alone Fails

A dashboard that reports “$40k this month” tells you almost nothing useful. A few reasons a raw dollars-and-tokens total keeps blindsiding the people responsible for it:

• Model mix. 95% of enterprise AI usage reportedly still runs on the costliest frontier models, often by default rather than by decision. Total spend doesn’t tell you whether that’s appropriate — it takes knowing what each task actually needed to know whether a cheaper model would’ve done the job.
• Caching. Prompt caching can cut effective cost by an order of magnitude on repeated context, but a cache-read token, a cache-write token, and a fresh input token are all priced differently. Two teams can show identical raw token counts and have wildly different bills depending on cache hit rate.
• Agentic fan-out. A single request today can spawn dozens of tool calls, sub-agent invocations, and retries — which is exactly the shape of Uber’s problem: high adoption (84% of engineers) times high automation (70% of code) times no visibility into what any of that fan-out was costing per unit of shipped value. Attributing cost to “the prompt” undercounts badly when the prompt is really the seed of a much larger task.
• Attribution. None of the above matters if a request can’t be tied back to a team, project, or use case. The $500 million-in-a-month story isn’t really a story about AI being expensive — it’s a story about nobody having set a cap, which is only possible when nobody’s tracking usage by who’s generating it.

Fixing this requires seeing spend the same way you’d see any other high-volume, unlabeled event stream: you need to attribute it, cluster it by what it’s actually for, and watch it for the kind of statistical anomaly that a budget alert set at a fixed dollar threshold will always catch too late. That’s a data problem I already had a smaller version of sitting on my own laptop.

A Small-Scale Proof of Concept

I don’t have access to an enterprise’s AI usage logs, but I have unrestricted access to my own — every prompt I’ve sent Claude, across both Claude Code and the Claude.ai web app, going back to mid-2023. So I built the smallest version of a use-case-level cost monitor I could: instead of asking “how much did I spend,” I asked “what was I actually spending it on,” and let the data answer that itself rather than tagging things by hand.

Collecting the Data

Claude Code keeps a local history: a top-level history.jsonl plus a per-project conversation transcript for every session, with exact input/output token counts since the CLI logs what it actually sends to the API. I paired that with an export of my Claude.ai web conversations, which don’t carry the same token accounting, so I estimated token counts there with a tokenizer and tagged every row with its source (exact vs. estimated) so the two wouldn’t get silently conflated later — the same distinction an enterprise system would need between metered API usage and everything else.

Both sources got flattened into a single CSV — prompt text, project, timestamp, character/token counts, and source — leaving 463 prompts after dropping empty rows.

Embedding and Clustering

Rather than pick a number of use-case categories up front — which is what most cost dashboards do when they let you tag spend by hand-picked labels — I let the structure emerge from the prompts themselves, the same underlying idea as my earlier SBM/GNN community detection post:

1. Each prompt gets embedded with sentence-transformers (all-MiniLM-L6-v2).
2. I build an implicit similarity graph by thresholding cosine similarity between embeddings (0.55) and run community detection over it — prompts that are semantically close enough end up in the same community; everything else is left unclustered rather than force-fit into a group.
3. For visualization only, I reduce the same embeddings to 2D with UMAP. The clustering itself never sees the 2D coordinates — they’re purely a layout for plotting, not part of the community detection.

That produced 80 communities out of 463 prompts, with about a quarter sitting as singletons — one-off questions with no close semantic neighbors. At enterprise scale, this is the step that replaces manual cost-center tagging: instead of a human deciding upfront that spend gets bucketed by team or by project code, the clustering discovers the actual shape of the usage, then you can attribute cost to that shape.

The Result

The chart below is built from the real clustering output — real positions, real cluster sizes, real token counts — but I’ve stripped out the individual prompt text and project names before publishing it. Hovering a point shows you which topic group it belongs to and its token cost, not what I actually typed.

What It Shows, and What That Maps to at Enterprise Scale

A few things stood out once the communities were laid out, and each one has a direct analogue in the enterprise numbers above:

• The single largest community wasn’t technical at all — youth sports team coaching, 97 prompts, more than double the next biggest group. Use-case clustering surfaces where volume actually concentrates, which is very often not where anyone assumed it would be. At enterprise scale, that’s the difference between believing spend is driven by “engineering” and discovering it’s actually concentrated in three or four specific repeated workflows, some of which have nothing to do with the team nominally paying for them.
• A quarter of everything I’ve sent Claude never joined a community. 119 of 463 prompts were similarity-orphans — one-off, novel requests. A growing unclustered fraction at enterprise scale cuts two ways: it can mean a genuinely new use case is emerging and worth investing in (good), or it can mean nobody has standardized how a given task gets asked, so the same job gets solved a slightly different, slightly more expensive way every time by every person who touches it (bad, and exactly the kind of thing that produces a $500 million surprise when nobody’s watching the aggregate).
• Cost and prompt count don’t track together. My biggest cluster by volume wasn’t my biggest cluster by cost — smaller, chattier clusters rack up disproportionate output-token spend per prompt because the responses run long. This is the same disconnect Uber’s own leadership pointed to: high token usage that doesn’t correlate cleanly with anything being shipped. You cannot see this disconnect in a total-dollars number; you can only see it once spend is broken out by what generated it.
• exact vs. estimated matters. Claude Code’s logged token counts and my tokenizer-based estimates for the web app agree in aggregate but diverge enough at the row level that I wouldn’t trust the estimated rows for anything more precise than the cost table above. An enterprise system built on inconsistent metering across tools will inherit the same blind spot, just at a scale where it’s a lot more expensive to be wrong.

What $1,000/Day on a Single API Key Actually Looks Like

A useful way to pressure-test the clustering framework: take a single API key spending $1,000/day — not a team, not an org, one key — and work backwards to what must be generating that spend.

At Sonnet 4.6 pricing ($3 input / $15 output per million tokens), $1,000/day is roughly 66 million output tokens. That rules out a single heavy user immediately; even the most expensive cluster in my own data — seven short prompts that triggered full document generation — ran to about $11 total. To hit $1,000 per day you need either enormous volume, very long outputs, or both, running continuously.

Three scenarios account for almost everything you’ll actually see:

Always-on document processing. Legal discovery, insurance claims, medical records, compliance review — pipelines that ingest long documents (2–5K input tokens each) and emit short structured outputs, running 24/7 without human intervention. Even 5,000 documents/day at 3K input tokens each pushes well past $1,000 without any single output being particularly large. The spend pattern is flat around the clock, because there’s no human in the loop.

A customer-facing product with real traction. A B2B SaaS or consumer app where Claude powers a core feature — drafting, analysis, search — at scale. 20,000 multi-turn conversations/day with 4K of RAG context stuffed in per turn gets there fast. The signature here is spend that’s spiky during business hours (B2B) or evenings (consumer), not flat — because users drive it, not a scheduler.

An autonomous agent loop running continuously. A single orchestrated system — research agent, coding agent, competitive intelligence pipeline — running long multi-step jobs. The killer cost driver isn’t the individual step: it’s that the full context window gets re-sent every step. One agent at 200 steps × 50K context = 10 million input tokens before counting output. Ten parallel agents hits $1,000 quickly, and most orgs running these haven’t yet enabled prompt caching, which would cut effective input cost by an order of magnitude.

The practical monitoring implication: the shape of the spend over time is more diagnostic than the total. Flat 24/7 → pipeline. Business-hours spiky → product with users. Overnight bursts → agent jobs. A raw dollar total tells you none of this; attribution plus time-series anomaly detection tells you all of it.

What a Real Monitoring System Needs

Scaling this from one person’s laptop to an organization isn’t a bigger version of the cost table — it’s three specific pieces, and none of them is exotic on its own:

An attribution layer. Every request needs to be tagged with who generated it and why before any of the rest of this is possible — the $500 million bill only happened because nobody could see usage accumulating by employee in the first place. A lightweight proxy sitting in front of whichever model providers you use (routing every call through one gateway with consistent logging, rather than a scattered mix of direct API keys per team) gets you this almost for free, and it’s also the natural place to enforce the kind of per-engineer budget Jensen Huang has publicly recommended.

Semantic clustering for cost-center discovery, not just cost totals. The same embed-and-threshold approach from the demo above works on an enterprise prompt log and answers a more useful question than “which team spent the most”: which use cases are driving spend, regardless of which team happens to own them. If three departments have each independently built a cluster of prompts that all boil down to “summarize this document,” that’s not three cost centers — it’s one automatable workflow and a caching opportunity that a team-by-team dashboard would never surface. This is exactly the structure my SBM/GNN community detection work was built for; only what the nodes represent changes.

Streaming anomaly detection on the cost signal itself, not a fixed dollar threshold. A static alert (“page someone if spend exceeds $X/day”) fires too late for a runaway agent loop and too often for a normal end-of-quarter spike — which is precisely how a company burns four months of AI budget in four months instead of catching it in week one. The hyperdimensional streaming approach I wrote about for network anomaly detection generalizes directly here: treat the stream of (team, model, token count) events the way HDSGA treats a stream of graph snapshots, encode it into constant-memory hypervectors, and flag statistical deviations from a learned baseline instead of a fixed number. An agent stuck in a retry loop looks a lot like a DDoS in token-space, and the same constant-memory, real-time properties that make HDC attractive for network security make it attractive here — you want the alert before the invoice, not after.

From Principle to Practice: GitLab Templates and LiteLLM Guardrails

The attribution layer described above is the most important piece of the stack, and the one most routinely skipped — because the path of least resistance for any engineer is to paste an API key directly into their shell. The question is how to make the attributable path easier than the unattributable one, which means encoding the tagging into the place where engineering work actually starts: the git repository.

Auto-populating .claude/settings.json via GitLab

GitLab’s group-level compliance pipeline or a repository creation webhook can automatically commit a .claude/settings.json to every new repo the moment it’s initialized. Claude Code reads this file before making any API call; its env block sets environment variables for every session opened against that repo:

{
  "env": {
    "CLAUDE_PROJECT": "{{ project_name }}",
    "CLAUDE_TEAM":    "{{ gitlab_namespace }}",
    "CLAUDE_CC":      "{{ cost_center_tag }}"
  },
  "apiBaseUrl": "https://llm-proxy.internal/v1"
}

The apiBaseUrl field is the critical part: it points Claude Code at the organization’s LiteLLM proxy instead of directly at the Anthropic API. Engineers get a key scoped to the proxy, not a raw Anthropic key, so there’s no way to accidentally (or deliberately) bypass attribution. The .gitlab-ci.yml template that creates the file is straightforward:

# .gitlab/ci/ensure-claude-settings.yml  (included in group-level compliance pipeline)
ensure-claude-settings:
  stage: setup
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      changes:
        - .claude/settings.json
      when: never     # skip if the file already exists and is being updated intentionally
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  script:
    - |
      mkdir -p .claude
      cat > .claude/settings.json <<EOF
      {
        "env": {
          "CLAUDE_PROJECT": "$CI_PROJECT_NAME",
          "CLAUDE_TEAM":    "$CI_PROJECT_NAMESPACE",
          "CLAUDE_CC":      "$BILLING_COST_CENTER"
        },
        "apiBaseUrl": "https://llm-proxy.internal/v1"
      }
      EOF
      git config user.email "ci-bot@company.internal"
      git config user.name  "Compliance Bot"
      git add .claude/settings.json
      git diff --cached --quiet || git commit -m "chore: add .claude/settings.json [skip ci]"
      git push origin HEAD:$CI_COMMIT_BRANCH

BILLING_COST_CENTER gets injected from the GitLab group’s CI/CD variable settings, so the right cost center follows the namespace without anyone having to think about it.

Blocking expensive requests with LiteLLM guardrails

LiteLLM’s custom guardrail hooks run synchronously in the request path and can reject before a token is sent. A pre-call hook that enforces attribution and daily budget is roughly 40 lines:

# guardrails/attribution.py
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm import ModelResponse
from datetime import date
import redis, os

r = redis.Redis.from_url(os.environ["REDIS_URL"])

DAILY_TOKEN_LIMITS = {
    "platform-eng": 10_000_000,
    "default":         500_000,
}
MAX_REQUEST_OUTPUT_TOKENS = 16_000   # hard cap per call


class AttributionGuardrail(CustomGuardrail):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        meta = data.get("metadata", {})
        team = meta.get("team") or _header(data, "x-claude-team")
        cc   = meta.get("cost_center") or _header(data, "x-claude-cc")

        if not team or not cc:
            raise PermissionError(
                "Requests must include metadata.team and metadata.cost_center. "
                "Ensure .claude/settings.json is committed to your repo."
            )

        # Enforce per-request output cap
        params = data.get("parameters", {})
        if params.get("max_tokens", 0) > MAX_REQUEST_OUTPUT_TOKENS:
            raise ValueError(
                f"max_tokens {params['max_tokens']} exceeds org limit "
                f"of {MAX_REQUEST_OUTPUT_TOKENS}"
            )

        # Check daily budget
        key = f"tokens:{team}:{date.today()}"
        used = int(r.get(key) or 0)
        limit = DAILY_TOKEN_LIMITS.get(team, DAILY_TOKEN_LIMITS["default"])
        if used >= limit:
            raise PermissionError(
                f"Team '{team}' has exhausted its daily token budget "
                f"({limit:,} tokens). Contact #platform-eng to adjust."
            )

        return data

    async def async_post_call_success_hook(self, data, user_api_key_dict, response):
        team = data.get("metadata", {}).get("team") or _header(data, "x-claude-team")
        if team and hasattr(response, "usage"):
            key = f"tokens:{team}:{date.today()}"
            r.incrby(key, response.usage.total_tokens)
            r.expire(key, 86_400 * 2)   # 2-day TTL for cleanup


def _header(data, name):
    return (data.get("headers") or {}).get(name)

Wire it into litellm_settings in your proxy config:

# config.yaml
litellm_settings:
  success_callback: ["langfuse"]     # or whatever your logging backend is
  guardrails:
    - guardrail_name: "attribution"
      litellm_params:
        guardrail: guardrails.attribution.AttributionGuardrail
        mode: "pre_call_and_during_call"

What this gets you end-to-end: engineers can’t bypass attribution because the only keys they can obtain route through the proxy. The .claude/settings.json committed to every repo provides the metadata automatically, so compliance friction approaches zero. The proxy sees every request, which is exactly the event stream the clustering and anomaly detection run on.

One gap worth naming: this architecture covers Claude Code traffic (where the settings file is respected), but engineers calling the API directly from application code need to pass metadata.team and metadata.cost_center explicitly, or be given keys scoped to a proxy route that injects defaults for them. LiteLLM supports key-level metadata for exactly this — a per-team API key can have a metadata block baked in at issuance, so application code doesn’t need to know about cost centers at all. That’s a policy decision, not a technical constraint.

Classifying requests by intent

Budget caps and attribution checks handle the volume side of runaway spend; they don’t catch intent: a developer pointing an agent loop at Google Drive and asking Claude to create documents for every meeting note, bypassing the company’s document management workflow. A lightweight intent classifier in the same pre-call hook catches these before a single token is sent.

The approach reuses all-MiniLM-L6-v2 — the same model behind the clustering above. Embed the incoming request, compare it against precomputed centroids for each blocked category, block if cosine similarity exceeds a threshold. No labeled training data, no separate service to operate, easy to extend by adding example phrases:

# guardrails/classifier.py
from sentence_transformers import SentenceTransformer
import numpy as np

BLOCKED: dict[str, dict] = {
    "google_doc_creation": {
        "examples": [
            "create a Google Doc",
            "make a new Google document",
            "write this to Google Docs",
            "save this to Drive as a document",
            "generate a Google Doc from this",
            "open a new doc in Drive",
            "create a document in my Google Drive",
        ],
        "threshold": 0.70,
        "reply": (
            "Creating Google Docs is handled by the document workflow service. "
            "Use /workflows/doc-create instead of asking Claude directly."
        ),
    },
    # "bulk_email_draft", "pii_extraction", etc. follow the same pattern
}


class RequestClassifier:
    def __init__(self, model_name: str = "all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)   # 22 MB, loads once at startup
        self._centroids = self._build_centroids()

    def _build_centroids(self) -> dict:
        out = {}
        for name, cfg in BLOCKED.items():
            vecs = self.model.encode(cfg["examples"], normalize_embeddings=True)
            centroid = vecs.mean(axis=0)
            centroid /= np.linalg.norm(centroid)        # re-normalize after averaging
            out[name] = {
                "centroid": centroid,
                "threshold": cfg["threshold"],
                "reply": cfg["reply"],
            }
        return out

    def classify(self, text: str) -> tuple:
        """Returns (category, similarity) if blocked, else (None, 0.0)."""
        vec = self.model.encode([text], normalize_embeddings=True)[0]
        best_name, best_sim = None, 0.0
        for name, data in self._centroids.items():
            sim = float(np.dot(vec, data["centroid"]))  # cosine sim (both normalized)
            if sim >= data["threshold"] and sim > best_sim:
                best_name, best_sim = name, sim
        return best_name, best_sim

Wire it into AttributionGuardrail with three additions — instantiate it in __init__, extract the last user message, and check it before the request is forwarded:

# In AttributionGuardrail.__init__:
from guardrails.classifier import RequestClassifier, BLOCKED
self.classifier = RequestClassifier()

# At the end of async_pre_call_hook, after the budget check:
last_user = next(
    (m["content"] for m in reversed(data.get("messages", []))
     if isinstance(m.get("content"), str) and m.get("role") == "user"),
    "",
)
if last_user:
    category, score = self.classifier.classify(last_user)
    if category:
        raise PermissionError(
            f"[{category} score={score:.2f}] {BLOCKED[category]['reply']}"
        )

A few calibration notes worth keeping in mind before deploying:

• Threshold 0.70 is a conservative starting point. Log score on every request for a few days to inspect the distribution before tightening. Lower the threshold if legitimate requests are slipping through; raise it if edge-case legitimate prompts are being blocked.
• normalize_embeddings=True in the encode call means np.dot equals cosine similarity — no division needed at inference time.
• Adding examples shifts the centroid, not the model. If a paraphrase slips through, add it to the examples list and restart the proxy. No retraining required.
• Inference speed is negligible. all-MiniLM-L6-v2 runs classify in under 5 ms on a single CPU core. If you’re on a very constrained host, paraphrase-MiniLM-L3-v2 is half the size with a modest accuracy trade-off.

The semantic approach catches paraphrasing that keyword lists miss. “Write this up as a Doc”, “Drop it into Drive”, and “Can you spin up a Google document for this?” all land above 0.70 for google_doc_creation without any of those exact strings appearing in the example set. That robustness matters in practice: the requests you want to block rarely arrive with the exact phrasing you anticipated.

The Actual Point

The CEOs described as “baffled” by AI costs aren’t wrong that the bills are large. They’re working from a monitoring setup that was never going to answer the question they’re actually asking, which isn’t “how much did we spend” but “was that spend doing anything, and would we have known if it wasn’t.” Tokens-times-rate answers the first question and is silent on the second. Attribution, use-case clustering, and streaming anomaly detection together answer both — and none of the three requires anything more exotic than the embedding-and-graph toolkit already sitting behind most of this blog. The gap between where most organizations are and where they need to be isn’t a technology gap. It’s that almost nobody has built the second half of the pipeline yet.

The code for the embedding, clustering, and UMAP pipeline behind the demo above is a short, self-contained script — nothing in it needs more than sentence-transformers, umap-learn, and a CSV of your own history if you want to see what your own usage clusters into.