The RAG Pattern That Stops LLMs From Inventing Fields That Don't Exist | SuperML.dev — AI/ML Architecture for Enterprise

NEW SuperML.dev repo is now available! » AgenticAI

Login

📬 Subscribe<br>× 📩

SuperML.dev Newsletter

Get weekly ML & AI tutorials, tools, and project updates — straight to your inbox. No spam, unsubscribe anytime.

Subscribe Now →

AI & Machine Learning<br>The RAG Pattern That Stops LLMs From Inventing Fields That Don't Exist<br>Standard RAG retrieves documents and lets the LLM generate field names freely — which means it will invent fields your schema never had. AK-RAG (Attribute Knowledge RAG) indexes your governed attribute catalog instead, and the LLM can only emit attribute_ids that actually exist. Here's the architecture and why it matters for regulated-industry AI.<br>Jul 6, 2026<br>crazyaiml

Share this article<br>Comments Shorts

Share:

Table of Contents

Every team building RAG for a regulated enterprise hits the same failure mode eventually. The system retrieves the right context — clinical guidelines, product documentation, data dictionaries — and the LLM produces a confident, fluent response that references fields, thresholds, and identifiers that don’t exist in the actual data model. In a consumer app this is a nuisance. In a bank’s credit underwriting pipeline, a healthcare payer’s cohort identification system, or a compliance team’s AML query layer, it is a governance failure. The LLM invented a field. That field never went through data governance. The output is meaningless at best and misleading at worst.

Standard RAG was not designed to prevent this. It retrieves document chunks and hands them to the LLM as context. The LLM then generates free-form text, including any field names, identifiers, and thresholds it infers from that context. There is no guarantee that what the LLM generates corresponds to anything that actually exists in your data catalog. Prompt-level mitigations — instructions to only use fields from the provided context, few-shot examples, output validation — help at the margins but don’t solve the structural problem. The problem is that the retrieval unit is wrong.

Attribute Knowledge RAG (AK-RAG) is a reference architecture that fixes this at the retrieval unit level. Instead of embedding documents, it indexes your governed attribute catalog as individual knowledge objects — one embedding per attribute, not one embedding per document page. The LLM’s job is phrase extraction and clarification dialog, not field name generation. The only identifiers that can appear in the final output are attribute_id values that exist in the indexed catalog. The LLM cannot invent fields because field selection happens through a governed retrieval and classification pipeline, not through free generation.

Architecture

AK-RAG has two distinct pipelines: an ingestion pipeline that builds the attribute index, and a query pipeline that translates natural language into a governed DSL. Both are designed around a core principle: the attribute catalog is the knowledge layer, not the document corpus .

Ingestion Pipeline

The ingestion pipeline takes your enterprise attribute metadata — typically maintained in Excel, CSV, or exposed via an API — and transforms it into a searchable, versioned index:

Excel / CSV / API<br>Contract Validation & Normalization<br>- Split synonyms<br>- Parse ranges and thresholds<br>- Coerce types, standardize units/dates<br>NDJSON (One Attribute = One Document)<br>┌──────────────────────────────────────┐<br>│ Embedding Provider │<br>│ sentence-transformers (dev) │<br>│ openai · gemini · bedrock (prod) │<br>└──────────────────────────────────────┘<br>┌──────────────────────────────────────┐<br>│ Search / Index Backend │<br>│ local BM25 + token vector + RRF │ ← offline, no cluster (dev)<br>│ opensearch BM25 + HNSW kNN + RRF │ ← production<br>│ faiss / chroma │ ← dense vector alternatives<br>└──────────────────────────────────────┘<br>Each attribute becomes a single NDJSON document. Here’s what that looks like for a clinical attribute in the healthcare domain:

"attribute_id": "clinical.hba1c",<br>"type": "numeric",<br>"business_name": "HbA1c Level",<br>"technical_field": "hba1c_pct",<br>"domain": "clinical",<br>"definition": "Most recent HbA1c percentage on record. Indicates glycemic control...",<br>"synonyms": ["glycated hemoglobin", "blood sugar control", "a1c", "hemoglobin a1c"],<br>"operators": [">", ">=", ", ", "="],<br>"unit": "%",<br>"example_values": [7.0, 8.0, 9.0, 10.0],<br>"governance": {<br>"phi": true,<br>"hipaa_category": "clinical",<br>"de_identification_required": false,<br>"minimum_cell_size": 11,<br>"allowed_channels": ["care_management", "quality_reporting", "analytics"],<br>"consent_required": false<br>},<br>"embedding_text": "numeric | HbA1c Level | hba1c_pct | glycated hemoglobin | blood sugar control | a1c | glycemic control | diabetes | percentage | clinical"<br>The embedding_text field is hand-composed to maximize retrieval signal: it concatenates the type, business name, technical field name, synonyms, definition keywords, and domain tags. This...