Evidence First, AI Second: Deterministic Pydantic Validation Guards for Healthcare LLMs
Eliminating medical hallucinations by forcing LLM reasoning through structured JSON schemas and relational database ground-truth.
1. The Probabilistic Fallacy in Healthcare
When developers connect a conversational LLM directly to healthcare or pharmaceutical queries, they commit a fundamental category error: treating a probabilistic next-token generator as an authoritative reference manual.
Even state-of-the-art models exhibit a 2-5% hallucination rate on pharmacology queries—inventing non-existent brand names, conflating active ingredients with commercial additives, or fabricating contraindication warnings. In a clinical or consumer medicine environment, a 2% failure rate is not an acceptable tolerance; it is a critical safety liability.
The core engineering philosophy of MediGuard AI is simple: 'Evidence first. AI second.' An LLM must never be allowed to invent a medication name, dosage schedule, or clinical restriction. It must only translate verified facts from an authoritative database into clear, accessible language.
2. Relational Ground Truth Architecture
The system is divided into two distinct components: the Ground Truth Layer and the Translation Layer.
The Ground Truth Layer is a normalized PostgreSQL database containing verified pharmaceutical entities imported directly from national regulatory circulars. It models ActiveIngredients, BrandNames, Manufacturers, ScheduledDosages, and ContraindicationMatrices.
When a user searches for a medication (e.g., 'Napa Extra' or 'Paracetamol + Caffeine'), the query is resolved deterministically via PostgreSQL Trigram fuzzy search. Only after a confirmed database record is located does the system proceed.
3. The Pydantic Validation Gateway
The LLM is invoked through structured JSON output mode, constrained by a strict Pydantic model. Crucially, the Pydantic schema includes custom field validators that verify every generated attribute against the retrieved database entity.
If the LLM introduces a drug name that does not match the exact registered UUID, or includes dosage units not present in the official formulary, Pydantic immediately throws a `ValidationError` and halts rendering.
from pydantic import BaseModel, field_validator
from uuid import UUID
from typing import List
from mediguard.models import MedicineRegistry
class VerifiedMedicineGuidance(BaseModel):
generic_name: str
verified_brand_id: UUID
indications: List[str]
contraindications: List[str]
is_prescription_only: bool
patient_friendly_summary: str
@field_validator("verified_brand_id")
@classmethod
def verify_database_entity(cls, v: UUID) -> UUID:
# Assert that the UUID corresponds to a verified pharmaceutical entity
if not MedicineRegistry.objects.filter(id=v, is_verified=True).exists():
raise ValueError(f"Entity UUID {v} does not exist in the verified registry.")
return v
@field_validator("is_prescription_only")
@classmethod
def match_schedule_status(cls, v: bool, info) -> bool:
# Cross-check prescription schedule against relational record
brand_id = info.data.get("verified_brand_id")
if brand_id:
record = MedicineRegistry.objects.get(id=brand_id)
if record.requires_prescription != v:
raise ValueError("Model hallucinated the legal prescription schedule.")
return v4. Fail-Safe Datasheet Fallback
If the LLM fails schema parsing, times out, or attempts to hallucinate an unverified claim, MediGuard AI triggers an immediate deterministic fallback: it bypasses the AI completely and presents the raw, verified pharmaceutical datasheet directly from PostgreSQL.
The user receives 100% accurate, regulator-approved information, with a transparent UI indicator explaining that raw clinical data is displayed. Trust is preserved, and safety is never compromised.
Core Engineering Takeaways
- •Never allow an LLM to generate medical or financial assertions without schema-enforced ground truth.
- •Separate the Ground Truth Layer (PostgreSQL) from the Translation Layer (LLM).
- •Use Pydantic field validators to cross-reference AI output attributes with database records.
- •Implement automatic fallbacks to raw clinical datasheets when validation fails.
View the complete architectural case study and source implementation.
Continue reading technical architecture insights.