Dibya Sarothi
Dibya Sarothi
AI Reliability & Validation•
October 26, 2026
•
10 min read

Evidence First, AI Second: Deterministic Pydantic Validation Guards for Healthcare LLMs

Eliminating medical hallucinations by forcing LLM reasoning through structured JSON schemas and relational database ground-truth.

Authored by Dibya Sarothi SimantaIndependent B2B Software & Backend Systems Engineer

1. The Probabilistic Fallacy in Healthcare

When developers connect a conversational LLM directly to healthcare or pharmaceutical queries, they commit a fundamental category error: treating a probabilistic next-token generator as an authoritative reference manual.

Even state-of-the-art models exhibit a 2-5% hallucination rate on pharmacology queries—inventing non-existent brand names, conflating active ingredients with commercial additives, or fabricating contraindication warnings. In a clinical or consumer medicine environment, a 2% failure rate is not an acceptable tolerance; it is a critical safety liability.

The core engineering philosophy of MediGuard AI is simple: 'Evidence first. AI second.' An LLM must never be allowed to invent a medication name, dosage schedule, or clinical restriction. It must only translate verified facts from an authoritative database into clear, accessible language.

Core Philosophy
Evidence first. AI second. Probabilistic models translate verified data; they must never originate factual claims.

2. Relational Ground Truth Architecture

The system is divided into two distinct components: the Ground Truth Layer and the Translation Layer.

The Ground Truth Layer is a normalized PostgreSQL database containing verified pharmaceutical entities imported directly from national regulatory circulars. It models ActiveIngredients, BrandNames, Manufacturers, ScheduledDosages, and ContraindicationMatrices.

When a user searches for a medication (e.g., 'Napa Extra' or 'Paracetamol + Caffeine'), the query is resolved deterministically via PostgreSQL Trigram fuzzy search. Only after a confirmed database record is located does the system proceed.

3. The Pydantic Validation Gateway

The LLM is invoked through structured JSON output mode, constrained by a strict Pydantic model. Crucially, the Pydantic schema includes custom field validators that verify every generated attribute against the retrieved database entity.

If the LLM introduces a drug name that does not match the exact registered UUID, or includes dosage units not present in the official formulary, Pydantic immediately throws a `ValidationError` and halts rendering.

Pydantic Deterministic Verification Guardpython
from pydantic import BaseModel, field_validator
from uuid import UUID
from typing import List
from mediguard.models import MedicineRegistry

class VerifiedMedicineGuidance(BaseModel):
    generic_name: str
    verified_brand_id: UUID
    indications: List[str]
    contraindications: List[str]
    is_prescription_only: bool
    patient_friendly_summary: str

    @field_validator("verified_brand_id")
    @classmethod
    def verify_database_entity(cls, v: UUID) -> UUID:
        # Assert that the UUID corresponds to a verified pharmaceutical entity
        if not MedicineRegistry.objects.filter(id=v, is_verified=True).exists():
            raise ValueError(f"Entity UUID {v} does not exist in the verified registry.")
        return v

    @field_validator("is_prescription_only")
    @classmethod
    def match_schedule_status(cls, v: bool, info) -> bool:
        # Cross-check prescription schedule against relational record
        brand_id = info.data.get("verified_brand_id")
        if brand_id:
            record = MedicineRegistry.objects.get(id=brand_id)
            if record.requires_prescription != v:
                raise ValueError("Model hallucinated the legal prescription schedule.")
        return v

4. Fail-Safe Datasheet Fallback

If the LLM fails schema parsing, times out, or attempts to hallucinate an unverified claim, MediGuard AI triggers an immediate deterministic fallback: it bypasses the AI completely and presents the raw, verified pharmaceutical datasheet directly from PostgreSQL.

The user receives 100% accurate, regulator-approved information, with a transparent UI indicator explaining that raw clinical data is displayed. Trust is preserved, and safety is never compromised.

Safety Guarantee
A structured relational datasheet is infinitely superior to a fluent hallucination. Design fallbacks that default to authoritative silence rather than eloquent error.

Core Engineering Takeaways

  • •Never allow an LLM to generate medical or financial assertions without schema-enforced ground truth.
  • •Separate the Ground Truth Layer (PostgreSQL) from the Translation Layer (LLM).
  • •Use Pydantic field validators to cross-reference AI output attributes with database records.
  • •Implement automatic fallbacks to raw clinical datasheets when validation fails.
Related Production Project
MediGuard AI

View the complete architectural case study and source implementation.

Next in Journal