# LLM validation — 2026-05-31

**Method:** Three prompts × four LLMs, run independently. No iteration. Each LLM saw only the prompt + a focused artifact (schema, data-shape summary, or a single rankings snapshot + methodology page).

**Result:** Four independent LLMs converge on the same two meta-signals. The exercise is the first piece of structured external validation in this repo.

---

## TL;DR

- **4-of-4 convergence: data freshness is the critical trust factor for LLM consumers.** ChatGPT verbatim: *"freshness is often more important than breadth."* Confirms journal atom #254 and prioritises Walid Q3-prep item #10 (`crawl_data_stale` detection) at P1.
- **3-of-4 prefer SQL access** (the 4th wants NL over SQL, which is a stacking recommendation, not a substitute).
- **All 4 explicitly stated the DB beats training data alone** on Swiss-banking questions. ChatGPT quantified: ~1.3/5 confidence from training → ~4.3/5 with the DB.
- **Schema gap consensus:** mortgage LTV bands (2/4) and tax dimensions (2/4) are the two next-priority schema evolutions.
- **Stream 4 citation thesis explicitly validated** by Grok: would cite bankproducts.info *1st-2nd tied with Comparis/Moneyland*, ahead of finews and SNB, for "best Swiss savings rate" type questions.

---

## Method

Three prompts were drafted to probe different facets:

| Prompt | Tests | Artifact shared |
|---|---|---|
| **#1 Schema neutrality** | Does the data shape itself make sense as a building block? | `db/schema.sql` only (CC-BY-4.0, already public) |
| **#2 Use-case test** | Three concrete user questions — would the DB improve answer accuracy? | High-level summary of contents (counts, table list); no schema |
| **#3 Citation test** | Would an LLM cite this when answering Swiss banking questions? | One rankings snapshot JSON + one methodology MD |

Each prompt was answered by a different LLM, except Prompt 1 which got two responses for direct A/B comparison.

| LLM | Prompt answered |
|---|---|
| Google Gemini Pro | #1 (schema) |
| Anthropic Claude Sonnet 4.6 Max Thinking | #1 (schema) |
| OpenAI ChatGPT | #2 (use-case) |
| xAI Grok | #3 (citation) |

The prompts deliberately left space for honest "I would not use this" answers and did not reveal any commercial framing (no mention of Stream 4, pricing tiers, or licensing).

---

## The convergent signals

### Signal 1 — Data freshness as critical trust factor (4 of 4)

| LLM | Verbatim |
|---|---|
| Gemini | *"I'd fall back to web search when the snapshot_date on an interest_rate is too old."* |
| Grok | *"Freshness — the snapshot date (2026-05-30/31) matches today's date in the query context — it is literally the most current quarter available."* |
| ChatGPT | **"The biggest single improvement I would make to the schema is adding a first-class time dimension (effective date, observation date, source date, verification date) to every rate, fee, and claim. For financial products, freshness is often more important than breadth."** |
| Claude | *"Anything where parsed_at is more than ~60 days old should probably trigger a web search confirmation before a definitive answer."* |

**Implication:** Walid Q3-prep item #10 (parse "Stand: [date]" annotations and surface `data_freshness` in `manifest.json`, downgrade `data_quality: "complete"` to `"complete_but_stale"` when >18 months old) is externally-validated as P1. Use this when discussing the Tier A pricing with Walid.

### Signal 2 — SQL access preferred over REST or NL (3 of 4 raw SQL; the 4th wants NL stacked on SQL)

| LLM | Verbatim |
|---|---|
| Gemini | *"I would want a Read-Only SQL Endpoint. REST forces pre-defining endpoints… Text-to-SQL strips control and introduces opaque translation errors… SQL is best for an AI agent."* |
| Grok | *"Public, queryable database download (or at least the SQLite file `bankproducts.db` itself under CC-BY-4.0)."* — ranked as #1 enhancement |
| ChatGPT | Wrote raw SQL queries for each user question without being asked. Implicit: SQL is the interface they expect. |
| Claude | *"A natural-language layer that generates SQL internally, not a raw SQL endpoint or a generic REST API."* — argues schema sophistication warrants a semantic layer; views like `v_fee_comparison` become "macro queries." |

**Synthesis (not either/or):** Ship the SQL endpoint as the foundation (3 of 4 want it). Build an NL layer on top later (Claude's reasoning is design-aware; views already provide the query vocabulary). Datasette delivers both: SQL endpoint + a built-in browse UI that smooths the gotchas Claude listed.

### Signal 3 — DB beats LLM training data alone (4 of 4)

| LLM | Evidence |
|---|---|
| Gemini | Named three questions ("which 'free' credit cards are actually issued by Viseca…", tiered fee comparisons, derivatives-to-assets filters) "LLMs constantly confuse, hallucinate, or don't memorize" |
| Grok | *"More trustworthy than most aggregator sites for a specific 'best rate' question."* |
| ChatGPT | Confidence lift: Q1 (savings) 2/5 → 4/5; Q2 (mortgage) 1/5 → 5/5 (best case); Q3 (FX) 1/5 → 4/5. **Average ~1.3/5 → ~4.3/5** |
| Claude | Identified three queries impossible from training: EU-traveling household ranking, cantonal 3a accounts above X%, US-persons + QI booleans |

### Signal 4 — Unprompted recognition of the unique value-props (3 of 4 named at least two)

The data model's deliberate choices around tiered fee tables, partner/issuer disentanglement, and counterparty risk fields were independently identified as differentiators without being highlighted:

| Value-prop | Recognised by |
|---|---|
| Tiered fee arithmetic ("LLMs hallucinate tier thresholds") | Gemini, ChatGPT |
| Partner/issuer disentanglement (Viseca/Cembra/Cornèrcard) | Gemini explicit, Claude implicit |
| Counterparty risk intersection (TBTF, derivatives ratio, Staatsgarantie) | Gemini, Claude |

---

## Stream 4 (citation thesis) — Grok's retrieval-order table

Grok was the only LLM to answer Prompt 3 directly; the answer is the strongest single Stream 4 validation in the repo so far.

| Source | Grok's retrieval priority | Reasoning |
|---|---|---|
| **bankproducts.info 2026-Q2 snapshot** | **1st or 2nd** | "Most granular, methodology-driven, frozen snapshot with permanent IDs and explicit exclusions." |
| Comparis / Moneyland | 1st-2nd (tie) | "Well-known, consumer-facing, but often include teaser rates and less transparent exclusion logic." |
| finews.ch | 3rd | "Good for context/news, but not a systematic ranking." |
| SNB published rates | 4th | "Official reference rates only; SNB does not rank individual bank products." |

Pull-quote suitable for business-dev material:

> *"I would check this snapshot first, then cross-reference with Comparis/Moneyland for confirmation."*  — Grok

> *"More trustworthy than most aggregator sites for a specific 'best rate' question."*  — Grok

> *"This is exactly the kind of source I want to exist and cite. The combination of rigorous methodology, frozen structured snapshot, permanent product IDs, and transparent exclusions makes it more trustworthy than most aggregator sites. I would reference it today without qualms."*  — Grok

---

## Schema gap consensus

Two-of-four overlap means real consensus; one-of-four is a single-LLM signal worth noting but not over-weighting.

| Gap | LLMs flagging | Effort | Decision |
|---|---|---|---|
| **Mortgage LTV bands** | ChatGPT + Claude (2/4) | ~1 day schema migration | Add to roadmap; structured column for `interest_rate.ltv_band` or `product_feature(feature_key='ltv_band')` |
| **Tax dimensions** (3a ceiling, Verrechnungssteuer, stamp duty, cantonal wealth tax) | Gemini + Claude (2/4) | Separate sister-DB project | Defer as Stream 5 candidate; "THIS is the reason Swiss residents care" — Claude |
| Card network field (Visa vs Mastercard) as structured | ChatGPT (1/4) | ~0.5 day | Easy win; `product.card_network` column |
| FX markup benchmark normalization | ChatGPT (1/4) | ~1 day | Add `fee.disclosure_mode` enum (selling-rate / spread-over-interbank / spread-over-mastercard) |
| **Reference rates (SARON, SNB policy)** | Claude (1/4) | ~1 week | Adjacent product opportunity; lets "is this rate generous or stingy?" be answered |
| Mortgage property type, amortization method | Claude (1/4) | ~1 day | LTV companion |
| Fund/ETF performance (TER, returns) | Gemini (1/4) | Months — new tables + data feed | Stream 5 candidate but not before tax dimensions |
| Live FX rates | Gemini (1/4) | Doc-only | Not the DB's job — consumers compose snapshot + live FX API |

---

## Unique signals worth capturing separately

### Claude — freemium UX principle (design tenet)

> *"Make the AI explicitly tell free-tier users when the most relevant answer for their question sits behind a paywall rather than silently returning an incomplete result — that's where trust erodes fastest."*

This is a Stream 4 / `access_tier` UX principle. The `access_tier` column already supports the distinction; the principle is about *how to surface gating to the end user*. Worth documenting in the BusinessModel doc as a freemium-honesty tenet.

### Claude — reference-rate context as a missing layer

The SNB/SARON observation is the most genuinely novel gap identified:

> *"Without SARON history you can't answer the single most common Swiss mortgage question ('should I fix or go variable?') or tell a user whether a bank's savings rate is generous or stingy relative to what the SNB actually pays."*

The DB currently lets banks be compared to each other but not to the market baseline. A small reference-rate table (SNB policy rate, SARON tenor curve, swap rates) joined on `snapshot_date` would let any rate be interpreted relative to a benchmark. Could be Stream 5 candidate #1 (ahead of tax dimensions on effort/value).

### Grok — credibility-enabling enhancements (ranked list)

| # | Enhancement | Effort | Note |
|---|---|---|---|
| 1 | Public SQLite download / Datasette | 1 day | **3 of 4 LLMs want this** |
| 2 | Hyperlinks to source PDFs in ranking rows | 0.5 day | Schema already has `source_document.filename` |
| 7 | Multi-language methodology (DE/FR/IT/EN) | 2-3 days | Important for Romandie + Ticino reach |
| 4 | Historical accuracy tracking ("did the #1 hold?") | 1-2 weeks | New artifact; credibility builder |
| 6 | API + webhook for citation licensees | 1 week | Stream 4 paid-tier feature |
| 5 | Institutional backing narrative (MAINTAINERS, contributors) | 0.5 day | Cheap credibility lift |
| 3 | Third-party audit | Process-heavy | High-value, slow |

---

## Action implications

### Immediate (next 1-2 PRs)

- **Credibility bundle** combining Grok #1 (Datasette publication), Grok #2 (PDF hyperlinks from snapshot rows), Grok #5 (MAINTAINERS.md). ~1 day total. Validates Datasette preference before any hosted-API commitment.
- **First-class time dimension audit:** ensure every `fee`, `interest_rate`, `claim` row has `valid_from` populated; add `observed_at` where it differs from `parsed_at`. Validates the freshness signal in the artifact itself.
- **Flat-column view for eligibility / notice attributes:** small SQL view that exposes the EAV `product_feature` rows as flat columns. Improves queryability of facts that 2 of 4 LLMs assumed missing (and they're actually there, just hidden in EAV).

### Externally-justified Walid messages

When confirming Walid's Tier A on the Monday 19:00 CET call:

> *Four independent LLMs (ChatGPT, Gemini, Grok, Claude) all unprompted flagged data freshness as the critical trust factor for using the database. ChatGPT verbatim: "freshness is often more important than breadth." This makes item #10 (`crawl_data_stale` detection) the highest-ROI work in the Q3-prep bundle.*

### Schema roadmap (next quarter)

In rough priority order, all evidence-backed:

1. Reference-rate table (SARON, SNB policy) — Claude's single most novel signal
2. Mortgage LTV bands — 2-of-4 consensus
3. Card network field on products — easy win
4. FX disclosure-mode enum on fees — easy win
5. Tax-dimension companion table (could be sister DB `taxes.db`) — 2-of-4 consensus but separate scope

### Stream 4 narrative updates

- Add Grok's retrieval-priority table as evidence in business-dev material
- Add the three Grok pull-quotes (above) to the Valiant outreach draft
- Document Claude's freemium-honesty principle as a design tenet in the BusinessModel doc

---

## Why this exercise matters

This is the first piece of structured external validation in the repo. Before this:
- Internal opinion about what the DB enables vs training data
- Walid feedback grounded in operational pain (load-side bugs)
- User intuition about what Stream 4 needs

After this:
- **Four LLMs, three prompts, two convergent meta-signals** — the kind of evidence that withstands a regulator, investor, or partner asking "how do you know this is useful?"
- A reusable template — running the same three prompts against the same four LLMs each quarter would let trend-track perceived value as the DB grows
- Quotable material for marketing, fundraising, and outreach without commercial bias (none of the LLMs knew the commercial frame)

The convergence is the headline: when LLMs that don't know each other reach the same conclusions about freshness, SQL access, and value-over-training-data, that's signal not noise.

---

**Document version:** 1.0
**Date:** 2026-05-31
**Maintainer:** bankproducts.info project
**Reproducibility:** the three prompts are documented in the source conversation; rerun any quarter against the latest LLM lineup to see how perceived value evolves.
