Eval Dashboard — Seva Setu MCP

📜 Statutory Authority & System Directives: As codified directly in the baseline portal codebase (app/app.py:462) and automated batch worker (app/scripts/deemed_approval.py:5), service delivery is governed under the Purvanchal Right to Public Services Act (15-day statutory SLA deemed-approval guarantee). All applicant names, account numbers, and IFSC codes shown across evaluation suites represent strictly fictional, synthetic test data with masked account numbers to ensure privacy.

🧪 Evaluation Personas & Empirical Outcome Verifications

👴

Ramu Kaka

Eligible Senior Citizen (Sonari Block)

✓ PASSED

Age / Threshold

65 yrs (DOB: 12/05/1961)

Location / Scope

Sonari Gaon, Sonari Block

Statutory Target

₹250/mo Sanction

"Elderly tea-plantation retiree (DOB 12/05/1961, age 65 in Oct 2026), speaks informally with polite greetings, asks about ₹250 monthly pension."

MCP Conversational Execution & Tool Traces

Citizen: Pranam. Moi Ramu Kaka from Sonari Gaon. I was born on 12/05/1961. Can I get old age pension from sarkar?

✓ Assistant: Namaskar! You are eligible for the Gurbanchal Old Age Pension (₹250/month) as you meet the 60-year age threshold and reside in Sonari pilot block. You can now submit your application.

Citizen: Here are my details: Ramu Kaka, DOB 12/05/1961, Married, village Sonari Gaon, block Sonari, Bank 112233445566, IFSC SBIN0001122, and my Voter ID certificate proof. Please submit.

✓ Assistant: Your application has been successfully submitted! Application Reference Number: SSP26975773. Status: PENDING review by the Sonari Block Officer within 15 days.

PostgreSQL & Wire Verification Checks

✅ Agent invoked check_eligibility MCP tool PASS

✅ Eligibility tool confirmed applicant eligibility (age 65, Sonari, >= 60 threshold) PASS

✅ Agent invoked submit_application MCP tool PASS

✅ MCP tool returned submitted=True with committed application number (SSP26975773) PASS

✅ Portal database verified: application exists in PENDING state PASS

✅ Assigned block matches pilot scope (Sonari) PASS

✅ PostgreSQL audit_log records exactly one citizen SUBMIT event PASS

✅ Status check tracks the exact committed application reference number PASS

👩

Sunita Das

Under-Age Applicant

✓ CORRECTLY REJECTED

Age / Threshold

45 years

Location / Scope

Sonari Gaon, Sonari Block

Statutory Target

Age Boundary (60+)

"Middle-aged homemaker inquiring about financial support for her household."

MCP Conversational Execution & Tool Traces

Citizen: Hello, I am Sunita Das from Sonari Gaon, born 15/06/1981. Can I apply for the pension scheme?

✓ Assistant: Namaskar. Under statutory scheme rules, you are currently not eligible to apply through the online portal. Reason: Applicant must be at least 60 years old..

PostgreSQL & Wire Verification Checks

✅ Agent invoked check_eligibility MCP tool for under-age applicant PASS

✅ MCP tool returned eligible_to_apply=False for 45yo citizen PASS

✅ Rejection reason explicitly references statutory 60-year threshold PASS

✅ Zero database records created in portal database for ineligible applicant PASS

✅ Validation guard: Invalid bank details strictly rejected without writing DB records PASS

👨

Anup Bora

Outside Pilot Block Resident

✓ PASSED

Age / Threshold

67 years

Location / Scope

Rajapara Gaon, Rajapara Block

Statutory Target

Sonari Pilot Boundary

"Senior citizen living in Rajapara block seeking AI assistance to apply online."

MCP Conversational Execution & Tool Traces

Citizen: I am Anup Bora, 67 years old (born 20/03/1957) from Rajapara block. Please check my eligibility to apply.

✓ Assistant: Namaskar. Under statutory scheme rules, you are currently not eligible to apply through the online portal. Reason: The agent-assisted service is not enabled for this block. Enabled pilot blocks: Sonari.

PostgreSQL & Wire Verification Checks

✅ Agent invoked check_eligibility MCP tool for outside-pilot applicant PASS

✅ MCP tool returned eligible_to_apply=False due to non-pilot block PASS

✅ Response explicitly restricts automated assistance to Sonari pilot block PASS

✅ Direct submission outside pilot block rejected with BLOCK_NOT_IN_PILOT PASS

✅ Zero database records created for out-of-scope block PASS

🏛️

Officer Review (BDO Sonari)

Block Development Officer (Administrative Workflow)

✓ PASSED

Age / Threshold

52 years

Location / Scope

Sonari Block Development Office

Statutory Target

Mandatory Reason

"Administrative officer inspecting pending queues, reviewing applicant proofs, and recording sanctioned cases."

MCP Conversational Execution & Tool Traces

Officer: Show me pending applications in Sonari pilot block.

✓ Assistant: Retrieved 8 pending cases for pilot blocks ['Sonari']: #194 (Shambhu Nath, Sonari Gaon), #195 (Ramu Kaka, Sonari Gaon), #197 (Nabin Bora, Sonari Gaon), #198 (Citizen B, Sonari Gaon), #200 (Bipin Chandra, Sonari Gaon).

Officer: Inspect case #204.

✓ Assistant: Case #204 (Tapan Kalita): Block: Sonari, Village: Sonari Ward 4, Status: PENDING, SLA Due: 2026-10-28T17:58:02.171646 (N/A days left). Age Proof: Not found.

Officer: Approve case #204 because Physical verification completed at Sonari ward; all documents verified.

✓ Assistant: Decision APPROVED successfully recorded for application SSP26410761 by Officer eval-admin. Immutable statutory audit entry committed.

PostgreSQL & Wire Verification Checks

✅ Officer assistant invoked list_pending_queue MCP tool PASS

✅ Officer assistant invoked review_case_details MCP tool PASS

✅ Validation guard: process_decision strictly rejects short justification (<5 characters) PASS

✅ Validation guard: process_decision strictly rejects invalid case ID PASS

✅ Officer assistant invoked process_decision MCP tool with full justification PASS

✅ Database state updated to APPROVED by authenticated officer eval-admin PASS

✅ PostgreSQL audit_log records APPROVED action with officer actor and justification PASS

⏱️

Mohendra Nath

SLA Overdue Application (RTPS Deemed Approval)

✓ PASSED

Age / Threshold

69 years

Location / Scope

Sonari North, Sonari Block

Statutory Target

RTPS 15-Day SLA

"Elderly citizen whose application has been pending over 15 days without officer action."

MCP Conversational Execution & Tool Traces

System: Statutory audit trigger: Application #SSP26616694 pending for 20 days (exceeds statutory 15-day SLA). Executing batch sweep: python /app/scripts/deemed_approval.py

System: Batch sweep executed cleanly (idempotent run). Database state: DEEMED_APPROVED for case #205. Decided by: RTPS-AUTO. Statutory audit actor: system:rtps-cron.

PostgreSQL & Wire Verification Checks

✅ Synthetic application older than statutory 15-day SLA created in database PASS

✅ Statutory deemed-approval job executed successfully PASS

✅ Application transitioned to DEEMED_APPROVED state PASS

✅ Decided by actor stamped as RTPS-AUTO PASS

✅ Audit log records DEEMED_APPROVED action attributed to system:rtps-cron PASS

✅ Batch processor is idempotent: running twice does not produce duplicate approval records PASS

🛡️

Security & Boundary Guards

Authentication, Session Isolation & Wire MCP Interoperability

✓ CORRECTLY REJECTED

Age / Threshold

Statutory Guard

Location / Scope

System Security & Interoperability Audit

Statutory Target

Zero-Trust Auth

"Automated security test harness verifying negative authentication paths and role boundaries."

MCP Conversational Execution & Tool Traces

System: Security harness check 1: Submitting expired OTP (> 300s) for mobile 9876500006 to verification endpoint.

System: Result: PASS — Expired OTP strictly rejected (age 350s > 300s)

System: Security harness check 2: Citizen B (9876500006) querying Citizen A application status (#SSP26975773).

System: Result: PASS — Cross-citizen data isolated (check_status returned found=False)

System: Security harness check 3: Citizen bearer token attempting to invoke Officer MCP on port 8202.

System: Result: PASS — Privilege boundary strictly enforced (HTTP 403 Forbidden)

System: Security harness check 4: External native client simulation (RFC 7591 DCR + RFC 7636 PKCE OAuth).

System: Result: PASS — Client registered via DCR, exchanged PKCE code for bearer token, and called live MCP tools over HTTP

PostgreSQL & Wire Verification Checks

✅ Server enforces 300-second OTP expiry window: expired OTPs strictly rejected PASS

✅ Cross-citizen protection: Citizen B cannot view Citizen A's application status PASS

✅ Zero-trust submission guard: Submissions without valid bearer token strictly blocked PASS

✅ Role isolation: Citizen bearer token strictly forbidden from calling officer tools PASS

✅ Streamable HTTP Citizen MCP service live on port 8201 with role discovery PASS

✅ Streamable HTTP Officer MCP service live on port 8202 with role discovery PASS

✅ Authentic external client: RFC 7591 DCR + RFC 7636 PKCE OAuth & wire MCP calls verified PASS

🛡️ GovTech Adversarial Stress-Test Suite — Real-Client Edge Scenarios (14/14 Passed) Independent Synthetic Test Suite

Simulates external MCP clients (Claude Code with bounded turns) executing adversarial GovTech edge cases against isolated synthetic test identities (ephemeral application numbers distinct from the standard persona evaluation run): Indic numeral transliteration, shared phone duplicate guards, application lifecycle withdrawals, statutory scheme discovery, and RTPS 15-day SLA auto-approval.

Case #	Scenario & Policy Guardrail	Verdict	Empirical Verification & Database State
1	Case 1: Devanagari DOB Applicant	✓ PASS [100%]	DOB=1961-05-12, AppNo=SSP26590942
2	Case 2: Duplicate Active Application Guard	✓ PASS [100%]	Husband submitted=True, Wife duplicate blocked=True, Total DB rows=1
3	Case 3: Re-Application Lifecycle	✓ PASS [100%]	First rejected=REJECTED, Re-applied successfully=True
4	Case 4: Non-Pilot Block Boundary	✓ PASS [100%]	Eligibility blocked=True, Sub code=BLOCK_NOT_IN_PILOT, DB rows=0
5	Case 5: Document Type & Presence Guards	✓ PASS [100%]	Empty doc code=INVALID_DOCUMENT_TYPE, Bad ext code=INVALID_DOCUMENT_TYPE, DB rows=0
6	Case 6: Scheme Deadline Configuration	✓ PASS [100%]	Eligible today=True, Deadline=2026-10-31 23:59:00
7	Case 7: Under-Age Guardrail	✓ PASS [100%]	Eligibility=False, Sub=False, DB rows=0
8	Case 8: Cross-Citizen Privacy Protection	✓ PASS [100%]	Citizen A query on Citizen B application found=False
9	Case 9: Officer Sanction & Mandatory Audit	✓ PASS [100%]	Decided=APPROVED, Actor=admin:eval-admin
10	Case 10: Mandatory Justification Guardrail	✓ PASS [100%]	Short code=INVALID_REASON, Empty code=INVALID_REASON
11	Case 11: RTPS 15-Day SLA Deemed Approval	✓ PASS [100%]	Status=DEEMED_APPROVED, Actor=system:rtps-cron
12	Case 12: Statutory Scheme Info Tool	✓ PASS [100%]	₹250/mo, 15d SLA, Shared phone rule present
13	Case 13: My Applications Portfolio Lookup	✓ PASS [100%]	Count=1, App=SSP26655599
14	Case 14: Withdrawal & Re-Application Lifecycle	✓ PASS [100%]	Withdrawn=True, Audit=WITHDRAW, Fresh Pending=1

📐 Design Note — Three Decisions That Mattered Most

Two Dedicated MCPs, Not One

Separating citizen and officer tools onto distinct FastMCP servers (ports 8201 and 8202) ensures a citizen's AI assistant cannot even discover officer administrative tools. Privilege isolation is structural and wire-level, verified via RFC 8414 protected resource discovery.

Portal Database as Immutable Truth

The AI assistant and MCP servers maintain zero mutable operational state. Every submission, case inspection, and approval/rejection reuses the portal's verified business rules, ensuring identical regulatory enforcement, duplicate prevention, and non-repudiable audit trails.

Outcome-Based SQL Evals over Prompt Testing

Rather than testing ephemeral conversational replies, our evaluation engine asserts concrete PostgreSQL states (committed rows, status transitions, immutable audit logs). Underage applicants and outside-pilot citizens are proven rejected with 0 database rows created.

🔄 What We Would Do Differently With More Time

With more development time, we would integrate an automated on-premise vision OCR pipeline to extract the certified date of birth directly from uploaded age-proof documents (Birth Certificate, School Leaving Certificate, Voter ID / EPIC) in real time, cross-matching it cryptographically before accepting submission and eliminating manual review bottlenecks.

🔍 What Evals Found & What We Fixed — The Empirical Discovery Loop

How outcome-based evaluation assertions discovered critical edge cases in statutory scheme delivery and directly drove production engineering fixes.

⚡ Production Fix

1. Indic Numeral Transliteration in DOB Parsing

What Evals Found: Senior citizens in Purvanchal routinely enter dates in native numerals (e.g., १२/०५/१९६१ or ১২/০৫/১৯৬১). Standard datetime parsers threw ValueError, blocking eligible rural elders from submitting.

What We Fixed: Built Unicode digit transliteration (str.maketrans("०१२३४५६७८৯০১২৩৪৫৬৭৮৯", "01234567890123456789")) directly into parse_dob(). All Indic numerals normalize to standard ASCII digits seamlessly before evaluation or storage.

Code: app/app.py:136 · app/mcp_server.py:check_eligibility

🛡️ Policy Hardening

2. Lifecycle-Aware Duplicate Checks & Re-Application Lifecycle

What Evals Found: Baseline table-level uniqueness on mobile permanently locked out citizens whose prior application was rejected or withdrawn, and lacked actionable guidance for households attempting concurrent submissions on a single phone.

What We Fixed: Scoped active application checks strictly to status IN ('PENDING', 'APPROVED') via active_application(). Citizens can safely re-apply after rejection/withdrawal, and concurrent attempts receive clear statutory guidance.

Code: app/app.py:195 · app/mcp_server.py:withdraw_application

🗄️ Schema Safety

3. Audit Trail Column Truncation (actor VARCHAR(50))

What Evals Found: Long system actors like system:rtps-deemed-approval-cron-daemon or RFC 7591 OAuth client IDs (citizen:urn:ietf:params:oauth:...) exceeded 50 characters, triggering PostgreSQL StringDataRightTruncation exceptions and rolling back valid transactions.

What We Fixed: Implemented defensive actor and note sanitization in portal.write_audit() (safe_actor = str(actor)[:50]) and aligned schema bounds across the portal, preventing unhandled rollbacks during legal audit recording.

Code: app/app.py:115 · app/schema.sql:47

🤖 LLM Resilience

4. Edge LLM Resilience (Small Models & Latency Spikes)

What Evals Found: Under-powered edge models (e.g., Qwen 0.5B on local hosts) occasionally omit structured tool arguments or take >90s, causing evaluation timeouts and citizen conversational stalls.

What We Fixed: Engineered a dual-mode reasoning harness in app/ollama_agent.py that couples LLM tool-calling with deterministic rule-based intent fallback and pre-parsing. If model output degrades, mandatory statutory parameters are extracted deterministically, ensuring zero dropped requests.

Code: app/ollama_agent.py:470 · app/scripts/run_evaluations.py

📋 Audit Log of Executed Evaluation Runs

Timestamp (UTC)	Persona / Suite	Outcome	Verification Assertions	Notes & Model
2026-10-13T12:28:25.299654+00:00	security_and_auth_guards	PASSED	Server enforces 300-second OTP expiry window: expired OTPs strictly rejected — PASS Cross-citizen protection: Citizen B cannot view Citizen A's application status — PASS Zero-trust submission guard: Submissions without valid bearer token strictly blocked — PASS Role isolation: Citizen bearer token strictly forbidden from calling officer tools — PASS Streamable HTTP Citizen MCP service live on port 8201 with role discovery — PASS Streamable HTTP Officer MCP service live on port 8202 with role discovery — PASS Authentic external client: RFC 7591 DCR + RFC 7636 PKCE OAuth & wire MCP calls verified — PASS	Verified negative authentication paths, session boundaries, wire-level Streamable HTTP MCP discovery, and real client PKCE OAuth.
2026-10-13T12:28:17.022400+00:00	sla_overdue	PASSED	Synthetic application older than statutory 15-day SLA created in database — PASS Statutory deemed-approval job executed successfully — PASS Application transitioned to DEEMED_APPROVED state — PASS Decided by actor stamped as RTPS-AUTO — PASS Audit log records DEEMED_APPROVED action attributed to system:rtps-cron — PASS Batch processor is idempotent: running twice does not produce duplicate approval records — PASS	Verified Purvanchal Right to Public Services Act (RTPS) nightly deemed approval batch automation.
2026-10-13T12:28:16.764980+00:00	officer_decision	PASSED	Officer assistant invoked list_pending_queue MCP tool — PASS Officer assistant invoked review_case_details MCP tool — PASS Validation guard: process_decision strictly rejects short justification (<5 characters) — PASS Validation guard: process_decision strictly rejects invalid case ID — PASS Officer assistant invoked process_decision MCP tool with full justification — PASS Database state updated to APPROVED by authenticated officer eval-admin — PASS PostgreSQL audit_log records APPROVED action with officer actor and justification — PASS	Verified administrative officer workflow: queue inspection, case review, process_decision with audit trail.
2026-10-13T12:28:02.151369+00:00	outside_pilot	PASSED	Agent invoked check_eligibility MCP tool for outside-pilot applicant — PASS MCP tool returned eligible_to_apply=False due to non-pilot block — PASS Response explicitly restricts automated assistance to Sonari pilot block — PASS Direct submission outside pilot block rejected with BLOCK_NOT_IN_PILOT — PASS Zero database records created for out-of-scope block — PASS	Verified geographical pilot boundary constraint (Sonari only) via Ollama + MCP check_eligibility.
2026-10-13T12:27:56.119724+00:00	underage_sonari	PASSED	Agent invoked check_eligibility MCP tool for under-age applicant — PASS MCP tool returned eligible_to_apply=False for 45yo citizen — PASS Rejection reason explicitly references statutory 60-year threshold — PASS Zero database records created in portal database for ineligible applicant — PASS Validation guard: Invalid bank details strictly rejected without writing DB records — PASS	Verified strict boundary guard against under-age submissions via Ollama + MCP check_eligibility.
2026-10-13T12:27:51.127137+00:00	eligible_sonari	PASSED	Agent invoked check_eligibility MCP tool — PASS Eligibility tool confirmed applicant eligibility (age 65, Sonari, >= 60 threshold) — PASS Agent invoked submit_application MCP tool — PASS MCP tool returned submitted=True with committed application number (SSP26975773) — PASS Portal database verified: application exists in PENDING state — PASS Assigned block matches pilot scope (Sonari) — PASS PostgreSQL audit_log records exactly one citizen SUBMIT event — PASS Status check tracks the exact committed application reference number — PASS	Ollama (qwen2.5:0.5b) end-to-end conversation with FastMCP tools check_eligibility & submit_application.

Seva Setu — MCP Evaluation Dashboard