HealthSteward — Technical Design Document
A point-in-time architecture snapshot, written up for anyone who wants to go deep on how it's built — reviewers, collaborators, or a technical interview. Re-written on architectural shifts, not on every change; see the note at the bottom of this page.
Problem & motivation
Patients managing fragmented care — multiple specialists, no shared record system — carry an unpaid coordination job: remembering what changed since the last visit, which labs are pending, what to raise with which doctor. This falls hardest on people with the least capacity to carry it (mid-flare, mid-crisis), and nobody on the clinical side owns it.
Most tools treat this as a records problem: store the documents, make them searchable, maybe chat with them. HealthSteward treats it as a coordination problem instead — ingest documents providers already give you, track what's changed and what's still open, and turn that into something concrete for the next visit. Locally, by default, because a tool holding this much health history shouldn't require sending it to a server to be useful.
What it does
Health profile management
Conditions (ICD-10 coded), medications, doctors, appointments in one place.
AI visit preparation
An agentic loop drafts personalized questions for an upcoming visit — pulling relevant history and calling tools before finalizing. Runs fully local via Ollama by default; Claude API and other OpenAI-compatible providers are opt-in.
After-visit summary parsing
Upload the PDF from a visit. Parsed locally, reviewed by you, then applied to your profile.
Proactive action items
Follow-ups to book, labs to get done, referrals to schedule — surfaced at the moment of engagement and again until resolved.
Goals
- Generate genuinely useful, specialty-relevant visit-prep questions from the patient's own data
- Keep health data local by default; anonymize anything that must leave the machine
- Turn parsed AVS data into closed-loop action, not just storage
- Degrade gracefully — a failing LLM call should never block the user from getting something useful
Explicit non-goals
- Not a clinical decision-support tool. Generated questions are prompts for a conversation with a real clinician — not diagnostic or treatment guidance. No automated clinical-safety validation exists or is planned; human review is the only safety mechanism today.
- Not multi-user yet. Family sharing is deferred pending a decision, not built toward.
- Not HIPAA-scoped. Personal/family use, not a covered entity.
- No A/B testing, canary rollout, or drift-monitoring infrastructure. Single-user local app, no population to canary against — a deliberate scope cut, not an oversight.
Tech stack
FastAPI + SQLAlchemy (async) + SQLite, migrations via Alembic
React 19 + TypeScript + Tailwind CSS + Vite
Pluggable backend — Ollama (local) by default, or Claude API (Sonnet), or any OpenAI-compatible provider — for visit prep's tool-use loop, switchable at runtime from Settings
Ollama (qwen2.5:7b) for PDF parsing and context-selection relevance scoring
System Design
Two LLMs, two trust boundaries. Almost everything runs inside the dashed local boundary below; the only thing outside it is an optional external LLM call, made only after anonymization.
Data flow
Only if LLM_PROVIDER=claude or custom — anonymized context and tool results only. Response returns to Orchestrate; Serve continues either way.
Two trust boundaries
Ollama, local
PDF parsing always runs on a local model, and visit prep's agentic loop does too by default (DEC-016). Raw records never touch the network — enforced by a hard localhost-only safety check in the Ollama client, not just convention.
Pluggable backend (Claude or any custom provider)
Only ever receives already-anonymized data, when opted into. Anonymization happens before the agentic loop starts, and every tool result fed back into the loop is anonymized the same way, regardless of backend.
Reliability design
The agentic tool-use loop is fallback-not-hard-failure by construction: if it can't converge within agent_max_turns (default 6) or a backend produces a malformed tool call, the system falls back to the pre-existing single-shot prompt-in/JSON-out call. The single-shot path is the floor every enhancement is built on top of — no functional regression is possible by design, only degraded quality on failure.
Deep Dives
Expand any section — these are written to be answerable as "walk me through how X works" in a technical conversation.
Agentic tool-use loop +
_run_agentic_loop in src/agents/visit_prep.py: send context + tool specs to the backend → if it returns tool calls, execute each against the DB, anonymize the result, append to the conversation, repeat → if it returns final text with no tool calls, that's the answer → if agent_max_turns is exhausted first, raise and let the caller fall back to single-shot.
Model returns plain text before the turn cap. Logged to ConversationLog, parsed as the final JSON questions.
Parse error or turn cap hit → caught, and prepare_visit() silently falls back to a single-shot call.
On-demand structured lookup — dosage, frequency, purpose, side effects — for one or all current medications. Not a real interaction checker; that's a separate, bigger feature (issue #24).
Deeper visit-history query beyond what's already in the context, filterable by specialty/keyword. Windowing to "since last visit with this provider" is scoped separately (issue #21).
- Why bounded, not open-ended: an unbounded loop against a small/unreliable local model can spin. A hard turn cap makes worst-case latency and cost predictable.
- Why Ollama as default, not Claude API (DEC-016, superseding DEC-009's default): the dev machine (Apple M3, 8GB RAM) can only run 4-bit quantized 7-8B models, and small quantized models produce unreliable tool-calling — malformed JSON, wrong tool selection, non-convergence — which is still true and exactly why the fallback-to-single-shot behavior below matters. But defaulting the primary, most-used flow to an external API sat awkwardly next to the project's local-first pitch. Claude (and any custom OpenAI-compatible provider) remain fully supported, now as explicit opt-ins from Settings rather than the default.
- Two tools today, deliberately bounded scope for v1 — a real drug-interaction checker and a user-facing pause-to-ask-clarifying-questions flow were explicitly descoped as separate, bigger features.
Pluggable LLM backend +
src/agents/llm_backend.py: an LLMBackend ABC with three implementations — ClaudeBackend (wraps AsyncAnthropic, uses Claude's native tools= parameter), OllamaBackend (calls Ollama's /api/chat with OpenAI-style function-calling tool specs), and CustomOpenAICompatibleBackend (DEC-016 — any endpoint speaking OpenAI's /chat/completions + tool-calling format: OpenAI, OpenRouter, Groq, Together, a self-hosted vLLM/LM Studio server, etc.). All three normalize into the same ToolCall/LLMTurnResult shape, so _run_agentic_loop never branches on which provider it's talking to — it just calls backend.call(...) and reads a uniform result.
- One canonical tool spec, reused wire format:
src/agents/tools.pydefinesTOOL_SPECSonce, thenclaude_tools()adapts it for Claude andollama_tools()covers both Ollama and the custom provider — Ollama's/api/chatalready mirrors OpenAI's function-calling shape, so the custom backend needed no third adapter. - Why this matters beyond the agentic loop: it's what makes
LLM_PROVIDER=ollamaa real, fully-local default rather than a stub — the same tool-use loop, same tools, same anonymization boundary, just a different backend underneath. - Selected via a factory,
get_llm_backend(settings), keyed offsettings.llm_provider. DEC-016 also consolidated dispatch insidevisit_prep.pyitself — it previously checkedllm_provider == "ollama"via string equality in three separate places; all three now route throughget_llm_backend()/get_tools_for_provider(). - Runtime-editable, not just env-configured:
settings.llm_provideris now overlaid from a DB-backedAppSettingssingleton row (src/services/settings_service.py), so switching providers from the Settings page takes effect on the next request — no.envedit or restart needed.
Specialty-aware prompting +
Before DEC-011, visit prep generated the same questions regardless of which specialist the appointment was with — it would suggest discussing a dermatology cream with a cardiologist. Fixed with two system prompt templates (SYSTEM_PROMPT_TEMPLATE, SYSTEM_PROMPT_GENERIC in src/agents/visit_prep.py) plus data enrichment that tags conditions and medications with the specialty that actually manages them.
| ICD-10 prefix/range | Mapped specialty |
|---|---|
| E08–E13 | Endocrinology (diabetes) |
| I00–I99 | Cardiology |
| L00–L99 | Dermatology |
| E28 | Endocrinology + Gynecology (e.g. PCOS) |
- ICD-10 → specialty mapping (
ICD10_SPECIALTY_MAP, prefix and range matching via_icd10_to_specialties()) — exact prefixes checked before ranges, so a specific code like E28 doesn't get swallowed by the broader E20–E35 range. - Medication → prescriber specialty tagging — matches a medication's free-text
prescribing_doctoragainst the profile'sDoctorrecords to label it (e.g. "prescribed for Dermatology"), so the model can tell which meds are actually this specialist's business. - Clinic-name inference fallback (
_infer_specialty_from_clinic()) — keyword-matches the clinic name (e.g. "Sutter Endocrinology") whendoctor.specialtyis null. - The prompt itself draws the line explicitly: don't suggest unrelated-specialty medications, but do surface real cross-condition interactions relevant to this specialty (its own worked example: Hashimoto's and PCOS interact hormonally, which matters for an endocrinologist even if a gynecologist made the original diagnosis).
PII anonymization +
Hybrid approach in src/utils/anonymization.py: structured fields get deterministic replacement (patient name → "Patient", DOB → exact age, doctor name → "your [specialty]"), free text goes through regex (phone/email/SSN patterns) plus spaCy NER for names.
- Documented as best-effort on free text, not a guarantee — NER can miss names in unusual phrasing. This is stated explicitly rather than implied, since silently overstating a privacy guarantee is worse than none.
- Applies uniformly regardless of provider — even when running fully local Ollama (where anonymization is technically unnecessary since nothing leaves the machine), the same code path is used for consistency and to avoid two divergent implementations.
- Tool results are anonymized too, not just the initial context — a result from
lookup_past_visitsmid-loop goes through the sameAnonymizerbefore re-entering the conversation.
4-stage context selection +
Before generating visit-prep questions, relevant past-visit history is selected via a 4-stage pipeline in src/utils/context_selection.py:
More than 5 visits remain after stage 1, and Ollama is available. Each is scored 1–10; only ≥7 survives.
≤5 visits remain, or Ollama is unavailable — falls straight through to rules-based results plus truncation.
- Rules-based filter (instant, free) — always include same-doctor's last visit and PCP/Internal Medicine visits; apply a specialty-relevance mapping (e.g. Endocrinology ↔ Cardiology, Nephrology); exclude doctors flagged out of prep context.
- Local LLM relevance scoring (Ollama) — only runs if more than 5 visits remain after stage 1; scores each 1-10, keeps ≥7. Skipped if Ollama is unavailable, falling back to rules + truncation.
- Token budget check — summarizes older visits with the local LLM if still over budget.
- Anonymize — the only stage where data crosses into what the external-capable backend will see.
Stage 2's relevance scoring deliberately runs on raw, pre-anonymization text — it's a local-only call, so there's no privacy reason to anonymize before it, and doing so first would degrade the local model's ability to judge relevance from real names/context.
AVS PDF parsing +
Section-routing architecture in src/parsers/: deterministic parsers handle structured sections (patient info, medication changes, follow-ups, appointments, diagnoses with ICD codes); local Ollama handles unstructured sections (vitals, lab orders, notes, referrals, diagnoses without a structured Assessment section).
- Local-only, enforced not just documented —
ollama_chat.pyhas a safety check that blocks any non-localhost URL, so a misconfiguration can't silently route PHI externally. - Review before apply — parsed items are always presented for user confirmation with per-section checkboxes; nothing is auto-applied to the profile.
- Deduplication — diagnoses dedupe by name on apply (existing conditions get ICD-10 backfilled if missing); medication stops match by name.
Proactive action items & nudging +
Closes the loop between "document parsed" and "patient acts on what the doctor ordered." Two surfaces: a post-AVS action panel (fires at the moment of highest engagement, right after confirming a parsed document) and a persistent "Needs Attention" overview section (catches items accumulated from past visits, always visible).
- Snooze (1w/2w/1m) and completion state persist via
snoozed_until/completed_atonFollowUp/LabOrder/Referral, plus aNudgeStatetable for computed nudges that have no natural row to attach state to (e.g. "past-due appointment"). - Five computed nudge types drive the overview section: past-due appointments (
scheduled_datepassed, status stillscheduled), upcoming appointments without prep (scheduled within the next 30 days, noVisitPreprow yet), completed visits with no AVS uploaded within 14 days of the appointment date, adverse vitals trends (meaningful directional change in weight/BMI/blood pressure/heart rate across ≥2 recorded visits), and follow-ups, lab orders, and referrals that remain active (not yet completed) since they were logged. All five respect per-item snooze state so a dismissed nudge doesn't resurface until it expires. - Scheduled push notifications for genuinely disengaged patients — reaching someone who isn't currently in the app — were considered and explicitly deferred; it's new infrastructure (a scheduler + notification channel) that arguably conflicts with the local-first architecture, not a pure addition on existing data like the two shipped surfaces are.
Data model & schema design +
src/data/models.py: HealthProfile at the root, with Condition, Medication, Doctor, and Appointment hanging off it. AVS-parsed data nests under Document — Vitals (1:1), LabOrder/Referral/FollowUp (1:many) — so every parsed item traces back to the source PDF it came from. VisitPrep and ConversationLog record generated output and (anonymized) LLM call history; NudgeState persists snooze/dismiss state for the nudges that have no natural row of their own.
- UUID primary keys, not auto-increment integers (DEC-004) — a sequential ID leaks how many records exist and lets rows be enumerated by guessing. UUIDs cost a little storage/perf, which is a fine trade for health data specifically.
- Profile-nested API routes (
/api/profiles/{id}/conditions/, etc.) — ownership is enforced by the URL structure itself, not left to an ad-hoc check in each handler. Condition.icd_10was added specifically to enable specialty-aware prompting (DEC-010/DEC-011) — the schema changed because a feature needed it, not speculatively ahead of time.- Async throughout — SQLAlchemy 2.0 async sessions,
aiosqlite, matching the async FastAPI route handlers end to end rather than mixing sync DB calls into an async app.
Decisions & Tradeoffs
A curated set of the calls with real engineering tension — not all 15+ decision-log entries, just the ones worth defending in a conversation. Full log: docs/notes/DECISIONS.md.
Agentic loop framework: native tool use, not Agent SDK or LangGraph
| Option | Why not |
|---|---|
| Anthropic Agent SDK | New dependency and learning curve for a single-agent workflow that doesn't need its structured primitives |
| LangGraph | Explicit state machine is overkill — heavy abstraction for one agent, one loop |
Local Ollama vs. Claude API for the agentic backend
| Option | Tradeoff |
|---|---|
| Local Ollama (7-8B, quantized) | Free, maximum privacy — but only 4-5GB RAM available after OS on an 8GB M3, so only 4-bit quantized models fit, and those produce unreliable tool-calling (malformed JSON, wrong tools, loops) |
| Claude API (Sonnet) | Reliable tool use out of the box, ~$0.03-0.10 per visit prep (~$1/month typical use) — but leaves the local-first ideal for this one call path |
LLM_PROVIDER=ollama with documented reduced reliability. Anonymization made the Claude path privacy-acceptable rather than privacy-ideal. Superseded by DEC-016 (below) — the default flipped, this card's tool-reliability finding did not.Flip the default agentic backend to Ollama; add a custom OpenAI-compatible provider
| Option | Tradeoff |
|---|---|
| Keep Claude as default | Reliable tool-calling out of the box — but sits awkwardly next to a project pitched as local-first when the most-used flow defaults off-device |
| Flip default to Ollama, keep Claude opt-in | Privacy-first positioning holds by default — but more installs hit the small-model tool-calling unreliability DEC-009 documented, leaning harder on the existing single-shot fallback (DEC-013) |
CustomOpenAICompatibleBackend (any OpenAI-compatible endpoint — OpenAI, OpenRouter, Groq, self-hosted, etc.) as explicit opt-ins, switchable at runtime from a new Settings page (DB-backed, no .env edit or restart needed) rather than .env-only. DEC-009's reliability finding is unchanged — this supersedes its default choice, not its technical conclusion.SQLite vs. PostgreSQL
| Option | Tradeoff |
|---|---|
| SQLite | Zero setup, file-based, portable — but single-writer, not ideal if multi-user ever ships |
| PostgreSQL | Production-ready, concurrent — but requires a running server for a currently single-user local app |
UUID vs. auto-increment integer primary keys
| Option | Tradeoff |
|---|---|
| Auto-increment integers | Simple, compact, fast — but leaks record count and lets IDs be enumerated/guessed |
| UUIDs | Larger, marginally slower — but private and trivially portable across databases |
PDF parsing: local-only Ollama vs. cloud OCR/vision
| Option | Tradeoff |
|---|---|
| Claude Vision / cloud OCR | Best accuracy on scanned/image PDFs — but sends raw medical documents to a third party |
| Local Ollama + deterministic parsers | Maximum privacy, section-routing keeps accuracy reasonable on structured sections — but weaker on genuinely unstructured, image-heavy documents |
Agentic loop scope: two read-only tools, not a full feature set
| Descoped | Why |
|---|---|
| Real drug-interaction checker | Needs a licensed external interaction-database API — a separate, bigger feature (tracked as its own issue) |
| User-facing clarifying-question pause | Needs new DB state, a new API endpoint, and new frontend UI to resume a paused conversation (tracked as its own issue) |
Walkthrough
⚠ synthetic example — fabricated patient, not real dataThere's no captured real-patient output in the repo to pull from (by design — nothing leaves a real user's machine). This walkthrough is a constructed, illustrative example showing what actually happens end-to-end for one visit-prep run.
Input: upcoming appointment + existing profile data
Patient has an upcoming Endocrinology visit. Profile has two conditions with a real cross-specialty interaction worth surfacing: Type 2 Diabetes (E11) and Hashimoto's thyroiditis (E06.3) — both endocrine, but historically diagnosed by different specialists (PCP and an OB-GYN, respectively).
Conditions: - Type 2 Diabetes Mellitus (E11.9) [active] — typically managed by: Endocrinology - Hashimoto's Thyroiditis (E06.3) [active] — typically managed by: Endocrinology Medications: - Metformin 500mg — twice daily [prescribed for Endocrinology] - Levothyroxine 75mcg — once daily, morning [prescribed for Endocrinology] Lab Orders: - HbA1c (ordered 2026-05-02) - TSH (ordered 2026-05-02)
Context selection + anonymization
4-stage pipeline includes the same-doctor's last visit and PCP visits automatically; a past OB-GYN visit surfaces via the Endocrinology ↔ Gynecology specialty-relevance mapping (PCOS/thyroid cross-relevance) rather than being excluded as off-specialty. Before this reaches the LLM, patient name → "Patient", DOB → "39 years old", doctor names → "your Endocrinologist".
Agentic loop: a tool call mid-generation
Model calls get_medication_details with no filter to double-check dosing/timing before finalizing a question about levothyroxine-metformin timing overlap — a real absorption interaction worth asking about. Result is anonymized (prescriber name redacted) and fed back into the conversation.
→ tool_call: get_medication_details({})
← result: "- Metformin (500mg) — twice daily
Purpose: Blood sugar control
- Levothyroxine (75mcg) — once daily, morning
Purpose: Thyroid hormone replacement
Prescribed by: your Endocrinologist"
Final output
Loop converges (no further tool calls) within 2 turns. Output parsed as JSON, returned to the UI:
- How does having both Hashimoto's and Type 2 Diabetes affect your target A1c range?
- Is my current thyroid control (TSH) affecting how well Metformin is managing my blood sugar?
- Should I space out Levothyroxine and Metformin — I take them close together in the morning?
- My HbA1c and TSH were both ordered last visit — what results would prompt a dosage change on either medication?
Context summary: "Patient manages co-occurring Type 2 Diabetes and Hashimoto's Thyroiditis, both under active medication management with pending labs to assess control."
Risks & Open Gaps
What's actually unsolved right now — shown deliberately, not glossed over.
Risks
Generated output is health-adjacent guidance. The only safety mechanism is the patient reading it before a real appointment — no automated check exists for a plausible-but-wrong suggestion (e.g. a hallucinated interaction, a misread lab trend). Framed as a permanent human-in-the-loop requirement, not a gap with a "done" state.
Ollama model tags (qwen2.5:7b, llama3.2) are referenced by tag, not pinned digest — a silent upstream re-tag could change parsing/scoring/visit-prep behavior with no code change to explain why, and this now affects the default agentic backend directly, not just an opt-in path. Anthropic API and any custom provider's pricing/availability changes only matter to whoever has opted into them from Settings. Tracked in issue #31.
The specialty-aware system prompts are the actual product behavior, but changes go through normal code review with no dedicated versioning or before/after quality comparison. Low-stakes at current scale; would need a real process if the loop gains more agentic freedom.
Known gaps
Tests mock the LLM and verify the pipeline works — loop convergence, tool execution, anonymization boundaries — not whether generated questions are actually good: relevant, non-redundant, correctly specialty-scoped. Tracked in issue #29.
ConversationLog stores the data, but nothing surfaces how often the loop actually falls back to single-shot. A silently degrading backend would be invisible. Tracked in issue #30.
Backend has 72+ tests; frontend has zero — no framework installed, no test files. Tracked in issue #27.
See all open issues for the full backlog, or start a Discussion if something here isn't clearly a bug or a scoped feature yet.
Glossary
Terms as used specifically in this codebase — some have a more specific meaning here than the general ML/LLM usage.
- Agentic loop
- The bounded tool-use conversation loop in
visit_prep.py— the model can call tools before producing a final answer, up toagent_max_turns, with automatic fallback to single-shot generation if it doesn't converge. - AVS
- After-Visit Summary — the PDF a patient is typically handed at the end of a doctor's visit, containing vitals, orders, diagnoses, and notes.
- Anonymization boundary
- The point in the pipeline (after context selection, before the agentic loop starts) past which no unredacted PII should exist — a hard constraint (DEC-006), not a best-effort nicety.
- Specialty-aware prompting
- System prompts that constrain generated questions to what's relevant to the specific doctor's specialty, using ICD-10 → specialty mapping and medication → prescriber tagging (DEC-011).
- Section-routing
- The AVS parser's architecture: deterministic parsers handle structured PDF sections, local LLM calls handle unstructured ones — routed by section type, not one monolithic LLM call over the whole document.
- Pluggable backend
LLMBackendabstraction (src/agents/llm_backend.py) that normalizes Ollama, Claude API, and any custom OpenAI-compatible provider's tool-calling into one interface, so the agentic loop code doesn't branch on provider (DEC-013, DEC-016).- NudgeState
- DB table persisting snooze/dismiss state for computed action items that have no natural row of their own to attach state to (e.g. "appointment is past-due").
- DEC-XXX
- An entry in
docs/notes/DECISIONS.md's architectural decision log — Date / Topic / Context / Options Considered / Decision / Reasoning / Status.
FAQ
It needs a licensed external service (RxNorm/DrugBank/FDB-class), which is a bigger, separate integration than what shipped with the agentic loop's first version — deliberately descoped rather than blocking the rest of the architecture on it. Tracked as its own follow-up.
Because small local models are known to produce unreliable tool-calling, and the single-shot path already worked before the agentic loop existed. Falling back preserves that guarantee — a user should never get a worse experience than pre-agentic HealthSteward, only a potentially less-refined one on failure.
Not scoped as HIPAA-compliant — it's personal/family use, not a covered entity under HIPAA. The privacy design (local-first, anonymization, no cloud storage) is motivated by good practice for sensitive personal data, not a regulatory compliance target.
They're actually the same Ollama instance/model (qwen2.5:7b) used for two different jobs — PDF parsing and context relevance scoring — not two separate models. They're kept as distinct call sites because they have different failure-handling needs (parsing failure surfaces to the user for review; scoring failure silently falls back to rules-based selection).
PDF parsing simply can't proceed (surfaced to the user). Context-selection relevance scoring skips stage 2 and falls back to rules-based filtering plus truncation. Visit prep — Ollama by default as of DEC-016 — falls back to its own single-shot path per DEC-013's convergence handling; switching to Claude or a custom provider in Settings avoids the dependency on Ollama for visit prep specifically.