Triage Threshold Tuning Runbook¶

Owner: Platform Ops
Last updated: 2026-05-07
Admin UI: /admin/triage
Related ADR: ADR-0026 (matching framework), Issue #768

Overview¶

Triage thresholds control when and how patients advance through the triage layer and are routed to MSO (Medical Second Opinion) consultations. All thresholds are stored as Flagsmith per-identity overrides (keyed by tenant ID) and can be tuned without a code deploy.

Every change is audited: - Event row (EventType.ADMIN_FLAG_IDENTITY_OVERRIDE) — real-time event bus - AuditLog row (action=triage_threshold.update) — queryable from /admin/audit-log

Thresholds Reference¶

Section A — Triage Layer Advance¶

What it does: Minimum confidence score from the triage layer required to auto-advance a patient past triage to provider matching.

When to adjust: - Lower (e.g. 0.5): More patients advance automatically. Use when intake conversion is too low and clinical data is high-quality (high-context markets, existing FHIR records). - Higher (e.g. 0.85): Fewer patients advance automatically; more trigger a human review. Use when the match quality is poor due to sparse intake data.

Expected impact: Each 0.1 decrease adds ~12–18% more cases to the automated advance pool (based on 2026-Q1 data). Monitor match_confidence distribution in Metabase before and after.

Baseline: 0.70 (calibrated on 180 completed cases, 2026-Q1).

Section B — MSO Routing¶

mso_completion_threshold¶

Minimum intent-completion score (from the MSO offer agent) required before an MSO consultation is offered to the patient.

When to adjust: - Lower (e.g. 0.6): Offer MSO earlier in the patient journey, when less information is available. May increase MSO volume but reduce specialist consultation quality. - Higher (e.g. 0.9): Only offer MSO when the patient profile is nearly complete. Reduces volume, increases quality.

Baseline: 0.80.

mso_pfs_skip_threshold¶

If the patient's procedure-free survival (PFS) signal is above this value (in days), the PFS branch of the MSO offer is skipped. Useful when the PFS signal is unreliable for certain procedure categories.

When to adjust: - Increase: Fewer patients are routed through the PFS branch — use if PFS data quality is low for a specific tenant's case mix. - Decrease (0): All patients go through the PFS branch — maximum coverage.

Baseline: 60 days.

mso_eligible_decision_stages¶

JSON array of patient decision stages that qualify for MSO routing.

Valid values: initial_inquiry, exploring_options, comparing_options, decided_on_destination, post_consultation.

When to adjust: - Add exploring_options to capture patients earlier in the journey. - Remove decided_on_destination if that stage has low MSO conversion. - Start with the narrowest set and expand based on conversion data.

Baseline: ["comparing_options", "decided_on_destination"].

mso_max_offers_per_case¶

Maximum number of MSO consultations a single case may receive. Set to 0 to disable MSO for this tenant entirely.

When to adjust: - Increase: High-complexity case mix (e.g. oncology, cardiac) may benefit from multiple specialist opinions. - Decrease / 0: Disable for tenants not yet contracted for MSO.

Baseline: 3.

Section C — Prompt Versions¶

These two flags must be changed together. The admin UI enforces pair validation and warns when the combination is unsupported. Saving an unsupported pair requires checking "I understand — save anyway" (force_save=true) which is recorded in the audit log.

When to adjust: - Prompt version upgrades are coordinated with the AI team. - Always test 3 live conversations after updating (per feedback_agent_chat_sacrosanct.md). - Roll back by setting the flags back to v4 / v1 (no code change needed).

Rollback¶

All thresholds are Flagsmith flags — rollback means setting the flag value back to the previous value. No code deploy or migration needed.

1. Open /admin/triage in the admin portal.
2. Set the affected flag(s) to the previous value.
3. Click Save.
4. Verify the AuditLog shows the revert action.

For prompt version rollback: set prompt_version=v4 + triage_layer_context_version=v1.

Monitoring¶

After any threshold change, verify in:

• Metabase: match_confidence distribution, MSO offer volume, advance rate.
• Langfuse: Triage agent prompt version tag on new traces.
• Audit log: /admin/audit-log — filter by action=triage_threshold.update.

Expect a 10–30 minute lag before new flag values are picked up (Flagsmith cache TTL = 60s; triage runs are async).

Related¶

• docs/adr/0026-matching-framework-architecture.md — parameter registry pattern
• docs/reference/feature-flags.md — all Flagsmith flags
• docs/runbook/triage-v3-baseline-test.md — baseline conversation quality test