Results, Limitations & Personas

Author: Uday Tamma | Document Version: 1.0 | Date: January 06, 2026 at 11:33 AM CST

Load Test Results

Test Configuration

Observed Performance

Latency Breakdown

Total P99: 106ms
├── Feature computation (Redis):  ~50ms (47%)
├── Risk scoring (detection):     ~20ms (19%)
├── Policy evaluation:            ~10ms (9%)
├── Evidence capture (async):     ~20ms (19%)
└── Network/serialization:        ~6ms  (6%)

Key Insight: Redis velocity lookups dominate latency at 47% of total. At scale, this is the first optimization target.

Capacity Projection

Identified Bottleneck: At ~4,000 RPS in ramp-up testing, PostgreSQL evidence writes saturated connection pool, increasing tail latency. This confirms the planned path to scale via sharding/replicas and event-sourced evidence ingestion.

Replay Validation

Using synthetic historical data with known fraud labels:

Summary:

• Detection rate: 72-94% depending on attack type
• False positive rate: 1.3-5% depending on scenario
• Card testing attacks have highest detection confidence
• Velocity attacks show strong detection with rule-based approach

Policy Impact Simulation (Replay)

Comparing baseline rules vs platform policy on 1M synthetic transactions:

The replay engine and economics framework provide finance and risk partners a quantified view of trade-offs before changes are deployed.

Limitations

Infrastructure Limitations

Data Limitations

Model Limitations

Operational Limitations

Honest Assessment

What This Proves:
  ✓ Architecture meets latency requirements
  ✓ Detection logic catches known fraud patterns
  ✓ Evidence capture is comprehensive
  ✓ Policy engine is configurable
  ✓ System handles expected load

What This Does Not Prove:
  ✗ Performance under real production traffic
  ✗ Detection accuracy on real fraud (vs synthetic)
  ✗ ML model performance (not yet implemented)
  ✗ Operational readiness (no real incidents yet)
  ✗ Economic impact (no real financial data)

Personas & Dashboard Usage

Persona 1: Fraud Analyst

Role: Reviews flagged transactions, makes manual decisions, investigates patterns

Primary Dashboard Panels:

Workflow:

1. Check Review Queue
   └── Sort by priority (HIGH first)
   └── Filter by amount (high value first)

2. For each case:
   └── View transaction details (decision, scores, detectors fired, policy version)
   └── Review triggered signals and feature snapshot
   └── Check customer history
   └── Make decision: APPROVE / DECLINE / ESCALATE
   └── Annotate with disposition (confirmed fraud, friendly fraud, service issue)

3. Bulk actions:
   └── Add device to blocklist
   └── Add card to blocklist
   └── Flag user for enhanced monitoring

4. End of shift:
   └── Review queue age metrics
   └── Ensure nothing >4h old

Key Decisions:

• Accept/decline individual transactions
• Add entities to blocklists
• Escalate suspicious patterns to Risk Lead
• Annotate cases with dispositions (feeds back into model training labels)

Persona 2: Risk Lead / Fraud Manager

Role: Sets strategy, monitors KPIs, adjusts thresholds, manages team

Primary Dashboard Panels:

Workflow:

1. Morning Review:
   └── Check 24h approval rate
   └── Review any after-hours alerts
   └── Compare block rate to baseline

2. Weekly Metrics Review:
   └── Fraud rate trend (30d lag)
   └── False positive estimate
   └── Dispute outcomes
   └── Threshold performance

3. Threshold Adjustment:
   └── Run replay simulation on proposed change
   └── Review projected impact
   └── If acceptable: Apply via Policy Settings
   └── Monitor for 48h post-change

4. Incident Response:
   └── Spike in block rate? Check for attack or bug
   └── Drop in approval rate? Check threshold misconfiguration
   └── Latency spike? Escalate to Engineering

Key Decisions:

• Threshold adjustments (friction/review/block levels)
• Policy rule additions or modifications
• Escalation to Engineering or Security
• Resource allocation (analyst coverage)

Persona 3: SRE / On-Call Engineer

Role: Maintains system reliability, responds to alerts, handles incidents

Primary Dashboard Panels:

Workflow:

1. Alert Response:
   └── Check alert source and severity
   └── Verify via dashboard (not just alert)
   └── Follow runbook for specific alert type

2. Latency Spike Response:
   └── Check Redis latency panel
   └── Check PostgreSQL latency panel
   └── Identify bottleneck component
   └── Scale or restart as needed

3. Safe Mode Activation:
   └── Automatic if error rate >5%
   └── Manual if component failure detected
   └── Notify Fraud Ops (decisions will be conservative)
   └── Document reason and duration

4. Post-Incident:
   └── Collect metrics from incident window
   └── Write post-mortem
   └── Update runbooks if needed

Key Alerts:

Dashboard Mapping

Demo Dashboard (dashboard.py) - Current Implementation

Production Dashboard Needs (Gap Analysis)

This document provides an honest assessment of what the system proves and does not prove, mapping dashboards to real user personas and their workflows.