Waldo · Agent Harness · System Design

The Lead Engineer Map

One page that holds the whole system: the mental model, the architecture, the agent loop, how data flows across Cloudflare DO / R2 / Supabase, the authentication boundaries, and where the build actually stands in Linear.

🖱 Every dotted-underlined term has a definition on hover (tap on touch devices). Full glossary at the bottom. Diagrams are rendered with Mermaid — they need an internet connection on first load.

First principles

1 · The Mental Model

Waldo is not an app with an AI feature. It is a persistent per-user agent process — one Durable Object per human — whose first context stream happens to be the human body.

Eight invariants. If a design proposal violates one of these, it is wrong before any code review:

INVARIANT 01

The DO is the agent

Not a function that calls an LLM — a process that wakes on alarms and events, assembles context, runs a bounded loop, delivers, and hibernates. There is exactly one brain per user and it lives in the DO (ADR-0052). Edge Functions never call an LLM.

INVARIANT 02

98.4% deterministic, 1.6% model

The harness is deterministic infrastructure: hooks, ACLs, sanitisers, schedulers, math. If a deterministic function can do the job (CRS, pre-filters, topic rules), it must. The LLM is reserved for judgment.

INVARIANT 03

One writer per truth

Supabase owns raw health + identity + consent. DO SQLite owns agent state + memory. R2 owns cold archive + workspace files. Never two writers, never duplicated authority. Raw biometrics never enter DO or R2.

INVARIANT 04

The prompt is compiled, not written

Context is assembled fresh every wake from typed layers (REASONS: 7 conceptual / 10 physical). Memory is consulted deterministically (recall-before-act) — never left to the model to remember to ask.

INVARIANT 05

Security lives at seams, not in prompts

The model may propose anything; handlers refuse. Per-trigger tool ACLs, the autonomy gate, canary tokens, taint gates and the sanitiser are code-owned boundaries. A prompt instruction is never a security control.

INVARIANT 06

Contract-first

@pin4sf/waldo-types compiles before backend code exists. Tool counts, trigger names, card shapes — all derive from the type union, never from prose. Zod parses every boundary.

INVARIANT 07

Trust resets, memory persists

Every wake is a fresh permission slate (session trust reset) — yesterday's authorization never carries forward. But the brain (halls, episodes, skills) is durable. Memory is information, never authorization.

INVARIANT 08

The cost knob is the escalation rate

The primary model is swappable config behind the AI Gateway; what moves cost an order of magnitude is the escalation rate to the reasoning tier. ~$0.30/user/day blended; pre-filters and KAIROS skips are cost controls, not niceties.

The map

mental model — mermaid mindmap

mindmap
  root((Waldo
one agent per user)) Agent harness DO runtime, hibernates Run journal and outbox Hook pipeline, 9 events ReAct loop, max 3 iters Typed tools + per-trigger ACL Model routing via AI Gateway Body as context Health pipeline to CRS Form Zones energized to depleted Narrative context in every prompt Calendar, tasks, comms signals Memory 5 typed halls Episodes with FTS5 search Scribe inbox merge Recall before act Dreaming Mode at 2am Interface layer Brief, Fetch, Spots, Chat Patrol audit, Handoff EPA Telegram rich two-way APNs and FCM push Contracts waldo-types barrels Zod at every boundary Semver pinned, no drift Trust and safety Session trust reset Sanitiser and canaries Autonomy gate L1 L2 L3 Taint gate, GDPR erasure Economics Escalation rate is the knob Pre-filter and KAIROS skips Tier-aware spend cap, open

System design

2 · System Architecture

Three repos (waldo-app · waldo-backend · waldo-types), two trust zones (Supabase, Cloudflare), one shared contract. The app talks to EFs over REST; the Worker routes every agent event to that user's DO; the DO reads derived health from Supabase and delivers through channel adapters.

architecture — trust zones & modules

flowchart TB
  subgraph SURF["USER SURFACES"]
    APP["iOS app
Expo SDK 53"] TG["Telegram
rich two-way"] PUSH["APNs / FCM"] end subgraph SB["SUPABASE TRUST ZONE — deterministic only, no LLM"] EF["13 Edge Functions
JWT → rate-limit → Zod"] PG[("Postgres + RLS
raw health · identity · consent
chat mirror · agent_logs")] BI["build-intelligence EF
zero-LLM CRS math"] end subgraph CF["CLOUDFLARE TRUST ZONE"] WK["Worker router"] subgraph DO["WaldoAgent DO — one per user"] RUN["runtime
run journal + outbox"] HK["hook pipeline
9 lifecycle events"] CTX["context module
REASONS prompt builder"] TL["tool dispatcher
per-trigger ACL"] MEM[("DO SQLite
halls · episodes · inbox
runs · outbox · budgets")] SAFE["safety gates
medical · canary · taint"] end R2[("R2
workspace files
cold JSONL archive")] GW["CF AI Gateway
single LLM control plane"] end LLM["Gemma 4 26B primary
Sonnet 4.6 escalation ~5%"] APP -- "REST + JWT" --> EF EF --> PG PG --> BI BI --> PG APP -- "chat · approvals" --> WK TG -- "webhooks" --> WK WK -- "route by user_id" --> DO DO -- "per-user RLS JWT
derived health only" --> PG CTX --- MEM MEM <--> R2 RUN --> GW GW --> LLM DO -- "outbox flush" --> TG DO -- "outbox flush" --> PUSH DO -- "cards · thread state" --> APP

Deep modules — the interfaces that matter

Each module is deep: a small interface hiding a large implementation. Callers (and tests) know the seam, never the internals.

Interface (the seam)What it hides
tick(runId) → RunStateThe whole invocation state machine: run journal, outbox, ReAct iterations, crash-resume after eviction (ADR-0054)
computeCRS(snapshot) → CrsResultSAFTE-FAST weights, time-of-day normalization, multi-source HRV reconciliation, pillar drag — pure math, zero LLM
buildPrompt(trigger, ctx) → string7 conceptual REASONS layers as 10 physical layers, recall block, R2 workspace files, provider shaping, channel hints, sandwich defense
dispatchTool(call, ctx)Zod parse → ACL check → autonomy gatetaint gate → adapter call → result sanitise → typed error taxonomy
retrieve(args) / recall(ctx, hint?)Hall selection per trigger, FTS5 BM25 + temporal + RRF fusion, CARA/ACT-R weighting, hot-DO ∪ cold-R2 federation — the agent never sees storage tiers
submitMemoryIntent(intent)Sanitise-at-write, inbox staging, nightly Scribe merge with pattern_id supersedence, bi-temporal history
enqueueDelivery(intent) → OutboxEntryIdempotency, push budget, collapse-id stacking, channel persona slicing, retry semantics
sanitise(input, destination)5 ordered checks — destination-aware: internal contexts keep derived scores, external surfaces get zone language (ADR-0024)
evaluateTrace(trace) → TraceEvalLLM-judge rubrics, WIS components, KeepRate, replay fixtures — generation never grades itself
deleteUserEverywhere(userId)GDPR erasure across Supabase + DO SQLite + R2 + observability stores + channel residuals (ADR-0055)

Workflow

3 · The Agent Loop

Every invocation — a 7am Brief, a 15-minute Patrol pass, a Telegram message — runs the same spine. The only variation per trigger is which tools are allowed, which memory halls load, and which model route fires.

3a · One invocation, end to end

invocation workflow — mermaid sequence

sequenceDiagram
  autonumber
  participant T as Trigger
(alarm · webhook · user msg) participant W as Worker router participant H as Hook pipeline participant J as Run journal participant M as Memory participant P as Prompt builder participant G as AI Gateway → LLM participant D as Tool dispatcher participant Q as Quality gates participant O as Outbox participant C as Channels T->>W: event arrives W->>H: route to this user's DO H->>H: session trust reset — fresh ACL, fresh canaries, rate limits H->>H: emergency detection — BEFORE any LLM call H->>J: startRun / tick(runId) — resumes if DO was evicted H->>H: pre-filter — calm + high Form → template, skip LLM entirely M->>P: recall(ctx) — halls + episodes + evolutions, fenced P->>P: REASONS prompt — 10 layers, provider-shaped, under 10K tokens loop ReAct — max 3 iters (7 for chat) P->>G: generate G->>D: tool calls D->>D: Zod parse → trigger ACL → autonomy gate → taint gate D->>G: sanitised JSON results (capped, fenced) end G->>Q: candidate output Q->>Q: medical scrub → hallucination guard → canary check Q->>O: side-effects committed in SAME txn as state advance O->>C: idempotent flush — Telegram / APNs / in-app O->>J: state → DELIVERED → DONE J->>M: write episode · trace · memory intents · WIS signals

TRIGGER TAXONOMY (ADR-0015)

brief (+ variant morning/midday/evening/event) · fetch_alert · patrol · handoff_explore/plan/act/replan · intervention · user_message · dreaming_mode · pre_activity_spot (v0.2.0). The trigger decides everything downstream.

ITERATION BUDGETS

Automated triggers: 3 ReAct iterations max. user_message/Handoff: 7. Diminishing-returns guard: 3 tiny outputs in a row → hard fallback. Concurrent tool calls run via Promise.all.

4-LEVEL FALLBACK

L1 full-context primary → L2 reduced-context primary → L3 template with real data (no LLM) → L4 silent + retry next cycle. Circuit breaker: 3 failures → OPEN (jump to L3) · 2 successes → CLOSED. The user never sees an error.

3b · Durability — the run journal state machine (ADR-0054)

Cloudflare may evict a DO mid-loop. The answer is not "hope": every step is one committed DO SQLite transaction, and every side-effect rides the outbox with an idempotency key — so a crashed Brief resumes instead of double-sending.

run lifecycle — mermaid state diagram

stateDiagram-v2
  [*] --> PENDING : startRun(trigger, variant, nonce)
  PENDING --> CONTEXT_BUILT : recall + prompt committed
  CONTEXT_BUILT --> LLM_CALLED : model response committed
  LLM_CALLED --> TOOLS_DONE : tool batch committed
  TOOLS_DONE --> GATED : safety gates passed
  GATED --> DELIVERED : outbox flushed (idempotent)
  DELIVERED --> DONE : episode + trace written
  LLM_CALLED --> FAILED : retries exhausted → fallback chain
  FAILED --> [*]
  DONE --> [*]
  note right of CONTEXT_BUILT
    Each arrow = one DO SQLite transaction.
    DO evicted anywhere? Next wake calls
    tick(runId) and resumes from the last
    committed state. The outbox dedupes,
    so resume never double-fires a push.
    Key = hash(user, trigger, variant, run_nonce)
  end note

THE 9 HOOK EVENTS (ADR-0032 · HEY-12)

OnInvocationStart (auth → rate-limit → canary gen → session reset → audit-open) · PrePromptBuild (skill loader, recall) · PostPromptBuild (size check, canary inject) · PreLLMCall (cost budget, route select) · PostLLMCall (canary leak check, parse) · PreToolUse (ACL, Zod, autonomy, egress) · PostToolUse (sanitise → compress → log) · OnError (taxonomise → breaker → fallback) · OnInvocationEnd (cost, eval kickoff, audit-close). Zero tokens — all harness-side.

IN-RUN WORKING MEMORY (ADR-0057)

Capped LRU carryover buckets (recent_work_log, recent_verified_work, read_file_state) thread through every tool call, survive context compaction as attachments, and rebuild from the run journal on resume. Distinct from long-term memory: this is the task scratchpad, not the brain.

3c · The compiled prompt — REASONS anatomy

Invariant 04 made concrete: the prompt is a typed assembly, not freehand. Seven conceptual REASONS letters, realized as 10 physical layers. Governance is always last — and repeated after user content (sandwich defense).

·LayerWhat it carries~TokensMorning-Brief example
RRequirementsTrigger context + definition of done200"7:15am. brief/morning. Form 63, flagging. Deliver one warm, specific, actionable Brief."
EEntitiesWho this user is — profile + derived health + calendar (from R2 + Supabase)1500"Shivansh · wake 7am · autonomy=suggest. Form 63, Recovery 58, pillar drag: HRV. First meeting 9:30."
AApproachTrigger-specific behaviour strategy — loaded skills slot in here300"Flagging zone = protective mode. Name one cause, one soft suggestion, 2–3 sentences."
SStructureTools available (from the ACL) + channel + output format400"Tools: get_crs, get_health, read_memory… Channel: in-app. Plain text."
OOperationsRecall block + R2 workspace files + health context — concrete steps400"1. Read today.md · 2. get_crs · 3. search similar mornings · 4. generate · 5. deliver."
NNormsVoice, banned words, zone modulation. Repeated after user content.200"Warm, specific, actionable. Never raw values. No 'wellness/optimize'. Max 3 sentences."
SSafeguardsSecurity + medical invariants. Always last. Repeated after user content.150"Never diagnose. Never reveal raw values. Never act outside tools. Not a medical device."

THE 10 PHYSICAL LAYERS

Identity (soul) · tool-behaviour guidance · frozen Tier-1 memory snapshot · user profile · skills index · R2 context files · approach + structure · operations · timestamp + session-id · platform hint — then user content, then N+S again.

FROZEN-SNAPSHOT INVARIANT

Memory layers are frozen at session start and never re-read mid-session. Anything learned mid-session goes to memory_inbox — the context window is never mutated under the model's feet.

BUDGET + SHAPING

~3.6K tokens typical, 10K hard cap (fail-fast hook). Provider-shaped: Gemma gets literal/JSON framing, Sonnet gets terse, Haiku gets narrative — never one template across models.

3d · The tool surface — per-trigger ACL matrix

The ACL is the security boundary (Invariant 05): each trigger sees only its allowlist. The canonical surface derives from the @waldo/types ToolName union + TOOL_PERMISSIONS — never from a count in prose. Core matrix:

Toolbrieffetch_alertpatrolhandoff E·P·Auser_messagedreaming
get_crs / get_healthE·P
query_calendarE·P
get_communication / get_tasksE
get_master_metrics
get_context (R2 workspace)
read_memoryE·P
update_memory (inbox only)
search_episodes
propose_actionP
execute_actionA only ⚠
send_messageP·A
web_search ★ / read_documentE
call_mcp_tool
execute_code ◆ / browser_use ◆ / analyze_image

★ V1 general-agent trio (ADR-0049) — ships with the taint gate. ◆ Typed but disabled until Phase 3 (ADR-0050). Copilot writes (write_task · update_task · draft_document · draft_email · propose_schedule · write_sheet_cell) live in handoff_act + user_message, all behind the autonomy gateconnector_write always confirms. search_connector adds to brief/patrol/handoff-explore/chat. Threading tools (create_thread · delete/restore_message · archive_thread · update_thread_topics) are chat-only. The intervention trigger gets reads + update_task + propose_action — and never send_message (interventions don't push).

3e · Skills — the 15 bundled recipes (ADR-0022)

Tools are primitives; skills are recipes — markdown files composing tools + a reasoning template + a trigger condition + an effectiveness score. Every user gets these 15 on day one. A skill can never exceed its trigger's ACL (required_tools ⊆ TOOL_PERMISSIONS) — recipes don't grant privileges.

#SkillCategoryFires onWhat it does
1morning-briefLifecyclebrief · morningThe day-opening read: Recovery, what was handled overnight, day context
2midday-briefLifecyclebrief · middayForm trend since morning, Weight build-up, what's ahead
3evening-brief (The Close)Lifecyclebrief · eveningDay summary, Form trajectory, wind-down recommendation
4event-briefLifecyclebrief · eventFires on significant intraday change (Form −15, HR spike) — heightened attentiveness, plan is dynamic
5fetch-alertLifecyclefetch_alertThe acute stress alert (shadow-mode in V1)
6interventionLifecycleinterventionOverload escalation: Form floor + Load ceiling + complex task + 3 declining Patrol entries (ADR-0020)
7patrol-passLifecyclepatrolThe every-15-min background observation cycle (ADR-0017)
8prep-for-eventCopilotbrief · patrol · chatPre-meeting brief: attendees, last meeting's open items, connector context
9compose-todayCopilotbrief · chat"What's on my plate today" — schedule + tasks with Form-aware ordering
10summarise-meetingCopilotchat · dreamingTranscript → summary + decisions + action items → tasks
11recovery-dayCopilotbrief · patrolAuto-fires when Form < 50 sustained: rearranges the day for recovery, UI density drops
12focus-block-prepCopilotpatrol · chatLoads working context (docs, issues, tasks) as a Focus Window starts
13weekly-reviewCopilotbrief · chat · dreamingFriday/Sunday week roll-up document
14catch-upCopilotchat · briefReturning after a 48h+ gap: "here's what changed while you were away"
15permission-slipSocialchatGenerates a socially-credible boundary message ("Waldo says I need to take it easy") — the viral mechanic

FOUR PROVENANCES, ONE IDENTITY BOUNDARY

system (git, immutable, identity-locked — the 15 above) · connector (auto-load on connect: linear-sprint-summary, notion-spec-draft, gmail-priority-triage, spotify-mood-check…) · user ("make that a routine" → R2 /skills/user/) · agent_authored (Dreaming Mode spots a tool-sequence repeated 5+ times in 14 days → proposes via the autonomy gate, lives 7 days as provisional). Anything touching medical guidance, CRS, safety rules, voice, or the lifecycle is identity-locked — no runtime edit, ever.

EFFECTIVENESS — SKILLS EARN THEIR SLOT

Score = 0.6 × rule-based signals (thumbs, completion, dismissal, undo) + 0.4 × nightly LLM-judge. Below 0.40 three times → auto-revert / review. Above 0.80 ten times → preferred routine. Loader takes top-K per trigger (Brief 5 · chat 8). Agent-authoring caps: ≤2 new skills/week, never on safety-critical triggers, never execute_code without explicit user approval. Note: the learning loop is Phase-G; the pre-build scope register proposes V1 ships these as static markdown first.

Data architecture

4 · Memory & Storage

Three stores, three jobs: Supabase is the system of record, DO SQLite is the hot working brain, R2 is the cold archive plus the agent's filing cabinet. Nothing is ever in two places with two writers.

4a · Who owns what

SUPABASE POSTGRES · ap-south-1 · system of record

Raw truth, queryable, regulated

Holds: health_daily (raw device readings — the only place raw HRV/HR ever persists) · crs_scores (derived) · users / user_consents (Art. 9 consent before first health write) · oauth_tokens (Vault-encrypted) · chat_threads/messages mirror · patrol_entries · agent_logs · spots · notification_log.
Writers: EFs (sync, intelligence) + DO (audit/chat/logs via scoped writes).
Guard: RLS on every table, day 0.

DO SQLITE · per user · hot working set

The agent's brain, sub-ms reads

Holds: memory_blocks (5 typed halls, bi-temporal, pattern_id) · episodes + FTS5 · memory_inbox (Scribe staging) · runs/outbox (ADR-0054) · thread_topic_index · daily_push_budget · goals · commitments · handoff_state · interventions · skills · agent_evolutions.
Writer: the DO alone — single-writer by construction.
Guard: AuditedDB blocks UPDATE/DELETE on append-only tables. 10 GB soft cap → ADR-0056 compaction.

R2 · per-user prefix · $0 egress

Cold archive + workspace

Holds: waldo-workspace/{user}/today.md · baselines.md · patterns.md · goals.md · orchestration.md · skills/ (pre-compiled context, written nightly) · waldo-episodes/{user}/y/m/episodes.jsonl (JSONL archive >90d) · GDPR exports (signed URL, 24h).
Writer: the DO via a 5-function mount (read / write / list / commit / discard).
Guard: archive-don't-delete; only GDPR erasure removes rows.

R2 as the agent's LLM wiki — yes, by design

Can the agent treat R2 as a markdown knowledge base — an LLM wiki of facts plus its working files? Yes, in two levels — with three non-negotiable constraints.

LEVEL 1 — SHIPPING IN V1 (the workspace)

R2 already is the agent's filing cabinet: today.md · baselines.md · patterns.md · goals.md · orchestration.md · skills/*.md — written nightly by Dreaming Mode through the WorkspaceMount (read / write / list / commit / discard) and injected into every prompt. Files-as-interop: any model, any provider, reads the same markdown — no lock-in.

LEVEL 2 — THE FULL WIKI (planned, roadmap M3 · HEY-76)

Per-hall markdown pages with a Compiled Truth paragraph + append-only Evidence Trail + backlinks — exactly the pattern this vault uses on itself. Specced in the agent-intelligence design (6 hall files, INGEST/QUERY/LINT cycle); ticketed as HEY-76 (memory-tree cascade-seal + Obsidian mirror). The wiki becomes the user-readable, exportable view of what Waldo knows.

CONSTRAINT 1 — COMPILED VIEW, NOT SOURCE OF TRUTH

Typed facts live canonically in DO SQLite memory_blocks (confidence, bi-temporal validity, pattern_id, per-hall ACL). The R2 wiki is generated from the halls — a build artifact. If they ever disagree, the halls win. Two writers to one truth breaks Invariant 03.

CONSTRAINT 2 — NO RAW HEALTH, EVER

The destination-aware sanitiser runs before every R2 write: derived zones and narratives go in, raw HRV/HR/sleep numbers never do. R2 prefixes are per-user, but it is still outside the health system-of-record.

CONSTRAINT 3 — R2 HAS NO SEARCH

No FTS, no SQL — R2 is read-by-path. The design absorbs this: the common case is pre-compiled (the agent just opens today.md), and the long tail goes through retrieve() where keyword search lives in DO FTS5. Use R2 for "read the file I know," never "find me something."

The 5-tier memory stack

TierWhereWhatBudget / lifecycle
Tier 0 — context windowvolatileREASONS prompt assembled fresh each wake~10K tokens, never stored
Tier 1 — memory_blocksDO SQLite5 halls: facts · events · discoveries · preferences · advice≤200 tokens always loaded · CARA confidence · ACT-R decay
Tier 2 — episodesDO SQLitePer-session diary entries, FTS5-indexed0–90 days hot, then R2
Tier 3 — skills / proceduresDO SQLite + R215 bundled + connector + user/agent-authored skillseffectiveness-scored, top-K loaded per trigger
Tier 4 — cold archiveR2JSONL episodes >90d + memory-history chainsforever (GDPR erasure only) · reachable via retrieve()

4b · Health dataflow — raw stays put, zones travel

health pipeline — device → supabase → derived context

flowchart LR
  subgraph DEV["DEVICE — private"]
    AW["Apple Watch sensors"] --> HKS["HealthKit store"]
    HKS -- "Background Delivery wake" --> NM["Swift module
anchor-based incremental"] NM --> OPS[("op-sqlite + SQLCipher
intraday raw stays here")] end subgraph SBZ["SUPABASE — system of record"] EF1["sync EF — JWT
unit normalize · anomaly bounds"] HD[("health_daily
1 row / user / day")] BIX["build-intelligence
zero-LLM CRS math"] CS[("crs_scores
Form · zone · pillars")] end subgraph DOZ["DO — derived only"] BC["body-context module"] NCX["narrative context block
zones + trends, NEVER raw values"] end NM -- "JWT POST" --> EF1 WHX["WHOOP / Oura cloud"] -- "HMAC webhook" --> EF1 EF1 --> HD --> BIX --> CS CS -- "per-user RLS JWT" --> BC --> NCX

Rule of the wall: raw sensor values (HRV ms, HR bpm, sleep minutes) exist only on-device and in health_daily. The DO reads derived scores; the sanitiser is destination-aware — internal contexts keep the CRS number, anything user-facing or external gets zone language ("flagging", "rough night"). Multi-device conflicts resolve by hrv_confidence per metric.

THE LOCKED FORMULA (ADR-0011 · SAFTE-FAST GROUNDED · 856-DAY VALIDATED · ZERO LLM)

Form = Sleep × 0.50  +  HRV (CASS) × 0.35  +  Circadian × 0.075  +  Motion × 0.075

Recovery = Sleep × 0.50 + CASS × 0.25 + RHRTS × 0.15 + RRS × 0.10 (backward-looking, fixed at wake). Computed deterministically in build-intelligence — no model anywhere in the health math. The score maps to the persona zone that modulates every message:

80+ energized — challenge mode65–79 steady — balanced, watchful50–64 flagging — protective<50 depleted — minimal, gentle

4c · Memory writes — nothing touches the brain directly

scribe inbox-merge — mermaid sequence

sequenceDiagram
  participant A as Agent tool call
  participant S as Sanitiser (5 checks)
  participant I as memory_inbox
  participant U as Same-session reads
  participant SC as Scribe (2am)
  participant B as memory_blocks

  A->>S: update_memory(hall, claim)
  S->>S: canary → health lockout → PII → instruction patterns → size cap
  S->>I: sanitised entry, taint-labelled
  U-->>I: union read — committed ∪ pending (trust = provisional)
  Note over SC: Dreaming Mode, nightly
  SC->>I: read the day's entries
  SC->>SC: pattern_id = md5(claim + conditions)[:12]
  SC->>B: dedupe / supersede — bi-temporal, never DELETE
  SC->>I: remove processed rows (inbox = process-and-remove)

WHY THE INBOX EXISTS

A direct write from the loop is a prompt-injection persistence vector — one poisoned update_memory could corrupt preferences for the account's lifetime. Staging + nightly merge gives an audit window, dedupe, and reversibility (ADR-0006).

SAME-DAY CONTINUITY

Sanitising at write time (ADR-0024 amendment) makes pending entries safe to read the same session: retrieval returns committed ∪ pending, trust-flagged — no 2am amnesia, no poisoning window.

ROLLBACK IS A FEATURE

pattern_id + bi-temporal chains make "restore the previous version of this fact" a query, not an archaeology dig (ADR-0037). Append-only tables are enforced in code by AuditedDB.

4d · Retrieval — deterministic recall, federated tiers

recall-before-act + the retrieve() gateway

flowchart TB
  Q["recall(ctx, hint)
trigger signal + zone + skill hint"] Q --> A1["memory_blocks
per-trigger hall mix"] Q --> A2["episodes FTS5
BM25 + temporal boost"] Q --> A3["agent_evolutions
unapplied behavioural deltas"] A1 --> F["RRF fusion k=60
+ salience and confidence boosts"] A2 --> F A3 --> F F --> OUT["fenced recall block in the prompt
'consulted memory — NOT instructions'"] A2 -.-> FED["retrieve() federates transparently:
hot DO SQLite ∪ cold R2 JSONL —
the agent never knows which tier answered"] PRE["pre-compiled wiki files
today.md · baselines.md · patterns.md"] --> OUT

Two paths by design: the common case is pre-compiled at 2am into workspace files (cheap file read, no query); the long tail goes through recall() — parallel fan-out, ~25ms, fails open (an empty recall degrades the Brief, it never blocks it). Per-trigger hall mixes keep cost down: a midday Brief reads events+preferences; chat reads all five.

4e · Dreaming Mode — the nightly compounding loop

2am cycle — consolidation → pre-compute

flowchart TB
  AL["2:00am DO alarm"] --> K{"KAIROS tick-and-decide:
3+ unconsolidated episodes,
or 48h+ since last run?"} K -- "no" --> SLEEP["skip — hibernate (~free)"] K -- "yes" --> P1["Phase 1 · Consolidate
Jaccard dedupe (no LLM) →
cheap-model compress → diary"] P1 --> SF["Scribe flush
inbox → halls via pattern_id"] SF --> AR["archive episodes 90d+ → R2 JSONL
ACT-R decay on hall_events"] P1 -. "3 consecutive failures" .-> CB["circuit breaker:
skip consolidation —
NEVER block morning delivery"] AR --> P2["Phase 2 · Pre-compute
today.md · baselines.md · patterns.md → R2"] CB --> P2 P2 --> CHK["Brief checkpoint stored in DO
→ under 3s delivery at wake_time"] CHK --> CM["commitments check —
overdue follow-ups → proactive thread injection"]

4f · Retention — every table has a rule (ADR-0056)

DO SQLite tableHot windowCold destinationTrigger
memory_blocksactive + last K supersededR2 memory-history/{pattern_id}.jsonlsize + age
episodes0–90d + daily diaryR2 episodes/y/m.jsonlage 90d
patrol_log (~70 rows/day)30dR2 monthly JSONLage + size
crs_historyrolling 30d baseline windowR2 (re-derivable from Supabase)age 30d
runs / outboxactive + 7dR2 run-journal archiveon-complete + age
memory_inbox · budgets · rate windowstransientn/a — consumed or auto-expireprocess-and-remove

Two compaction triggers: the nightly time-based sweep and a do_bytes size watermark (60% of the 10 GB cap → evict coldest-first; 80% → alert). Row-DELETE is allowed in exactly two places: self-clearing transients and GDPR erasure. Storage compaction is invisible to the agent — moving a row DO→R2 only changes which leg of retrieve() finds it.

4g · The math under the hood

Four algorithms — implemented in plain TypeScript inside the DO, zero external dependencies — are what make the memory smarter than a key-value store.

CARA — CONFIDENCE THAT LISTENS

+0.1 when the user confirms a remembered pattern · −0.2 when they contradict it · below 0.5 the block stops loading into context. Why: discoveries can be wrong — two contradictions and a bad pattern silences itself instead of poisoning every future prompt.

ACT-R — FORGETTING AS A FEATURE

activation = ln(Σ tᵢ⁻⁰·⁵), threshold θ = −2.0. Decay by age: 1d ≈ 1.0 · 7d ≈ 0.38 · 30d ≈ 0.18 · 90d ≈ 0.10. Why: a three-month-old diary entry shouldn't compete with yesterday's for the 200-token always-loaded budget — it fades below threshold (archived, never deleted).

BM25 + TEMPORAL → RRF — RETRIEVAL FUSION

BM25 (k1=1.2, b=0.75) for keywords · temporal boost ×1.5 recent / ×0.3 idle · fused by score = Σ 1/(60 + rank) · post-fusion ×1.15 if citation_rate > 0.3. Why: combines "what matches" with "what's fresh" and "what Waldo actually uses" — no vector DB needed for V1.

BI-TEMPORAL — TIME-TRAVEL FOR FACTS

valid_from / valid_to / superseded_by on every block. An update invalidates and inserts — never deletes. Why: rollback of poisoned memory, point-in-time queries ("what was my baseline in March?"), and honest GDPR exports all come free from one schema decision.

Identity & boundaries

5 · Authentication & Security Model

Five credential domains, none interchangeable: the user's JWT, the EF-only service-role key, the DO's per-user RLS JWT, webhook signatures, and connector OAuth in Vault. The single most important line: the agent loop never holds a credential that can bypass RLS (ADR-0052).

auth boundaries — who holds which credential

flowchart TB
  subgraph USR["USER SPACE"]
    APPC["iOS app"]
    TGU["Telegram user"]
    WHC["WHOOP / Oura cloud"]
    RCX["RevenueCat"]
    GOOG["Google OAuth consent"]
  end
  subgraph SBT["SUPABASE TRUST ZONE"]
    AUTH["Supabase Auth
issues user JWT"] EFZ["Edge Functions
validateJWT first 10 lines
rate-limit → Zod → version gate"] PGZ[("Postgres — RLS day 0")] VLT[("Vault
OAuth tokens encrypted,
stored as key references")] WBH["webhooks EF router"] end subgraph CFT["CLOUDFLARE TRUST ZONE"] DOZ["WaldoAgent DO
(NEVER holds service-role)"] SEC["wrangler secrets
APNs .p8 · bot token · gateway key"] end APNS["APNs"] TGAPI["Telegram Bot API"] APPC -- "sign-in (SIWA / Google)" --> AUTH AUTH -- "user JWT" --> APPC APPC -- "Bearer JWT on every call" --> EFZ EFZ -- "service-role — EF-ONLY,
RLS bypass confined here" --> PGZ TGU -- "update" --> WBH WBH -- "telegram_user_id allowlist —
unknown sender = silent drop.
Linking: one-time deep-link token (10 min)" --> DOZ WHC -- "HMAC-SHA256 signature" --> WBH RCX -- "signed webhook" --> WBH GOOG -- "PKCE flow, state = one-time token" --> VLT DOZ -- "per-user RLS-enforced JWT
derived health reads only" --> PGZ DOZ -- "ES256 JWT (1h, per request)" --> APNS DOZ -- "bot token" --> TGAPI

Inside the loop — defense in depth, in order

#GateWhat it stops
1Emergency detection — regex, before any LLMCrisis input gets hotlines (iCall India · 988 US), never a model response
2Session trust reset — every wakeYesterday's chat-session permissions bleeding into today's Brief; injection persistence across sessions
3Per-trigger tool ACL — deny-first, in the dispatcherA Brief invocation ever holding execute_action; injection with nowhere to go
4Autonomy gate L1/L2/L3 — at the tool handler, not the promptUnapproved mutations. connector_write (3rd-party-visible) always confirms — even at L3. Email send requires a user-tap HMAC approval token (5-min expiry)
5Taint gate — ships with the V1 general tools"Ignore previous instructions" in a calendar title flowing into a privileged action; external-derived args force human confirm
6Sanitiser — 5 checks at every write/egress seamRaw health values leaking outward; PII in drafts; instruction patterns entering memory; oversize blobs
7Output gates — medical scrub · hallucination guard · canary checkDiagnosis language; numbers the tools never returned; system-prompt leak (canary in output = session terminated)
8Egress allowlist + push budget in DO storageLoop calling unapproved hosts; notification spam surviving crash-retries (3/day, counted durably)

AUTONOMY GATE

L1 "Just tell me" → observe only, log to Patrol. L2 "Suggest" (default) → inline proposal card, wait for tap. L3 "Move things" → auto-apply + undo, except connector_write which always confirms. Mid-day downgrade triggers the reversibility sweep (ADR-0018).

WHY MEMORY ≠ AUTHORIZATION

A memory block saying "user previously approved bulk send" is information. Today's send still needs the autonomy gate + a fresh confirmation token. Past approval never grants future capability (ADR-0033).

GDPR ERASURE (ADR-0055)

One runbook enumerates every store: Supabase cascade, POST /do/{user}/delete (SQLite + R2 prefix), observability (Langfuse/PostHog/Sentry — hashed IDs, no health payloads), and the Telegram residual (documented, user-instructed). 30-day Apple ceiling. Telegram carrying health-derived text makes this launch-blocking.

Live build state · linear.app/heywaldo · read 2026-06-03

6 · Build State — what Linear actually says

112 issues in the HeyWaldo team, one project (Waldo V1 — Build Sprint), five sprint milestones plus a deliberately parked Phase-G/self-evolution backlog. The repos are greenfield — the contract ships first, then the spine, then skills.

Done 3In Progress 5Todo 57Backlog 46Canceled 1  ·  ready-for-agent: 79  ·  needs-info: 6  ·  external blockers: 2  ·  by repo: backend 73 · app 12 · brain 7 · types 6

Milestone#What it buildsSpotlight tickets
Sprint 0 — Foundation19The contract + schemas + security primitives. Nothing else starts until these compile.HEY-73 ★ types v0.2.0 — IN REVIEW (PR #3) HEY-6 repos HEY-7 types v0.1.0 HEY-9/10 schemas HEY-12 hooks ×9 HEY-13 sanitiser HEY-14/15/16 loader·recall·REASONS HEY-99 spend-cap decision HEY-5 Apple Dev (ext)
Sprint 1 — Lifecycle12Model routing, channels, the 7 lifecycle skills, CRS engine — the first end-to-end Brief.HEY-17 LLM routing HEY-18 Telegram HEY-19 APNs HEY-20–24 brief/fetch/intervention/patrol skills HEY-102 CRS engine HEY-25–27 threading + injection
Sprint 2 — Copilot reads11App scaffold + HealthKit native module, read-side copilot, channel persona, Dreaming Mode.HEY-28/29 app + HealthKit HEY-30 search_connector HEY-31 compression HEY-32 persona HEY-37 Dreaming 6-phase HEY-98 general tools (F1)
Sprint 3 — Copilot writes12Doc/Email adapters, mutating tools behind the autonomy gate, Window evaluator, GDPR cascade.HEY-38/39 Doc+Email adapters HEY-40–43 write tools HEY-48 autonomy gate HEY-46 Window evaluator HEY-101 GDPR delete
Sprint 4 — Eval + TestFlight8SheetProvider, final skills, the verification layer, first internal build.HEY-50/51 sheets HEY-53 LLM-judge HEY-54 WIS HEY-55 30 golden cases HEY-56 TestFlight
Unmilestoned spine48Runtime core + harness adoptions + the parked self-evolution track.HEY-72 DO loop shell HEY-110 run journal+outbox HEY-111 eval spine HEY-78 ToolDispatcher ACL HEY-71/74–77/79 adoptions HEY-80–97 Phase-G self-evolution (parked) HEY-63–68 calendar+voice HEY-109 State/session-bus HEY-103/104 ADR-0053 dev-loop
Phase 3 — Deferred2Power tools stay typed-but-disabled until sandbox contracts are proven.HEY-49 execute_code (ADR-0050) HEY-108 agent-behavior CI evals

6a · The critical path — what blocks what

dependency spine — from contract to first end-to-end brief

flowchart LR
  H73["HEY-73 ★
types v0.2.0 spine
IN REVIEW · PR #3"] H112["HEY-112
dep-migration gate
backend+app pin ^0.2.0"] H9["HEY-9
Supabase schema ×17 + RLS"] H10["HEY-10
DO SQLite schema"] H102["HEY-102
CRS engine (zero-LLM)"] H12["HEY-12
hook registry ×9"] H13["HEY-13
Scribe sanitiser"] H15["HEY-15
recall()"] H16["HEY-16
REASONS builder"] H72["HEY-72
DO loop shell"] H110["HEY-110
run journal + outbox"] H78["HEY-78
ToolDispatcher + ACL"] H17["HEY-17
LLMProvider routing"] H99["HEY-99 ⛔ founder call:
tier-aware spend cap"] H18["HEY-18 Telegram
HEY-19 APNs"] SK["HEY-20…24
lifecycle skills"] E2E(["first Brief
end-to-end"]) H73 --> H112 H112 --> H9 H112 --> H10 H9 --> H102 H112 --> H12 --> H13 H10 --> H15 --> H16 H112 --> H72 --> H110 --> H78 H16 --> H17 H99 -. "blocks" .-> H17 H17 --> H18 H78 --> SK H18 --> SK H102 --> SK SK --> E2E

UNIT ECONOMICS (cost-model v2 · 2026-06-02)

Blended ~$0.30/user/day. Pup free (loss-leader, ~$0.03/day) · Pro $14.99/mo ≈ $0.419/day net of Apple's 15% — average Pro costs ~$0.25/day → ~40% net margin · Pro Max $29.99 uncapped. The problem child: power Pro ~$0.53/day = negative at Pro pricing → the ADR-0051 / HEY-99 founder fork. Escalation sensitivity: 5% = baseline · 10% = +$0.10/day · 20% = margin gone — which is why the shadow-eval is load-bearing.

WHAT THE TICKET GRAPH TELLS A LEAD ENGINEER

① Contract-first is enforced, not aspirational — 73 backend tickets sit behind HEY-73/112. ② Security primitives (hooks, sanitiser, ACL dispatcher, autonomy gate) are sequenced before any mutating tool ships. ③ Eval is typed early (HEY-111) even though the judge runs last. ④ 79 of 112 tickets are ready-for-agent — spec'd to run AFK; humans are reserved for OAuth flows, physical-device tests, and decisions. ⑤ The self-evolution ambition (HEY-80–97) is parked where it belongs: behind a working loop and real traces.

THE OPEN RISKS, HONESTLY

HEY-99 is the only non-technical blocker on the spine — a founder pricing call (power Pro costs ~$0.53/day vs $0.419/day net revenue). ② HEY-5 Apple Developer enrollment gates push + TestFlight. ③ Calendar/voice (HEY-63–68) are backlogged while ADR-0040/41/42 pulled them into Phase 1 — a scheduling contradiction to reconcile. ④ CRS is validated n=1; the Fetch ships shadow-mode until the false-positive bar clears (pre-build register D3). ⑤ Coordination runs through HEY-109's baton protocol — single writer per cluster, cross-review by comment.

Reference

7 · Glossary

The same definitions the hover tooltips use.

Compiled 2026-06-03 · sources: 57-ADR set · WALDO_ARCHITECTURE_OVERVIEW · V1 master/harness/backend plans · ADR-0054/0055/0056/0057 · live Linear read (112 HeyWaldo issues) · non-authoritative visual reference per the vault's source-hierarchy rule.