Quantago — Feature Request: AI Memory & Cost Efficiency

Universal Reason LLC · May 2026 · Internal Engineering

Executive Summary

Quantago's conversational AI (strategy discussion, code updates with Joco) currently has no memory between sessions. Every conversation restart re-sends the full context from scratch, paying full input token prices on every turn. This feature request defines three complementary improvements that together reduce AI inference costs by an estimated 60–80% while making the assistant significantly more useful across sessions.

Problem: Users talk with the AI for 2–30 minutes, leave, return hours later. On return: full conversation history re-sent at full token price — no caching benefit. No continuity between sessions: AI has no memory of past strategy decisions. Prompt caching is not yet implemented. Estimated current cost: $30–60/month at typical usage intensity.

Feature 1: Two-Layer Conversation Memory

Overview

Replace raw conversation history re-transmission with a two-layer memory architecture: a permanent compressed summary stored in Neon, and a recent message buffer in Cloudflare KV that stays warm within the model's cache TTL window.

Architecture

Layer 1 — Permanent Summary (Neon DB)

Compressed history of all past sessions for each user
Updated at end of each session or when a gap > 55 minutes is detected
Target size: 1–2k tokens regardless of full conversation length
Generated by gpt-5.4-mini (cheap summarization model)
Preserves: strategy decisions made, code changes, open tasks, current strategy state

Layer 2 — Recent Messages (Cloudflare KV)

Raw messages from the current/recent session
KV TTL set to 2 hours (buffer above model's 1-hour cache TTL)
Kept hot while user is actively coding or discussing
Cleared and folded into Layer 1 summary on gap detection

Session Lifecycle

On each API call, the system assembles context dynamically:

Step	Condition	Context Sent	Token Cost
1. Active session	Gap < 55 min	Summary + KV cache hits + new message	Near zero (cache)
2. User returns	Gap > 55 min	Summary + new message only	Summary size only (~1–2k tokens)
3. Long session	Never leaves	Grows but cached per turn	Cheap per turn
4. First message	No history yet	System prompt + new message	One-time cost

Gap Detection & Summarization

When last_active timestamp exceeds 55 minutes:

Fetch recent messages from KV
Call gpt-5.4-mini to merge into existing summary (max 1,500 token output)
Store updated summary in Neon conversation_summaries table
Clear recent messages from KV
Update last_active timestamp

Database Schema

Neon Tables Required:
  conversations          — full message history (append only, audit trail)
  conversation_summaries — compressed context per user, versioned
  strategy_snapshots     — strategy state at each version (ties to code commits)

Implementation Sketch

async function buildConversationContext(userId: string, newMessage: string) {
  const summary = await neon`
    SELECT content FROM conversation_summaries
    WHERE user_id = ${userId}
    ORDER BY created_at DESC LIMIT 1
  `
  const recentMessages = await kv.get(`session:${userId}:messages`)
  const lastActivity = await kv.get(`session:${userId}:last_active`)
  const gapMinutes = (Date.now() - lastActivity) / 1000 / 60
  const cacheWarm = gapMinutes < 55

  if (!cacheWarm && recentMessages) {
    await summarizeSession(userId, summary, recentMessages)
  }

  return buildMessages({ summary, recentMessages: cacheWarm ? recentMessages : null, newMessage })
}

async function summarizeSession(userId: string, currentSummary: string, recentMessages: string) {
  const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
  const result = await openai.chat.completions.create({
    model: "gpt-5.4-mini",
    max_tokens: 1500,
    messages: [{
      role: "user",
      content: `Update this strategy conversation summary.

Existing summary: ${currentSummary}

New conversation to incorporate:
${recentMessages}

Preserve: decisions made, code changes, open tasks, strategy state.
Be concise. Max 1500 tokens.`
    }]
  })

  await neon`
    INSERT INTO conversation_summaries (user_id, content, created_at)
    VALUES (${userId}, ${result.choices[0].message.content}, NOW())
  `
  await kv.delete(`session:${userId}:messages`)
}

Feature 2: Prompt Caching Implementation

Overview

Anthropic's prompt caching allows stable content to be stored server-side and billed at ~10% of normal input token price on cache hits. This is currently not implemented in Quantago — every API call pays full input price for the entire context including the system prompt, strategy rules, and conversation history that hasn't changed since the last call.

What Needs to Be Built

Add cache_control: { type: "ephemeral" } breakpoints to the Anthropic API message assembly
Ensure the Anthropic client is initialised with the direct API (not Bedrock — caching is not available on Bedrock)
Validate cache hit rates via Anthropic's usage response fields (cache_creation_input_tokens, cache_read_input_tokens)

Cache Hit Validation

const response = await anthropic.messages.create({ ... })

// Log cache performance
console.log({
  inputTokens: response.usage.input_tokens,
  cacheCreated: response.usage.cache_creation_input_tokens,
  cacheRead: response.usage.cache_read_input_tokens,
})
// cache_read_input_tokens > 0 = cache hit, billed at ~10% price
// cache_creation_input_tokens > 0 = first call, full price, cached for next

Feature 3: Structured Prompt Cache Breakpoints

Overview

Implement explicit Anthropic cache_control breakpoints on every stable context layer. This ensures repeated content (strategy constitution, session summary, recent history) is billed at cache-read rates (~10% of input price) rather than full input price on every API call.

Constraint: Anthropic supports a maximum of 4 cache breakpoints per request.

Cache Breakpoint Allocation

Breakpoint	Content	Stability	Est. Size
#1	Strategy constitution (rules, risk params, prop firm constraints)	Permanent	~3–5k tokens
#2	Permanent summary from Neon (Layer 1)	Per session end	~1–2k tokens
#3	Recent messages from KV (Layer 2)	Within 55 min window	~5–20k tokens
#4	Reserved	Future use	—

Cache TTL Alignment

Anthropic prompt cache TTL: 5 minutes (resets on each API call)
For active sessions with turns < 5 min apart: cache stays continuously warm
For gaps > 5 min but < 55 min: KV has messages but cache may miss — re-caches on next call
For gaps > 55 min: summarize and start fresh — cache TTL irrelevant
Note: If using DeepSeek direct API instead, TTL is 1 hour — same architecture applies

Implementation Sketch

function buildMessages({ summary, recentMessages, newMessage }) {
  return [
    // Breakpoint 1: strategy constitution — never changes
    {
      role: "user",
      content: [{
        type: "text",
        text: QUANTAGO_SYSTEM_PROMPT,
        cache_control: { type: "ephemeral" }
      }]
    },
    // Breakpoint 2: permanent summary — changes per session end
    {
      role: "user",
      content: [{
        type: "text",
        text: summary.content,
        cache_control: { type: "ephemeral" }
      }]
    },
    // Breakpoint 3: recent messages — warm if < 55 min
    ...(recentMessages ? [{
      role: "user",
      content: [{
        type: "text",
        text: recentMessages,
        cache_control: { type: "ephemeral" }
      }]
    }] : []),
    // Live tail: no cache_control
    {
      role: "user",
      content: [{ type: "text", text: newMessage }]
    }
  ]
}

Implementation Notes

cache_control: { type: 'ephemeral' } added to each stable layer in the messages array
Live tail (current user message) always sent without cache_control
Completed turns get cache breakpoints added retroactively before next call goes out
Compatible with claude-sonnet-4-6 and claude-haiku-4-5 for main conversation; gpt-5.4-mini for summarization (OpenAI API)
Not compatible with Bedrock — keep AI calls on direct Anthropic API

Cost Impact Analysis

Scenario	Current (no caching)	After Implementation	Saving
Active session, 20 turns	$0.80–$1.60	$0.08–$0.16	~90%
User returns after 2hr gap	$0.40–$0.80 (full history)	$0.02–$0.05 (summary only)	~95%
Monthly (typical Joco usage)	$30–$60	$5–$12	~75–80%
Summarization overhead	—	$0.20–$0.50/mo (gpt-5.4-mini calls)	Net positive

Implementation Plan

Phase 1 — Database & KV Setup

Add conversation_summaries table to Neon via Drizzle migration
Add strategy_snapshots table
Set up Cloudflare KV namespace: QUANTAGO_SESSION_CACHE
Define KV key schema: session:{userId}:messages and session:{userId}:last_active

Phase 2 — Prompt Caching Implementation

Verify AI calls are on direct Anthropic API (not Bedrock)
Add cache_control: { type: "ephemeral" } to system prompt in message assembly
Log cache_creation_input_tokens and cache_read_input_tokens from API responses
Validate cache hit rate over first week of usage

Phase 3 — Context Assembly Service

Build buildConversationContext(userId, newMessage) function
Implement gap detection logic (55-minute threshold)
Wire full cache breakpoint stack (Feature 3) into message assembly
Add last_active timestamp update on every API call

Phase 4 — Summarization Pipeline

Add OpenAI SDK dependency (openai npm package)
Build summarizeSession(userId) function using gpt-5.4-mini
Trigger on gap detection and on explicit session end
Store versioned summaries in Neon (keep last 5 per user)
Summary prompt must preserve: decisions, code changes, open tasks, strategy state

Phase 5 — Testing & Validation

Unit test gap detection edge cases (exactly 55 min, 54 min, 56 min)
Integration test cache breakpoint assembly with Anthropic API
Cost validation: compare token bills before/after on identical sessions
Regression test: summary quality — Joco should not need to re-explain past decisions

Dependencies & Constraints

Item	Detail
Anthropic API	Direct API only (not Bedrock) — required for prompt caching
Cache breakpoints	Max 4 per request — allocation defined above
Cloudflare KV	Already in stack — new namespace required
Neon / Drizzle	Already in stack — 2 new tables required
OpenAI SDK	`npm install openai` — required for gpt-5.4-mini summarization
gpt-5.4-mini	OpenAI API key required — summarization only, not main conversation
Out of scope	Trading decision AI — separate system, not affected

Quantago · Universal Reason LLC · Confidential · May 2026

Executive Summary​

Feature 1: Two-Layer Conversation Memory​

Overview​

Architecture​

Session Lifecycle​

Gap Detection & Summarization​

Database Schema​

Implementation Sketch​

Feature 2: Prompt Caching Implementation​

Overview​

What Needs to Be Built​

Cache Hit Validation​

Feature 3: Structured Prompt Cache Breakpoints​

Overview​

Cache Breakpoint Allocation​

Cache TTL Alignment​

Implementation Sketch​

Implementation Notes​

Cost Impact Analysis​

Implementation Plan​

Phase 1 — Database & KV Setup​

Phase 2 — Prompt Caching Implementation​

Phase 3 — Context Assembly Service​

Phase 4 — Summarization Pipeline​

Phase 5 — Testing & Validation​

Dependencies & Constraints​

Executive Summary

Feature 1: Two-Layer Conversation Memory

Overview

Architecture

Session Lifecycle

Gap Detection & Summarization

Database Schema

Implementation Sketch

Feature 2: Prompt Caching Implementation

Overview

What Needs to Be Built

Cache Hit Validation

Feature 3: Structured Prompt Cache Breakpoints

Overview

Cache Breakpoint Allocation

Cache TTL Alignment

Implementation Sketch

Implementation Notes

Cost Impact Analysis

Implementation Plan

Phase 1 — Database & KV Setup

Phase 2 — Prompt Caching Implementation

Phase 3 — Context Assembly Service

Phase 4 — Summarization Pipeline

Phase 5 — Testing & Validation

Dependencies & Constraints