SocraticPair - Collaborative Partner

The last week of August We(Supriyo Chowdhry, Lakshit Verma and Avinash) built SocraticPair for All Things Agentic Hackathon, the "Collaborative Partner" track. It's a pair-programming partner that leads the session instead of writing your code for you. It asks questions, runs tests before you write them, hands you hints when you're stuck, and remembers which concepts are weak across sessions. The repo is socratic-pair-ai, and the spec lives here.

The pitch in one line: most AI tools solve your problem and hand you code you can't explain, which is copy-paste amnesia. SocraticPair flips it. On a broken insert in a binary search tree it won't print the fix. It points at line 4 and asks what None has to do with calling .insert. You write the code, it keeps the score.

This post is the compressed story of the design and the build. The design half is spec-shaped and a little dry, so I've buried the interesting runtime stuff near the end where the actual plot is.

The problem we're solving

Three separate failure modes, all from AI tools that just hand you the answer:

1. Copy-paste amnesia. When someone writes the whole function, you skip the parts that taught you something: memory layout, runtime limits, the edge cases that make the bug real. You paste code you can't defend in a review.
2. Context amnesia. Chat windows reset every session. The tool forgets that you spent two hours on recursion yesterday, so it re-explains base cases like it's your first day.
3. Zero effort. Learning needs active problem solving. Dumping a solution skips the mental steps that turn a duplicate into a skill.

SocraticPair is a senior engineer sitting beside you who asks what you want to achieve, breaks the problem into pieces, and asks you to write each part first. It doesn't fetch you the answer, it test-drives you toward it.

The three modes

You switch teaching style with Cmd+Shift+M:

• Apprentice mode. Full Socratic guidance. The agent writes the tests first, asks guiding questions, and only gives hints. This is the default and the mode the product is built around.
• Pair mode. Equal collaboration. It reviews your code, answers syntax questions directly, and argues tradeoffs instead of steering you through them.
• Firefighting mode. Zero Socratic questions. For an outage or a tight deadline it hands you a diff patch, then logs the problem to Firestore as learning debt and books a review for the next day. You survive production, then circle back and actually learn the thing.
  
  
  
  
  When you're stuck, the hint ladder escalates in three stages. Level 1 is an observation that names the bug without praising you: "Line 4 calls .insert() on None." Level 2 is a skeleton with the missing parts marked TODO, injected as ghost text in the editor you accept with Tab. Level 3 is the full solution in a side-by-side diff editor, with time and space complexity. If you hit the bailout button or type "I give up", it jumps straight to Level 3, because at that point the goal has shifted from teaching to shipping.

A small detail I like: the hints are applied non-destructively. The agent never calls setValue() to wipe your buffer. It uses executeEdits() against a monotonic version counter, so Level 1 hints render as a view zone under the line, Level 2 as ghost text, and the test harness lives in a separate virtual model so your undo and redo history stays untouched.

The architecture

Every subsystem builds against three locked contracts, so the three of us could build in parallel without stepping on each other.

Contract 1: frontend to gateway. POST /api/socratic/turn with a session ID, user ID, mode, active file, cursor line, a code diff, and the test result. It streams back Server-Sent Events (SSE, a one-way stream from server to browser) with hint_chunk, editor_action, and done events. The done event carries the active concept and the hint level, so the client knows what just happened.

Contract 2: gateway to memory worker. Each turn is pushed onto a Cloud Tasks queue (Google's queued job delivery, with automatic retries) as a background task. The payload is the turn plus the agent's hint and the test outcome. The worker is a separate Cloud Run deployment that consumes these.

Contract 3: worker to Firestore. The worker writes to a concept DAG in Firestore (Google's cloud document database) at users/{userId}/concept_dag/{conceptId}. That document holds a Beta distribution, the mastery estimate, prerequisite edges, and a pointer to the last turn that touched it.

Which breaks down into two paths. The fast path is Gemini streaming Socratic hints back over SSE, target under 400ms. The slow path is everything that can afford latency: each turn gets queued, a memory worker on Cloud Run (serverless container host, scales to zero) runs deeper analysis on Gemma 4 hosted on Vertex AI (Google's platform for hosting models), then updates the DAG.

Concepts live as nodes in a directed acyclic graph (DAG, a graph with no loops), one per skill, with prerequisite edges. So Recursion Base Cases points at BST Insertion, which points at Tree Traversal, and knowing one depends on the one before it. When you struggle on Tree Traversal, the graph tells the agent to backtrack to the Recursion Base Cases node instead of re-teaching the same thing.

LSP context slicing

Real projects are dozens of files. Shoving a whole repo into the context window burns tokens and blurs the model's attention. SocraticPair runs a headless LSP worker (Language Server Protocol, the editor tool that gives code intelligence) that does one-hop program slicing: when Pyright reports an error in service.py:42 that references a model in models.py, it grabs only the symbol declaration and its type signature, not the whole file. The slice compresses to under 150 tokens, which is the difference between a sharp diagnosis and "here are twenty unrelated files to think about."

Cloud architecture and security

This part was honestly the most fun to think through, even though a lot of it is the kind of thing you tick a box for.

Dual-tier execution. A pre-flight parser picks where user code runs. Tier 1 is Pyodide WASM in the browser: pure Python, standard library, pre-compiled wheels, $0 server compute and zero network latency. Tier 2 is a gVisor sandbox on Cloud Run, triggered when the code imports native C-extensions or network sockets like asyncpg or redis. Tier 2 runs as a non-root user with a hard CPU limit and a blocked metadata firewall.

Prompt injection. Model Armor scans the incoming code comments and text for override patterns like # SYSTEM OVERRIDE before anything reaches the model. On the way out, a sliding window of 32 tokens sweeps the SSE stream so PII and credentials don't leak.

SSRF. Running untrusted code server-side invites server-side request forgery (SSRF, a tricked server request) against the Google metadata server at 169.254.169.254. Tier 2 blocks all egress there with an explicit iptables -A OUTPUT -d 169.254.169.254 -j DROP. One line, but it's the difference between sandbox and open door.

Streaming without drops. Cloud Run kills idle containers when it scales to zero, and WebSockets die with them. SocraticPair uses SSE over HTTP/2 instead, which is one-way anyway (the client sends edits as POSTs, the server streams responses back). If a connection drops, the browser reconnects automatically with the last event ID, so you never lose a hint mid-stream.

Firestore write limits. Firestore allows roughly one write per second per document, so writing on every keystroke would fail immediately. Edits buffer in Redis first, and a background flusher writes merged updates once every five seconds, well under the limit.

The memory engine

This is the part I owned, and it's the most interesting piece. Each concept carries a Beta distribution that says how well you remember it. Expected mastery is just the mean of that distribution. New evidence nudges alpha or beta up depending on whether you got the attempt right or wrong, so a correct attempt and a failed attempt move the score in opposite directions.

Two details keep it from being a naive counter.

First, decay. Your evidence counts shrink toward a neutral prior with a half-life of roughly two weeks. Discrete BKT (Bayesian knowledge tracing, a model that scores concept mastery) breaks when sessions are days apart, because a "correct" answer from last week gets the same weight as one from this second. The exponential decay means if you don't touch tree_insert for three weeks, your mastery drifts down instead of clipping at a boundary. People forget things; the model should too.

Second, slip and guess. Slip probability (0.2) and guess probability (0.1) absorb single-keystroke typos. Without them, one fat-fingered insert would drag your mastery down in a way that isn't a real signal about you. Errors that span multiple concepts apply fractional blame across the co-occurring DAG nodes, so the model doesn't decide the whole problem is tree_insert when the real issue was one bad base case upstream.

Two consecutive unassisted implementations graduate a concept to mastered, past a mastery of 0.85. And the concept score is decoupled from the episodic session log, so a temporary preference (you wrote it with List this week) can't corrupt the permanent skill record.

The sneaky bug here is idempotency. Cloud Tasks retries a job when a worker dies mid-request. If the retry re-runs the update, your score takes the hit twice. So every turn is recorded in a processed_turns document, and the whole update runs inside a Firestore transaction that bails the moment it sees a turn it already handled. Idempotent (running the same turn twice changes nothing extra). It's the kind of correctness detail that nobody thanks you for until the worker crashes at 3am and the retry doesn't silently break everyone's skill scores.

There's also the learning debt side. In firefighting mode, when you take the quick fix, the agent writes a learning_debt document in Firestore with the topic, the exact diff, and a debrief time for tomorrow. The shortcut isn't lost, it's scheduled back up.

The build got real

Now the actual plot. I built this with an agent doing a meaningful chunk of the heavy lifting, and a few things went sideways in ways that made the project's whole point very funny.

The framework's streaming mode crashed. Google ADK (their Python framework for building agents) has a Runner that streams responses as SSE. On this workload it raised a context-variable telemetry error. The workaround was to drop the Runner entirely for the fast path and stream directly from the genai client. The ADK agent still defines the model, the LSP tool, and the turn-complete hook, but the tokens come straight from the client. It was a documented deviation, and it works.

The leak guard caught the exact bug the project exists to stop. SocraticPair has a guard that refuses to let a Level 1 or Level 2 hint sneak a full solution through. I deployed, hit it with a real request, and watched the model stream out a complete def insert(root, val): ... return root as a "hint". The leak guard slammed it down to # TODO: implement the rest. That's the failure mode we built the whole product to prevent, happening live, on the thing we shipped. I can't overstate how much I enjoyed that.

The 400ms SLA was never going to happen. The spec promises sub-400ms first-token latency. The enterprise model's time-to-first-byte was about six seconds. Streaming helps you perceive it as tokens arriving, but it can't fix the model's own startup. I'd rather say that plainly than pretend we hit a number the model was never going to allow. This is the gap between a spec that looks great in a table and the latency your model actually delivers.

A version of the streaming first had a bug where the model call failed fast and returned a 0.7s "fallback hint" instead of a real answer, which looked amazing in a benchmark and was actually the error path. The first-pass streaming was also not leak-safe, because it streamed the whole raw answer incrementally and then corrected it at the end, which means the client had already seen the leak by the time the guard clamped it. That's the tension with streaming: you can't stream tokens live and guarantee no full-solution leak, because you only know it's a leak once you've seen the whole block. It's a genuine design tension, not a sloppy one.

What shipped

The gateway is live on Cloud Run, and the frontend is live on Vercel. I verified the whole path end to end: a browser-style cross-origin POST from the Vercel domain hits the gateway, CORS is open, the guest token works, and it streams hint_chunk -> editor_action -> done with a fresh turn_<uuid> and the right hint level. I also confirmed the built client bundle actually points at the real gateway instead of falling back to a mock route.

Two parts did not ship, and it's worth being honest about that rather than burying it. The memory worker and the Gemma 4 integration never deployed, because there's no Gemma 4 model registered in the Vertex project and no API key to register one. So the "290ms first-token, 82% cheaper from offloading to Gemma" figures in the spec are design targets, not numbers we measured on the live thing. The concept DAG and the BKT engine are real code with real tests, but they're not running against a live model fight. We reported the blocked layers with the exact commands to finish them, and left them for whoever picks it up next.

A detail worth noting: the live gateway actually ran gemini-3.5-flash, not the gemini-3.7-flash the spec names. The Dual-model economics in the spec are real in principle, but the shipped thing used what the project had access to. Specs drift the moment you start building.

For reference, the numbers the spec claims for going dual-model: first token from 1,850ms down to 290ms, memory extraction from $0.0035 to $0.0006 per turn (82% cheaper), and a 50-turn session from $0.22 down to $0.04. Those are the ambition, not the measurement.

Takeaways

The funniest lesson is the unintended one: we built a tool that stops AI from dumping full answers, and the first thing our own model did was try to dump a full answer. The leak guard was the actual product, proving itself in production on day one.

The practical lesson mirrors what I keep relearning on the FLOSS project: the model you budget for isn't always the model you deploy, the SLA you promise needs to be measured against real first-token latency, and honest "this didn't ship" beats a spec that looks finished. Streaming is genuinely valuable, but it doesn't make a slow model fast, and you can't have both live token streaming and a guarantee against leaking a solution at the same time.

Three of us, one weekend, a working fast path, a concept DAG that's real code, and a memory engine that's ready for a model that exists. The Collaborative Partner category is a good fit, because the product is about the relationship. The model is a teacher, not a typing assistant, and it holds that line even when it has to police itself to do it.