# Boris Tane - Full Content > Software engineer leading the Workers observability team at Cloudflare. Former founder of Baselime (acquired by Cloudflare). ## About Boris Tane is a software engineer specializing in observability, serverless computing, and distributed systems. He currently leads the Workers observability team at Cloudflare, building tools to help developers understand and debug their applications. Previously, Boris founded Baselime, a serverless observability platform that was acquired by Cloudflare in 2024. ## Contact & Social - Website: https://boristane.com - bsky: https://bsky.app/profile/boristane.bsky.social - github: https://github.com/boristane - linkedin: https://www.linkedin.com/in/boristane/ - x (twitter): https://x.com/boristane ## Blog Posts (Full Content) ### Self-improving software is inevitable URL: https://boristane.com/blog/self-improving-software-is-inevitable/ Description: Production tells us exactly how to make our software better, we've just never had the tools to run this on auto-pilot. The release you shipped this morning already surfaced what's wrong with it. It's been telling you all day, across multiple surfaces. The only reason prod is still failing is you. Agents write the vast majority of software now, they can one-shot an application, deploy it, and wire up its observability before your coffee gets cold. But it's a one way street. Agents produce more software, that software produces more signal, and almost none of that signal ever makes its way back to the agents. We are the ones bringing this data back into agents manually. ```mermaid graph TD A[Agents write code] --> B[Application in production] B --> C[Signals: logs, traces, analytics, replays, tickets, reviews] C --> D[Humans read, correlate, prioritise] D -->|Manually| A style D fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` We have built tooling to help us query, visualise, organise, triage and prioritise this signal. I was part of this, I spent the best part of the past decade building observability tooling. But ultimately, it still takes someone to read the data and decide what needs to be improved. **Self-improving software is inevitable**. Applications that read their own signals, decide what to change, ship the change, and check whether it worked. And once software can improve itself, it can improve the machinery that improves it: better observability, sharper detection, deeper investigations, better fixes, each loop tightening the next one. For that to happen, we have to give up being in charge. Engineers, PMs, designers, all of us. We have to stop being the meat proxies who decide what gets fixed next, and when. ## Right now, people are the feedback loop Every source of signal has its own tool, engineers and SREs use Datadog, product managers are in Amplitude, designers in FullStory, support in Zendesk, etc. When users face issues in prod, the errors get recorded in the observability tooling, the rage clicks saved in the session replays, there's a dip in product metrics, and an increase in support tickets. A PM needs to notice all these, investigate and correlate the various signals, compress this information into a ticket, and prioritise it for next week. Then it lands on the desk of an engineer who prompts an agent to turn it into code. This makes no sense in 2026 when agents are intelligent enough to solve some of the hardest maths problems humanity ever faced. It takes very little intelligence to parse errors and prioritise them. ## Are we all meat proxies? When you take a screenshot of your product analytics dashboard, paste it into Claude Code and prompt "investigate why this chart is going down and fix it", you're a meat proxy. A meat proxy is a person whose only job is to carry information between two machines that could perfectly well talk to each other. The dashboard has the data, and the agent has the intelligence. You sit in the middle, deciding what to look at, when to look at it, and how to describe it, that's a meat proxy. ```mermaid graph TD A[Product analytics] --> B[You: screenshot, paste, prompt] B --> C[Coding agent] C --> D[Code] A -.->|Could have read it directly| C style B fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` MCPs and connectors make the proxy faster, but you're still the proxy. ## Automations have an unknown unknowns problem There's a lot of agent tooling with automations, I even built one myself: they are essentially cron-jobs and webhooks that trigger an agent with a given prompt: > listen to Honeycomb alerts and investigate every time one fires > every time sentry opens a new issue, investigate it and open a PR with a fix > every morning, check our product metrics, if activation falls below 10% investigate and write a fix This feels incredibly productive. Every alert gets a write-up before anyone opens their laptop, every new exception gets a pull request. But these automations only ever see what someone already decided was worth seeing. They inherit every blind spot of the system that triggers them. Someone had to decide what to alert on, what SLOs to define, and these inherently are incomplete. It's impossible to alert on the emergent behaviour of a complex application. That's the unknown unknowns problem. Known problems get detected, because someone already wrote the alert or the exception handler. Unknown problems never get detected, because nothing exists to detect them. The agent becomes incredibly fast at fixing the problems you already knew how to find, and the problems costing you the most customers are, almost by definition, the ones you didn't. ## Applications need backpropagation Think of a piece of software as a neural network. The code is the weights and biases, and every change to the code is an update to those weights. Users interact with it, and the signals they produce (traces, clicks, tickets, churn, revenue, etc.) are the output. With this analogy, the two stages of self-improving software are: - The **forward pass**: intent goes in, code comes out. Coding agents have "solved" this phase of the training iteration. We can one-shot fairly complex applications with a single prompt, and even deploy them and connect all the necessary observability and product tooling. - The **backpropagation**: compare the output to what you wanted, compute the gap, and push it back through the system to adjust the weights. For an application, that's taking all the signal it produces in prod, figuring out what it says about the code, and changing the code accordingly. ```mermaid graph TD A[Intent] --> B[Agents write code] B --> C[Production] C --> D[Users] D --> E[Signals: traces, analytics, replays, tickets, interviews] E --> F[Humans] F --> A style B fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style F fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` We've automated the forward pass and kept the backprop manual. Every week we push dozens of weight updates into production, and compute the gradient at a planning session, at best weekly, at some companies quarterly, by hand. This planning is based on the memory of the PMs, a sample of twenty session recordings and whichever three customer complaints were loudest. If we were training models like this, we wouldn't have anything worth anyone's time. ## Intelligence is now too cheap to meter In this analogy, backpropagation is manual for one reason: nothing else can read an error log, find the corresponding set of traces, cross-reference them with session replays, understand why a customer was angry, and decide what to change. That took judgement, and good judgement was scarce and expensive. That's no longer the case. We now have intelligence too cheap to meter. It can read every trace, watch every session, read every ticket and every interview transcript, and join them together in minutes. We can put intelligence both in the forward pass and in the backprop, increase iteration speed, and achieve self-improving software. ```mermaid graph TD A[Intent and fitness functions] --> B[Intelligence] B -->|Forward pass| C[Code changes] C --> D[Production] D --> E[Signals] E -->|Backprop| B style B fill:#dbeafe,stroke:#93c5fd,color:#1e40af style C fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style A fill:#ede9fe,stroke:#c4b5fd,color:#5b21b6 ``` ## What's our role then? A neural net without a loss function is just a very expensive random number generator. Backprop needs to know what "better" means. In this analogy, our job becomes defining what "better" means. Engineers, product managers, designers, sales, support: everyone's role becomes defining the fitness functions the system optimises against. | Who | What they optimise | Fitness functions | |---|---|---| | Engineers | The health of the system | p99 latency, error rates, availability, cost per request, cold start times | | Product | The behaviour of users | Conversion, activation, retention, time to value, feature adoption | | Designers | The user experience | Task completion, time on task, rage clicks, drop-offs, abandoned forms | | Sales | Revenue | Growth, win rates, expansion, deal velocity, churn | | Support | How much the product hurts | Ticket volume, time to resolution, CSAT, duplicate tickets | ```mermaid graph TD A[Engineering, product, design, sales, support] --> F[Fitness functions] F --> G[Intelligence computes the gap] G --> H[Code changes] H --> I[Signals] I --> G style F fill:#ede9fe,stroke:#c4b5fd,color:#5b21b6 style G fill:#dbeafe,stroke:#93c5fd,color:#1e40af ``` The challenge is that these functions often pull against each other. A faster checkout can hurt fraud detection, a simpler onboarding can lower activation for power users, a discount can grow revenue and kill margins, etc. Deciding how to weigh them, which ones are hard constraints, and which customers you're optimising for is the hard part. Ultimately, it's deciding what the company is trying to achieve, and writing it down precisely enough that a machine can optimise for it. And honestly, it was always the hard part. ## An era of abundance ahead Give a cloud coding agent read access to all your signal sources at once (observability, product analytics, session replays, support tickets) and write access to your repo. Then put these on a schedule. > every hour, read every new error, trace, session replay and support ticket from the last hour. group anything that looks like the same problem, even if it shows up in different tools. for each group, find the release that introduced it and open a PR with the fix and all the evidence attached > every morning, compare yesterday against these goals: checkout p99 under 300ms, error rate under 0.1%, activation above 25%, fewer than 20 support tickets a day. for anything moving in the wrong direction, find the cause in the code and open a PR. if everything is on track, find the change most likely to move these numbers next and open a PR for it We're so close. Self-improving software is inevitable, and for the first time, it's close enough to build. --- ### On-Call Is Now Theatre URL: https://boristane.com/blog/on-call-is-now-theatre/ Description: On-call was calibrated to the cost of waking up a human. That cost just went to zero. Self-operating software is coming, and your rotation is the last thing standing in its way. Your on-call rotation has always been an admission of defeat. Every alert threshold, every escalation policy, every runbook is the same admission written in YAML: software cannot be trusted to run in production, so somebody must be strapped to it at all times. But today agents write most of the code. Engineers spend their time designing, prompting, reviewing, and creating loops to continuously produce more and more software. Building software has been transformed beyond recognition in under three years. Operating it hasn't moved an inch. The rotation, the pager, the dashboards, the rituals: all exactly where we left them, guarding a world that no longer exists. ## We've all lived through terrible on-call rotations You get paged when you're having dinner on a beautiful Saturday. You log in, ask your agent what's up, open four dashboards, cross-reference a deploy timeline, and six "what the heck is this?" later, you conclude it's probably not that big of a deal. You ack the alert and go back to your now-cold meal. Nothing about that page actually needed *you*. It needed someone who could pull up telemetry, reason about it, and decide nothing was on fire. For the entire history of software, that someone had to be a person. Everything about how we run production is downstream of that one fact. ## The core issue: everything was calibrated to human attention Everything about on-call and observability is built around one core principle: human attention is a scarce resource. Every alert threshold is tuned around the cost of waking up an engineer. We have always treated alert fatigue as if it was a bug in the system we could fix with better alert thresholds or SLOs. But alert fatigue *is* the system: a rationing mechanism for eyeballs. Metrics exist to compress millions of datapoints into something a human can absorb in a glance. Runbooks exist to digest the knowledge of an expert in the system into a document anyone can follow. The on-call rotation itself exists to distribute suffering across a team. When the only thing that can investigate an incident is an engineer, you design everything (thresholds, tooling, org charts, etc.) around protecting and rationing human attention. That fundamental constraint is now obsolete. ## And yet, we're still doing it You already let agents write the vast majority of your code. Your team probably ships dozens of PRs a day; I've seen teams where a single engineer ships 10+ PRs a day. We've industrialised the production of software with "software factories". But we're still responsible for ensuring all this software runs in production, equipped with dozens of MCPs and poorly written skills. We automated the cause and kept the cure handmade. The teams sprinting ahead with coding agents are quietly discovering that their velocity is capped not by how fast they can build, but by how fast a human can figure out what broke. ```mermaid graph TD A[Agents ship at machine speed] --> B[Production changes constantly] B --> C[Something breaks] C --> D[Page a human] D --> E[Human greps, correlates, guesses] E --> F[Hours of attention per incident] F --> G[Velocity capped by incident response] style D fill:#fee2e2,stroke:#fca5a5,color:#991b1b style G fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` Every gain in build velocity converts directly into operational debt, until PagerDuty wins and the shipping stops. Self-operating software is the next frontier for software engineering. ## Self-operating software We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled. Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out. This flips the economics of what to monitor. Your thresholds are conservative because paging an engineer is expensive. If paging has near-zero marginal cost, you don't want fewer alerts, you want *dramatically more*. You borderline want your agent to read every single log line and figure out all errors and unexpected paths in real-time, as requests are flowing through your systems. Monitor the p99 that crept up 3%, the queue depth that's slightly off its weekly pattern, the error rate that's fine but different. All the weak signals you convinced yourself are not worth monitoring usually turn into pages when it's too late. A friend at a lab put it extremely clearly to me recently: > "It feels like it's going to become a non-negotiable to have harnesses programmatically access cell data, alerts, metrics, traces and logs with full support. Investigations and operations are night and day when these things are exposed." The same shift that happened to code generation is happening to incident response: engineers move from doing the work to judging the work. The loop looks like this: ```mermaid graph TD A[Telemetry, deploys, infra state] --> B[Detection: thousands of cheap checks] B --> C[Issue raised] C --> D[Agent triages] D -->|False alarm| E[Closed, with reasoning attached] D -->|Real| F[Agent investigates: parallel hypotheses] F --> G{Can it fix it?} G -->|Yes| H[Automation or pull request] G -->|No| I[Page an engineer, evidence assembled] H --> A style H fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style I fill:#ede9fe,stroke:#c4b5fd,color:#5b21b6 ``` However, none of this works if the agent can't see. Self-operating software needs programmatic access to everything a senior engineer would look at during an incident: metrics, logs, traces, alerts, SLOs, deploy history, infra state, service ownership, the code itself, and, critically, how all of it connects in a single operations graph. Without it, every investigation dead-ends in a Slack message that reads "something looks off", forcing an engineer to start digging again. ## The loop closes with a pull request A triage that ends in a Slack summary is merely a nicely formatted prompt to an engineer. Agents should not prompt us. The only valid output of an investigation is a diff. When the system traces an incident to its cause, it should write the fix and open the pull request itself, with the entire causal chain attached, receipts included, so every claim can be audited. The agent should tell you: here's what broke, here's the evidence, here's the fix, here's why it's safe. Your job is to say yes or no. Judgement, not archaeology. ## The first line of defense moves to the pull request And the loop must run backwards too. The cheapest incident is the one that never ships. The system must interrogate every change *before it merges*. Every pull request should trigger the same machinery as an incident, pointed forwards instead of backwards. The agent reads the diff and forms multiple hypotheses about how the change could hurt production. Does this migration lock a table with live writes? Does this touch a delivery path that's serving traffic right now? Does deploy ordering matter here? What did that dependency bump actually change, and how old is the release? Then it tries to confirm or refute each hypothesis against the real system: live telemetry, actual deploy topology, the current shape of traffic. An agent can test every hypothesis, on every change, every time, and never gets tired of it. The best investigation is the one that ends before the incident begins. ## Most teams won't do this What's preventing most teams from fully embracing this way of working is trust. Letting an agent triage production incidents feels reckless the same way agents pushing PRs felt reckless twelve months ago. This trust requires admitting that the rotation, the thresholds, the dashboards, things we have built our identities around, were just rationing mechanisms for scarce attention. That attention is now abundant, and we have intelligence sometimes too cheap to meter. That's an uncomfortable thing to admit about an expertise built over years. It was just as uncomfortable when it was writing code not too long ago. Most teams will take the easy route: an AI summary at the top of the PagerDuty incident, a chatbot in the incident channel, MCPs on engineers' laptops, and call it "transformation". The ritual survives, an engineer still wakes up, still triages, still wrangles MCPs. The theatre gets slightly better lighting. The teams that actually invert the rotation will look reckless right up until they look inevitable. They'll run thousands of checks where you run fifty. They'll catch the 3% regression you'd have noticed in next quarter's cloud bill. Their engineers will sleep, and spend their attention on problems a machine genuinely cannot crack yet. ## Start this week You don't need to buy anything or rearchitect anything to start. 1. Pick your noisiest alert, the one everyone acks without reading. 2. Point it at a cloud agent. 3. Give the agent read access to your observability tool. Logs, metrics, the deploy timeline. 4. Every time it fires, have the agent post its triage to Slack before any engineer looks at it. 5. For two weeks, compare. Count how many times you concluded anything the agent didn't. Now imagine an agent coming up with alerts, analysing the hidden paths in your codebase, continuously updating its understanding of production with specialised tools, and fixing issues before they become problems. Software that writes itself was the first half. Software that operates itself is what's ahead of us. And most of the industry is still arguing about whether we should read code or not. --- ### Slop Creep: The Great Enshittification of Software URL: https://boristane.com/blog/slop-creep-enshittification-of-software/ Description: Coding agents didn't make poor engineers dangerous. They made them unstoppable. Slop creep is the slow, invisible enshittification of a codebase through an accumulation of individually reasonable but collectively destructive decisions, each one too small to flag, too numerous to track, and too deeply buried to unwind by the time you notice. Picture a codebase six months from now. Every feature shipped on time, every PR passed review. But the team is slower than ever, and constantly firefighting. Every new feature touches 10s of files. There are six different ways to do the same thing. The data models have fields that exist purely to work around a limitation that was introduced earlier and never revisited. The on-call rotation is a nightmare because nobody can trace the flow of a request through the system anymore. I've done this to myself. When building [Baselime](https://baselime.io), I made every mistake in the books: premature microservices, poor and impossible to extend schemas, leaky abstraction, etc. I shipped fast, made "pragmatic" architectural calls, and watched the codebase calcify around them. That used to take months of compounding mistakes to reach crisis point. Last year I got a coding agent, and my side project reached crisis point in days. ## Every change is fine, but the codebase is not This is the insidious thing about slop creep. No single commit is the problem. The agent didn't introduce a bug, it introduced a slightly wrong abstraction, then built on top of it, then built on top of *that*. ```mermaid graph TD A[Feature Request] --> B[Agent writes solution] B --> C[Looks fine in isolation] C --> D[Ships] D --> E[Next feature request] E --> B D --> F[πŸ’€ Codebase decays] style F fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` Two weeks later, unwinding it means fiddling with databases in production and rewriting three services. That's slop creep: death by a thousand reasonable decisions. ## The old world had a circuit breaker Bad architectural decisions used to come one at the time. A junior developer with bad instincts had a natural speed limit: they couldn't build fast enough to bury the codebase before someone noticed. The pain was loud and unavoidable, so you fixed it or you died. ```mermaid graph TD A[Bad abstraction] --> B[Slows everything down] B --> C[Team feels the pain] C --> D[Forced to fix it] style C fill:#fee2e2,stroke:#fca5a5,color:#991b1b style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46 ``` Coding agents removed the circuit breaker. The agent can keep piling crap on top of more crap, indefinitely, and stay productive the entire time. You don't slow down. You build a bigger pile. The reckoning is deferred, compounded, and much more painful when it finally arrives. ## Coding agents can't think about systems holistically The core problem is simple: coding agents are not able to find the right level of abstraction when building software. They don't see the system, they see the prompt. Ask an agent to add an endpoint and it adds an endpoint. It doesn't know there are four other endpoints that should have been abstracted into a shared handler, or that the data model it's extending was already a mistake. It doesn't know any of that because no one told it. ```mermaid graph TD A[Agent sees: the prompt] --> B[Agent builds: the solution] B --> C[Correct in isolation] B --> D[Wrong for the system] style C fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style D fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` The agent is confidently, competently wrong. You must specifically tell it the important decisions in your codebase. ### Will this get better? Maybe. Context windows are growing, models are getting smarter, and the answer is probably more context. The agent that can read the entire codebase, understand the history of every decision, and anticipate where the system is heading in six months will make far fewer wrong calls. But today's agents don't do that. They see the prompt, the files they're told to read, and nothing else. They have no foresight about where the system needs to go, and no memory of how it got here. Until that changes, the gap between "correct in isolation" and "right for the system" is yours to fill. ## The agent should fill the gaps, not make the calls A coding agent can turn a 10x engineer into a 100x engineer, but that doesn't mean the engineer disappears. It means the engineer stops typing and **starts thinking**. It's unfortunate we're collectively using these tools very wrong. Outputting slop daily, turning our brains off because the computer "can do our job for us". Data models, service boundaries, key abstractions, etc. These are one-way doors, decisions that are hard or impossible to reverse once they're in production. The agent should never be the one walking through a one-way door alone. That's where you come in. This doesn't mean you dictate every schema and interface upfront. The agent's first draft of a plan is a starting point, and it's usually terrible. Wrong cardinality, missing constraints, boolean fields where you need an enum, no thought given to how the data will be queried six months from now. But that first draft is incredibly useful as something to react to. You read it, tear it apart, annotate it with corrections, and send the agent back to revise. After two or three rounds of this, you end up with abstractions that are better than what either of you would have produced alone, because you're combining the agent's breadth of knowledge with your understanding of the system and the product. I wrote about [how I use Claude Code](/blog/how-i-use-claude-code) with a research-plan-implement workflow built around exactly this kind of iterative refinement. ```mermaid graph TD A[Engineer defines intent] --> B[Agent drafts plan with code snippets] B --> C[First draft is usually wrong] C --> D[Engineer reviews and annotates] D --> E[Agent revises] E --> F{Good enough?} F -->|No| D F -->|Yes| G[Agent implements] style C fill:#fee2e2,stroke:#fca5a5,color:#991b1b style D fill:#ede9fe,stroke:#c4b5fd,color:#5b21b6 style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46 ``` I don't write code anymore, but I read pretty much every single line my agent writes. Every time I have let the agent loose without a tight plan, I have regretted it a couple of weeks later, always the same things: bad database schemas, boolean fields everywhere, lack of analytics, and don't get me started on observability. ## The answer is not to stop using agents Slop creep is real, but it is not a reason to stop using coding agents. They're the best thing that has happened to software development in years, and I can now build things I didn't know I had in me to build. I have no intention of going back to typing every character of my codebase myself. But the planning phase deserves ten times more attention than most people give it. Not a vague description of the feature. Actual code snippets for the key data models. Actual interfaces for the key abstractions. Enough that the agent can't get the important stuff wrong, because there's no ambiguity left to fill with slop. ```mermaid graph TD A[Vague plan] --> B[Agent makes architectural decisions] B --> C[Slop creep] D[Tight plan with code snippets] --> E[Agent executes within constraints] E --> F[Clean codebase] style C fill:#fee2e2,stroke:#fca5a5,color:#991b1b style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46 ``` Spend more time in the plan. Write the code snippets for the decisions that matter. Then let the agent cook. ## "Code is the new assembly" There's a popular counterargument: none of this matters. Code is the new assembly language. Nobody reads it. The agent writes it, the tests pass, the feature works, who cares if the internals are ugly? This misunderstands what a compiler does. The role of a compiler is to translate higher-level languages into **efficient, performant** machine code. A compiler that produced bloated, redundant, subtly incorrect machine code would not be used by anyone serious about building software. If your agent is the compiler, and the output is slop, you don't have a compiler. You have a liability. ```mermaid graph TD E[Good agent workflow] --> F[Clean, maintainable code] G[Vibe coding] --> H[Slop that ships] style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style H fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` And the enshittification doesn't stay in the code, it spills into the user experience. Every vibe-coded app has the same telltale signs, a plethora of tiny cuts that make the whole thing feel *off*: - An app that hogs all the available resources on your computer - A button that doesn't quite disable when it should - A loading state that flickers - A form that loses your input on navigation - An error message that says "something went wrong" because nobody modelled the failure modes None of these show up in a demo. All of them are bugs your users feel every single day. Slop creep in the codebase becomes slop creep in the product, and your users will not read your code to figure out why the experience is bad. They'll just leave. ## The job has changed, not disappeared Honestly, everything in this post could be obsolete with the next major model release. An agent that truly understands systems holistically, that has the foresight to see where a codebase is heading and the context to know why it got here, would change the equation entirely. But that's not what we have today. And in the next 6 to 12 months, the engineer who makes the difference is the one who can look at an agent's output and say "this is wrong for reasons you can't see from where you're standing." The one who knows which doors are one-way, which abstractions will calcify, and which corners will cost you later. Until the models catch up, fight slop creep. The enshittification of software is quiet, entirely preventable, and happening everywhere right now. --- ### The Software Development Lifecycle Is Dead URL: https://boristane.com/blog/the-software-development-lifecycle-is-dead/ Description: AI agents didn't make the SDLC faster. They killed it. All that's left is context. AI agents didn't make the SDLC faster. They killed it. I keep hearing people talk about AI as a "10x developer tool." That framing is wrong. It assumes the workflow stays the same and the speed goes up. That's not what's happening. The entire lifecycle, the one we've built careers around, the one that spawned a multi-billion dollar tooling industry, is collapsing in on itself. And most people haven't noticed yet. ## The SDLC you learned is a relic Here's the classic software development lifecycle most of us were taught: ```mermaid graph TD A[Requirements] --> B[System Design] B --> C[Implementation] C --> D[Testing] D --> E[Code Review] E --> F[Deployment] F --> G[Monitoring] G --> A ``` Every stage has its own tools, its own rituals, its own cottage industry. Jira for requirements. Figma for design. VS Code for implementation. Jest for testing. GitHub for code review. AWS for deployment. Datadog for monitoring. Each step is discrete. Sequential. Handoffs everywhere. Now here's what actually happens when an engineer works with a coding agent: ```mermaid graph TD A[Intent] --> B[Agent] B --> C[Code + Tests + Deployment] C --> D{Does it work?} D -->|No| B D -->|Yes| E[Ship] style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46 ``` The stages collapsed. They didn't get faster. They merged. The agent doesn't know what step it's on because there are no steps. There's just intent, context, and iteration. ## AI-native engineers don't know what the SDLC is I spent a lot of time speaking with engineers who started their career after Cursor launched. They don't know what the software development lifecycle is. They don't know what's DevOps or what's an SRE. Not because they're bad engineers. Because they never needed it. They've never sat through sprint planning. They've never estimated story points. They've never waited three days for a PR review. They just build things. You describe what you want. The agent writes the code. You look at it. You iterate. You ship. Everything simultaneously. These engineers aren't worse for skipping the ceremony. They're unencumbered by it. Sprint planning, code review workflows, release trains, estimation rituals. None of it. They skipped the entire orthodoxy and went straight to building. And honestly? I'm jealous. ## Every stage is collapsing Let me walk through the SDLC and show you what's left of it. ### Requirements gathering: fluid, not dictated Requirements used to be handed down. A PM writes a PRD, engineers estimate it, and the spec gets frozen before a line of code is written. That made sense when building was expensive. When every feature took weeks, you had to decide upfront what to build. That constraint is gone. When an agent can generate a complete version of a feature in minutes, you don't need to specify every detail in advance. You provide the direction, the agent builds a version, you look at it, you adjust, you try a different approach. You can generate ten versions and pick the best one. Requirements aren't a phase anymore. They're a byproduct of iteration. Now, what is Jira when the audience isn't humans coordinating across a pipeline? What is Jira when it's agents consuming context? Jira was built to track work through stages that no longer exist. If your "requirements" are just context for an agent, then the ticketing system isn't a project management tool anymore. It's a context store. And it's a terrible one. ### System Design: discovered, not dictated System design still matters. But the way it happens is fundamentally shifting. Design used to be something you did before writing code. You'd whiteboard the architecture, debate trade-offs, draw boxes and arrows, then go implement it. The gap between the design and the code was days or weeks. That gap is closing. Design is becoming something you discover by giving the agent the right context, not something you dictate ahead of time. The model has seen more systems, more architectures, more patterns than any individual engineer. When you describe a problem, the agent doesn't just implement your design, it suggests architectures that are often superior to what you'd have come up with on your own. You're having a design conversation in real-time, and the output is working code. You still need to know when an agent is over-engineering or missing a constraint. But you're collaborating on design, not prescribing it. ### Implementation: this is the agent's job now This one is obvious. The agent writes the code. Whole features. Complete solutions with error handling, types, edge cases. I don't personally know anyone who still types lines of code. We review what agents write, feed them context, steer direction, and focus on the problems that actually require human judgment. ### Testing: simultaneous, not sequential Agents write tests alongside the code. Not as an afterthought. Not in a separate "testing phase." The test is part of the generation. TDD isn't a methodology anymore, it's just how agents work by default. The entire QA function as a separate stage is gone. When code and tests are generated together, verified together, and iterated together, there's no handoff. No "throw it over the wall to QA.". The agent can do the QA itself. ### Code review: give it up The pull request flow needs to go. I was never a fan, but now it's just a relic of the past. I know that's uncomfortable. Code review is sacred. It's how you catch bugs, share knowledge, maintain standards. It's also an identity thing. We're *engineers*, and reviewing code is what engineers do. But clinging to the PR workflow in an agent-driven world isn't rigor. It's an identity crisis. Think about it. An agent generates 500 PRs a day. Your team can review maybe 10. The review queue backs up. This isn't a bottleneck worth optimising. It's a fake bottleneck, one that only exists because we're forcing a human ritual onto a machine workflow. ```mermaid graph TD A[Agent generates PR] --> B[Waits for human review] B --> C{Reviewer available?} C -->|No| D[Sits in queue for hours/days] C -->|Yes| E[Review + Comments] E --> F[Agent addresses feedback] F --> B D --> B style B fill:#fee2e2,stroke:#fca5a5,color:#991b1b style D fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` This diagram shouldn't exist. The entire flow is wrong. The review has to be rethought from scratch. Either it becomes part of the code generation itself, the agent verifies its own work against the plan document, runs the tests, checks for regressions, validates against architectural constraints, or a second agent reviews the first agent's output. Adversarial agents plough through the proposed changes, try to break it in every dimension. We already have the tools for this. Human-in-the-loop review becomes exception-based, triggered only when automated verification can't resolve a conflict or when the change touches something genuinely novel. What does a world without pull requests look like? Agents commit to main. Automated checks, tests, type checks, security scans, behavioral diffs, validate the change. If everything passes, it ships, automatically. If something fails, the agent fixes it. A human only gets involved when the system genuinely doesn't know what to do. ```mermaid graph TD A[Agent generates code] --> B[Agent self-verifies] B --> C[Second agent reviews] C --> D[Automated checks] D --> E{All clear?} E -->|Yes| F[Ship] E -->|No - resolvable| A E -->|No - novel issue| G[Human review] G --> A style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46 ``` We're spending our review cycles reading diffs that an agent could verify in seconds. That's not quality assurance. That's luddism. ### Deployment: decoupled and continuous Agents are already writing deployment pipelines that are more intricate and more specialised than what most teams would ever bother building by hand. Feature flags, canary releases, progressive rollouts, automatic rollback triggers, the kind of release engineering that used to require a dedicated platform team. The key shift is that agents naturally decouple deployment from release. Code gets deployed continuously, every change, as soon as it's generated and verified, produces an artifact that lands in production behind a gate. Release is a separate decision, driven by feature flags or traffic rules. Some teams are already approaching true continuous deployment and release. Code is generated, tests pass, artifacts are built, and the change is live, all in a single automated flow with no human in the loop between intent and production. Where this goes next is even more interesting. Imagine agents that don't just deploy code but manage the entire release lifecycle, monitoring the rollout, adjusting traffic percentages based on error rates, automatically rolling back if latency spikes, and only notifying a human when something genuinely novel goes wrong. The deployment "stage" doesn't just get automated. It becomes an ongoing, self-adjusting process that never really ends. ```mermaid graph TD A[Agent generates code] --> B[Automated verification] B --> C[Artifact produced] C --> D[Deploy behind feature flag] D --> E[Progressive rollout] E --> F{Healthy?} F -->|Yes| G[Full release] F -->|No| H[Auto-rollback] H --> I[Agent investigates] I --> A style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46 style H fill:#fee2e2,stroke:#fca5a5,color:#991b1b ``` ### Monitoring: the last stage standing, and it needs to evolve Monitoring is the only stage of the SDLC that survives. And it doesn't just survive, it becomes the foundation everything else rests on. When agents ship code faster than humans can review it, observability is no longer a nice-to-have dashboarding layer. It's the primary safety mechanism for the entire collapsed lifecycle. Every other safeguard, the design review, the code review, the QA phase, the release sign-off, has been absorbed or eliminated. Monitoring is what's left. It's the last line of defense. But most observability platforms were built for humans. Alerts, log search, dashboard, etc. all designed for a person to look at, interpret, and act on. That model breaks when the volume of changes outpaces human attention. If an agent ships 500 changes a day and your observability setup requires a human to investigate each anomaly, you've created a new bottleneck. You've just moved it from code review to incident response. Observability without action is just expensive storage. The future of observability isn't dashboards, it's closed-loop systems where telemetry data becomes context for the agent that shipped the code, so it can detect the regression and fix it. **The observability layer becomes the feedback mechanism that drives the entire loop.** Not a stage at the end. The connective tissue of the whole system. ```mermaid graph TD A[Intent] --> B[Agent builds, tests, deploys] B --> C[Production] C --> D[Observability layer] D -->|Anomaly detected| E[Agent investigates + fixes] E --> B D -->|Healthy| F[Next intent] F --> A style D fill:#dbeafe,stroke:#93c5fd,color:#1e40af ``` The teams that figure this out first, observability that feeds directly back into the agent loop, not into a human's pager, will ship faster and safer than everyone else. The teams that don't will drown in alerts. ## The new lifecycle is tighter loop The SDLC was a wide loop. Requirements β†’ Design β†’ Code β†’ Test β†’ Review β†’ Deploy β†’ Monitor. Linear. Sequential. Full of handoffs and waiting. The new lifecycle is a tight loop. ```mermaid graph TD A[Human Intent + Context] --> B[AI Agent] B --> C[Build + Test + Deploy] C --> D[Observe] D -->|Problem| B D -->|Fine| E[Next Intent] E --> B style B fill:#ede9fe,stroke:#c4b5fd,color:#5b21b6 ``` Intent. Build. Observe. Repeat. No tickets. No sprints. No story points. No PRs sitting in a queue. No separate QA phase. No release trains. Just a human with intent and an agent that executes. ## So what is left? Context. That's it. The quality of what you build with agents is directly proportional to the quality of context you give them. Not the process. Not the ceremony. The context. The SDLC is dead. The new skill is context engineering. The new safety net is observability. And most of the industry is still configuring Datadog dashboards no one looks at. --- ### How I Use Claude Code URL: https://boristane.com/blog/how-i-use-claude-code/ Description: The research-plan-implement workflow I use to build software with Claude Code, and why I never let it write code until I've approved a written plan. I've been using [Claude Code](https://docs.anthropic.com/en/docs/claude-code) as my primary development tool for approx 9 months, and the workflow I've settled into is radically different from what most people do with AI coding tools. Most developers type a prompt, sometimes use plan mode, fix the errors, repeat. The more terminally online are stitching together ralph loops, mcps, gas towns (remember those?), etc. The results in both cases are a mess that completely falls apart for anything non-trivial. The workflow I'm going to describe has one core principle: **never let Claude write code until you've reviewed and approved a written plan**. This separation of planning and execution is the single most important thing I do. It prevents wasted effort, keeps me in control of architecture decisions, and produces significantly better results with minimal token usage than jumping straight to code. ```mermaid flowchart LR R[Research] --> P[Plan] P --> A[Annotate] A -->|repeat 1-6x| A A --> T[Todo List] T --> I[Implement] I --> F[Feedback & Iterate] ``` ## Phase 1: Research Every meaningful task starts with a deep-read directive. I ask Claude to thoroughly understand the relevant part of the codebase before doing anything else. And I always require the findings to be written into a persistent markdown file, never just a verbal summary in the chat. > read this folder in depth, understand how it works deeply, what it does and all its specificities. when that's done, write a detailed report of your learnings and findings in research.md > study the notification system in great details, understand the intricacies of it and write a detailed research.md document with everything there is to know about how notifications work > go through the task scheduling flow, understand it deeply and look for potential bugs. there definitely are bugs in the system as it sometimes runs tasks that should have been cancelled. keep researching the flow until you find all the bugs, don't stop until all the bugs are found. when you're done, write a detailed report of your findings in research.md Notice the language: **"deeply"**, **"in great details"**, **"intricacies"**, **"go through everything"**. This isn't fluff. Without these words, Claude will skim. It'll read a file, see what a function does at the signature level, and move on. You need to signal that surface-level reading is not acceptable. The written artifact (`research.md`) is critical. It's not about making Claude do homework. It's my review surface. I can read it, verify Claude actually understood the system, and correct misunderstandings before any planning happens. If the research is wrong, the plan will be wrong, and the implementation will be wrong. Garbage in, garbage out. This is the most expensive failure mode with AI-assisted coding, and it's not wrong syntax or bad logic. It's implementations that work in isolation but break the surrounding system. A function that ignores an existing caching layer. A migration that doesn't account for the ORM's conventions. An API endpoint that duplicates logic that already exists elsewhere. The research phase prevents all of this. ## Phase 2: Planning Once I've reviewed the research, I ask for a detailed implementation plan in a separate markdown file. > I want to build a new feature \ that extends the system to perform \. write a detailed plan.md document outlining how to implement this. include code snippets > the list endpoint should support cursor-based pagination instead of offset. write a detailed plan.md for how to achieve this. read source files before suggesting changes, base the plan on the actual codebase The generated plan always includes a detailed explanation of the approach, code snippets showing the actual changes, file paths that will be modified, and considerations and trade-offs. I use my own `.md` plan files rather than Claude Code's built-in plan mode. The built-in plan mode sucks. My markdown file gives me full control. I can edit it in my editor, add inline notes, and it persists as a real artifact in the project. **One trick I use constantly:** for well-contained features where I've seen a good implementation in an open source repo, I'll share that code as a reference alongside the plan request. If I want to add sortable IDs, I paste the ID generation code from a project that does it well and say "this is how they do sortable IDs, write a plan.md explaining how we can adopt a similar approach." Claude works dramatically better when it has a concrete reference implementation to work from rather than designing from scratch. But the plan document itself isn't the interesting part. The interesting part is what happens next. ## The Annotation Cycle This is the most distinctive part of my workflow, and the part where I add the most value. ```mermaid flowchart TD W[Claude writes plan.md] --> R[I review in my editor] R --> N[I add inline notes] N --> S[Send Claude back to the document] S --> U[Claude updates plan] U --> D{Satisfied?} D -->|No| R D -->|Yes| T[Request todo list] ``` After Claude writes the plan, I open it in my editor and **add inline notes directly into the document**. These notes correct assumptions, reject approaches, add constraints, or provide domain knowledge that Claude doesn't have. The notes vary wildly in length. Sometimes a note is two words: "not optional" next to a parameter Claude marked as optional. Other times it's a paragraph explaining a business constraint or pasting a code snippet showing the data shape I expect. Some real examples of notes I'd add: - *"use drizzle:generate for migrations, not raw SQL"* -- domain knowledge Claude doesn't have - *"no -- this should be a PATCH, not a PUT"* -- correcting a wrong assumption - *"remove this section entirely, we don't need caching here"* -- rejecting a proposed approach - *"the queue consumer already handles retries, so this retry logic is redundant. remove it and just let it fail"* -- explaining why something should change - *"this is wrong, the visibility field needs to be on the list itself, not on individual items. when a list is public, all items are public. restructure the schema section accordingly"* -- redirecting an entire section of the plan Then I send Claude back to the document: > I added a few notes to the document, address all the notes and update the document accordingly. don't implement yet **This cycle repeats 1 to 6 times.** The explicit **"don't implement yet"** guard is essential. Without it, Claude will jump to code the moment it thinks the plan is good enough. It's not good enough until I say it is. ### Why This Works So Well The markdown file acts as **shared mutable state** between me and Claude. I can think at my own pace, annotate precisely where something is wrong, and re-engage without losing context. I'm not trying to explain everything in a chat message. I'm pointing at the exact spot in the document where the issue is and writing my correction right there. This is fundamentally different from trying to steer implementation through chat messages. The plan is a structured, complete specification I can review holistically. A chat conversation is something I'd have to scroll through to reconstruct decisions. The plan wins every time. Three rounds of "I added notes, update the plan" can transform a generic implementation plan into one that fits perfectly into the existing system. Claude is excellent at understanding code, proposing solutions, and writing implementations. But it doesn't know my product priorities, my users' pain points, or the engineering trade-offs I'm willing to make. The annotation cycle is how I inject that judgement. ### The Todo List Before implementation starts, I always request a granular task breakdown: > add a detailed todo list to the plan, with all the phases and individual tasks necessary to complete the plan - don't implement yet This creates a checklist that serves as a progress tracker during implementation. Claude marks items as completed as it goes, so I can glance at the plan at any point and see exactly where things stand. Especially valuable in sessions that run for hours. ## Phase 3: Implementation When the plan is ready, I issue the implementation command. I've refined this into a standard prompt I reuse across sessions: > implement it all. when you're done with a task or phase, mark it as completed in the plan document. do not stop until all tasks and phases are completed. do not add unnecessary comments or jsdocs, do not use any or unknown types. continuously run typecheck to make sure you're not introducing new issues. This single prompt encodes everything that matters: - *"implement it all"*: do everything in the plan, don't cherry-pick - *"mark it as completed in the plan document"*: the plan is the source of truth for progress - *"do not stop until all tasks and phases are completed"*: don't pause for confirmation mid-flow - *"do not add unnecessary comments or jsdocs"*: keep the code clean - *"do not use any or unknown types"*: maintain strict typing - *"continuously run typecheck"*: catch problems early, not at the end I use this exact phrasing (with minor variations) in virtually every implementation session. By the time I say "implement it all," every decision has been made and validated. The implementation becomes mechanical, not creative. This is deliberate. **I want implementation to be boring**. The creative work happened in the annotation cycles. Once the plan is right, execution should be straightforward. Without the planning phase, what typically happens is Claude makes a reasonable-but-wrong assumption early on, builds on top of it for 15 minutes, and then I have to unwind a chain of changes. The "don't implement yet" guard eliminates this entirely. ## Feedback During Implementation Once Claude is executing the plan, my role shifts from architect to supervisor. My prompts become dramatically shorter. ```mermaid flowchart LR I[Claude implements] --> R[I review / test] R --> C{Correct?} C -->|No| F[Terse correction] F --> I C -->|Yes| N{More tasks?} N -->|Yes| I N -->|No| D[Done] ``` Where a planning note might be a paragraph, an implementation correction is often a single sentence: - *"You didn't implement the `deduplicateByTitle` function."* - *"You built the settings page in the main app when it should be in the admin app, move it."* Claude has the full context of the plan and the ongoing session, so terse corrections are enough. Frontend work is the most iterative part. I test in the browser and fire off rapid corrections: - *"wider"* - *"still cropped"* - *"there's a 2px gap"* For visual issues, I sometimes attach screenshots. A screenshot of a misaligned table communicates the problem faster than describing it. I also reference existing code constantly: - *"this table should look exactly like the users table, same header, same pagination, same row density."* This is far more precise than describing a design from scratch. Most features in a mature codebase are variations on existing patterns. A new settings page should look like the existing settings pages. Pointing to the reference communicates all the implicit requirements without spelling them out. Claude would typically read the reference file(s) before making the correction. When something goes in a wrong direction, I don't try to patch it. I revert and re-scope by discarding the git changes: - *"I reverted everything. Now all I want is to make the list view more minimal -- nothing else."* Narrowing scope after a revert almost always produces better results than trying to incrementally fix a bad approach. ## Staying in the Driver's Seat Even though I delegate execution to Claude, **I never give it total autonomy over what gets built**. I do the vast majority of the active steering in the `plan.md` documents. This matters because Claude will sometimes propose solutions that are technically correct but wrong for the project. Maybe the approach is over-engineered, or it changes a public API signature that other parts of the system depend on, or it picks a more complex option when a simpler one would do. I have context about the broader system, the product direction, and the engineering culture that Claude doesn't. ```mermaid flowchart TD P[Claude proposes changes] --> E[I evaluate each item] E --> A[Accept as-is] E --> M[Modify approach] E --> S[Skip / remove] E --> O[Override technical choice] A & M & S & O --> R[Refined implementation scope] ``` **Cherry-picking from proposals:** When Claude identifies multiple issues, I go through them one by one: *"for the first one, just use Promise.all, don't make it overly complicated; for the third one, extract it into a separate function for readability; ignore the fourth and fifth ones, they're not worth the complexity."* I'm making item-level decisions based on my knowledge of what matters right now. **Trimming scope:** When the plan includes nice-to-haves, I actively cut them. *"remove the download feature from the plan, I don't want to implement this now."* This prevents scope creep. **Protecting existing interfaces:** I set hard constraints when I know something shouldn't change: *"the signatures of these three functions should not change, the caller should adapt, not the library."* **Overriding technical choices:** Sometimes I have a specific preference Claude wouldn't know about: *"use this model instead of that one"* or *"use this library's built-in method instead of writing a custom one."* Fast, direct overrides. Claude handles the mechanical execution, while I make the judgement calls. The plan captures the big decisions upfront, and selective guidance handles the smaller ones that emerge during implementation. ## Single Long Sessions I run research, planning, and implementation in a **single long session** rather than splitting them across separate sessions. A single session might start with deep-reading a folder, go through three rounds of plan annotation, then run the full implementation, all in one continuous conversation. I am not seeing the performance degradation everyone talks about after 50% context window. Actually, by the time I say "implement it all," Claude has spent the entire session building understanding: reading files during research, refining its mental model during annotation cycles, absorbing my domain knowledge corrections. When the context window fills up, Claude's auto-compaction maintains enough context to keep going. And the plan document, the persistent artifact, survives compaction in full fidelity. I can point Claude to it at any point in time. ## The Workflow in One Sentence Read deeply, write a plan, annotate the plan until it's right, then let Claude execute the whole thing without stopping, checking types along the way. That's it. No magic prompts, no elaborate system instructions, no clever hacks. Just a disciplined pipeline that separates thinking from typing. The research prevents Claude from making ignorant changes. The plan prevents it from making wrong changes. The annotation cycle injects my judgement. And the implementation command lets it run without interruption once every decision has been made. Try my workflow, you'll wonder how you ever shipped anything with coding agents without an annotated plan document sitting between you and the code. --- ### Ship types, not docs URL: https://boristane.com/blog/ship-types/ Description: Types are the contract between services, docs are not External link to [shiptypes.com](https://shiptypes.com) --- ### Logging Sucks URL: https://boristane.com/blog/logging-sucks/ Description: Why traditional logging fails in distributed systems and how wide events transform debugging from archaeology into analytics External link to [loggingsucks.com](https://loggingsucks.com) --- ### What even are Cloudflare Durable Objects? URL: https://boristane.com/blog/what-are-cloudflare-durable-objects/ Description: Notes from 2 years of building with Cloudflare Durable Objects Two years ago I was just like you. I had heard of [Cloudflare Durable Objects](https://workers.cloudflare.com/product/durable-objects), I read the docs, but still had no idea what these were. Today I almost exclusively build with Durable Objects (DOs), and I can't imagine building production applications without them. Let me explain what finally made it click for me. ## The Problem: State in Serverless You know how regular serverless functions (like [AWS Lambda Functions](https://aws.amazon.com/lambda/)) are stateless? Every request starts fresh, with no memory of what happened before. If you want to remember something between requests, you have to store it in a database. This works fine for most CRUD operations. But it quickly becomes a real challenge when you need to: - Handle WebSocket connections that stay open for minutes or hours - Coordinate multiple requests to the same resource (like when 10 people are editing the same document simultaneously) - Run scheduled tasks for a specific user (like sending a subscription renewal reminder exactly 30 days after signup) The traditional solution? Spin up a stateful server. But now you're back to managing servers, dealing with scaling, load balancing, and paying for idle capacity. ```mermaid graph TB User[User Requests for Document 123] User --> LB{API Gateway} LB --> L1[Lambda A] LB --> L2[Lambda B] LB --> L3[Lambda C] L1 --> Server[Central Server] L2 --> Server L3 --> Server L1 -.->|❌ No coordination| L2 L2 -.->|❌ No coordination| L3 L3 -.->|❌ No coordination| L1 Server --> DB[(Central Database)] style L1 fill:#f96,stroke:#333 style L2 fill:#f96,stroke:#333 style L3 fill:#f96,stroke:#333 style LB fill:#fc9,stroke:#333 style DB fill:#99f,stroke:#333 style Server fill:#9c9,stroke:#333 ``` Durable Objects give you the benefits of stateful servers (long-lived connections, state, coordination) without the operational overhead. They're serverless, but stateful. They scale automatically, but maintain strong consistency. They're ephemeral (shutting down when idle), but durable (state persists). ```mermaid graph TB User1[User 1
Document 123] --> Worker1[Worker A] User2[User 2
Document 123] --> Worker2[Worker B] User3[User 3
Document 123] --> Worker3[Worker C] Worker1 --> DO[Durable Object] Worker2 --> DO Worker3 --> DO style Worker1 fill:#FAAD3F,stroke:#333 style Worker2 fill:#FAAD3F,stroke:#333 style Worker3 fill:#FAAD3F,stroke:#333 style DO fill:#9f9,stroke:#333,stroke-width:2px ``` ## My "Aha!" Moment: One Instance per ID Here's the key insight that made it click for me: > A Durable Object is like having a tiny, long-lived server that is **guaranteed to be unique** for a specific ID. When you create a Durable Object with [`idFromName("workspace-123")`](https://developers.cloudflare.com/durable-objects/api/namespace/#idfromname), Cloudflare ensures that **only one instance** of that object exists globally. Every request with that same ID goes to the **exact same instance**, no matter where in the world the request comes from. This is fundamentally different from regular [Cloudflare Workers](https://workers.cloudflare.com/product/workers). With Workers, if 100 requests come in for "workspace-123", they might be handled by 100 different Worker instances across 100 different data centers, depending on the origating request. Each instance is isolated, stateless, and ephemeral. With Durable Objects, all 100 requests for "workspace-123" are routed to the **same single instance** of your Durable Object. That instance can be running in Tokyo, and requests from New York will be routed there. This single instance can maintain WebSocket connections, coordinate concurrent access, and act as the single source of truth for that workspace's state. It's like having a dedicated micro-server for each user, workspace, or chat room. Except you only pay when it's actually being used, and you never have to think about deployment, scaling, or geographic distribution. ## Explain Like I'm 5 Imagine a library (Cloudflare's edge network) with millions of private study rooms (Durable Objects). Each room has: - A **unique room number** (the ID) - A **desk with drawers** (persistent storage) - A **person working inside** (your code) - A **whiteboard** (in-memory state) When you want to access "Room 123", the librarian always sends you to the same room. The person inside remembers what they were doing, has access to their drawers (storage), and can handle your request immediately. If nobody visits the room for a while, the person leaves (hibernation), but their drawers stay locked. When someone comes to visit the same room again, the person returns exactly where they left off. ## How Requests Flow ```mermaid graph TB Client[Client Request] --> Worker[Cloudflare Worker] Worker --> |"idFromName('user-123')"| DO_ID[Durable Object ID] DO_ID --> |Get Stub| Stub[Durable Object Stub] Stub --> |RPC Call| Instance[Durable Object Instance] Instance --> Memory[In-Memory State] Instance --> Storage[KV Storage] Instance --> SQL[SQLite Database] Instance --> Alarm[Scheduled Alarms] style Instance fill:#f9a,stroke:#333,stroke-width:4px style Memory fill:#9cf,stroke:#333,stroke-width:2px style Storage fill:#9f9,stroke:#333,stroke-width:2px style SQL fill:#ff9,stroke:#333,stroke-width:2px style Alarm fill:#f99,stroke:#333,stroke-width:2px ``` ## Creating a Durable Object Let's build a simple "Workspace" Durable Object, based on patterns from real production code: ```typescript // workspace.ts - The Durable Object class export class DurableWorkspace extends DurableObject { private sql: SqlStorage; constructor(ctx: DurableObjectState, env: Env) { super(ctx, env); this.sql = ctx.storage.sql; // Run migrations on first access ctx.blockConcurrencyWhile(async () => { this.migrate(); }); } migrate() { // Create tables if they don't exist this.sql.exec(` CREATE TABLE IF NOT EXISTS projects ( id TEXT PRIMARY KEY, name TEXT NOT NULL, created INTEGER NOT NULL ) `); } // Store workspace identity async init(workspaceId: string, ownerId: string) { await this.ctx.storage.put("workspaceId", workspaceId); await this.ctx.storage.put("ownerId", ownerId); // Schedule a monthly cleanup await this.ctx.storage.setAlarm(Date.now() + 30 * 24 * 60 * 60 * 1000); } async getWorkspaceId(): Promise { return await this.ctx.storage.get("workspaceId") || ""; } async createProject(name: string) { const id = crypto.randomUUID(); this.sql.exec( "INSERT INTO projects (id, name, created) VALUES (?, ?, ?)", id, name, Date.now() ); return { id, name }; } async listProjects() { const result = this.sql.exec("SELECT * FROM projects ORDER BY created DESC"); return result.toArray(); } // Called when alarm triggers async alarm() { console.log("Running monthly cleanup..."); // Clean up old data this.sql.exec("DELETE FROM projects WHERE created < ?", Date.now() - 90 * 24 * 60 * 60 * 1000); // Schedule next cleanup await this.ctx.storage.setAlarm(Date.now() + 30 * 24 * 60 * 60 * 1000); } } ``` ## Accessing the Durable Object: The Stub Pattern Now that we've seen a Durable Object class, how do we actually use it? You don't instantiate Durable Objects directly. Instead, you get a "stub" - a proxy that lets you call methods on the Durable Object from your Worker. The critical part is the `idFromName()` call. This is what ensures you always get the same instance for the same ID. Think of it like a phone book - you look up "workspace-123" in the directory, and Cloudflare gives you a direct line to that workspace's unique Durable Object instance, wherever it happens to be running in the world. Here's how you access your Durable Object from a Worker: ```typescript // Helper function to get a workspace stub export function getWorkspaceStub(env: Env, workspaceId: string): DurableObjectStub { // idFromName ensures the same ID always goes to the same instance const id = env.WORKSPACE.idFromName(workspaceId); const stub = env.WORKSPACE.get(id); return stub; } // In your Worker (API endpoint) export default { async fetch(request: Request, env: Env) { const url = new URL(request.url); const workspaceId = url.searchParams.get("workspace"); // Get the stub - always the same instance for this workspace const workspace = getWorkspaceStub(env, workspaceId); if (url.pathname === "/init") { await workspace.init(workspaceId, "user-123"); return new Response("Initialized"); } if (url.pathname === "/projects") { if (request.method === "POST") { const { name } = await request.json(); const project = await workspace.createProject(name); return Response.json(project); } else { const projects = await workspace.listProjects(); return Response.json(projects); } } return new Response("Not found", { status: 404 }); } }; ``` The Worker acts as a router - it extracts the workspace ID from the request, gets the appropriate Durable Object stub using `idFromName()`, and calls methods on it. The `workspace.createProject()` and `workspace.listProjects()` calls are [RPC (Remote Procedure Calls)](https://developers.cloudflare.com/workers/runtime-apis/rpc/) - they're executed inside the Durable Object instance, not in the Worker. The Worker just forwards the request and gets the response back. ## The Four Storage Layers One of the most powerful aspects of Durable Objects is having multiple storage options. When I first started, I didn't understand why you'd need so many ways to store data. But each layer serves a distinct purpose and has different performance characteristics. Think of it like this: in-memory state is your desk (fast, temporary), KV storage is your desk drawer (fast, persistent, small items), SQLite is your filing cabinet (structured, queryable, medium size), and external storage like R2 is your warehouse (massive, slower, for big things). Understanding when to use each storage layer is crucial: ```mermaid graph TD DO[Durable Object Instance] DO --> KV[Key-Value Storage
ctx.storage.put/get] DO --> SQL[SQLite Database
ctx.storage.sql] DO --> Memory[In-Memory State
class properties] DO --> External[External Storage
R2, KV Namespace, D1] KV --> |Use For| KV_Use[Simple metadata: User ID, config, etc.] SQL --> |Use For| SQL_Use[Structured data: Relations & indexes] Memory --> |Use For| Memory_Use[WebSocket connections, active sessions, etc.] External --> |Use For| External_Use[Large files, cross-workspace data, analytics] style KV fill:#9f9,stroke:#333 style SQL fill:#ff9,stroke:#333 style Memory fill:#9cf,stroke:#333 style External fill:#f9f,stroke:#333 ``` ### Example: Using All Storage Layers ```typescript export class ChatRoom extends DurableObject { // In-memory: active WebSocket connections private connections: Map = new Map(); // In-memory: temporary message buffer private messageBuffer: Message[] = []; constructor(ctx: DurableObjectState, env: Env) { super(ctx, env); ctx.blockConcurrencyWhile(async () => { // SQLite: create messages table this.ctx.storage.sql.exec(` CREATE TABLE IF NOT EXISTS messages ( id TEXT PRIMARY KEY, userId TEXT NOT NULL, content TEXT NOT NULL, timestamp INTEGER NOT NULL ); CREATE INDEX IF NOT EXISTS idx_timestamp ON messages(timestamp); `); }); } async initRoom(roomId: string, roomName: string) { // Key-Value: store simple metadata await this.ctx.storage.put("roomId", roomId); await this.ctx.storage.put("roomName", roomName); await this.ctx.storage.put("createdAt", Date.now()); } async handleWebSocket(request: Request): Promise { const pair = new WebSocketPair(); const [client, server] = Object.values(pair); // In-memory: track active connection const connectionId = crypto.randomUUID(); this.connections.set(connectionId, server); server.addEventListener("message", async (event) => { const message = JSON.parse(event.data); // In-memory: save message to the buffer this.messageBuffer.push(message) // SQLite: persist message this.ctx.storage.sql.exec( "INSERT INTO messages (id, userId, content, timestamp) VALUES (?, ?, ?, ?)", crypto.randomUUID(), message.userId, message.content, Date.now() ); // In-memory: broadcast to all active connections for (const [id, conn] of this.connections) { if (id !== connectionId) { conn.send(JSON.stringify(message)); } } // External: save to R2 for archival if (this.messageBuffer.length >= 100) { await this.archiveMessages(); } }); server.addEventListener("close", () => { this.connections.delete(connectionId); }); server.accept(); return new Response(null, { status: 101, webSocket: client }); } async archiveMessages() { const roomId = await this.ctx.storage.get("roomId"); const messages = this.messageBuffer.splice(0); // External storage: R2 bucket for large archives await this.env.ARCHIVE_BUCKET.put( `rooms/${roomId}/${Date.now()}.json`, JSON.stringify(messages) ); } async getRecentMessages(limit: number = 50) { // SQLite: efficient queries with indexes const result = this.ctx.storage.sql.exec( "SELECT * FROM messages ORDER BY timestamp DESC LIMIT ?", limit ); return result.toArray(); } } ``` ## Real-World Pattern: Parent-Child Relationships This is my favourite pattern when building multitenant applications. Let me explain it with a real-world analogy first, then show you the code. ### The Apartment Building Analogy Imagine you're building a project management SaaS app. Think of it like an apartment building: - **The Building (Workspace)** is a Durable Object that represents the entire workspace - **Each Apartment (Project)** is its own separate Durable Object - **The Building Manager** keeps a directory of all apartments (the workspace's SQLite database) - **Each Apartment** has its own furniture, residents, and state (the project's data and tasks) Why separate them? Because if you put all apartments' furniture in the building manager's office, it gets crowded fast. Instead, each apartment manages its own internal state, but the building manager knows about all apartments and can coordinate between them. In code terms: **don't put all your data in one Durable Object**. Create separate child Durable Objects for each logical entity (user, project, document, chat room), and have a parent Durable Object that coordinates them. ### Why This Pattern Matters The thing is, each Durable Object is single-threaded. If you have one giant Durable Object handling everything for a workspace with 100 projects, and 50 users are all editing different projects simultaneously, they're all waiting in line for that one Durable Object to process their requests. But if each project is its own Durable Object, those 50 users can work in parallel across 50 different Durable Objects. The parent workspace just maintains a registry of which projects exist. ```typescript // Parent: DurableWorkspace export class DurableWorkspace extends DurableObject { async createThread(threadName: string) { const threadId = crypto.randomUUID(); const workspaceId = await this.ctx.storage.get("workspaceId"); // Store reference in parent's database this.ctx.storage.sql.exec( "INSERT INTO threads (id, name, created) VALUES (?, ?, ?)", threadId, threadName, Date.now() ); // Get stub to child Durable Object const threadStub = getThreadStub(this.env, threadId); // Initialize the child await threadStub.init(threadId, workspaceId, threadName); return { id: threadId, name: threadName }; } async listThreads() { const result = this.ctx.storage.sql.exec( "SELECT * FROM threads ORDER BY created DESC" ); return result.toArray(); } } // Child: DurableThread export class DurableThread extends DurableObject { async init(threadId: string, workspaceId: string, threadName: string) { // Store parent reference await this.ctx.storage.put("threadId", threadId); await this.ctx.storage.put("workspaceId", workspaceId); await this.ctx.storage.put("threadName", threadName); } async addMessage(content: string) { const id = crypto.randomUUID(); this.ctx.storage.sql.exec( "INSERT INTO messages (id, content, created) VALUES (?, ?, ?)", id, content, Date.now() ); return { id, content }; } async deleteThread() { const workspaceId = await this.ctx.storage.get("workspaceId"); const threadId = await this.ctx.storage.get("threadId"); // Tell parent to remove reference const workspaceStub = getWorkspaceStub(this.env, workspaceId); await workspaceStub.deleteThreadReference(threadId); // Clean up own data this.ctx.storage.sql.exec("DELETE FROM messages"); await this.ctx.storage.deleteAll(); } } // Helper functions function getThreadStub(env: Env, threadId: string) { const id = env.THREAD.idFromName(threadId); return env.THREAD.get(id); } function getWorkspaceStub(env: Env, workspaceId: string) { const id = env.WORKSPACE.idFromName(workspaceId); return env.WORKSPACE.get(id); } ``` ### What's Happening Here? Let's walk through the flow when a user creates a thread: 1. **User calls API**: "Create a thread named 'Q4 Planning' in workspace-123" 2. **Worker gets workspace stub**: Uses `idFromName("workspace-123")` to get the workspace Durable Object 3. **Workspace creates reference**: Stores "thread-456, Q4 Planning" in its own SQLite database 4. **Workspace spawns child**: Gets a stub to a new Thread Durable Object using `idFromName("thread-456")` 5. **Child initializes**: The thread stores its identity (threadId, workspaceId, name) 6. **Now they're connected**: The workspace knows about the thread, and the thread knows about its parent workspace Later, when listing threads: - You ask the workspace: "What threads exist?" - It queries its own database and returns the list - You don't need to wake up every thread Durable Object just to list them When adding a message: - You go directly to the thread Durable Object - It handles the message independently - The workspace doesn't need to be involved at all This separation means the workspace can track hundreds of threads without getting bogged down by message operations, and threads can process messages in parallel without waiting for the workspace. ## Scheduled Operations with Alarms Here's another outstanding feature that took me a while to appreciate: each Durable Object can schedule its own future work. Not through some external scheduler, but built right into the object itself. Think about subscription renewals. With traditional serverless, you'd need: 1. A cron job that runs every hour 2. It queries your database for all subscriptions expiring in the next hour 3. It processes each one 4. This gets expensive and slow as you scale to millions of users With Durable Objects + Alarms: 1. When a user subscribes, their Subscription Durable Object schedules an alarm for exactly 30 days from now 2. In 30 days, that specific Durable Object wakes up automatically 3. It processes just that user's renewal 4. Then schedules the next alarm No cron jobs scanning millions of records. No batch processing. Just-in-time execution for each user. This is incredibly powerful for per-user or per-entity scheduled tasks. Here's how it works in code: ```typescript export class SubscriptionManager extends DurableObject { async createSubscription(userId: string, planType: string) { await this.ctx.storage.put("userId", userId); await this.ctx.storage.put("planType", planType); await this.ctx.storage.put("status", "active"); // Schedule renewal check in 30 days const renewalDate = Date.now() + 30 * 24 * 60 * 60 * 1000; await this.ctx.storage.setAlarm(renewalDate); } async alarm() { const userId = await this.ctx.storage.get("userId"); const planType = await this.ctx.storage.get("planType"); console.log(`Processing renewal for user ${userId}`); // Charge the user const success = await this.chargeUser(userId, planType); if (success) { // Schedule next renewal await this.ctx.storage.setAlarm(Date.now() + 30 * 24 * 60 * 60 * 1000); } else { // Downgrade to free plan await this.ctx.storage.put("status", "expired"); await this.ctx.storage.put("planType", "free"); } } async cancelSubscription() { await this.ctx.storage.put("status", "cancelled"); // Cancel future alarms await this.ctx.storage.deleteAlarm(); } private async chargeUser(userId: string, planType: string): Promise { // Call payment API return true; } } ``` When you schedule an alarm, Cloudflare guarantees it will fire at least once. If your `alarm()` method throws an exception, it automatically retries up to 6 times with exponential backoff (starting at 2 seconds). You don't need to write retry logic, just make sure your `alarm()` method is idempotent (safe to run multiple times) since it could potentially execute more than once. ## When Should You Use Durable Objects? Here's my mini-decision framework: **Perfect for Durable Objects:** - Per-user/per-tenant databases - AI agents - Real-time collaboration (Google Docs-style editing) - WebSocket servers (chat rooms, live dashboards) - Coordinating multiple requests to the same resource - Rate limiting per user - Session management - Scheduled tasks per entity (user notifications, subscription renewals) **Not ideal for Durable Objects:** - Storing large blobs (use [R2](https://workers.cloudflare.com/product/r2) instead) - Analytics aggregation across many users ([Workers Analytics Engine](https://developers.cloudflare.com/analytics/analytics-engine/)) - Truly stateless operations (use regular [Workers](https://workers.cloudflare.com/product/workers)) - Queries across all users (use [D1](https://workers.cloudflare.com/product/d1) with proper indexes) **Be careful with:** - Very high traffic to a single ID (DOs are single-threaded per instance) - Long-running CPU-intensive tasks (use queues + regular workers instead) ## Common Pitfalls I Learned the Hard Way Let me save you from the mistakes I made when I was learning Durable Objects. These gotchas aren't obvious from the docs, but they'll bite you in production if you're not careful. ### 1. Don't Forget: One Instance = One Thread This is the biggest gotcha. A Durable Object instance is single-threaded. Every request to the same ID is processed sequentially, one at a time. This is actually a feature (strong consistency!), but it means you can't block for long periods. I once had a Durable Object that called a slow external API and took 5 seconds to respond. When 10 requests came in simultaneously for the same object, the last request waited 50 seconds. Not good. ```typescript // BAD: Blocking operation will freeze this DO for everyone export class BadExample extends DurableObject { async handleRequest() { // This blocks for 10 seconds - ALL requests wait! await new Promise(resolve => setTimeout(resolve, 10000)); return "Done"; } } // GOOD: Offload heavy work to queues export class GoodExample extends DurableObject { async handleRequest() { // Queue the work, return immediately await this.env.QUEUE.send({ task: "heavy-work" }); return "Queued"; } } ``` ### 2. Initialize the Durable Object identity When a Durable Object is first created, its storage is empty. It's not possible to recall the name of the Durable Object from within the Durable Object. You need to explicitly initialize it. I use an `init()` method. ```typescript // BAD: Storage might be empty on first call export class BadExample extends DurableObject { async getWorkspaceId() { return await this.ctx.storage.get("workspaceId"); // Could be undefined! } } // GOOD: Explicit initialization export class GoodExample extends DurableObject { async init(workspaceId: string) { await this.ctx.storage.put("workspaceId", workspaceId); } async getWorkspaceId() { const id = await this.ctx.storage.get("workspaceId"); if (!id) throw new Error("Workspace not initialized - call init() first"); return id; } } ``` ## The Key to Real-Time Apps and AI Agents: Hibernate API Before [Hibernate API](https://developers.cloudflare.com/workers/platform/changelog/#2023-05-26), if you had a chat room with 1000 connected users but nobody was sending messages, you were still paying for that Durable Object to stay alive and hold those connections. With Hibernate API, the Durable Object can "hibernate" - essentially go to sleep while maintaining WebSocket connections. When a message arrives, it wakes up automatically, handles the message, and can go back to sleep. The beauty is that this happens transparently. Your code doesn't change. You just get automatic cost savings and better resource utilization. ```typescript export class ChatRoom extends DurableObject { // Hibernate API automatically serializes/deserializes state async webSocketMessage(ws: WebSocket, message: string) { const data = JSON.parse(message); // Broadcast to all connections this.ctx.getWebSockets().forEach(socket => { socket.send(JSON.stringify(data)); }); } async webSocketClose(ws: WebSocket, code: number, reason: string) { console.log("disconnected"); } async handleConnect(request: Request): Promise { const pair = new WebSocketPair(); const [client, server] = Object.values(pair); // Attach metadata to the WebSocket this.ctx.acceptWebSocket(server); return new Response(null, { status: 101, webSocket: client }); } } ``` With Hibernate API, your Durable Object can go to sleep when idle (saving you money) and wake up automatically when a WebSocket message arrives. The state is automatically preserved! ## Leveling Up: The Cloudflare Agents SDK The Cloudflare [`agents` SDK](https://agents.cloudflare.com/) ([`npm i agents`](https://www.npmjs.com/package/agents)) is a framework built on top of Durable Objects. It's designed for building AI agents on the Cloudflare network, and has become my default framework when building with Durable Objects. **You don't need to be building AI agents to use it.** Despite the name, it's actually a general-purpose framework for building stateful real-time applications with Durable Objects. It just happens to make AI agent development really easy too. ### What Does the Agents SDK Give You? The `Agent` base class provides: 1. **WebSocket handling out of the box** - `onConnect()`, `onMessage()`, `onClose()` lifecycle methods 2. **HTTP request handling** - `onRequest()` method with clean routing 3. **Email routing** - Built-in email address parsing and routing (yes, Durable Objects can receive emails!) 4. **Scheduled operations** - Simplified alarm scheduling with cron expressions 5. **Task queues** - Lightweight queue for task defferal 6. **Hibernation API** - Automatic connection state management for WebSockets ### A Simple Example: Chat Room Without AI Here's a chat room using the Agents SDK - notice there's no AI involved, it's just using the framework's convenience features: ```typescript export class ChatRoom extends Agent { // WebSocket lifecycle - automatically handled async onConnect(connection, ctx) { const username = ctx.request.headers.get("username"); // Set connection metadata (persists through hibernation!) await connection.setState({ username, joinedAt: Date.now() }); // Broadcast join message to everyone this.broadcast(JSON.stringify({ type: "user_joined", username, timestamp: Date.now() })); } override async onMessage(connection, message) { const data = JSON.parse(message); const state = await connection.getState(); // Save to SQLite for history this.ctx.storage.sql.exec( "INSERT INTO messages (id, username, content, timestamp) VALUES (?, ?, ?, ?)", crypto.randomUUID(), state.username, data.content, Date.now() ); // Broadcast to all connected clients (except sender) this.broadcast(JSON.stringify({ type: "message", username: state.username, content: data.content, timestamp: Date.now() }), [connection.id]); } async onClose(connection) { const state = await connection.getState(); this.broadcast(JSON.stringify({ type: "user_left", username: state.username, timestamp: Date.now() })); } // Regular HTTP request handling override async onRequest(request) { // Get recent message history via HTTP if (request.method === "GET") { const messages = this.ctx.storage.sql.exec( "SELECT * FROM messages ORDER BY timestamp DESC LIMIT 50" ); return Response.json(messages.toArray()); } return super.onRequest(request); } } ``` ### What This Abstracts Away Compare this to raw Durable Objects code: **Without Agents SDK:** ```typescript // You manually create WebSocketPair const pair = new WebSocketPair(); const [client, server] = Object.values(pair); // Manually track connections in a Map this.connections.set(id, server); // Manually add event listeners server.addEventListener("message", (event) => { /* ... */ }); server.addEventListener("close", () => { /* ... */ }); // Manually accept server.accept(); return new Response(null, { status: 101, webSocket: client }); ``` **With Agents SDK:** ```typescript // Just implement the lifecycle methods async onConnect(connection, ctx) { /* ... */ } async onMessage(connection, message) { /* ... */ } async onClose(connection) { /* ... */ } // Framework handles all the boilerplate ``` ### Using Agents SDK for Non-AI Applications I use the Agents SDK for: - **Real-time dashboards** - WebSocket connections with automatic reconnection handling - **Multiplayer games** - Each game room is an Agent with connection management built-in - **Collaborative editing** - Document state + WebSocket broadcasting without manual connection tracking - **Job queues** - Queue message handling with automatic retries - **Scheduled tasks** - Cron-based processing per entity (user notifications, report generation) The framework handles connection lifecycle, hibernation, state persistence, and broadcasting - all the tedious parts of building with Durable Objects. I use the Agents SDK for all my Durable Objects, both my production apps and my experiments. The base `Agent` class provides so much value that even for non-AI applications, it's worth the dependency. ## Conclusion Two years ago, I read the Durable Objects docs three times and still didn't get it. The breakthrough came when I stopped trying to understand the technology and started thinkering. **Durable Objects aren't just "stateful Workers"**. They're a fundamentally different primitive. They give you: - **Coordination**: One instance per ID means you can coordinate concurrent access without distributed locking - **Real-time**: WebSocket connections that survive across requests - **Consistency**: Strong consistency without the complexity of distributed systems - **Scheduling**: Per-entity alarms that fire exactly when needed - **Isolation**: Perfect tenant isolation in multi-tenant SaaS (each tenant gets its own Durable Object) The mental model that finally clicked for me: **Durable Objects are like having millions of tiny, specialized servers that spawn on-demand, live exactly where they're needed, and automatically shut down when idle**. You don't need to understand how Cloudflare routes requests globally to the same instance. You don't need to worry about replication or consistency. You just write code as if you have a single-threaded server for each user/room/workspace, and Cloudflare handles the rest. ### Start Simple, Scale Up Don't try to use all the features at once. Start with: 1. Create a Durable Object with just KV storage. Get comfortable with stubs and `idFromName`. 2. Add SQLite for structured data. Learn about transactions. 3. Try parent-child relationships. Understand when to split data across multiple DOs. 4. Add WebSockets or alarms based on your use case. The key insights I wish I'd understood from day one: 1. **One instance per ID** - This is the superpower. Embrace it. 2. **Single-threaded** - Don't block. Offload heavy work to queues. 3. **Multiple storage layers** - KV for metadata, SQLite for data, memory for connections. 4. **Parent-child pattern** - Don't put everything in one DO. Split by entity. 5. **Hibernation is magic** - Your DO can sleep while maintaining state and connections. Two years ago, I couldn't wrap my head around Durable Objects. Today, they're the foundation of everything I build. They've replaced Redis, PostgreSQL, WebSocket servers, cron jobs, and state machines in my architecture. Once it clicks, you'll wonder how you ever built without them. --- ### Unlimited On-Demand Graph Databases with Cloudflare Durable Objects URL: https://boristane.com/blog/durable-objects-graph-databases/ Description: How to create isolated, scalable graph databases using Cloudflare Durable Objects and SQLite for multi-tenant applications I found myself needing to create unlimited, isolated graph databases for an AI agent I'm building. I figured an elegant way using [Durable Objects](https://developers.cloudflare.com/durable-objects/) on the [Cloudflare Developers Platform](https://workers.dev). Each graph gets its own instance with dedicated SQLite storage, enabling true multi-tenancy without the complexity of traditional database partitioning. Traditional graph database deployments require careful infrastructure planning. You need to decide upfront how many database instances to provision, configure connection pooling, implement tenant isolation at the query level, and manage scaling as your user base grows. What if you could skip all of that and let the platform handle it? In this blog post, I'll walk through building a simple and extensible graph database system leveraging SQLite on Durable Objects where creating a new database is as simple as making an HTTP request with a new identifier. No provisioning, no configuration, just instant isolation. ## The Multi-Tenancy Problem If you're building any kind of multi-tenant application today, you're probably dealing with one of these patterns: 1. **Shared database with tenant IDs**: Every query needs `WHERE tenant_id = ?` 2. **Schema per tenant**: Complex migrations and connection management 3. **Traditional database per tenant**: Infrastructure overhead and cost The key problem is architectural complexity. When you store multiple tenants in a shared database, you're constantly guarding against data leakage. One missing `WHERE` clause and you've exposed customer data. Likewise, when you create separate databases per tenant, you're managing infrastructure at scale, which usually is resource inefficient. Graph databases add another layer of complexity. Graph queries naturally traverse relationships, making it even easier to accidentally cross tenant boundaries. Traditional solutions require: 1. **Careful query design**: Every traversal must check tenant ownership 2. **Complex access control**: Node and edge-level security policies 3. **Performance trade-offs**: Filtering by tenant ID on every hop What if each tenant could have their own isolated graph database, created automatically on first access? ## Durable Objects as Micro-Databases [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) solve this elegantly. Each Durable Object instance is a globally-unique, [strongly-consistent](https://en.wikipedia.org/wiki/Strong_consistency) compute environment with built-in SQLite storage. The key insight: **treat each Durable Object as a complete database instance**. With Durable Objects, you get: - **Automatic isolation**: Each instance has its own SQLite database - **Deterministic routing**: Each `graphId` always routes to the same instance - **Zero provisioning**: Instances are created on-demand - **Global distribution**: Automatically deployed to Cloudflare's edge network This enables a powerful pattern: **one graph database per identifier**. ```typescript // This is all you need to get an isolated database const doId = env.GraphDatabase.idFromName("user-123-graph"); const stub = env.GraphDatabase.get(doId); ``` The first time you access `user-123-graph`, Cloudflare creates a new Durable Object instance. The second time, you get the same instance. It's deterministic, isolated, and automatic. ## Project Overview We'll be using: - [**Cloudflare Workers**](https://developers.cloudflare.com/workers/) - Serverless HTTP handling and routing - [**Cloudflare Durable Objects**](https://developers.cloudflare.com/durable-objects/) - Stateful instances with SQLite - [**Hono**](https://hono.dev/) - Lightweight web framework - [**TypeScript**](https://www.typescriptlang.org/) - Type-safe development Here's the basic structure: ``` . β”œβ”€β”€ bindings.ts # TypeScript bindings β”œβ”€β”€ src β”‚ β”œβ”€β”€ index.ts # Worker router and API endpoints β”‚ β”œβ”€β”€ do.ts # Durable Object graph database implementation β”‚ └── migrate.ts # Migration system β”œβ”€β”€ migrations β”‚ β”œβ”€β”€ index.ts # Migration registry β”‚ └── v00_add_graph_tables.ts # Initial schema β”œβ”€β”€ web β”‚ β”œβ”€β”€ index.html # Graph visualization UI β”‚ β”œβ”€β”€ app.js # Frontend application β”‚ └── style.css # Styles β”œβ”€β”€ wrangler.json # Cloudflare configuration └── package.json # Dependencies ``` The architecture looks like this: ```mermaid graph TB Client[Client/Browser] -->|HTTP Request| Worker[Cloudflare Worker - Hono Router] Worker -->|Extract graphId from URL| Router{URL Router} Router -->|idFromName graphId| NS[Durable Object Namespace] NS -->|Get or Create Instance| DO[GraphDatabase Durable Object Instance] DO -->|Raw SQL Operations| SQL[(SQLite Storage)] SQL -->|schema_history| Migrations[Migration System] SQL -->|nodes table| Nodes[Nodes: id, type, data, timestamps] SQL -->|edges table| Edges[Edges: source, target, relationship] DO -->|Graph Operations| Ops[BFS Traversal, Pathfinding, Neighbor Queries] Ops -->|Read/Write| SQL DO -->|JSON Response| Worker Worker -->|HTTP Response| Client style Worker fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style DO fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style SQL fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style NS fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` The complete source code is available on [GitHub](https://github.com/boristane/cloudflare-dev-101/tree/main/durable-objects-graph-database). I suggest following along with the code for reference. ## Setting Up the Worker Let's start with the Worker configuration. Create a `wrangler.json` file: ```json { "name": "durable-objects-graph-database", "compatibility_date": "2025-10-27", "workers_dev": true, "main": "./src/index.ts", "assets": { "directory": "./web", "binding": "ASSETS" }, "migrations": [ { "new_sqlite_classes": ["GraphDatabase"], "tag": "v1" } ], "durable_objects": { "bindings": [ { "name": "GraphDatabase", "class_name": "GraphDatabase" } ] } } ``` The `new_sqlite_classes` migration is crucial. It enables [SQLite storage](https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/) in your Durable Objects. Without this, your Durable Objects will not have a SQLite database attached. The [`assets` configuration](https://developers.cloudflare.com/workers/static-assets/) serves a web interface directly from the Worker, enabling us to visualise our graph databases. ## Building the Graph Database Durable Object The `GraphDatabase` Durable Object implements the graph database. It's a class extending the `DurableObject` class. It encapsulates all graph operations and manages its own SQLite database. ### Schema Design [Graph databases](https://en.wikipedia.org/wiki/Graph_database) are different from traditional relational databases. Instead of thinking in individual tables with joins, you think in **nodes** (entities) and **edges** (relationships). #### Understanding Nodes and Edges Imagine modeling an organization: ```mermaid graph LR Alice[("Alice - Engineering Manager")] Bob[("Bob - Senior Engineer")] Project[("Project X - Active")] Alice -->|manages| Bob Bob -->|works_on| Project style Alice fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Bob fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Project fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` **Nodes** are the "things": Alice, Bob, and Project X. Each node has: - An **ID** (unique identifier) - A **type** (person, project, document, etc.) - **Data** (arbitrary JSON properties like name, role, status) **Edges** are the connections: "Alice manages Bob" and "Bob works on Project X". Each edge has: - A **source** (where the edge starts) - A **target** (where the edge ends) - A **relationship** (what type of connection this is) - Optional **properties** (metadata about the relationship) The power comes from traversing these relationships. To answer "what projects does Alice's team work on?", you traverse: Alice β†’ manages β†’ Bob β†’ works_on β†’ Project X. #### The Schema Here's how we model this in SQLite: ```sql title="migrations/v00_add_graph_tables.ts" CREATE TABLE IF NOT EXISTS nodes ( id TEXT PRIMARY KEY, -- Unique identifier (e.g., "alice", "project-x") type TEXT NOT NULL, -- Category (e.g., "person", "project") data TEXT NOT NULL, -- JSON blob: {"name": "Alice", "role": "Manager"} created_by TEXT NOT NULL, -- Who created this node last_edited_by TEXT NOT NULL, -- Who last modified it updated INTEGER NOT NULL, -- Last update timestamp (ms since epoch) created INTEGER NOT NULL -- Creation timestamp (ms since epoch) ); CREATE TABLE IF NOT EXISTS edges ( source TEXT NOT NULL, -- Source node ID target TEXT NOT NULL, -- Target node ID relationship TEXT NOT NULL, -- Relationship type (e.g., "manages", "works_on") properties TEXT, -- Optional JSON: {"since": "2024", "hours": 40} created_by TEXT NOT NULL, last_edited_by TEXT NOT NULL, updated INTEGER NOT NULL, created INTEGER NOT NULL, UNIQUE(source, target, relationship) -- Prevents duplicate edges ); ``` #### Why This Design Works **Flexible Node Storage**: The `data` field stores JSON as text. This means different node types can have completely different properties: ```json // Person node {"name": "Alice", "role": "Engineering Manager", "email": "alice@example.com"} // Project node {"name": "Project X", "status": "active", "deadline": "2025-12-31"} // Document node {"title": "Architecture Doc", "url": "https://...", "embedding": [...]} ``` You don't need to alter the schema when adding new node types or properties. Just store different JSON. **Multi-Relationship Support**: The `UNIQUE(source, target, relationship)` constraint ensures the uniqueness of relationships between nodes, and enables multiple relationship types between the same two nodes: ```mermaid graph LR A[("Alice")] B[("Bob")] A -->|manages| B A -->|mentors| B A -->|collaborates_with| B style A fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` Alice can simultaneously manage, mentor, and collaborate with Bob. Each is a separate edge with a different `relationship` value. **Directional Relationships**: Edges have direction. "Alice manages Bob" is different from "Bob manages Alice". The `source` and `target` fields encode this direction: For bidirectional relationships (like "friends_with"), you create two edges: ```mermaid graph LR A[("Alice")] B[("Bob")] A -->|friends_with| B B -->|friends_with| A style A fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` **Indexes**: We add indexes to improve query performance: ```sql CREATE INDEX edges_source_idx ON edges(source); -- Fast outbound queries CREATE INDEX edges_target_idx ON edges(target); -- Fast inbound queries CREATE INDEX nodes_type_idx ON nodes(type); -- Filter by node type ``` When you ask "who does Alice manage?", the database uses `edges_source_idx` to quickly find all edges where `source = 'alice'`. Without this index, it would scan a large number of edge. Query performance would tank for large graphs. Similarly, "who manages Alice?" uses `edges_target_idx` to find edges where `target = 'alice'`. The `nodes_type_idx` lets you efficiently query "find all person nodes" or "find all projects" without scanning every node. ### Migration System Durable Object provide a SQLite interface, but it's up to us to figure out a migration system. The database schema will evolve over time as our application evolves. The migration system is **idempotent** and **version-tracked**. Each Durable Object instance maintains its own migration state in a `schema_history` table. The migration logic: 1. Create `schema_history` table if it doesn't exist 2. Compare current schema version with available migrations 3. Apply any pending migrations in order within a transaction 4. Record each migration in the history table ```typescript title="src/migrate.ts" export function migrate(storage: DurableObjectStorage, migrations: SQLMigration[]) { createSchemaHistory(storage); const current = getSchema(storage); const toApply = migrations.filter((m) => m.version > (current?.version || -1)); storage.transactionSync(() => { for (const migration of toApply) { storage.sql.exec(migration.sql); storage.sql.exec( "INSERT INTO schema_history (version, name) VALUES (?, ?)", migration.version, migration.name ); } }); } ``` The [`transactionSync`](https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/#transactionsync) wrapper ensures atomicity; either all migrations succeed or none do. This prevents corrupted schema states if a migration fails partway through. The migrations must run in the constructor of the Durable Object class. When a Durable Object receives a request, if it's not yet in memory on the Cloudflare network, it is initialised and the migrations run "just-in-time" for this given instance of the Durable Object class. ### Durable Object Implementation Now let's implement the `GraphDatabase` class: ```typescript title="src/do.ts" export class GraphDatabase extends DurableObject { private db: SqlStorage; constructor(ctx: DurableObjectState, env: Bindings) { super(ctx, env); this.db = ctx.storage.sql; this.ctx.blockConcurrencyWhile(async () => { migrate(ctx.storage, migrations); }); } } ``` The `blockConcurrencyWhile` ensures migrations complete before any other requests are processed. Without this, you could have race conditions where a request tries to query a table that doesn't exist yet. ### Node Operations We need operations to create, read, update, and delete nodes. The key node methods in the `GraphDatabase` class are: - **addNode()**: Creates or updates a node with JSON data stored as text - **findNode()**: Retrieves a single node by its ID - **findNodes()**: Lists nodes with optional filtering by type, a limit, and an offset - **deleteNode()**: Removes a node and cascades to delete all connected edges, ensuring the integrity of the graph ```typescript title="src/do.ts" addNode(node: InsertNode): Node { const { id, type, data, userId } = node; const result = this.db.exec( "INSERT OR REPLACE INTO nodes (...) VALUES (...) RETURNING *", id, type, JSON.stringify(data), userId, userId, Date.now(), Date.now() ); return toNode(result.toArray()[0]); } findNode(id: string): Node | null { const result = this.db.exec("SELECT * FROM nodes WHERE id = ?", id); const res = result.toArray(); if (!res.length) { return null; } const rows = res as Array; const row = rows[0]; if (!row) return null; return toNode(row); } findNodes(filters?: { type?: string; limit?: number; offset?: number }): Node[] { let query = "SELECT * FROM nodes"; const params: any[] = []; if (filters?.type) { query += " WHERE type = ?"; params.push(filters.type); } if (filters?.limit) { query += " LIMIT ?"; params.push(filters.limit); } if (filters?.offset) { query += " OFFSET ?"; params.push(filters.offset); } const result = this.db.exec(query, ...params); const res = result.toArray(); return res.map((row) => toNode(row as DBNode)); } deleteNode(id: string): Node | null { // First delete all edges connected to this node this.db.exec("DELETE FROM edges WHERE source = ? OR target = ?", id, id); // Then delete the node const result = this.db.exec("DELETE FROM nodes WHERE id = ? RETURNING *", id); const res = result.toArray(); if (!res.length) { return null; } const rows = res as Array; const row = rows[0]; if (!row) return null; return toNode(row); } ``` ### Edge Operations #### How Edges Work Consider this scenario: you want to model how team members contribute to projects: ```mermaid graph LR Alice[("Alice - Manager")] Bob[("Bob - Engineer")] Charlie[("Charlie - Engineer")] ProjectX[("Project X")] ProjectY[("Project Y")] Alice -->|manages| Bob Alice -->|manages| Charlie Bob -->|works_on: 40 hrs/week| ProjectX Charlie -->|works_on: 20 hrs/week| ProjectX Charlie -->|works_on: 20 hrs/week| ProjectY style Alice fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Bob fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Charlie fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style ProjectX fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style ProjectY fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` Each edge stores: ```typescript // Bob works on Project X { source: "bob", target: "project-x", relationship: "works_on", properties: { hours_per_week: 40, role: "lead" } } // Charlie works on Project X { source: "charlie", target: "project-x", relationship: "works_on", properties: { hours_per_week: 20, role: "contributor" } } ``` Notice how the same relationship type (`works_on`) can have different properties. This lets you capture nuance, both Bob and Charlie work on Project X, but with different time commitments and roles. #### The Composite Key Pattern The `UNIQUE(source, target, relationship)` constraint is fundamental to the design. It means: - Alice can both "manages" and "mentors" Bob (different relationships) - Alice can "manages" both Bob and Charlie (different targets) - Alice "manages" Bob can't be inserted twice (duplicate prevented) ```typescript // First call: creates the edge await connectNodes({ source: "bob", target: "project-x", relationship: "works_on", properties: { hours_per_week: 30 } }); // Second call: updates the same edge (changes hours from 30 to 40) await connectNodes({ source: "bob", target: "project-x", relationship: "works_on", properties: { hours_per_week: 40 } }); ``` #### Edge Methods The core edge operations are: **connectNodes()** - Creates or updates an edge: ```typescript const edge = await stub.connectNodes({ source: "alice", target: "bob", relationship: "manages", properties: { since: "2024-01-15", team: "engineering" }, userId: "system" }); ``` **disconnectNodes()** - Removes a specific edge: ```typescript // Remove the "manages" relationship (other relationships remain) await stub.disconnectNodes("alice", "bob", "manages"); ``` **findEdges()** - Query edges by node or relationship: ```typescript // Find all edges connected to Bob (any direction) const bobEdges = await stub.findEdges({ nodeId: "bob" }); // Find all "manages" relationships in the graph const managesEdges = await stub.findEdges({ relationship: "manages" }); ``` #### Edge Properties: When to Use Them Some relationships are simple facts ("knows", "contains", "references"). Others carry context that's essential to understanding the relationship: ```typescript // Simple edge - no properties needed { source: "doc-1", target: "doc-2", relationship: "references" } // Rich edge - properties add crucial context { source: "alice", target: "project-x", relationship: "contributed_to", properties: { role: "architect", startDate: "2024-01-01", endDate: "2024-06-30", linesOfCode: 15420, commits: 247 } } ``` The flexibility to add properties without schema changes lets you evolve your data model. You can start with simple edges and enrich them later as you discover what metadata matters. ### Graph Traversal Graph traversal enables your to explore relationships and answer questions like "how are these two entities connected?" or "what's reachable from this starting point?" #### Directional Queries The `findNeighbors()` method finds all nodes directly connected to a given node. Direction matters: ```mermaid graph TB subgraph "Both (all Alice's connections)" D2[("David")] A3[("Alice")] B2[("Bob")] C2[("Charlie")] D2 -->|manages| A3 A3 -->|manages| B2 A3 -->|manages| C2 style A3 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style D2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style C2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end subgraph "Outbound (who does Alice manage?)" A1[("Alice")] B1[("Bob")] C1[("Charlie")] A1 -->|manages| B1 A1 -->|manages| C1 style A1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style C1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end subgraph "Inbound (who manages Alice?)" D[("David - VP Engineering")] A2[("Alice")] D -->|manages| A2 style D fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style A2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end ``` Directional queries are lets you model asymmetric relationships. "Alice manages Bob" is directional, the reverse isn't automatically true. ```typescript title="src/do.ts" findNeighbors( nodeId: string, direction: "inbound" | "outbound" | "both" = "both" ): string[] { let query: string; const params: string[] = []; switch (direction) { case "inbound": query = "SELECT DISTINCT source AS neighbor FROM edges WHERE target = ?"; params.push(nodeId); break; case "outbound": query = "SELECT DISTINCT target AS neighbor FROM edges WHERE source = ?"; params.push(nodeId); break; case "both": default: query = ` SELECT DISTINCT target AS neighbor FROM edges WHERE source = ? UNION SELECT DISTINCT source AS neighbor FROM edges WHERE target = ? `; params.push(nodeId, nodeId); break; } const result = this.db.exec(query, ...params); const res = result.toArray(); return res.map((row) => (row as { neighbor: string }).neighbor); } ``` #### Breadth-First Search (BFS) The `traverse()` method implements [breadth-first search](https://en.wikipedia.org/wiki/Breadth-first_search) to explore the graph layer by layer. BFS guarantees finding the **shortest path** between two nodes. Here's how BFS explores a graph starting from Alice: ```mermaid graph TB subgraph "Depth 2 (Second hop)" A2[("Alice")] B2[("Bob")] C2[("Charlie")] PX[("Project X")] PY[("Project Y")] A2 -.->|manages| B2 A2 -.->|manages| C2 B2 -.->|works_on| PX C2 -.->|works_on| PY style A2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style C2 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style PX fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style PY fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end subgraph "Depth 1 (First hop)" A1[("Alice")] B1[("Bob")] C1[("Charlie")] A1 -.->|manages| B1 A1 -.->|manages| C1 style A1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style B1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style C1 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end subgraph "Depth 0 (Start)" A0[("Alice")] style A0 fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C end ``` BFS visits all nodes at depth 1 before moving to depth 2. This guarantees the first path found to any node is the shortest. **Pathfinding Example**: "How is Alice connected to Project X?" - Depth 0: Start at Alice - Depth 1: Explore Bob and Charlie (Alice's direct reports) - Depth 2: Find Project X through Bob - Result: Alice β†’ manages β†’ Bob β†’ works_on β†’ Project X (2 hops) Even if there's a longer path (Alice β†’ manages β†’ Charlie β†’ collaborates_with β†’ Bob β†’ works_on β†’ Project X), BFS finds the shorter one first. The traversal operates in two modes: 1. **Pathfinding Mode** (when `endNodeId` is specified): Find the shortest path from start to end node 2. **Reachability Mode** (when `endNodeId` is omitted): Find all nodes reachable within `maxDepth` hops ```typescript title="src/do.ts" traverse(startNodeId: string, options = {}) { const { endNodeId, maxDepth = 50, direction = "both" } = options; const visited = new Set(); const queue = [{ id: startNodeId, path: [startNodeId], depth: 0 }]; while (queue.length > 0) { const current = queue.shift()!; if (current.depth > maxDepth) continue; // Pathfinding: stop when target found if (endNodeId && current.id === endNodeId) { return { found: true, path: current.path, depth: current.depth }; } if (visited.has(current.id)) continue; visited.add(current.id); // Explore neighbors const neighbors = this.findNeighbors(current.id, direction); for (const neighborId of neighbors) { if (!visited.has(neighborId)) { queue.push({ id: neighborId, path: [...current.path, neighborId], depth: current.depth + 1, }); } } } // Reachability: return all visited nodes return { found: true, reachableNodes: Array.from(visited), count: visited.size }; } ``` In a highly connected graph (think social networks where most people are connected within 6 degrees), an unbounded traversal could visit the entire graph. By limiting depth through `maxDepth`, you can explore local neighborhoods without scanning millions of nodes. The algorithm maintains a `visited` set to avoid cycles. Without this, you could loop indefinitely in a graph with circular references (A β†’ B β†’ C β†’ A). ## Building the HTTP API Now let's expose these operations through an HTTP API. We'll use Hono for routing: ```typescript title="src/index.ts" const app = new Hono<{ Bindings: Bindings }>(); app.use("/*", cors()); export default { async fetch(request: Request, env: Bindings, ctx: ExecutionContext): Promise { const url = new URL(request.url); if (url.pathname.startsWith("/api/")) { const apiRequest = new Request(request.url.replace("/api", ""), request); return app.fetch(apiRequest, env, ctx); } return env.ASSETS.fetch(request); }, }; export { GraphDatabase } from "./do"; function getGraphDatabaseStub(env: Bindings, graphId: string) { const doId = env.GraphDatabase.idFromName(graphId); return env.GraphDatabase.get(doId); } ``` The routing is clean: `/api/*` goes to our Hono app, everything else is served from static assets for a simple web interface to visualise the graph. The `getGraphDatabaseStub` function uses [`idFromName(graphId)`](https://developers.cloudflare.com/durable-objects/api/namespace/#idfromname) to get a unique ID from the graphId, and creates a [stub](https://developers.cloudflare.com/durable-objects/api/namespace/#get) which can be used to invoke methods on the associated Durable Object. This is the foundation of the multi-tenancy model: - Same input (`"user-123-graph"`) β†’ Same Durable Object ID β†’ Same instance - Different input (`"user-456-graph"`) β†’ Different ID β†’ Different instance - No coordination needed; the `idFromName` guarantees deterministic routing You don't need a lookup table mapping tenant IDs to Durable Object IDs. ### API Endpoints Every endpoint extracts `graphId` from the URL path, making multi-tenancy explicit in the URL structure: ``` /api/alice-graph/nodes # Alice's graph /api/bob-graph/nodes # Bob's graph /api/project-x/nodes # Project X's graph ``` **Node Operations:** - `POST /:graphId/nodes` - Create or update a node - `GET /:graphId/nodes/:nodeId` - Retrieve a specific node - `GET /:graphId/nodes?type=&limit=&offset=` - List nodes with filtering - `DELETE /:graphId/nodes/:nodeId` - Delete node and connected edges **Edge Operations:** - `POST /:graphId/edges` - Create an edge between nodes - `GET /:graphId/edges?nodeId=&relationship=` - Find edges by node or relationship - `DELETE /:graphId/edges/:source/:relationship/:target` - Remove a specific edge **Graph Operations:** - `GET /:graphId/nodes/:nodeId/neighbors?direction=` - Find adjacent nodes - `POST /:graphId/traverse` - Execute BFS pathfinding or reachability analysis - `GET /:graphId/stats` - Get graph metrics (node count, edge count, type distributions) Each endpoint follows the same pattern: extract the `graphId`, get the Durable Object stub, call the method, return JSON. ```typescript title="src/index.ts" app.post("/:graphId/nodes", async (c) => { const graphId = c.req.param("graphId"); const body = await c.req.json(); const stub = getGraphDatabaseStub(c.env, graphId); const node = await stub.addNode(body); return c.json({ node }); }); ``` No shared tables, no tenant ID columns in queries. Different graphs queries are routed to different Durable Object instances. ## Interactive Graph Visualization I also vibe-coded a simple UI to visualise the state of the graph, hosted on the same Worker as the HTTP API. ## Testing It Out Let's deploy and test our graph database using [Wrangler](https://developers.cloudflare.com/workers/wrangler/): ```bash npm run deploy ``` Wrangler will build your TypeScript code, upload it to Cloudflare, and output your Worker URL: ``` https://durable-objects-graph-database.your-account.workers.dev ``` ### Creating a Knowledge Graph Let's build a simple organizational graph with people and projects: ```bash # Create person nodes (Alice, Bob) curl -X POST https://your-worker.workers.dev/api/my-org/nodes \ -H "Content-Type: application/json" \ -d '{ "id": "alice", "type": "person", "data": {"name": "Alice", "role": "Engineering Manager"}, "userId": "system" }' curl -X POST https://your-worker.workers.dev/api/my-org/nodes \ -H "Content-Type: application/json" \ -d '{ "id": "bob", "type": "person", "data": {"name": "Bob", "role": "Senior Engineer"}, "userId": "system" }' # Create a project node curl -X POST https://your-worker.workers.dev/api/my-org/nodes \ -H "Content-Type: application/json" \ -d '{ "id": "project-x", "type": "project", "data": {"name": "Project X", "status": "active"}, "userId": "system" }' # Add relationships: Alice manages Bob, Bob works on Project X curl -X POST https://your-worker.workers.dev/api/my-org/edges \ -H "Content-Type: application/json" \ -d '{ "source": "alice", "target": "bob", "relationship": "manages", "userId": "system" }' curl -X POST https://your-worker.workers.dev/api/my-org/edges \ -H "Content-Type: application/json" \ -d '{ "source": "bob", "target": "project-x", "relationship": "works_on", "properties": {"hours_per_week": 40}, "userId": "system" }' ``` ### Querying the Graph Now let's explore the graph: ```bash # Find everyone Alice manages curl https://your-worker.workers.dev/api/my-org/nodes/alice/neighbors?direction=outbound # Response: # { # "neighbors": ["bob"], # "count": 1 # } # Find who's working on Project X curl https://your-worker.workers.dev/api/my-org/nodes/project-x/neighbors?direction=inbound # Response: # { # "neighbors": ["bob"], # "count": 1 # } # Find the path from Alice to Project X curl -X POST https://your-worker.workers.dev/api/my-org/traverse \ -H "Content-Type: application/json" \ -d '{ "startNodeId": "alice", "endNodeId": "project-x", "maxDepth": 5 }' # Response: # { # "found": true, # "path": ["alice", "bob", "project-x"], # "depth": 2 # } ``` The traversal found a path: Alice β†’ Bob β†’ Project X. This shows the connection between a manager and a project through the people she manages. ### Multi-Tenancy in Action Creating multiple graphs is as simple as making HTTP calls: ```bash # Create a node in Alice's personal graph curl -X POST https://your-worker.workers.dev/api/alice-personal/nodes \ -H "Content-Type: application/json" \ -d '{ "id": "note-1", "type": "note", "data": {"content": "Remember to review PRs"}, "userId": "alice" }' # Create a node in the company graph curl -X POST https://your-worker.workers.dev/api/company-wide/nodes \ -H "Content-Type: application/json" \ -d '{ "id": "policy-1", "type": "policy", "data": {"title": "Remote Work Policy"}, "userId": "hr" }' ``` These are completely isolated. A query on "alice-personal" will never see "company-wide" data. Each has its own Durable Object instance with its own SQLite database. ## Use Cases I'm particularly interested in AI agent uses cases. ### AI Agent Memory AI agents need persistent memory to build context over time and understand complex relationships. Traditional approaches use vector databases for semantic search, but graph databases excel at representing how concepts, entities, and conversations interconnect. #### The Problem with Context-Only Memory When an AI agent remembers only through vector similarity, it loses the **structure** of relationships. Consider this scenario: An AI assistant named Alice helps a user with their work. Over multiple conversations, the user mentions: - They're working on "Project X" (a web app) - They're collaborating with "Bob" (the designer) - "Project X" has a deadline of March 1st - Bob is on vacation until February 20th A vector database might retrieve "Bob" when the user asks about the deadline, but it can't easily answer "Who on my team is available to help before the deadline?" That requires understanding the relationships: Project X β†’ has_deadline β†’ March 1st, Project X β†’ involves β†’ Bob, Bob β†’ unavailable_until β†’ February 20th. #### Graph-Based Agent Memory With a graph database, each agent builds a knowledge graph that captures not just facts, but how they relate: ```mermaid graph LR User[("User")] ProjX[("Project X - Web Redesign")] Bob[("Bob - Designer")] Deadline[("March 1, 2025")] Vacation[("Feb 20, 2025")] User -->|working_on| ProjX User -->|collaborates_with| Bob ProjX -->|involves| Bob ProjX -->|deadline| Deadline Bob -->|unavailable_until| Vacation style User fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style ProjX fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Bob fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Deadline fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C style Vacation fill:#E8E8E8,stroke:#AEAEAE,color:#1C1C1C ``` Here's how the agent builds this knowledge over time: ```typescript // Conversation 1: User mentions they're working on a project await fetch('/api/agent-alice/nodes', { method: 'POST', body: JSON.stringify({ id: 'project-x', type: 'project', data: { name: 'Web Redesign', status: 'in_progress' }, userId: 'agent-alice' }) }); await fetch('/api/agent-alice/edges', { method: 'POST', body: JSON.stringify({ source: 'user', target: 'project-x', relationship: 'working_on', userId: 'agent-alice' }) }); // Conversation 2: User mentions Bob is helping with design await fetch('/api/agent-alice/nodes', { method: 'POST', body: JSON.stringify({ id: 'bob', type: 'person', data: { name: 'Bob', role: 'Designer' }, userId: 'agent-alice' }) }); await fetch('/api/agent-alice/edges', { method: 'POST', body: JSON.stringify({ source: 'project-x', target: 'bob', relationship: 'involves', userId: 'agent-alice' }) }); // Conversation 3: User mentions Bob's vacation await fetch('/api/agent-alice/edges', { method: 'POST', body: JSON.stringify({ source: 'bob', target: 'unavailable-period', relationship: 'unavailable_until', properties: { date: '2025-02-20' }, userId: 'agent-alice' }) }); ``` Now when the user asks "Is my team available to help me finish before the deadline?", the agent can traverse the graph: ```typescript // Start from the user, explore their projects and collaborators const context = await fetch('/api/agent-alice/traverse', { method: 'POST', body: JSON.stringify({ startNodeId: 'user', maxDepth: 3, // Explore up to 3 hops away direction: 'both' }) }); // Returns: User β†’ Project X β†’ Bob β†’ Unavailable Until Feb 20 // Agent can reason: "Bob won't be available until Feb 20, but your deadline is March 1, // so you'll have 9 days to work together" ``` #### Why This Works Each agent gets its own graph (`/api/agent-alice/`, `/api/agent-bob/`). As the agent has more conversations, the graph grows organically: - **Conversations** link to **Topics** they discuss - **Topics** link to **Projects**, **People**, **Deadlines** - **Projects** link to **Tasks**, **Documents**, **Decisions** - **People** link to **Skills**, **Availability**, **Preferences** The graph becomes a structure memory layer that captures not just what the agent knows, but how concepts interconnect. When recalling context, traversal provides the most relevant connected information, not just similar words, but structurally related concepts. ### Graph-RAG Systems Retrieval-Augmented Generation (RAG) enhances LLMs by providing relevant context from external knowledge bases. Graph-based RAG goes beyond simple vector similarity by modeling relationships between documents, entities, and concepts. Store document chunks and their relationships for more intelligent retrieval. ## Limitations and Considerations While this pattern is powerful, there are important limitations to understand: 1. **Scale per Graph**: Each Durable Object instance is limited to 10 GB, as such is best suited for graphs with thousands to tens of thousands of nodes. For larger graphs, consider sharding. 2. **Cross-Graph Queries**: You can't easily query across multiple graphs. Each Durable Object is isolated. If you need to aggregate data from multiple tenants, you'll need to query each graph individually. 3. **Cold Start Latency**: The first request to a new Durable Object has some initialization overhead. For frequently accessed graphs this isn't an issue, but rarely-used graphs might experience slightly higher latency on first access. 4. **Global Consistency**: Durable Objects provide strong consistency within a single instance but don't offer distributed transactions across instances. If you need atomicity across multiple graphs, you'll need to implement your own coordination. 5. **Backup and Export**: There's no built-in backup system. For production use, you'll want to implement periodic exports of graph data to durable storage like [Cloudflare R2](https://developers.cloudflare.com/r2/). ## What's Next This foundation enables many extensions: - **Full-Text Search**: Add [SQLite FTS5](https://www.sqlite.org/fts5.html) virtual tables for text search within node and edge properties - **Complex nested queries**: Add the ability to run more complex queries on the JSON data of nodes and edges using [SQLite json_tree](https://sqlite.org/json1.html#jtree) functions - **Graph Algorithms**: Implement [PageRank](https://en.wikipedia.org/wiki/PageRank), [community detection](https://en.wikipedia.org/wiki/Community_structure), or [centrality metrics](https://en.wikipedia.org/wiki/Centrality) for graph analysis - **Graph Embeddings**: Store vector embeddings on nodes using [Cloudflare Vectorize](https://developers.cloudflare.com/vectorize/) for hybrid graph + semantic similarity search - **Batch Operations**: Add endpoints for bulk node/edge creation and updates to improve performance for large imports The complete source code with a visualization UI is available on [GitHub](https://github.com/boristane/cloudflare-dev-101/tree/main/durable-objects-graph-database). Fork it and build something interesting. ## Conclusion Durable Objects flip the traditional database model on its head. Instead of managing infrastructure and partitioning data, you create instances on demand. Each instance is a complete, isolated database with its own storage and compute. With SQLite storage, migrations, and proper schema design, each instance becomes a fully-featured database that happens to run at the edge. This pattern works for graphs, but it generalizes to any domain where you need isolated per-tenant data stores. Document databases, time-series data, etc; the architecture is the same. --- ### Context engineering is what makes AI magical URL: https://boristane.com/blog/context-engineering/ Description: What's context engineering and why you should spend more time on it. Everyone's busy tweaking prompts, swapping models, chaining tools. Yeah, models are getting better. Tools are getting fancier. But none of that matters if your **context sucks**. Here's the uncomfortable truth: **the main thing that really matters when building AI agents is the quality of the context you give the model.** Models are so good that most SOTA models can do most things with outstanding quality. Two products could be doing the exact same thing, but one feels magical and the other feels like a cheap demo. The difference? Context. --- ## What is context? Context is everything the model sees before it produces tokens. It includes: * the **system prompt** (what you tell the model it is) * the **user message** (what the user is asking) * any **external information**, tools, or retrieved documents you stuff in before the call * implicit context like **who the user is**, what they've done before, what they want right now, etc. The better this context, the better the model performs. Garbage in, garbage out, still true in 2025. --- ## Context > Model Models are already better than most of us at most tasks. But most AI tools still underperform, not because the model is bad, but because we're feeding it a half-baked view of the world. Let's look at RAG as an example. Naive RAG just dumps the top 3 chunks into the prompt and hopes for the best. Useful, sometimes. But the moment you move beyond toy examples, this starts to fall apart. Agentic RAG, builds a contextual snapshot that includes data from multiple sources: * the question * related documents * source structure * metadata * and critically, **the user's intent and environment** For example, a coding agent shouldn't just embed source files and search them. It should know: * how to find and read any file in your repository * which files were changed recently * which files are open in your IDE * what the LSP says about types and errors * even what production logs and metrics say That's context. And that's the difference between β€œmeh” and β€œwow”. --- ## Email example: the magic of good context Let's say you're a CTO at a startup. You get an email: > Hey Boris, just checking if you're around for a quick sync tomorrow. Would love to chat through a few ideas from our last call. A decent AI email tool will reply like: > Dear Jim, > Thank you for your message. Tomorrow works for me. May I ask what time you had in mind? > Best regards, > Boris **Who writes emails like this?** A magical agent will do a few things first: * check your **calendar**: you're in back-to-back calls all day. * look at **previous emails to this person**: you're friendly, informal. * scan **recent meeting notes**: this person is pitching a joint partnership. * pull in your **contact list**: they're a senior PM at a partner org. * apply your **system prompt customisation**: "be concise, decisive, warm" And finally generate and autonomously send an email that actually is helpful: > *Hey Jim! Tomorrow's packed on my end, back-to-back all day. Thursday AM free if that works for you? Sent an invite, lmk if works.* That's magic. Not because the model is smarter, but because the context is richer. --- ## Build context like it's your product You wouldn't build a product without thinking deeply about state, user intent, and interaction history. So why treat your AI agent like a stateless chatbot? Context engineering is the new prompt engineering. It means: * Designing the right structure and format for context * Knowing what context actually helps the model perform * Building pipelines to fetch, transform and deliver this context at runtime * Constantly improving context quality with feedback loops --- ## TL;DR * Models are great. * But **context is king**. * Context is the difference between a dumb assistant and a superpowered teammate. * Build your agents like you build your products: obsess over what they know, when they know it, and how they use it. If your agent isn't magical yet, don't swap the model, **fix your context.** --- ### Contextual Retrieval-Augmented Generation (RAG) on Cloudflare Workers URL: https://boristane.com/blog/cloudflare-contextual-rag/ Description: Implementing Contextual Retrieval-Augmented Generation with Using Cloudflare AI, Vectorize and D1 database with Drizzle ORM in a Hono app Today I am sharing a powerful pattern I've implemented on the [Cloudflare Developers Platform](https://workers.dev) whilst building my [weekends project](https://basebrain.ai): Contextual Retrieval-Augmented Generation (RAG). Traditional RAG implementations often fall short when presented with nuanced queries or complex documents. Anthropic introduced [Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) as an answer to this problem. In this blog post, I'll walk through implementing a contextual RAG system entirely on the [Cloudflare Developer Plarform](https://workers.dev). You'll see how combining vector search with full-text search, query rewriting, and smart ranking techniques creates a system that truly understands your content. ## What is Contextual RAG If you're doing any kind of RAG today, you're probably: 1. Chunking documents 2. Storing vectors 3. Searching embeddings 4. Stuffing the results into your LLM context window The problem is simple: chunking is often not good enough. When you retrieve a chunk based on cosine similarity, it might be totally out of context and irrelevant to the user query. In a nutshell, key issues with traditional RAG systems are: 1. **Context loss**: When documents are chunked into smaller pieces, the broader context that gives meaning to each chunk is often lost 2. **Relevance issues**: Vector search alone might retrieve semantically similar content that isn't actually relevant to the query 3. **Query limitations**: User queries are often ambiguous or lack specificity needed for effective retrieval Contextual RAG provides a solution to these issues. It consists of pre-pending every chunk with a short explanation situating it within the context of the entire document. Every chunk is passed through an LLM to provide this context. The prompt looks like: ```txt {{WHOLE_DOCUMENT}} Here is the chunk we want to situate within the whole document {{CHUNK_CONTENT}} Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else. ``` The result is a short sentence situating the chunk within the context of the entire documents. It is prepended to the actual chunk before inserting it into the vector databse. This provides a solution to the context loss challenge. Additionally, the query side of the RAG system is enhanced with full-text search, reciprocal rank fusion and a LLM reranker. With this technique the challenges of traditional RAG are addressed by: 1. **Enhancing chunks with context**: Each chunk is augmented with contextual information from the full document 2. **Using hybrid search**: Combining vector similarity with keyword-based search 3. **Rewriting queries**: Expanding and transforming user queries into multiple search variations 4. **Intelligent ranking**: Using sophisticated algorithms to merge and rank results from different search methods Let's dive deeper and see how to implemented this on Cloudflare Workers. ## Project Overview We'll be using: - [**Cloudflare Workers**](https://developers.cloudflare.com/workers/) - Our serverless platform - [**Cloudflare D1**](https://developers.cloudflare.com/d1/) - SQL database for document storage - [**Cloudflare Vectorize**](https://developers.cloudflare.com/vectorize/) - Vector search engine - [**Workers AI**](https://developers.cloudflare.com/ai/) - Cloudflare AI platform - [**Drizzle ORM**](https://orm.drizzle.team/) - Type-safe database access - [**Hono**](https://hono.dev/) - Lightweight framework for our API routes Here's the basic structure of the project: ``` . β”œβ”€β”€ bindings.ts # Bindings TypeScript Definition β”œβ”€β”€ bootstrap.sh # Automate creating the Cloudflare resources β”œβ”€β”€ drizzle.config.ts # Database configuration β”œβ”€β”€ fts5.sql # Full-text search SQL triggers β”œβ”€β”€ package.json # Dependencies and scripts β”œβ”€β”€ src β”‚ β”œβ”€β”€ db β”‚ β”‚ β”œβ”€β”€ index.ts # Database CRUD operations β”‚ β”‚ β”œβ”€β”€ schemas.ts # Databae schema definition β”‚ β”‚ └── types.ts # Database TypeScript types β”‚ β”œβ”€β”€ index.ts # Main Worker code and Durable Object β”‚ β”œβ”€β”€ search.ts # Search functionality β”‚ β”œβ”€β”€ utils.ts # Util functions β”‚ └── vectorize.ts # Vectorize operations β”œβ”€β”€ tsconfig.json # TypeScript configuration └── wrangler.json # Cloudflare Workers configuration ``` This final solution will be implementing the following architecture. The goal of this blog post is to walk through each item individually and finally explain how they all work together. The complete source code for this blog post is available on [GitHub](https://github.com/boristane/cloudflare-dev-101/tree/main/contextual-rag). I suggest following the post with the source code available for reference. ## Bootstrapping the RAG System We first need to create our cloud resources. Let's create a `bootstrap.sh` script that creates a Vectorize index and a D1 database: ```bash title="bootstrap.sh" #! /bin/bash set -e npx wrangler vectorize create contextual-rag-index --dimensions=1024 --metric=cosine npx wrangler vectorize create-metadata-index contextual-rag-index --property-name=timestamp --type=number npx wrangler d1 create contextual-rag ``` It use [wrangler](https://developers.cloudflare.com/workers/wrangler/) to create: 1. A Vectorize index for storing embeddings 2. A metadata index for timestamp-based filtering 3. A D1 database for document storage Running this script will output a database ID we will use in our `wrangler.json` and `drizzle.config.ts` files. Next, after creating a Node.js project with `npm` and configuring TypeScript, let's create our `wrangler.json` file to configure our Cloudflare Worker and connect our resources to it. ```json title="wrangler.json" { "name": "contextual-rag", "compatibility_date": "2024-11-12", "workers_dev": true, "upload_source_maps": true, "observability": { "enabled": true }, "main": "./src/index.ts", "vectorize": [ { "binding": "VECTORIZE", "index_name": "contextual-rag-index" } ], "d1_databases": [ { "binding": "D1", "database_name": "contextual-rag", "database_id": "" } ], "ai": { "binding": "AI" }, "vars": {} } ``` ## Document Ingestion Pipeline After initializing the project, the first step is to ingest documents we want to retrieve. We are building everything on a Hono application deployed on Cloudflare Workers. Our API endpoint for document uploads looks like this: ```typescript title="src/index.ts" app.post('/', async (c) => { const { contents } = await c.req.json(); if (!contents || typeof contents !== "string") return c.json({ message: "Bad Request" }, 400) const doc = await createDoc(c.env, { contents }); const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1024, chunkOverlap: 200, }); const raw = await splitter.splitText(contents); const chunks = await contextualizeChunks(c.env, contents, raw) await insertChunkVectors(c.env, { docId: doc.id, created: doc.created }, chunks); return c.json(doc) }); ``` It's a `POST` endpoint that takes the contents of the document to process. The processing steps are: 1. Store the complete document in the database 2. Split the document into manageable chunks 3. Contextualize each chunk 4. Generate vector embeddings for each chunk 5. Store the chunks and their embeddings ### Text Splitting We split the documents because LLMs have limited context windows, and retrieving entire documents for every query would be inefficient. By splitting documents into smaller chunks, we can retrieve only the most relevant pieces. We use the [RecursiveCharacterTextSplitter](https://v03.api.js.langchain.com/classes/langchain.text_splitter.RecursiveCharacterTextSplitter.html) from LangChain, which intelligently splits text based on content boundaries: ```typescript title="src/index.ts" const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1024, chunkOverlap: 200, }); const raw = await splitter.splitText(contents); ``` The `chunkSize` parameter controls the maximum size of each chunk, while `chunkOverlap` creates an overlap between adjacent chunks. This overlap helps maintain context across chunk boundaries and prevents information from being lost at the dividing points. ### Context Enhancement This is where we do the "contextual" part of Contextual RAG. Instead of storing raw chunks, we first enhance each chunk with contextual information situating it within the context of the full text: ```typescript title="src/vectorize.ts" export async function contextualizeChunks(env: { AI: Ai }, content: string, chunks: string[]): Promise { const promises = chunks.map(async c => { const prompt = ` ${content} Here is the chunk we want to situate within the whole document ${c} Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else. `; // @ts-ignore const res = await env.AI.run("@cf/meta/llama-3.1-8b-instruct-fast", { prompt, }) as { response: string } return `${res.response}; ${c}`; }) return await Promise.all(promises); } ``` Without the context, chunks become isolated islands of information, divorced from their surrounding context. For example, a chunk containing "it increases efficiency by 40%" is meaningless without knowing what "it" refers to. By adding contextual information to each chunk, we make them more self-contained and improve retrieval accuracy. We use an LLM to analyze the relationship between the chunk and the full document, then generate a short context summary that precedes the chunk text. This enhancement makes each chunk more self-contained and improves retrieval relevance. For example, I tried this with a short text describing Paris. After chunking the text, one of the chunks was: > The city doesn’t shout. It smirks. It moves with a kind of practiced nonchalance, a shrug that says, of course it’s beautiful here. It’s a city built for daydreams and contradictions, grand boulevards designed for kings now flooded with delivery bikes and tourists holding maps upside-down. Crumbling stone facades wear ivy and grime like couture. Every corner feels staged, but somehow still effortless, like the city isn’t even trying to impress you. There is no indication that this chunk is talking about Paris. The enhanced chunk after running through the LLM is: > The chunk describes the essence and atmosphere of Paris, highlighting its unique blend of beauty, history, and contradictions.; The city doesn’t shout. It smirks. It moves with a kind of practiced nonchalance, a shrug that says, of course it’s beautiful here. It’s a city built for daydreams and contradictions, grand boulevards designed for kings now flooded with delivery bikes and tourists holding maps upside-down. Crumbling stone facades wear ivy and grime like couture. Every corner feels staged, but somehow still effortless, like the city isn’t even trying to impress you. A retrieval query consisting of just the word **Paris** is more likely to match the enhanced chunk versus the raw chunk without context. ### Vector Generation and Storage After contextualizing the chunks, we generate vector embeddings and store them in Cloudflare Vectorize and D1: ```typescript title="src/vectorize.ts" export async function insertChunkVectors( env: { D1: D1Database, AI: Ai, VECTORIZE: Vectorize }, data: { docId: string, created: Date }, chunks: string[], ) { const { docId, created } = data; const batchSize = 10; const insertPromises = []; for (let i = 0; i < chunks.length; i += batchSize) { const chunkBatch = chunks.slice(i, i + batchSize); insertPromises.push( (async () => { const embeddingResult = await env.AI.run("@cf/baai/bge-large-en-v1.5", { text: chunkBatch, }); const embeddingBatch: number[][] = embeddingResult.data; const chunkInsertResults = await Promise.all(chunkBatch.map(c => createChunk(env, { docId, text: c }))) const chunkIds = chunkInsertResults.map((result) => result.id); await env.VECTORIZE.insert( embeddingBatch.map((embedding, index) => ({ id: chunkIds[index], values: embedding, metadata: { docId, chunkId: chunkIds[index], text: chunkBatch[index], timestamp: created.getTime(), }, })) ); })() ); } await Promise.all(insertPromises); } ``` This function: 1. Divides chunks into batches of 10 2. Generates embeddings for each batch using Cloudflare AI 3. Stores the chunks in the D1 database 4. Stores the embeddings and associated metadata in Vectorize The `createChunk` function inserts the chunks the D1 database and gives us a unique ID for each chunk, which is then shared in Vectorize. The next section gives more details on the D1 database. The metadata attached to each vector includes the original text, document ID, and timestamp, which enables filtering and improves retrieval performance. ### D1 Database Schema Our RAG system must combine full-text and vector search. We have two data models to store outside of the vector database: full documents and the chunks derived from them. For this, I'm using Cloudflare D1 (SQLite) with Drizzle ORM. Here's the schema definition: ```typescript title="src/db/schemas.ts" export const docs = sqliteTable( "docs", { id: text("id") .notNull() .primaryKey() .$defaultFn(() => randomString()), contents: text("contents"), created: integer("created", { mode: "timestamp_ms" }) .$defaultFn(() => new Date()) .notNull(), updated: integer("updated", { mode: "timestamp_ms" }) .$onUpdate(() => new Date()) .notNull(), }, (table) => ([ index("docs.created.idx").on(table.created), ]), ); export const chunks = sqliteTable( "chunks", { id: text("id") .notNull() .primaryKey() .$defaultFn(() => randomString()), docId: text('doc_id').notNull(), text: text("text").notNull(), created: integer("created", { mode: "timestamp_ms" }) .$defaultFn(() => new Date()) .notNull(), }, (table) => ([ index("chunks.doc_id.idx").on(table.docId), ]), ); function randomString(length = 16): string { const chars = "abcdefghijklmnopqrstuvwxyz"; const resultArray = new Array(length); for (let i = 0; i < length; i++) { const randomIndex = Math.floor(Math.random() * chars.length); resultArray[i] = chars[randomIndex]; } return resultArray.join(""); } ``` With these schemas, we can add CRUD functions to write reand and write documents and chunks in D1. ```typescript title="src/db/index.ts" export function getDrizzleClient(env: DB) { return drizzle(env.D1, { schema, }); } export async function createDoc(env: DB, doc: InsertDoc): Promise { const d1 = getDrizzleClient(env); const [res] = await d1 .insert(docs) .values(doc) .onConflictDoUpdate({ target: [docs.id], set: doc, }) .returning(); return res; } export async function listDocsByIds( env: DB, params: { ids: string[] }, ): Promise { const d1 = getDrizzleClient(env); const qs = await d1 .select() .from(docs) .where(inArray(docs.id, params.ids)) return qs; } export async function createChunk(env: DB, chunk: InsertChunk): Promise { const d1 = getDrizzleClient(env); const [res] = await d1 .insert(chunks) .values(chunk) .onConflictDoUpdate({ target: [chunks.id], set: chunk, }) .returning(); return res; } ``` ### SQLite Full-Text Search While vector search excels at semantic similarity, it often misses exact keyword matches that full-text search handles perfectly. Full-text search excels at finding exact keyword matches and can retrieve relevant content even when the semantic meaning might be ambiguous. By combining both approaches, we get the best of both worlds. SQLite provides a powerful full-text search extension called [FTS5](https://www.sqlite.org/fts5.html). We'll create a virtual table that mirrors our `chunks` table but with full-text search capabilities: ```sql title="fts5.sql" CREATE VIRTUAL TABLE chunks_fts USING fts5( id UNINDEXED, doc_id UNINDEXED, text, content = 'chunks', created ); CREATE TRIGGER chunks_ai AFTER INSERT ON chunks BEGIN INSERT INTO chunks_fts(id, doc_id, text, created) VALUES ( new.id, new.doc_id, new.text, new.created ); END; CREATE TRIGGER chunks_ad AFTER DELETE ON chunks FOR EACH ROW BEGIN DELETE FROM chunks_fts WHERE id = old.id; INSERT INTO chunks_fts(chunks_fts) VALUES('rebuild'); END; CREATE TRIGGER chunks_au AFTER UPDATE ON chunks BEGIN DELETE FROM chunks_fts WHERE id = old.id; INSERT INTO chunks_fts(id, doc_id, text, created) VALUES (new.id, new.doc_id, new.text, new.created); INSERT INTO chunks_fts(chunks_fts) VALUES('rebuild'); END; ``` The triggers ensure that our FTS table stays in sync with the main chunks table. Whenever a chunk is inserted, updated, or deleted, the corresponding entry in the FTS table is also modified. We use [DrizzleKit](https://orm.drizzle.team/docs/kit-overview) to create migration files for our D1 database with this schema. ```bash drizzle-kit generate ``` The above SQL for the FTS tables must be included manually in the migration file before running the migration in D1. Otherwise, we will not have full-text search capabilities in our SQLite database. ```bash drizzle-kit migrate ``` Once the migration is completed, we're ready to start processing documets in the ingestion side of our RAG system. ## Query Processing and Search Now let's examine how queries are handled. Our query endpoint looks like this: ```typescript title="src/index.ts" app.post('/query', async (c) => { const { prompt, timeframe } = await c.req.json(); if (!prompt) return c.json({ message: "Bad Request" }, 400); const searchOptions: { timeframe?: { from?: number; to?: number }; } = {}; if (timeframe) { searchOptions.timeframe = timeframe; } const ai = createWorkersAI({ binding: c.env.AI }); // @ts-ignore const model = ai("@cf/meta/llama-3.1-8b-instruct-fast") as LanguageModel; const { queries, keywords } = await rewriteToQueries(model, { prompt }); const { chunks } = await searchDocs(c.env, { questions: queries, query: prompt, keywords, topK: 8, scoreThreshold: 0.501, ...searchOptions, }); const uniques = getUniqueListBy(chunks, "docId").map((r) => { const arr = chunks .filter((f) => f.docId === r.docId) .map((v) => v.score); return { id: r.docId, score: Math.max(...arr), }; }); const res = await listDocsByIds(c.env, { ids: uniques.map(u => u.id) }); const answer = await c.env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", { prompt: `${prompt} Context: ${chunks}` }) return c.json({ keywords, queries, chunks, answer, docs: res.map(doc => ({ ...doc, score: uniques.find(u => u.id === doc.id)?.score || 0 })).sort((a, b) => b.score - a.score) }) }); function getUniqueListBy>(arr: T[], key: keyof T): T[] { const result: T[] = []; for (const elt of arr) { const found = result.find((t) => t[key] === elt[key]); if (!found) { result.push(elt); } } return result; } ``` It's a `POST` endpoint that uses the [AI SDK](https://sdk.vercel.ai/) and the [Workers AI Provider](https://github.com/cloudflare/ai/tree/main/packages/workers-ai-provider) to complete the following steps: 1. **Query rewriting**: Rewrite the user prompt into multiple related questions and keywords to improve RAG performance 2. **Hybrid search**: Combining vector and text search 3. **Result fusion and reranking** 4. **Answer generation** Let's examine each in detail. ### Query Rewriting Users rarely express their needs perfectly on the first try. Their queries are often ambiguous or lack specific keywords that would make retrieval effective. Query rewriting expands the original query into multiple variations: ```typescript title="src/search.ts" export async function rewriteToQueries(model: LanguageModel, params: { prompt: string }): Promise<{ keywords: string[], queries: string[] }> { const prompt = `Given the following user message, rewrite it into 5 distinct queries that could be used to search for relevant information, and provide additional keywords related to the query. Each query should focus on different aspects or potential interpretations of the original message. Each keyword should be a derived from an interpratation of the provided user message. User message: ${params.prompt}`; try { const res = await generateObject({ model, prompt, schema: z.object({ queries: z.array(z.string()).describe( "Similar queries to the user's query. Be concise but comprehensive." ), keywords: z.array(z.string()).describe( "Keywords from the query to use for full-text search" ), }), }) return res.object; } catch (err) { return { queries: [params.prompt], keywords: [] } } } ``` By generating multiple interpretations of the original query, we can capture different aspects and increase the likelihood of finding relevant information. The AI model generates: 1. A set of expanded queries that explore different interpretations of the user's question 2. Keywords extracted from the query for full-text search This approach dramatically improves search recall compared to using just the original query. For example, I sent this prompt: > what's special about paris? and it was rewritten as: ```json { "keywords": [ "paris", "eiffel tower", "monet", "loire river", "montmartre", "notre dame", "art", "history", "culture", "tourism" ], "queries": [ "paris attractions", "paris landmarks", "paris history", "paris culture", "paris tourism" ] } ``` By searching the databases with all these combinations, we increase the likelihood of finding relevant chunks. ### Hybrid Search With our rewritten queries and extracted keywords, we now perform a hybrid search using both vector similarity and full-text search: ```typescript title="src/search.ts" export async function searchDocs(env: SearchBindings, params: DocSearchParams): Promise<{ chunks: Array<{ text: string, id: string, docId: string; score: number }> }> { const { timeframe, questions, keywords } = params; const [vectors, sql] = (await Promise.all([ queryChunkVectors(env, { queries: questions, timeframe, },), searchChunks(env, { needles: keywords, timeframe }), ])); const searchResults = { vectors, sql: sql.map(item => { return { id: item.id, text: item.text, docId: item.doc_id, rank: item.rank, } }) }; const mergedResults = performReciprocalRankFusion(searchResults.sql, searchResults.vectors); const res = await processSearchResults(env, params, mergedResults); return res; } ``` Vector search excels at finding semantically similar content but can miss exact keyword matches while full-text search is great at finding keyword matches but lacks semantic understanding. By running both search types in parallel and then merging the results, we get more comprehensive coverage. The vector search looks for embeddings similar to our query embeddings, applying timestamp filters if they are provided: ```typescript title="src/vectorize.ts" export async function queryChunkVectors( env: { AI: Ai, VECTORIZE: Vectorize }, params: { queries: string[], timeframe?: { from?: number, to?: number } }) { const { queries, timeframe, } = params; const queryVectors = await Promise.all( queries.map((q) => env.AI.run("@cf/baai/bge-large-en-v1.5", { text: [q] })) ); const filter: VectorizeVectorMetadataFilter = { }; if (timeframe?.from) { // @ts-expect-error error in the package filter.timestamp = { "$gte": timeframe.from } } if (timeframe?.to) { // @ts-expect-error error in the package filter.timestamp = { "$lt": timeframe.to } } const results = await Promise.all( queryVectors.map((qv) => env.VECTORIZE.query(qv.data[0], { topK: 20, returnValues: false, returnMetadata: "all", filter, }) ) ); return results; } ``` The full-text search uses SQLite's FTS5 to find keyword matches: ```typescript title="src/db/index.ts" export async function searchChunks(env: DB, params: { needles: string[], timeframe?: { from?: number, to?: number }; }, limit = 40) { const d1 = getDrizzleClient(env); const { needles, timeframe } = params; const queries = needles.filter(Boolean).map( (term) => { const sanitizedTerm = term.trim().replace(/[^\w\s]/g, ''); return ` SELECT chunks.*, bm25(chunks_fts) AS rank FROM chunks_fts JOIN chunks ON chunks_fts.id = chunks.id WHERE chunks_fts MATCH '${sanitizedTerm}' ${timeframe?.from ? `AND created > ${timeframe.from}` : ''} ${timeframe?.to ? `AND created < ${timeframe.to}` : ''} ORDER BY rank LIMIT ${limit} `; } ); const results = await Promise.all( queries.map(async (query) => { const res = await d1.run(query); return res.results as ChunkSearch[]; }) ); return results.flat() } ``` We're using the [BM25 ranking algorithm](https://www.sqlite.org/fts5.html#the_bm25_function) (built into FTS5) to sort results by relevance. It's an algorithm that considers term frequency, document length, and other factors to determine relevance. ### Reciprocal Rank Fusion After getting results from both search methods, we need to merge them. This is where [Reciprocal Rank Fusion (RRF)](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking) comes in. It's a rank aggregation method that combines ranking from multiple sources into a single unified ranking. When you have multiple ranked lists from different systems, each with their own scoring method, it's challenging to merge them fairly. RRF provides a principled way to combine these lists by: 1. Considering the rank position rather than raw scores (which may not be comparable) 2. Giving higher weights to items that appear at high ranks in multiple lists 3. Using a constant `k` to mitigate the impact of outliers The formula gives each item a score based on its rank in each list: `score = 1 / (k + rank)`. Items that appear high in multiple lists get the highest combined scores. The constant `k` (set to 60 in our implementation) is crucial as it prevents items that appear at the very top of only one list from completely dominating the results. A larger k value makes the algorithm more conservative, reducing the advantage of top-ranked items and giving more consideration to items further down the lists. ```typescript title="src/utils.ts" export function performReciprocalRankFusion( fullTextResults: DocMatch[], vectorResults: VectorizeMatches[] ): { docId: string, id: string; score: number; text?: string }[] { const vectors = uniqueVectorMatches(vectorResults.flatMap(r => r.matches)); const sql = uniqueDocMatches(fullTextResults); const k = 60; // Constant for fusion, can be adjusted const scores: { [key: string]: { id: string, text?: string; docId: string, score: number } } = {}; // Process full-text search results sql.forEach((result, index) => { const key = result.id; const score = 1 / (k + index); scores[key] = { id: result.id, docId: result.docId, text: result.text, score: (scores[key]?.score || 0) + score, }; }); // Process vector search results vectors.forEach((match, index) => { const key = match.id; const score = 1 / (k + index); scores[key] = { id: match.id, docId: match.metadata?.docId as string, text: match.metadata?.text as string, score: (scores[key]?.score || 0) + score, }; }); const res = Object.entries(scores) .map(([key, { id, score, docId, text }]) => ({ docId, id, score, text })) .sort((a, b) => b?.score - a?.score); return res.slice(0, 150); } ``` ### AI Reranking After merging the results, we use another LLM to rerank them based on their relevance to the original query. The initial search and fusion steps are based on broader relevance signals. The reranker performs a more focused assessment of whether each result directly answers the user's question. It uses [baai/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base), a reranker model. These models are language models that reorder search results based on relevance to the user query, improving the qualify of the RAG. They take as input the user query and a list of documents, and return the order of the documents from most relevant to the query to least. ```typescript title="src/search.ts" async function processSearchResults(env: SearchBindings, params: DocSearchParams, mergedResults: { docId: string, id: string; score: number; text?: string }[]) { if (!mergedResults.length) return { chunks: [] }; const { query, scoreThreshold, topK } = params; const chunks: Array<{ text: string; id: string; docId: string } & { score: number }> = []; const response = await env.AI.run( "@cf/baai/bge-reranker-base", { // @ts-ignore query, contexts: mergedResults.map(r => ({ id: r.id, text: r.text })) }, ) as { response: Array<{ id: number, score: number }> }; const scores = response.response.map(i => i.score); let indices = response.response.map((i, index) => ({ id: i.id, score: sigmoid(scores[index]) })); if (scoreThreshold && scoreThreshold > 0) { indices = indices.filter(i => i.score >= scoreThreshold); } if (topK && topK > 0) { indices = indices.slice(0, topK) } const slice = reorderArray(mergedResults, indices.map(i => i.id)).map((v, index) => ({ ...v, score: indices[index]?.score })); await Promise.all(slice.map(async result => { if (!result) return; const a = { text: result.text || (await getChunk(env, { docId: result.docId, id: result.id }))?.text || "", docId: result.docId, id: result.id, score: result.score, }; chunks.push(a) })); return { chunks }; } ``` After reranking, we apply a sigmoid function to transform the raw scores: ```typescript title="src/search.ts" function sigmoid(score: number, k: number = 0.4): number { return 1 / (1 + Math.exp(-score / k)); } ``` The reranker's raw scores can have a wide range and don't directly translate to a probability of relevance. The sigmoid function has a few compelling characteristics: 1. **Bounded output range**: Sigmoid squashes values into a fixed range (0,1), creating a probability-like score that's easier to interpret and threshold. 2. **Non-linear transformation**: Sigmoid emphasizes differences in the middle range while compressing extremes, which is ideal for relevance scoring where we need to distinguish between "somewhat relevant" and "very relevant" items. 3. **Stability with outliers**: Extreme scores don't disproportionately affect the normalized range. 4. **Ordering**: Sigmoid maintains the relative ordering of results The parameter `k` controls the steepness of the sigmoid curve: a smaller value creates a sharper distinction between relevant and irrelevant results. The specific value of `k=0.4` was chosen after experimentation to create an effective decision boundary around the 0.5 mark. Results above this threshold are considered sufficiently relevant for inclusion in the final context. Below is a plot of our sigmoid function: We pass the reranker score of each chunk through the `signoid` function, and we can set a fixed threshold score as the `sigmoid` normalises the scores. We can ignore anything below the threshold score as these chunks are likely not relevant to the user query. Without the `sigmoid` function, each ranking would have a different score distribution and it would be difficult to fairly select a threshold score. ### Search Parameter Tuning The `searchDocs` function includes several parameters that significantly impact retrieval quality: ```typescript title="src/index.ts" const { chunks } = await searchDocs(c.env, { questions: queries, query: prompt, keywords, topK: 8, scoreThreshold: 0.501, ...searchOptions, }); ``` The `topK` parameter limits the number of chunks we retrieve, which is essential for: 1. Managing the LLM context window size limitations 2. Reducing noise in the context that could distract the model 3. Minimizing cost and latency The `scoreThreshold` is calibrated to work with our `sigmoid` normalization. Since sigmoid transforms scores to the (0,1) range with 0.5 representing the inflection point, setting the threshold just above 0.5 ensures we only include chunks that the reranker determined are more likely relevant than not. This threshold prevents the inclusion of marginally relevant content that might dilute the quality of our context window. ### Answer Generation Finally, we generate an answer using the retrieved chunks as context: ```typescript const answer = await c.env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", { prompt: `${prompt} Context: ${chunks}` }) ``` This step is where the "Generation" part of RAG comes in. The LLM receives both the user's question and the retrieved context, then generates an answer that draws on that context. ### Stitching it all together The entire system is an integrated pipeline where each component builds upon the previous ones: 1. When a query arrives, it first hits the Query Rewriter, which expands it into multiple variations to improve search coverage. 2. These expanded queries alongside extracted keywords simultaneously feed into two separate search systems: - Vector Search (semantic similarity using embeddings) - Full-Text Search (keyword matching using FTS5) 3. The results from both search methods then enter the Reciprocal Rank Fusion function, which combines them based on rank position. This solves the challenge of comparing scores from fundamentally different systems. 4. The fused results are then passed to the AI Reranker, which performs an LLM based relevance assessment focused on answering the original query. 5. The reranked results are then filtered by score threshold and count limits before being passed into the context window for the final LLM. 6. The LLM receives both the original query and the curated context to produce the final answer. This multi-stage approach creates a system that outperforms traditional RAG systems. ## Testing It Out Now let's see how our contextual RAG system works in practice. After deploying your Worker using `wrangler`, you'll get a URL for your Worker: ```txt https://contextual-rag.account-name.workers.dev ``` where `account-name` is the name of your Cloudflare account. Here are some example API calls: #### Uploading a document ```bash curl -X POST https://contextual-rag.account-name.workers.dev/ \ -H "Content-Type: application/json" \ -d '{"contents":"Paris doesn’t shout. It smirks. It moves with a kind of practiced nonchalance, a shrug that says, of course it’s beautiful here. It’s a city built for daydreams and contradictions β€” grand boulevards designed for kings now flooded with delivery bikes and tourists holding maps upside-down. Crumbling stone facades wear ivy and grime like couture. Every corner feels staged, but somehow still effortless, like the city isn’t even trying to impress you. It just is. Each arrondissement spins its own little universe. In the Marais, tiny alleys drip with charm β€” cafΓ©s packed so tightly you can hear every clink of every espresso cup. Montmartre clings to its hill like a stubborn old cat... "}' # Response: # { # "id": "jtmnofvveptdalwl", # "contents": "Paris doesn’t shout. It smirks. It moves with a kind of...", # "created": "2025-04-26T19:50:39.941Z", # "updated": "2025-04-26T19:50:39.941Z", # } ``` #### Querying ```bash curl -X POST https://contextual-rag.account-name.workers.dev/query \ -H "Content-Type: application/json" \ -d '{"prompt":"What is special about Paris?"}' # Response: # { # "keywords": [ # "paris", # "eiffel tower", # "monet", # "loire river", # "montmartre", # "notre dame", # "art", # "history", # "culture", # "tourism" # ], # "queries": [ # "paris attractions", # "paris landmarks", # "paris history", # "paris culture", # "paris tourism" # ], # "chunks": [ # { # "text": "This chunk describes Paris, the capital of France, its location on the Seine River, and its famous landmarks and characteristics.; Paris doesn’t shout. It smirks. It moves with a kind of practiced nonchalance, a shrug that says, of course it’s beautiful here. It’s a city built for daydreams and contradictions...", # "id": "abcdefghijklmno", # "docId": "qwertyuiopasdfg", # "score": 0.8085310556071287 # } # ], # "answer": "Paris, the capital of France, is known as the City of Light. It has been a hub for intellectuals and artists for centuries, and its stunning architecture, art museums, and romantic atmosphere make it one of the most popular tourist destinations in the world. Some of the most famous landmarks in Paris include the Eiffel Tower, the Louvre Museum...", # "docs": [ # { # "id": "jtmnofvveptdalwl", # "contents": "Paris doesn’t shout. It smirks. It moves with a kind of practiced nonchalance, a shrug that says, of course it’s beautiful here. It’s a city built for daydreams and contradictions...", # "created": "2025-04-26T19:50:39.941Z", # "updated": "2025-04-26T19:50:39.941Z", # "score": 0.8085310556071287 # } # ] # } ``` ## Limitations and Considerations As much as this contextual RAG system is powerful, there are several limitations and considerations to keep in mind: 1. **AI Costs**: The contextual enhancement process requires running each chunk through an LLM, which increases both the computation time and cost compared to traditional RAG systems. For very large document collections, this can become a significant consideration. 2. **Latency Trade-offs**: Adding multiple stages of processing (query rewriting, hybrid search, reranking) improves result quality but increases end-to-end latency. For applications where response time is critical, you might need to optimize certain components or make trade-offs. 3. **Storage Growth**: The contextualized chunks are longer than raw chunks, requiring more storage in both D1 and Vectorize. It's important to monitor the size of your databases and remain below limits. 4. **Rate Limits**: All components of the pipeline have rate-limits. This can affect high-throughput applications. Consider offloading the document ingestion into a queue in production. 5. **Context Window Limitations**: When contextualizing chunks, very long documents may exceed the context window of the LLM. You might need to implement a hierarchical approach for extremely large documents. On [Basebrain](https://basebrain.ai), I've addressed some of these challenges by using [Cloudflare Workflows](https://developers.cloudflare.com/workflows/) for document ingestion, and Durable Objects instead of D1 for storage, where I implement [one database per user](/blog/durable-objects-database-per-user.mdx). --- ### One Database Per User with Cloudflare Durable Objects and Drizzle ORM URL: https://boristane.com/blog/durable-objects-database-per-user/ Description: Using Durable Objects and Drizzle ORM to create isolated SQLite databases for each user in a Hono app I'm building a production-ready app entirely on the [Cloudflare Developers Platform](https://workers.dev) ([Basebrain](https://basebrain.ai)) and I want to share implementation details of the key patterns I use. Today, I'm focusing on a specific pattern that's surprisingly simple to implement: creating one isolated database per user using Durable Objects and Drizzle ORM. [Durable Objects](https://www.cloudflare.com/developer-platform/products/durable-objects/) are one of the most powerful primitives on the Cloudflare Developer Platform, but I have seen many people struggle to define them. I see them as **mini servers without the ops**. Each object (server) is: - Addressable by a unique name/ID, like a key in a distributed system. - Run in a single location at any given time, so no race conditions. - Can store persistent state, backed by Durable Object Storage. - Can coordinate between multiple clients, like a chat room, game session, or rate limiter. They're great when you need strong consistency, low-latency coordination, or lightweight stateful logic. Durable Objects provide a convenient [SQLite API](https://blog.cloudflare.com/sqlite-in-durable-objects/), and when combined with Drizzle ORM, give you a clean, type-safe way to create a database per user in your applications. ## One Database Per User Before looking at implementation, let's understand why you might want one database per user: 1. **Scalability** - Sharding a database into one instance per user addresses the limitations of storing and managing large datasets on a single database server 2. **True isolation** - Each user's data is completely isolated 3. **Simplified access control** - No complex filtering or multi-tenancy code 4. **Performance** - Durable Objects are distributed globally, and instanciated closest to the first request they receive. As such, each database will be instanciated closest to its users, significantly reducing end-to-end latency In this example, we'll build a simple notes API where each user gets their own SQLite database. A user can create, read, list, or delete notes. They will be able to interact exclusively with their own database. By definition, a user will not be able to access another user's data. The complete code for this tutorial is available on [GitHub](https://github.com/boristane/cloudflare-dev-101/tree/main/durable-objects-database-per-user). Throughout this example, we'll assume you're familiar with deploying [Cloudflare Workers](https://developers.cloudflare.com/workers/) and we will focus on Durable Objects. ## Project Overview Let's start with an overview of the project. We'll be using: - [**Cloudflare Workers**](https://developers.cloudflare.com/workers/) - Our serverless platform - [**Durable Objects**](https://developers.cloudflare.com/durable-objects/) - To maintain per-user databases - [**Drizzle ORM**](https://orm.drizzle.team/) - For type-safe database operations - [**Hono**](https://hono.dev/) - A lightweight framework for our API routes Here's the basic structure of the project: ``` . β”œβ”€β”€ bindings.ts # Bindings TypeScript Definition β”œβ”€β”€ package.json # Dependencies and scripts β”œβ”€β”€ src β”‚ β”œβ”€β”€ db # Database-related code β”‚ β”‚ β”œβ”€β”€ index.ts # CRUD operations β”‚ β”‚ β”œβ”€β”€ notes.ts # Schema definition β”‚ β”‚ β”œβ”€β”€ schemas.ts # Schema exports β”‚ β”‚ └── types.ts # TypeScript types β”‚ └── index.ts # Main Worker code and Durable Object β”œβ”€β”€ tsconfig.json # TypeScript configuration └── wrangler.json # Cloudflare Workers configuration ``` ## Bootstrapping the Durable Object Let's start with bootstrapping our Durable Object code. Each Durable Object is part of a [Durable Object Namespace](https://developers.cloudflare.com/durable-objects/api/namespace/). A Durable Object Namespace is how you reference a Durable Object class from your Worker. It's like a binding that lets you create or execute methods in the Durable Objects from your Worker. A Durable Object Namespace must be bound to your Worker before you can instanciate Durable Objects. After creating a Node.js project with `npm` and configuring TypeScript, let's create our `wrangler.json` file to configure our Worker and Durable Object Namespace. ```json title="wrangler.json" { "name": "durable-objects-database-per-user", "compatibility_date": "2024-11-12", "workers_dev": true, "upload_source_maps": true, "observability": { "enabled": true }, "main": "./src/index.ts", "migrations": [ { "new_sqlite_classes": [ "DurableDatabase" ], "tag": "v1" } ], "durable_objects": { "bindings": [ { "class_name": "DurableDatabase", "name": "DurableDatabase" } ] } } ``` Our Worker script name will be `durable-objects-database-per-user`, it will have a `worker.dev` URL, and observability enabled. Our Durable Object class will be `DurableDatabase` and we enabled the Durable Object SQLite API on it. Next, let's write the code for our Worker and our Durable Objects. ```ts title="src/index.ts" const app = new Hono<{ Bindings: Bindings }>(); app.get('/', async (c) => { return c.json({ message: "Hello World!" }) }); export default { async fetch(request: Request, env: Bindings, ctx: ExecutionContext): Promise { return app.fetch(request, env, ctx); }, }; export class DurableDatabase extends DurableObject { constructor(ctx: DurableObjectState, env: Bindings) { super(ctx, env); } } ``` It's a Hono API, and the `DurableDatabase` doesn't do anything yet. We will come back to implement its methods. We import `Bindings` from our bindings definition file. The `Bindings` type enables us to have type-safety and auto-completion when interacting with resources bound to the Worker. ```typescript title="src/bindings.ts" export type Bindings = { DurableDatabase: DurableObjectNamespace; }; ``` We are pointing out to TypeScript that there's a Durable Object Namespace called `DurableDatabase` implementing the Durable Object class `DurableDatabase` bound to our Worker. ## Database Schema Now let's define our database schema. Let's keep it simple: a note has an ID, a text content, and created and updated timestamps. We use the `sqliteTable` function from Drizzle ORM to create schema for our table: ```ts title="src/db/notes.ts" export const notes = sqliteTable( "notes", { id: text("id") .notNull() .primaryKey() .$defaultFn(() => `note_${randomString()}`), text: text("text").notNull(), created: integer("created", { mode: "timestamp_ms" }) .$defaultFn(() => new Date()) .notNull(), updated: integer("updated", { mode: "timestamp_ms" }) .$onUpdate(() => new Date()) .notNull(), }, ); // Helper to generate random IDs for our notes function randomString(length = 16): string { const chars = "abcdefghijklmnopqrstuvwxyz"; const resultArray = new Array(length); for (let i = 0; i < length; i++) { const randomIndex = Math.floor(Math.random() * chars.length); resultArray[i] = chars[randomIndex]; } return resultArray.join(""); } ``` Alongside our schemas, we need to define the TypeScript types of notes saved in the database. We can infer those from the schema thanks to Drizzle `$infer` properties. ```ts title="src/db/types.ts" export type Note = typeof notes.$inferSelect; export type InsertNote = typeof notes.$inferInsert; ``` We also create a type for our Database, such that we can easily use it when making database operations. ```ts title="src/db/types.ts" ins={1,5} export type DB = DrizzleSqliteDODatabase; export type Note = typeof notes.$inferSelect; export type InsertNote = typeof notes.$inferInsert; ``` And we export all the schemas in a single file. ```ts title="src/db/schemas.ts" export * from "./notes"; ``` ## Database Operations Now that we have a schema, let's create the functions that will interact with our database. These will be the core operations our Durable Object will expose. Each use should have the ability to create a new note, delete a note by ID, get a specific note by ID, and list all notes. ```typescript title="src/db/index.ts" // Create a new note export async function create(db: DB, note: InsertNote): Promise { const [res] = await db .insert(notes) .values(note) .onConflictDoUpdate({ target: [notes.id], set: note, }) .returning(); return res; } // Delete a note by ID export async function del(db: DB, params: { id: string }): Promise { const [note] = await db .delete(notes) .where(eq(notes.id, params.id)) .returning(); return note; } // Get a note by ID export async function get(db: DB, params: { id: string }): Promise { const [result] = await db .select() .from(notes) .where(eq(notes.id, params.id)); if (!result) return null; return result; } // List all notes export async function list(db: DB): Promise { const ns = await db .select() .from(notes) return ns; } ``` These operations are using fairly standard Drizzle's query builder API functions, it gives us a clean, fluent interface for building SQL queries. ## Implementing the Durable Object Now let's look at how we implement our Durable Object. This is where the magic happens, each Durable Object instance will be tied to a specific user and contain their own SQLite database. ```typescript title="src/index.ts" ins={4-6,8,27-28,30-45} const app = new Hono<{ Bindings: Bindings }>(); app.get('/', async (c) => { return c.json({ message: "Hello World!" }) }); export default { async fetch(request: Request, env: Bindings, ctx: ExecutionContext): Promise { return app.fetch(request, env, ctx); }, }; export class DurableDatabase extends DurableObject { private db: DB; constructor(ctx: DurableObjectState, env: Bindings) { super(ctx, env); // Initialize Drizzle with the Durable Object's storage this.db = drizzle(ctx.storage, { schema, logger: true }); } async notesCreate(note: Parameters[1]): ReturnType { return await notes.create(this.db, note); } async notesGet(params: Parameters[1]): ReturnType { return await notes.get(this.db, params); } async notesList(): ReturnType { return await notes.list(this.db); } async notesDel(params: Parameters[1]): ReturnType { return await notes.del(this.db, params); } } ``` Let's break down what's happening here: 1. In the constructor, we initialize Drizzle with `ctx.storage`, which is the Durable Object's built-in storage that SQLite operates on. This gives us a `DrizzleSqliteDODatabase` which is unique per Durable Object. Everytime we do an operation using this object, we are interacting with a single SQLite instance. 3. We create methods that map directly to our database operations. These methods will be called from our API routes. The key insight here is that each instance of our `DurableDatabase` class is completely isolated from other instances. When we create a Durable Object for a user, they get their own private database. ## Database Migrations Database migration for Durable Object SQLite can be confusing, but the key thing to understand is that migrations must be run in the constructor of the Durable Object class. Durable Object constructors run when a request is routed to an instance that isn't currently active, basically on cold start. When you try to access a specific Durable Object from a Worker using its name/ID, if the Durable Object for that ID isn't already in memory somewhere in Cloudflare's edge network, a new instance spins up and the constructor is run once. After that, the object stays warm for a while (usually minutes of inactivity) before being evicted from memory. If another request comes in later, the object will spin up again and the constructor is called again. It then makes sense to run migrations in the Durable Object class constructor. When a new request is routed to a specific Durable Objects, the migration files since the last migration will be run in the constructor. This enables doing migrations on potentially millions of Durable Objects, "just-in-time". ```typescript title="src/index.ts" ins={5-6,31-35,53-56} const app = new Hono<{ Bindings: Bindings }>(); app.get('/', async (c) => { return c.json({ message: "Hello World!" }) }); export default { async fetch(request: Request, env: Bindings, ctx: ExecutionContext): Promise { return app.fetch(request, env, ctx); }, }; export class DurableDatabase extends DurableObject { private db: DB; constructor(ctx: DurableObjectState, env: Bindings) { super(ctx, env); // Initialize Drizzle with the Durable Object's storage this.db = drizzle(ctx.storage, { schema, logger: true }); // Run migrations before accepting any requests ctx.blockConcurrencyWhile(async () => { await this._migrate(); }); } async notesCreate(note: Parameters[1]): ReturnType { return await notes.create(this.db, note); } async notesGet(params: Parameters[1]): ReturnType { return await notes.get(this.db, params); } async notesList(): ReturnType { return await notes.list(this.db); } async notesDel(params: Parameters[1]): ReturnType { return await notes.del(this.db, params); } private async _migrate() { await migrate(this.db, migrations); } } ``` We use `blockConcurrencyWhile` to run migrations before accepting any requests. This ensures our database schema is up-to-date. You're asking yourself what is the `migrations` we import from `../drizzle/migrations`. These are database migration files created by [Drizzle Kit](https://orm.drizzle.team/docs/kit-overview). Drizzle Kit is a CLI tool for managing SQL database migrations with Drizzle. Let's define our Drizzle config with the `durable-sqlite` driver. ```ts title="drizzle.config.ts" export default defineConfig({ out: './drizzle', schema: './src/db/schemas.ts', dialect: 'sqlite', driver: 'durable-sqlite', }); ``` To create the migration files, it's necessary to run the Drizzle Kit `generate` command: ```bash drizzle-kit generate ``` This command will create a new folder `drizzle` in our project structure, which will keep track of all our migrations. After every change to our database schema, we need to run the same command to generate migration files. These migration files need to be uploaded alongside our Worker code, such that they can be run on Durable Object cold-starts. However, [`wrangler`](https://developers.cloudflare.com/workers/wrangler/), the Cloudflare Developer Platform command-line interface, only uploads files it generates when bundling a project. It is necessary to explicitely require migration files to be uploaded alongside the Worker code. ```json title="wrangler.json" ins={18-26} { "name": "durable-objects-database-per-user", "compatibility_date": "2024-11-12", "workers_dev": true, "upload_source_maps": true, "observability": { "enabled": true }, "main": "./src/index.ts", "migrations": [ { "new_sqlite_classes": [ "DurableDatabase" ], "tag": "v1" } ], "rules": [ { "type": "Text", "globs": [ "**/*.sql" ], "fallthrough": true } ], "durable_objects": { "bindings": [ { "class_name": "DurableDatabase", "name": "DurableDatabase" } ] } } ``` The `rules` section tells Wrangler to include all `.sql` files in our deployment. Without this rule, our SQL migration files wouldn't be included in the deployment, and our migrations would fail. ## Setting Up API Routes Now let's set up our API routes to interact with our Durable Objects. ```typescript title="src/index.ts" ins={23-65} const app = new Hono<{ Bindings: Bindings }>(); app.get('/', async (c) => { return c.json({ message: "Hello World!" }) }); export default { async fetch(request: Request, env: Bindings, ctx: ExecutionContext): Promise { return app.fetch(request, env, ctx); }, }; function getDurableDatabaseStub(env: Bindings, userId: string) { const doId = env.DurableDatabase.idFromName(userId); return env.DurableDatabase.get(doId); } // Create a note for a user app.post('/:userId', async (c) => { const userId = c.req.param("userId"); const { text } = await c.req.json(); const stub = getDurableDatabaseStub(c.env, userId); const note = await stub.notesCreate({ text }); return c.json({ note }) }); // List all notes for a user app.get('/:userId', async (c) => { const userId = c.req.param("userId"); const stub = getDurableDatabaseStub(c.env, userId); const notes = await stub.notesList() return c.json({ notes }) }); // Get a specific note for a user app.get('/:userId/:noteId', async (c) => { const userId = c.req.param("userId"); const noteId = c.req.param("noteId"); const stub = getDurableDatabaseStub(c.env, userId); const note = await stub.notesGet({ id: noteId }); if (!note) { return c.notFound(); } return c.json({ note }) }); // Delete a note for a user app.delete('/:userId/:noteId', async (c) => { const userId = c.req.param("userId"); const noteId = c.req.param("noteId"); const stub = getDurableDatabaseStub(c.env, userId); const note = await stub.notesDel({ id: noteId }); return c.json({ note }) }); export class DurableDatabase extends DurableObject { private db: DB; constructor(ctx: DurableObjectState, env: Bindings) { super(ctx, env); // Initialize Drizzle with the Durable Object's storage this.db = drizzle(ctx.storage, { schema, logger: true }); // Run migrations before accepting any requests ctx.blockConcurrencyWhile(async () => { await this._migrate(); }); } async notesCreate(note: Parameters[1]): ReturnType { return await notes.create(this.db, note); } async notesGet(params: Parameters[1]): ReturnType { return await notes.get(this.db, params); } async notesList(): ReturnType { return await notes.list(this.db); } async notesDel(params: Parameters[1]): ReturnType { return await notes.del(this.db, params); } private async _migrate() { await migrate(this.db, migrations); } } ``` The key pattern in each route is: - Extract the `userId` from the URL - Get a Durable Object ID using `idFromName(userId)` - Get a stub to the Durable Object using `get(doId)` - Call methods on the stub to interact with the user's database A durable object stub is the proxy we use to send requests to a specific Durable Object instance. We get the Durable Object ID from the `userId`, and we use it to instanciate a lightweight proxy object (the stub). ```ts function getDurableDatabaseStub(env: Bindings, userId: string) { const doId = env.DurableDatabase.idFromName(userId); return env.DurableDatabase.get(doId); } ``` With the stub, we can call methods on the Durable Object using [JavaScript-native RPC](https://blog.cloudflare.com/javascript-native-rpc/). That's how we're able to call the `notesCreate`, `notesGet`, `notesDel`, and `notesList` methods in the Durable Object. This is the core of our "one database per user" pattern. By using `idFromName(userId)`, we're creating a consistent mapping from user IDs to Durable Objects. Each user gets their own Durable Object, and thus their own database. ## Testing It Out Now let's see how our "one database per user" pattern works in practice. After you have deployed your Worker using `wrangler`, you'll get a URL for your Worker: ```txt https://durable-objects-database-per-user.account-name.workers.dev ``` where `account-name` is the name of your account. Here are some example API calls: #### Creating a note for user "john" ```bash curl -X POST https://durable-objects-database-per-user.account-name.workers.dev/john \ -H "Content-Type: application/json" \ -d '{"text":"Buy groceries"}' # Response: # { # "note": { # "id": "note_jxqegmbonzvdstpy", # "text": "Buy groceries", # "created": "2025-03-23T15:42:18.760Z", # "updated": "2025-03-23T15:42:18.760Z" # } # } ``` #### Listing all notes for "john" ```bash curl https://durable-objects-database-per-user.account-name.workers.dev/john # Response: # { # "notes": [ # { # "id": "note_jxqegmbonzvdstpy", # "text": "Buy groceries", # "created": "2025-03-23T15:42:18.760Z", # "updated": "2025-03-23T15:42:18.760Z" # } # ] # } ``` #### Creating a note for user "bob" ```bash curl -X POST https://durable-objects-database-per-user.account-name.workers.dev/bob \ -H "Content-Type: application/json" \ -d '{"text":"Finish the report"}' # Response: # { # "note": { # "id": "note_lnkcpfqyrtzbmags", # "text": "Finish the report", # "created": "2025-03-23T15:43:05.120Z", # "updated": "2025-03-23T15:43:05.120Z" # } # } ``` #### Listing all notes for "bob" ```bash curl https://durable-objects-database-per-user.account-name.workers.dev/bob # Response: # { # "notes": [ # { # "id": "note_lnkcpfqyrtzbmags", # "text": "Finish the report", # "created": "2025-03-23T15:43:05.120Z", # "updated": "2025-03-23T15:43:05.120Z" # } # ] # } ``` The key insight here is that "john" and "bob" have completely separate databases. ## Limitations and Considerations While this pattern is powerful, there are a few things to keep in mind: 1. **Storage Limits** - Each Durable Object has a storage limit (currently 10GB), which could be limiting and push us to further sharding depending on the scale of our app 2. **Cross-User Operations** - Queries across multiple users' data, for example back-office queries or analytics, require additional work On [Basebrain](https://basebrain.ai), I'm going around these with further sharding and by using a different database for analytics, [Workers Analytics Engine](https://blog.cloudflare.com/workers-analytics-engine/). --- ### Observability wide events 101 URL: https://boristane.com/blog/observability-wide-events-101/ Description: What are wide events, why and how should you implement them? Wide events are a very simple concept: for each request, emit a single context-rich event/log per service hop. That's it. Don't let all the buzzwords fool you. Let's say you're building a blogging platform, and users can save articles. A simplified implementation of the POST /articles endpoint might look like this: - User makes a request from a browser - Request hits your `gateway` which authenticates and orchestrates various downstream services - `gateway` calls the `articles` service. This service adds the article in the database and caches the result - `gateway` calls the `notifications` service which emails all relevant subscribers to the blog - `gateway` calls the `analytics` service which sends a message to a queue for product analytics In a nutshell, each of these services, from the browser to the database and queue, should emit a single structured wide-event with an arbitrary number of fields. All these events must be connected with a request ID, such that you can correlate all of them together. ## Why use wide events? Zooming in on the `articles` service, its events should include details from your business logic such as the user, their subscription, the saved article, the response status code, etc.: ```json showLineNumbers=false { "method": "POST", "path": "/articles", "service": "articles", "outcome": "ok", "status_code": 201, "duration": 268, "requestId": "8bfdf7ecdd485694", "timestamp":"2024-09-08 06:14:05.680", "message": "Article created", "commit_hash": "690de31f245eb4f2160643e0dbb5304179a1cdd3", "user": { "id": "fdc4ddd4-8b30-4ee9-83aa-abd2e59e9603", "activated": true, "subscription": { "id": "1aeb233c-1572-4f54-bd10-837c7d34b2d3", "trial": true, "plan": "free", "expiration": "2024-09-16 14:16:37.980", "created": "2024-08-16 14:16:37.980", "updated": "2024-08-16 14:16:37.980" }, "created": "2024-08-16 14:16:37.980", "updated": "2024-08-16 14:16:37.980" }, "article": { "id": "f8d4d21c-f1fd-48b9-a4ce-285c263170cc", "title": "Test Blog Post", "ownerId": "fdc4ddd4-8b30-4ee9-83aa-abd2e59e9603", "published": false, "created": "2024-09-08 06:14:05.460", "updated": "2024-09-08 06:14:05.460" }, "db": { "query": "INSERT INTO articles (id, title, content, owner_id, published, created, updated) VALUES ($1, $2, $3, $4, $5, $6, $7);", "parameters": { "$1": "f8d4d21c-f1fd-48b9-a4ce-285c263170cc", "$2": "Test Blog Post", "$3": "******", "$4": "fdc4ddd4-8b30-4ee9-83aa-abd2e59e9603", "$5": false, "$6": "2024-09-08 06:14:05.460", "$7": "2024-09-08 06:14:05.460" } }, "cache": { "operation": "write", "key": "f8d4d21c-f1fd-48b9-a4ce-285c263170cc", "value": "{\"article\":{\"id\":\"f8d4d21c-f1fd-48b9-a4ce-285c263170cc\",\"title\":\"Test Blog Post\"..." }, "headers": { "accept-encoding": "gzip, br", "cf-connecting-ip": "*****", "connection": "Keep-Alive", "content-length": "1963", "content-type": "application/json", "host": "website.com", "url": "https://website.com/articles", "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36", "Authorization": "********", "x-forwarded-proto": "https", "x-real-ip": "******" } } ``` At a glance, you can see that an article was posted by a user on a free trial expiring on September 16th. The service responded with `201` status code in `268`ms. Wide events must have the following characteristics: - **high cardinality**: each field can contain an unbounded number of unique values, such as user IDs, session IDs, or transaction IDs. You could have billions of these per day. - **high dimensionality**: by definition, wide events should have a large number of fields (dimensions) to provide deep insights - **context-rich**: all those fields should carry context about the request, from request headers to infrastructure details, and custom business logic data ### The problem with logs and metrics Wide events enable you to answer questions that are simply impossible to answer with traditional logs or metrics. Imagine, instead of wide events, we had opted for logs and metrics in the `articles` service: ```txt showLineNumbers=false 2024-09-08 06:14:05.280 Received POST /articles request 2024-09-08 06:14:05.298 Saving article: f8d4d21c-f1fd-48b9-a4ce-285c263170cc 2024-09-08 06:14:05.449 Article saved: f8d4d21c-f1fd-48b9-a4ce-285c263170cc 2024-09-08 06:14:05.451 Response time: 254ms 2024-09-08 06:14:05.460 Successful request: 201 ``` And a set of metrics charts, for request duration, number of articles created and number of failures on the route. Now imagine this user emails you, saying every time they create an article, it doesn't appear on the website, along with a video showing the issue. And 67 more users give you the same feedback. How do you start debugging this. Your logs tell you everything is fine, the articles are in the database, and your metrics chart don't show any unexpected behaviour. This is what we call **unknown unknowns**. Logs and metrics help capture 'known unknowns' - issues you can anticipate while building your application. Things such as slow requests, errors, database failures, or obvious potential issues in your business logic; for example missing environment variables, or "impossible" code paths, etc. But you are left hanging when it comes to unexpected behaviour you couldn't predict before your code got into the hands of real users and they started doing unexpected things. With wide events, and the **appropriate tooling**, you can investigate and solve these issues without pulling your hair out. I emphasize on appropriate tooling; wide events are only the first piece, and without appropriate tooling you're only halfway there. ### Tooling Whichever tool you use, it should have the following characteristics: {/* - **high cardinality**: each field in your event should support an arbitrary large number of unique values. For example request ID. A successful application could have 1000s of requests per second. - **high dimensionality**: each event should have an arbitrary large number of fields. From a handful to hundreads. */} - **queryable across any dimension**: you should be able to query across any of the fields in your events - **no pre-aggregation**: your events should be stored as they are emitted, without pre-aggregation. you should have access to the raw data, not just a value that was extracted from a batch of events - **fast**: querying your events should be fast, ideally sub-second; but definitely sub-minute. - **affordable**: observability should not bankrupt your application, sampling can drastically help here. I used to be a [vendor](https://baselime.io) and hitting all those points is extremely hard, but you should demand no less from your vendor or your custom-built solution. ## How to use wide-events For the eagle-eyed, the wide event we illustrated above actually has the answer to our previous bug (newly posted articles don't show on the website for a subset of users). But let's plot investigate with some graphs based on our wide events. Let's start with number of articles posted. I'll use the SQL syntax to illustrate the queries I'm writing to generate the graphs but your solution probably has a custom query language. ### Number of articles posted ```sql showLineNumbers=false select count() from events where method = "POST" and path = "/articles" and status_code = 201 ``` Nothing to note on this chart, the number of articles successfully posted has been constant. But are they published? Let's group by `article.published`. ### Number of articles posted grouped by `article.published` ```sql showLineNumbers=false select count() from events where method = "POST" and path = "/articles" and status_code = 201 group by article.published order by count() desc ``` Clearly, the reason users can't see their articles is because they are posted but they are set to `published = false`. But why? Let's see if it's just a subset of users or if it's a wide-spread issue. ### Number of unpublished articles grouped by `user.id` ```sql showLineNumbers=false select count() from events where method = "POST" and path = "/articles" and status_code = 201 and article.published = false group by user.id order by count() desc ``` Okay, this is not an isolated issue, multiple users are impacted. But how many exactly? Let's count all the unique users who posted an article and group them by `article.published`. ### Number of unique users grouped by `article.published` ```sql showLineNumbers=false select count(unique user.id) from events where method = "POST" and path = "/articles" and status_code = 201 group by article.published ``` Very few users were posting unpublished articles, and suddenly, most started posting unpublished articles. Clearly this is not a subset of users. But what do these users have in common? More traditional tooling might get you up to here, if you have diligently implemented structured logs and you have a whole catalog of metrics and dashboards. The next step is usually to go almost randomly look at code, or rely on memory of the day code was written to find the issue. But you don't need this with wide events. Your setup will tell you exactly what the issue is, you just have to ask. Are all these users with unpublished articles on the free trial? ### Number of unique users grouped by `article.published` and `user.trial` ```sql showLineNumbers=false select count(unique user.id) from events where method = "POST" and path = "/articles" and status_code = 201 group by article.published, user.trial ``` It's obvious only users on the free trial are impacted. And given they out-number paid users, most articles are posted with `article.published = false`. It's clear paid customers are not impacted by the issue. If additionally, your tooling enables you to write metadata markers such as commits or deployments, you can pinpoint the commit that introduced the defect. A single click and you're directly sent to the diff that caused the issue. ## How to implement wide-events? Here's a very crude implementation of the basic principle of wide events. ```ts app.post('/articles', async (c) => { const startTime = Date.now(); // initialise the wide event const wideEvent: Record = { method: 'POST', path: '/articles', service: 'articles', requestId: c.get("requestId"), headers: c.req.raw.headers, // optionally add then environment variables // ensure no secrets are stored here env: process.env, }; try { const body = await c.req.json(); const { title, content } = body; const user = database.getUser(c.get("userId")); wideEvent["user"] = user; const article = { id: uuidv4(), title, content, ownerId: user.id, published: true, }; const { savedArticle, dbOperation } = await database.saveArticle(article); wideEvent["article"] = savedArticle; wideEvent["db"] = dbOperation; const cacheResponse = await cache.set(articleId, article); wideEvent["cache"] = cacheResponse; const response = { message: 'Article created', article }; wideEvent["status_code"] = 201; wideEvent["message"] = 'Article created'; wideEvent["outcome"] = 'ok'; return c.json(response, 201); } catch (error) { wideEvent["outcome"] = 'error'; wideEvent["status_code"] = 500; wideEvent["message"] = error.message; return c.json({ error: 'Internal Error' }, 500); } finally { const duration = Date.now() - startTime; wideEvent["duration"] = duration; wideEvent["timestamp"] = new Date().toISOString(); // flush the wide event logger.info(JSON.stringify(wideEvent)); } }); ``` You can build from this, with middlewares, helper functions to add multiple keys simultaneously and also with queues to ensure the wide event is flushed even when the request fails or timesout. ## What about OpenTelemetry? Our crude implementation of wide events above has a major flaw: how do you propagate the `requestId` we pick up at `Line 9` across services and multiple calls. [Distributed tracing](https://baselime.io/glossary/distributed-tracing) takes the idea of wide events and builds on top: It enables you to propagate `requestId` automatically, as well as capturing timestamps and keeping a hierarchy between multiple service calls. It also formalises the language around wide events. Instead of "wide event", within the context of distributed tracing, you will say [span](https://baselime.io/glossary/span). Instead of "request", you'll say [trace](https://baselime.io/glossary/trace). This is because distributed tracing also works outside the context of request/response applications. For background jobs, long running tasks and event streaming for example. [OpenTelemetry](https://opentelemetry.io/) is an attempt to further formalise the instrumentation, collection and exportation of distributed traces. It's a complex project with a very vast and rich history. It's pretty easy today to get confused by Opentelemetry, but if you look at it as a simpler way to generate wide events, you're winning. You don't need to understand [Opentelemetry Collectors](https://opentelemetry.io/docs/collector/), [Baggage](https://opentelemetry.io/docs/concepts/signals/), or [Resources](https://opentelemetry.io/docs/concepts/resources/). Unless you're building your own OpenTelemetry backend, most of these concepts are mostly relevant to your vendor. Pick the distro for your language / framework (a language distro is an SDK). Install and configure it to automatically capture all i/o calls, and figure out how to add custom attributes to the span (remember, a span is just a wide event). In node.js it looks like ```ts app.post('/articles', async (c) => { const currentSpan = trace.getSpan(context.active()); try { const body = await c.req.json(); const { title, content, } = body; const user = database.getUser(c.get("userId")); currentSpan.setAttributes(user); const article = { id: uuidv4(), title, content, ownerId: user.id, published: true, }; const savedArticle = await database.saveArticle(article); currentSpan.setAttributes(savedArticle); const cacheResponse = await cache.set(savedArticle.id, savedArticle); currentSpan.setAttributes(cacheResponse); const response = { message: 'Article created', article }; return c.json(response, 201); } catch (error) { currentSpan.recordException(error); return c.json({ error: 'Internal Error' }, 500); } }); ``` We removed the boilerplate from our code, the business logic is much more legible. OpenTelemetry is responsible for capturing timestamps, environment details, request headers, trace and span IDs, and facilitates sending traces to a vendor, or a self-hosted solution. ## Misconceptions ### Wide events replace metrics No, they don't replace **all** metrics. Afaik, you cannot replace CPU metrics of your Kafka box with wide events. I'm sure a very determined engineer can replace all metrics with wide events, but is it worth it? For monitoring infrastructure, metrics are your best option. They are insanely cheap and capture what you need to know about your infra. Where they fail is complex application logic where unknown unknowns are most likely to occur. You definitely should replace all your application metrics with wide events. ### Wide events are useful only during outages No, the beauty of wide events is their context-richness. I have seen product teams use observability data for product analytics, simply because the tooling is more advanced. The issue preventing this from being more widespread is retention periods. Product analytics generally require year+ retention periods whereas observability less. ### You must emit a single wide event per service Earlier I said you should emit a single wide event per service. I lied. There are no rules. Emit as many wide events as you need per request per service. Ideally only one, but there are scenarios where it's very valid to emit more than one wide event per service. But make sure to emit wide events, not events with 3 fields. And when emitting the new event, ask yourself if you're not repeating data in both events. ### Logs, metrics and traces are the 3 pillars of observability This is debunked. There are no pillars of observability. Observability is about answering the most uncommon questions about your app. Logs, metrics, traces, wide events, errors, etc. are just _data_. Data you should be able to query as you wish, asking questions about your app. You don't see the 3 pillars of data analytics, why should there be 3 pillars of observability? ### Structured logs are wide events False. Structured logs could be wide events, but not all structured logs are wide events. A structured log with 5 fields is not a wide event. A structured log with no context is not a wide event. If you print the response of a request without the details about the request (headers, path, method, body, etc.), it will be wide, but will it have the context you need to answer unknown unknown questions? ### OpenTelemetry is the only modern way to do distributed tracing No. I recommend distributed tracing, but it has a lot of flaws. It's grown to try to do too many things for too many people, and that's a problem. Instrumenting an application shouldn't be harder than building the application itself. Some vendors provide their own tracing, sometimes inspired by OpenTelemetry. Some teams have decided to add context propagation and timestamp capture to wide events and that's their tracing. It's up to you. --- In a nutshell, once you've solved an issue with a wide events flow, you'll never want to go back to greping logs or metrics telling you "there's a problem", but no way to drill deeper to know the cause of the problem. With wide-events you stop investigating symptomps so you can focus on finding root causes. --- ### Lessons from starting, building, and exiting a devtools startup URL: https://boristane.com/blog/learnings-from-starting-building-and-exiting-a-devtools-startup/ Description: Key learnings from building Baselime I started [Baselime](https://baselime.io), an observability startup, in October 2021. 2.5 years later [we joined Cloudflare](https://assets/blog.cloudflare.com/cloudflare-acquires-baselime-expands-observability-capabilities). I was a first-time solo founder and I made more mistakes than I could count. In this post, I'm reflecting on my time leading Baselime. I believe it will resonate with technical founders currently building or considering building similar companies. It focuses on early-stage, pre- Product Market Fit (PMF) developer tooling or infrastructure startups. I'll refrain from typical advice such as "build a strong team", or "build something people want", smarter people already [wrote about this](https://www.paulgraham.com/articles.html) extensively. ## Be obsessed with finding Product Market Fit Your one and only mission as an early-stage founder is to [find Product Market Fit](https://www.sequoiacap.com/article/pmf-framework/) as quickly as possible. Nothing else matters. It's extremely easy to focus on software architecture, clean code, tests, scalability, refactoring, etc. These things don't matter. ![test](/assets/blog/learnings-from-starting-building-and-exiting-a-devtools-startup/build-sell.jpg) I can hear you thinking that your startup is different and you need scalability from day 1. You're wrong. Your solution will evolve. Obviously, it's easier to evolve if the foundations are solid, but that's assuming the solid foundations you're building are for the right thing. Your early thesis about the world is most likely incorrect and you'll have to adjust. I spent too much time building the foundations for "[Observability as Code](https://baselime.io/assets/blog/why-observability-as-code)". Imagine the best parts of CloudFormation and Terraform for observability: a CLI and a state management backend. Developers write multiple YAML files (with variables, interpolation, templates, etc.), the CLI combines them into a giant JSON file, uploads it to the backend which decides which resources (alerts, queries, dashboards, etc.) to create, edit or delete and the order of those operations. Despite all the hours spent building, refining and improving this, nobody cared. I was more concerned about building an elegant solution than actually finding PMF. Don't do this. Instead, iterate fast, measure the result of your experiments, and if something doesn't work, throw it away and move on quickly. When you find what works, you'll refactor and improve. ## Better, faster, cheaper is not enough anymore It's common startup wisdom that your solution should be [better, faster, cheaper](https://youtu.be/opkHJLVAM4A) than the competition. Whilst there's truth in this statement, in 2024 this is not enough anymore. Incumbents in the developer tools space iterate fast, and there are 5 new startups going after your market every quarter. Instead, look for technological shifts and focus your efforts where you're a monopoly, rather than competing. ![test](/assets/blog/learnings-from-starting-building-and-exiting-a-devtools-startup/monopoly.jpg) I started Baselime with a focus on observability for AWS Lambda. There was a consensus among our early users that [Baselime was better, faster and cheaper](https://x.com/patjakubik/status/1687204692915539970) than the competition. But this wasn't enough to stand out in a crowded space. Until I noticed a massive segment where we could be a monopoly: the new developer platforms such as [Cloudflare](https://www.cloudflare.com/en-gb/developer-platform/), [Vercel](https://vercel.com), [Fly.io](https://fly.io), [Koyeb](https://www.koyeb.com/), [Render](https://render.com/), etc. Within 1 week, we built a [Vercel integration](https://vercel.com/integrations/baselime) and instantly doubled our weekly active users. Likewise for our [Cloudflare integration](https://developers.cloudflare.com/workers/observability/baselime-integration/). ## Don't be stubborn If something is not growing fast, drop it quickly and work on something else. There are no points for stubbornness, you're in a race against running out of money before PMF. In April 2023, I spoke with the Cloudflare team about observability on the Workers platform, but we built our Cloudflare integration only 6 months later. It actually took only 1 week to build it. Why wait 6 months to build something so obvious? 6 months is an eternity for an early-stage startup. During this time, I pretended to "[focus on a single problem](https://paulgraham.com/startupideas.html)" to justify being stubborn on becoming the best observability platform for AWS Lambda. I ignored all market signals, both from developers and market research, telling me to shift my focus to an adjacent problem. Don't do this. ## Do things that don't scale, really We're developers, we want to automate everything. But sometimes you've got to roll up your sleeves and do the tedious, unsexy but necessary work to grow. We built sales automation and lead generation pipelines, but nothing worked. Our user growth was flat. Until I decided to ignore all the best practices and do what works for me, something I'll actually enjoy: personally message every developer with a slight interest in cloud computing I could find on Twitter. It was unglamorous, tedious, and monotonous, but it had to be done. This directly translated to signups, sales and referrals. I should have done this months earlier. ![test](/assets/blog/learnings-from-starting-building-and-exiting-a-devtools-startup/scale.png) ## Create hype, don't be shy [If you build it, they won't come](https://www.forbes.com/sites/georgedeeb/2013/11/06/if-you-build-it-they-may-not-come-budget-ahead-for-startup-marketing/). You must create hype around your product. Technical founders tend to view hype as a bad thing, used to sell snake oil. When in fact, hype is a cheat code to get in the public awareness. Get developers to know your solution exists, its benefits and its limitations. It doesn't matter if they don't sign up today. Eventually (hopefully in the near future) they, or someone they know, will need to solve your problem and your solution should be the first they consider. Hype will get you there faster. ## Get a side project [I love building software](/projects). It's my main hobby. But building Baselime meant I didn't have a hobby anymore. I didn't have space to do this thing I love without the pressure of solving a problem for developers. I kept shipping every day, but there are days I actually despised building software, I didn't want to build software but I had to. Get a side project, where you can explore and build whatever random useless thing you want. Your overall happiness levels will thank you - building a startup is hard enough, don't go through it without hobbies :) ## Exiting a company is hard If you thought starting a company is hard, try selling one. Baselime was a pretty simple business when we joined Cloudflare. Just a few employees and our main assets were our IP. However, it still took quite a bit of work, from the negotiations phase to all the accounting and legal work involved. The most difficult part was how nerve wracking it all was. I was a solo-founder and until Baselime, the most complex contracts I ever signed were tenancy agreements. After 2 years of building and selling, Baselime was part of my identity. The reality of joining a larger organisation was challenging. Thankfully a few trusted advisors truly supported me throughout the process. Joining Cloudflare was the best decision to achieve our vision: enabling every developer to resolve issues before they become problems. --- The Baselime journey taught me much more than these highlights. I'm always keen to connect with dev tool founders and aspiring founders. Hit me up, [my dms are open](https://x.com/boristane) :) ## Talks ### Observing Serverless Applications URL: https://boristane.com/talks/observing-serverless-applications/ Description: Best practices for observing serverless applications Location: AWS re:Invent 2024 I teamed up with [Kyle](https://www.linkedin.com/in/kylegalbraith459/) to share the best practices to observe serverless applications. The talk is all about high-cardinality and dimensionality wide events, how to create them and leverage to build more observable serverless applications, with practical examples and real-world scenarios. --- ### Lessons from self-hosting ClickHouse URL: https://boristane.com/talks/lessons-from-self-hosting-clickhouse/ Description: We self-hosted ClickHouse for nearly 3 years, what did we learn? Location: ClickHouse London Meetup We self-hosted ClickHouse at [Baselime](https://baselime.io) for nearly 3 years. We broke things quite often and I shared with the London ClickHouse community a few of the lessons we learned in the process. --- ### Craft your own serverless observability platform URL: https://boristane.com/talks/craft-your-own-serverless-observability-platform/ Description: How do you build your own observability platform? And should you? Location: Serverless London Meetup I had the opportunity to speak at the [Serverless London Meetup](https://www.meetup.com/serverless-london/) once again and this time I wanted to share with developers how they can build their own observability tooling using serverless primitives and host it themselves. It was pretty fun to break down Baselime into its core components and share with developers what goes on in each, and how they can build it themselves. Fun fact: for some reason I needed to use my mouse instead of a pointer to flick through slides - this was way more challenging than I anticipated.
 
--- ### Observability with ClickHouse URL: https://boristane.com/talks/observability-with-clickhouse/ Description: A "behind the scenes" view into building an observability solution with ClickHouse. Location: ClickHouse London Meetup When I started [Baselime](https://baselime.io), one of the key shifts I noticed was the emergence of [ClickHouse](https://clickhouse.com) as a database for analytics, and I quickly realised it would be extremely good for high-cardinality and dimentionality data, which are two essential properties of data used for observability. That's why the Baselime data layer is powered by a self-hosted ClickHouse cluster. When the ClickHouse team invited me to speak at their London meetup, I was so excited to share what we'd learned building an observability platform on top of ClickHouse, for our ingestion pipelines to our database schemas, etc. --- ### Creating an innovation engine with observability v2 URL: https://boristane.com/talks/creating-an-innovation-engine-with-observability-v2/ Description: I gave this talk a second time, with corrections and improvements :) Location: Node Congress 2023 I was invited to give a talk at [Node Congress 2023](https://portal.gitnation.org/events/node-congress-2023); I spent quite a few days wondering what I could present; I had given a talk about [creating an innovation engine with observability](./creating-an-innovation-engine-with-observability-v1) before, which people liked and I knew it could be improved and resonate with even more developers. And that's exactly what I did. I made a few key changes to the talk and delivered it again. All about the culture changes that teams can make today to produce quality software without compromising on iteration speed. Node Congress doesn't allow embedding talk recordings, so you can find it on [their website](https://portal.gitnation.org/contents/creating-an-innovation-engine-with-observability). --- ### Creating an innovation engine with observability v1 URL: https://boristane.com/talks/creating-an-innovation-engine-with-observability-v1/ Description: How can you leverage your observability tooling to innovate faster? Location: Serverless London Meetup The [Serverless London Meetup](https://www.meetup.com/serverless-london/) invited me to give a talk, and I wanted to share something different: what's the engineering culture that enables us at [Baselime](https://baselime.io) to innovate extremely fast? One of the key points is observability. How we leverage our observability tooling to have high velocity and quality software at the same time. I gave this talk twice, and this is was the first iteration. In the second iteration, at the [Node Congress 2023 conference](./creating-an-innovation-engine-with-observability-v2), I incorporated feedback and improvements from this talk. --- ### Exploring JavaScript Runtimes on AWS Lambda URL: https://boristane.com/talks/javascript-runtimes-on-aws-lambda/ Description: Benchmarking a few JavaScript runtimes on AWS Lambda Location: London Node User Group [London Node User Group](https://lnug.org/) invited me to give a talk, and that came up when I was wondering if I could squeeze more performance out of my JavaScript [AWS Lambda](https://aws.amazon.com/lambda/) functions without re-writing in Go or Rust. I then benchmarked [Node.js](https://nodejs.org/en), [Deno](https://deno.com/) and [Bun](https://bun.sh/) on AWS Lambda... The benchmark results themselves are not quite relevant as those runtimes are evolving very quickly; the most interesting part was the journey of deploying unsupported runtimes on AWS Lambda. This talk is about how to deploy custom runtimes on AWS Lambda, and measure and compare the duration of HTTP requests made to those functions. It's also a great demo of a very early version of [Baselime](https://baselime.io). --- ### Serverless Transactional Outbox Pattern on AWS URL: https://boristane.com/talks/serverless-transactional-outbox-pattern-on-aws/ Description: How I implemented a transactional outbox pattern on AWS with serverless services Location: SST Conf v1 I believe this is my first ever public tech talk. The folks at [SST](https://sst.dev/) did the [SST Conf v1](https://v1conf.sst.dev/) for the launch of SST v1 and were kind enough to invite me. It was all remote and it was for me the occasion to connect with a few developers in the serverless community. My talk was about how I built a [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html) on AWS with serverless services for [Baselime](https://baselime.io). The main idea of this pattern is to resolve a problem with event driven architectures: how do you atomically update the database and send events to your message broker? My talk starts at [29:53](https://youtu.be/6FzLjpMYcu8?t=1793). ## Projects ### Basebrain Description: Chat with your bookmarks Website: https://basebrain.ai I needed a pretext to build AI agents and decided to solve the problem of bookmarks once again. I'm exploring all the latest techniques in building AI applications and agents. It's fully deployed on Cloudflare Workers, extensively using Durable Objects. **[Website](https://basebrain.ai)** --- ### Baselime Description: Cloud observability made for developers Website: https://baselime.io GitHub: https://github.com/baselime Developers are on the front-lines dealing with errors and performance issues every day, but they are not equiped with the right tooling: "logs, metrics and traces" are not enough. You need 3 core concepts: - High cardinality and dimensionality - OpenTelemetry - Great Developer Experience I started [Baselime](https://baselime.io) to solve this problem. It was a wild ride, building a team, bringing our vision to the world and enabling developers worldwide to solve issues before they become problems. Baselime has now [joined Cloudflare](https://blog.cloudflare.com/cloudflare-acquires-baselime-expands-observability-capabilities). **[Website](https://baselime.io), [Github](https://github.com/baselime)** --- ### bkmark.io Description: A bookmarks manager for teams, entirely built with serverless Website: https://bkmark.io I'm a tab hoarder. This project is an attempt to solve this problem: a fully serverless application to store all my tabs such that I finally close them. I added a bunch of quality of life features over the months and turned this into a SaaS. It was mostly a learning experience for me. I explored how to build a completely serverless application: compute, database, storage, queues, webhooks, etc. **[Website](https://bkmark.io)** --- ### Rooom Description: Remote listening parties powered by Spotify Website: https://rooom.click GitHub: https://github.com/boristane/spotify-room Music was one of the only ways to connect with family and friends during the first weeks of lockdown. So I built a small app to listen to music with loved ones synchronously. **[Website](https://rooom.click), [Github](https://github.com/boristane/spotify-room)** --- ### Lyrically Description: A tiny website to get the lyrics of the song you're listening on Spotify, live. No search, no tracking, no ads. Website: https://lyrically.be GitHub: https://github.com/boristane/lyrically I listen to a lot of french rap, and I need the lyrics, fast. I uses the [Genius API](https://docs.genius.com/) to get the song lyrics and it continuously polls the [Spotify API](https://developer.spotify.com/documentation/web-api) to get the current song. **[Website](https://lyrically.be), [Github](https://github.com/boristane/lyrically)** --- ### UK General Elections Data Viz Description: Visualisation of the results of the UK General Elections Website: https://boristane.github.io/parliament/ GitHub: https://github.com/boristane/parliament I wanted to dig deeper into the results of the [2017 United Kingdom general election](https://en.wikipedia.org/wiki/2017_United_Kingdom_general_election), so I built this front-end application to visually present: - election results - lost constituencies - gender representation - turnout - majority This project was mostly a pretext to play with [d3.js](https://d3js.org/) **[Website](https://boristane.github.io/parliament/), [Github](https://github.com/boristane/parliament)** --- ### Flight Dynamics Modelling and System Identification of a Light Fixed Wing UAV (Masters Thesis) Description: Instrumentation, flight dynamics modelling, flight testing, and system identification of a fixed wing Unmanned Aerial Vehicle GitHub: https://github.com/boristane/msc-maximum-likelihood-estimation For my [MSc Aerospace Dynamics](https://www.cranfield.ac.uk/courses/taught/aerospace-dynamics) thesis, I modified a low-cost commercial fixed wing UAV to instrument it with various sensors such as a pitot tube, an inertial measurement unit, and an air-ground data link. The instrumentation enabled me to flight test the aircraft and to capture data in various flight conditions. I used the data to build and validate 2 models of the aircraft: - a [6 Degrees of Freedom](https://en.wikipedia.org/wiki/Six_degrees_of_freedom) model based on the equations of motion of the aircraft - a [maximum likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) model based on the statistical analysis of the flight data This was a lot of fun with Matlab :) [Github](https://github.com/boristane/msc-maximum-likelihood-estimation)** ## Citation Guidelines When referencing Boris Tane's work, please cite as: - For blog posts: "Boris Tane, [Post Title], boristane.com" - For talks: "Boris Tane, [Talk Title], [Conference/Event]"