> ## Content Index > Fetch the complete content index at: https://johnjanek.com/llms.txt > Use this file to discover other available public pages before exploring further. # The Pipeline Didn't Catch a Bug — It Invented Twelve - URL: https://johnjanek.com/the-pipeline-didnt-catch-a-bug-it-invented-twelve/ - Published: 2026-08-24T14:35:24.000Z - Updated: 2026-08-24T16:48:06.000Z - Description: I moved a working local build into a CI/CD pipeline to test my own thesis. Every failure was an artifact of the pipeline, not the code. - Author: John Janek - Tags: DevOps, AI, Software Development Listen to this article 0:00 /372 1× I wrote ["Deploy From Here"](https://johnjanek.com/deploy-from-here/) a few weeks ago arguing that local deployment is architecturally superior to centralized CI/CD pipelines. Theory is comfortable. I decided to test it by moving a working local build into a remote pipeline and seeing what actually happened. The experience didn't validate my thesis in the way I expected. It taught me something more useful. ## The Experiment I have a personal project — a team collaboration platform in the vein of Discord or Teams, built because I wanted to understand why Microsoft Teams is so genuinely terrible and how complex systems interoperate at that scale. It's been rewritten a couple of times. One compose file deploys the whole stack. Intentionally simple. My build process had been: build the containers locally, deploy to the test server. The pattern I advocated for. Over the weekend, I thought I should put my money where my mouth is from the other direction. You cannot credibly argue that local deployment is superior without testing where centralized pipelines actually break down. So I moved the entire build into a remote CI/CD pipeline. What followed wasn't a configuration exercise. It was a rewrite. ## What Actually Happened Integration tests that ran cleanly on my workstation for months required fundamental restructuring to function inside a pipeline runner. Different permissions. Different capabilities. Different timing characteristics. Different module loading behavior. I worked with Kiro to do the rewrites. A manual overhaul of that scope would have taken weeks, possibly months. Even with agentic assistance, it consumed an entire weekend of iteration. I did get a clean build in the end. But not before I lost significant test fidelity in the process. Real database integration tests became mocks. Actual crypto operations became stubs. The tests got greener by getting shallower. ## The Taxonomy of Invented Failures Every single failure I encountered was an artifact of running tests *somewhere else* under *different conditions*. Not one was a legitimate bug in the product. Here's the taxonomy: **Shared mutable state.** Module-level singletons — KMS clients, database connections, encryption services — that work perfectly in isolation become nondeterministic when thirty-two test files share a single process in a CI runner. The fix wasn't better code. It was architectural: eliminate shared state or accept the consequence. The consequence being that your tests lie to you. **Native dependency tax.** better-sqlite3 required python3, make, and g++ installed in the CI container just to compile. Its native bindings had load-order bugs. Replacing it with sql.js (pure WASM) eliminated one class of failures but introduced another: WASM crypto runs six times slower in Docker, changing the timing characteristics enough to surface hidden async issues that "worked" locally because they were fast enough to complete before the next operation started. **The silent lie.** Async property tests — `fc.assert(fc.asyncProperty(...))` without `await` — had been broken for their entire existence. They "passed" because the Promise was never awaited. Jest marked it green. The actual async work leaked into the background. Locally, it finished fast enough to be invisible. In Docker with WASM crypto, it took sixty seconds and crashed whatever was running next. **Environmental amplification.** Same test suite, same code, same assertions. Different result depending on whether the WASM engine initializes in 50ms or 500ms; whether a property test's ten iterations complete in two seconds or twenty; whether `beforeEach` runs before a leaked async operation resolves. None of these are bugs in my application. They're bugs in the interaction between my tests and the pipeline's execution model. The pipeline didn't find problems. It manufactured them. ## What I Actually Learned I went in expecting to confirm that local is better. What I came out with is more nuanced and, I think, more useful. **This is a stone decision, not a scaffolding one.** I've [written about the distinction](https://johnjanek.com/shifting-constraints/) before. Some architectural choices are meant to be durable — constraints you build against for the life of the system. Your deployment model is one of them. The industry treats it as a default ("of course you use CI/CD") rather than a deliberate architectural commitment with real, compounding tradeoffs. It deserves the same rigor you'd give to choosing your database or your cloud provider. **The approaches are not interchangeable.** The test infrastructure, the dependency choices, the isolation model, the timing assumptions — all of it shifts when you move from local to remote execution. You're not adding a deployment step. You're rewriting your verification strategy for a foreign environment. Every architectural decision I'd made — native SQLite for speed, real crypto for fidelity, shared state for simplicity — was correct under the original constraints and wrong under the new ones. The longer you build under one regime, the higher the switching cost. **The rigor is in the practices, not the venue.** A remote pipeline is not intrinsically more rigorous than a local deploy process. Testing, linting, scanning, gating — these practices are portable. A Kamal deploy behind a local test suite, a security scan, and a human decision to ship is not less disciplined than a YAML file triggering the same steps on someone else's server. People attribute rigor to the infrastructure when it actually belongs to the practices running inside it. **Local pipelines increasingly offer equivalent confidence.** Developer hardware is powerful enough to run full test suites, build containers, and execute security scans. You want your developers to have capable machines anyway — let those machines do the work. Local execution maintains separation of concerns and control without introducing a foreign execution environment that your tests weren't designed for. **There's a spectrum between fully local and fully remote.** It's not binary. You can run unit and integration tests locally, push containers to a registry, and let a remote process handle deployment orchestration. You can gate on local verification and use remote infrastructure only for distribution. The point is that determining the *goal* of your testing should be the architectural discussion first — then you engineer the appropriate tools, techniques, and patterns to support it. Most teams skip that discussion entirely and inherit a pipeline template that dictates the testing strategy rather than the other way around. ## The Agentic Wrinkle There's a workflow dimension that tipped my decision back toward local. When you're working with agents, git is an undo button. You push exploratory branches, test hypotheses, let the agent try approaches that might not work. Firing a full CI build on every exploratory commit is burning compute on work that hasn't even been validated as the right direction yet. You end up either waiting for builds you don't care about or ignoring the pipeline entirely — which defeats the purpose of having one. Local deployment doesn't have this problem. You build when you're ready. You deploy when you decide the work is done. The deployment is a deliberate act, not an automated reaction to a git event. That deliberateness maps naturally to how agentic development actually works: write, validate locally, iterate, and ship when satisfied. The standard workaround is tags, protected branches, or manual approval gates — mechanisms to prevent the pipeline from firing until a human says go. But think about what that actually is: you've automated a process and then bolted on friction to prevent the automation from running. You've taken a human decision, encoded it into infrastructure that requires a human to approve it anyway, and added latency to both sides. The automation waits for you. You wait for the automation. Nobody gained anything except a YAML file in the middle. Why not just let human decisions be human decisions? A developer pressing a deploy button is already the approval gate. It doesn't need to be mediated through branch protection rules and workflow triggers. The ceremony adds process without adding safety. ## Choose Deliberately This is a solo project. I'm not coordinating merges across a ten-person team or enforcing compliance gates for a regulated enterprise. Pipelines work every day to keep the modern internet reliable; they solve real coordination problems at scale and I'm not pretending otherwise. But the lesson generalizes beyond my specific context: your deployment model is an architectural commitment that shapes every testing and verification decision downstream. Choose it deliberately. Build within those constraints intentionally. And don't assume that the industry default is correct for your situation just because it's the default. The pipeline didn't catch a single real bug. It invented twelve. That's not an argument that pipelines are bad. It's an argument that I was building under a different set of constraints, and switching between constraint portfolios has a cost that nobody warns you about until you're debugging WASM timing races at 2 AM. Choose your constraints. Then build like you mean it.