Five bugs. Not one of them failed a test. All five failed in production.

That gap is the whole subject of this piece.

The mini-series so far has been about architecture. Deterministic input, typed context, an output layer that survives contact with Jira. Layers you can draw on a whiteboard. This one is the layer no whiteboard shows: the small, unglamorous decisions that hold up until real traffic leans on them. Operational discipline.

Each of these came with a lazy assumption that looked completely reasonable at the time. That is the pattern. The bug is never the interesting part. The assumption underneath it is.

Here are five.

1. An empty env var is not zero

Config loader read a numeric limit straight from the environment. Standard shape:

const maxSteps = Number(process.env.MAX_STEPS ?? DEFAULT)

Looks fine. It is not fine. In a Docker Compose setup an unset variable does not go missing, it arrives as an empty string. And Number("") is not NaN. It is 0.

So the fallback never fired. The variable was “present”, just empty, and every limit it fed quietly collapsed to zero. Playwright step budgets, timeouts, max contexts. All zeroed. Nothing threw. The system just started doing nothing, very fast, and the logs looked clean.

That is the worst kind of bug. Not a crash. A silence.

The fix was a small parser that treats empty, non-numeric, and less-than-or-equal-to-zero all as “use the fallback”:

function parsePositiveInt(raw, fallback) {
  const n = Number(raw)
  return Number.isFinite(n) && n > 0 ? n : fallback
}

Lesson: the boundary between “unset” and “set to garbage” is where config bugs live. ?? guards the first. It does nothing for the second. This one shipped in May.

2. The fix was a bigger number

A pre-flight check hit a /health/parity endpoint with a 15 second timeout. Reasonable. Health checks are supposed to be fast.

Except on a cold node with an empty cache, the honest answer took 20 to 25 seconds. So the check timed out, the task was marked failed, and the endpoint then answered correctly three seconds later, to nobody. The system was failing work over an endpoint that was, in fact, healthy. It was just healthy slowly, on first contact.

The fix was 60 seconds and a comment recording the real observed response time, so the next person does not “optimize” it back down.

// cold node + empty cache measured at 20-25s. do not lower.
const PARITY_TIMEOUT_MS = 60_000

Lesson: a timeout is a claim about how long the slowest legitimate case takes. If you have not measured that case, the number is a guess wearing a unit. This one was April.

3. The database will drop your connection, and it will not ask first

A pg.Pool with no error listener. Everything worked, because in development the database never goes away.

In production the database goes away all the time. A restart, a brief network blip, a routine failover. When an idle connection in the pool hit an error with nothing listening for it, the error went unhandled and took the whole process down with it. Live sessions that were mid-run just hung there until a heartbeat eventually timed them out. One idle socket, whole process gone.

pool.on('error', (err) => {
  logger.error({ err }, 'idle pg client error')
  // default pool reconnects on next query. do not rebuild that here.
})

The fix did not rebuild reconnection logic. The pool already reconnects on the next query. It just needed something to catch the idle error and refuse to escalate it into a crash.

Lesson: every dependency you did not write is a dependency that will fail without warning. Idle does not mean safe. If a connection can error while you are not looking at it, something has to be looking at it. This was May.

4. Read the body once

An endpoint consumed a webhook response and, further down, tried to read it again. Fetch response bodies are a stream. You get one read. The second one throws Body is unusable, and it throws it far away from the line that actually consumed the stream, so the stack trace points at the wrong crime scene.

The result was an integration that failed with a message describing nothing. Silence would almost have been more honest.

The fix is boring and correct: read the payload to text once, then parse from that.

const raw = await res.text()
const data = raw ? JSON.parse(raw) : null

One read, one string, everything downstream works from the string. Parse failures now say what they mean.

Lesson: a stream you can consume once will punish you for assuming otherwise, and it will do it at a distance. Boring, deterministic handling beats clever re-reads. This goes all the way back to March.

5. The dangerous one

The first four cost time. This one could have cost more.

Two endpoints took identifiers from the outside, a project name in one, a run id and task id in another, and fed them almost directly into filesystem paths via join, to read and write artifacts. Written out like that the problem is obvious. It was not obvious while writing it, because the inputs “came from our own system”, right up until you remember that the boundary of your own system is exactly the thing an attacker gets to lie about.

A crafted value with .. in it walks straight out of the intended directory. Read files outside the artifact tree. Write into the wrong one. Path traversal, the plain classic kind, in code that never looked risky because it never touched a database or a shell.

The fix was a hard gate before anything is joined to a base path: an allowlist and a regex for the well-known identifier, and a sanitizer for the free-form ones that strips .., restricts the character set, and caps the length.

function safeSegment(input) {
  const s = String(input).replace(/\.\./g, '').replace(/[^a-zA-Z0-9_-]/g, '')
  if (!s || s.length > 64) throw new Error('invalid path segment')
  return s
}

Lesson, and it is the one that ties the whole list together: input validation is not a database concern. It is a boundary concern. Any value that crosses from outside to inside is untrusted, including the values your own components hand each other, because “outside” is defined by where the data can be influenced, not by whose logo is on the service. This was May.

What the five have in common

None of these is clever. That is the point. There is no architecture in this article, because operational discipline is not architecture. It is the habit of not trusting the convenient assumption:

  • an unset variable is missing (no, it is an empty string, and empty is not zero)
  • a health check is fast (no, not on a cold start you never measured)
  • an idle connection is safe (no, it can die while you are not watching)
  • a response body is just data (no, it is a stream, and you get one read)
  • your own inputs are trusted (no, trust is about the boundary, not the brand)

You do not design your way out of these. You get bitten, you write the fix, and you write the comment so the next person does not un-fix it. Five gotchas, three months, one recurring shape: the assumption that felt too obvious to check was the one worth checking.

That is the whole layer. No repo this time, because there is nothing to clone. There is only the discipline, and you already have the five examples.

This is part of a mini-series on context-first QA. The architecture layers ship as public code. This one does not, because the lesson is the article.

Read in Polish: /pl/from-the-field/operational-discipline