You fixed that XSS bug three months ago. Today it's back — same field, same payload — and nobody noticed, because nothing in your pipeline remembers it was ever fixed. That's a security regression: a vulnerability you already closed quietly reopening, or a hardened config drifting back to weak.
They're more common than teams expect and they hide for months, because there's usually no automated check comparing today's state against "what we knew was good before."
Common causes of security regressions
Merge conflicts resolved insecurely
Developer A fixes an XSS vulnerability in the comment sanitization function. Developer B is working on a feature in the same file. When B merges, the conflict resolution accidentally reverts A's fix. Without a test specifically checking for XSS in comments, this passes CI.
Dependency downgrades
A major version upgrade of a framework introduces breaking changes. The team decides to pin back to the older version temporarily. The older version had a known CVE that the team had already moved past.
Configuration drift
Your production server had HSTS enabled and directory listing disabled. A new deployment of the same application uses a different base Docker image that doesn't carry over the nginx config. The next deployment is weaker than the previous one.
Test coverage gaps
The security tests in your test suite were written by the person who originally fixed the vulnerability. That person leaves. Six months later, a refactor changes the code path tested, but no one updates the test. The vulnerability returns; the test passes because it's no longer testing the right thing.
How semantic regression detection works
BattleTest stores a vector representation of every security finding. When a new scan runs, findings are compared against the historical record:
- If a new finding is semantically similar to a previously-resolved finding, it's flagged as a regression candidate
- The regression carries an additional penalty to the overall risk score
- The finding is annotated with "possible regression — similar to finding resolved on [date]"
This catches regressions that don't have an exact fingerprint match — a refactored function that reintroduces the same vulnerability class gets caught even though the code is different.
Diff-based monitoring for infrastructure
For live infrastructure, security regression detection works by diffing the current scan results against the previous scan:
- Endpoints that were previously clean but now have findings: NEW
- Findings that existed before and still exist: PERSISTENT
- Findings that previously existed but are now absent: RESOLVED
- Findings that look similar to previously resolved ones: REGRESSION
The REGRESSION category is what matters most — these are cases where your team already spent time fixing something and it came back.
Building regression detection into your process
For code
- Run a security scan on every PR (BattleTest does this automatically)
- Store findings with metadata: when first seen, when resolved, who fixed it
- On each new PR, compare new findings against the historical finding list
- Flag any new finding that matches a previously-resolved finding class
For infrastructure
- Run scheduled battletests on a fixed cadence (weekly minimum)
- Compare each scan result against the previous one
- Alert immediately on regressions; batch-report persistent findings
- Track resolution timelines for compliance evidence
The 4-hour cooldown and its purpose
BattleTest enforces a cooldown between on-demand battletests. A scan immediately after a deploy catches deployment regressions. A scan after the system has stabilised captures steady-state posture. Both signals are useful, and the cooldown is set with this in mind.