Insufficient Logging and Visibility
When something does go wrong in the pipeline, there is not enough evidence to work out what happened.
If this is new to you
Pipeline logging is almost always built for one purpose — working out why a build failed — and then quietly relied on for a completely different one: reconstructing what happened after a security incident.
Those need different data. Debugging wants the last error. Investigation wants to know who triggered what, when, with which permissions, and what changed. Logs that scroll past in a build view and expire with the retention window answer the first question and none of the second.
What to do about it
Try a concrete exercise: pick a date last month and reconstruct the deployments. What you cannot answer is your logging gap, stated precisely rather than vaguely.
Ship logs off the machine that produced them. This one is easy to postpone and it is the whole point — an incident on a host tends to destroy the evidence about that incident, and logs that live only on that host are lost exactly when they matter.
Log deployments as structured events with actor, version and outcome, kept separately from build output. Build logs are noise for this purpose.
Going deeper
Adopt an existing convention for metric and trace attribute names rather than inventing your own. OpenTelemetry's semantic conventions cost nothing at write time and make your data comparable, portable and understandable to somebody who did not write it.
One habit worth building: record absence as absence. If a provider did not report a token count, store null, not zero. "We did not measure" and "it was free" are different facts, and conflating them will eventually mislead somebody building a budget on your data.
Then read the logs occasionally. A control that silently stopped working looks exactly like a control that is working, right up until it doesn't.
Words used on this pagedefinitions
- Audit log
- A record of who did what and when, kept for reconstruction rather than debugging. Different data, different retention.
- Semantic conventions
- Agreed names for common attributes, so telemetry from different systems can be compared without translation.
- Retention window
- How long logs are kept before deletion. Usually shorter than the time it takes to discover an incident.
Why this matters
Pipeline logging is usually built for debugging failed builds, not for reconstructing events. Those need different data: debugging wants the last error, investigation wants who triggered what, when, with which permissions, and what changed. Logs that rotate away with a container answer neither question a week later.
What it looks like in practice
If any of these sound familiar, that is the point — most of them are things somebody did for a sensible reason on a busy afternoon.
- Build logs that exist only inside the CI system's retention window.
- No record of who triggered a deployment, only that one occurred.
- Audit logs that are never read, so a control that silently stopped working looks identical to one that is working.
- Application logs inside a container, lost on the next restart.
- Metrics with no attributes, so a spike cannot be attributed to a model, a route or a user journey.
Check your own pipeline
Questions you can actually answer, rather than a maturity score. If you cannot answer one of them, that is itself the finding — and a more useful one than a number.
Could you reconstruct last month's deployments?
Try it for a specific date. What you cannot answer is your logging gap, stated precisely.
Do your logs outlive the container that wrote them?
Restart something and look for the line you saw before. This is a two-minute test that frequently fails.
Are your metrics named so anyone else could read them?
Adopting an existing convention — OpenTelemetry's semantic conventions, for instance — costs nothing at write time and makes the data comparable and portable later.
How to fix it
- Ship logs off the host that produced them, so an incident on that host does not destroy the evidence about it.
- Log deployments as events with actor, version and outcome, separately from build output.
- Instrument with an existing semantic convention rather than inventing attribute names.
- Record the absence of data as absence. A token count the provider never reported is unknown, and storing it as zero fabricates a measurement.
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere
Risk taxonomy from OWASP Top 10 CI/CD Security Risks (Daniel Krivelevich and Omer Gil, September 2022, CC BY-SA 4.0). Explanations, checks, fixes and definitions written for this site.