The biggest security risk in modern application deployment might no longer just be found in the application code but rather in container images, specifically in the base images, the foundation with which software is now shipped.
A typical image pulled from Docker Hub today, when scanned, contains hundreds of packages with many known vulnerabilities. Most of which exist not because the applications need them, but because the base image was built for general use.
These packages often contain the operating system, system libraries, a shell, a package manager and debugging tools that most workloads will never invoke, while also creating potential exploitable paths for attackers.
In the case of a breach, smuggler tunnels -- like package managers -- can be used to install additional malicious software within the image. This is one piece of software most teams inherit without ever properly reviewing its contents.
In the past, developers had to constantly deal with hundreds of Common Vulnerabilities and Exposures (CVEs). Continuously fixing and patching them usually wastes engineering effort, slows deployments and turns into a race against the clock.
A better way to stay ahead and avoid spending all your time reacting to security incidents is to swap your base container images for hardened ones.
What are hardened images?
Unlike standard base images, which are built for flexibility rather than security, hardened images are a category of container images designed for a more secure software supply chain, with low-to-zero CVEs by design, thereby reducing the attack surface.
When you write FROM ubuntu:22.04 in your Dockerfile, it means you inherit every package, configuration file and binary that the image maintainer decided to include. The problem here is that, as far as production is concerned, most of what's included is deadweight.
A container running a Node application doesn't need a C compiler at runtime. A Node-hardened image will most likely contain just the Node runtime and exclude package managers, as well as tools like bash, curl, ssh, etc. -- though it might require a slightly different approach to building the images, such as installing dependencies in a separate stage in your Dockerfile. Ultimately, this makes the final image very secure with near-zero vulnerabilities.
The idea is that the final images should contain just what's needed to run your application. Today, hardened images are managed by several vendors, with a commitment to constant patching.
Why haven't hardened images always been the default
In the early days of container technologies, images were essentially lightweight virtual machines, with Docker's earliest versions optimized for portability and developer experience over security.
Developers simply ran docker run and exec on it, debugging interactively and installing packages on the fly, making containers accessible and easy to use, though this led to the fat image pattern.
Additionally, hardening is workload-specific, meaning a hardened Java image is quite different from a Golang version. Thus, significant investment is required to create and maintain it.
Unlike before, CVEs are now impossible to ignore. Organizations soon realized that independently patching general-purpose images was an endless battle.
This need has led to the rise of subscription-based image hardening services, offloading these burdens from infra teams.
Choosing a vendor
Over the past two years, hardened images have moved from a preference to an industry requirement, with vendors stepping in to provide solutions with different approaches, philosophies and business models. Some example vendors and products include the following.
BellSoft Hardened Images
BellSoft focuses on the runtime layer with an emphasis on enterprise Java stacks. These images are built on Alpaquita Linux and are available in both musl and glibc variants.
This dual-libc support is critical, as it enables you to switch to a more lightweight base without having to rewrite applications that strictly depend on the traditional GNU C libraries.
BellSoft also maintains the OS layer, container runtime and container image layer under a single SLA, so there is no need for multiple vendor contracts for corporate and procurement teams.
Chainguard Hardened Images
Chainguard adopts the distroless concept, but with a custom independent Linux distribution called Wolfi. With their AI-native build engine, they aggressively rebuild their catalog directly from fresh upstreams daily, aiming for a permanent near-zero CVE count.
Docker Hardened Images mirrors the distroless framework; however, it handles the “Pipeline Paradox”: The moment the image goes through your build pipeline, it's no longer a hardened image.
This is achieved by providing specific customizations for teams that might require them, such as adding root certifications, internal tools and permissions while keeping SLSA Level 3 provenance, security attestations and Docker SLA intact.
Docker's core DHI catalog is provided for free under a permissive Apache 2.0 license. It is best suited for teams that want a drop-in replacement for the standard base images and need flexibility.
Red Hat Universal Base Images (UBI) & Canonical Chiseled Ubuntu
Both Red Hat and Canonical approach image hardening differently by scaling down their enterprise-grade Linux while still retaining the underlying package management components. They are essentially combining the security of minimalist containers with the stability and compliance of a commercial OS.
When compared to pure distroless or Chainguard variants, they have a larger CVE footprint. Also, navigating licensing nuances can create operational friction.
Risks of using hardened containers
There are several risks and challenges to using hardened containers.
Debugging becomes harder. Without a shell present in the running container, developers can't run `docker exec -it <container> sh` or `kubectl exec -it <pod> -- sh` to investigate crashes. They have to debug from the outside looking in by attaching an administrative tool set to the running container's namespaces at runtime.
No server environment. In hardened containers, the server environment has been stripped away, so apps that install packages at runtime and those that need specific system tools will break.
Vendor dependency. Adapting third-party minimal base images means offloading responsibility for patching, scanning and updates to an external entity. This leaves room for operational vulnerabilities, including SLA and support delays, as well as breaking upstream changes.
Deciding to change your hardened base images isn't just a one-line code fix; it involves auditing what you have, testing for breaks and updating CI/CD to enforce the new standard.
Deciding to change your hardened base images isn't just a one-line code fix; it involves auditing what you have, testing for breaks and updating CI/CD to enforce the new standard.
How teams should adapt
Follow these best practices to adapt to the new dynamic.
Know what you have. Take an inventory of every base image used in production. Scan and record the result, as this will serve as a baseline for an argument for migration or prioritization.
Fix your debugging approach. For Kubernetes workloads, teams should set up ephemeral containers, invest in centralized logging and distributed tracing, and set up health endpoints.
Block the old images at runtime. Make sure to use admission controllers Kyverno or OPA Gatekeeper to prevent non-approved base images from reaching production.
Plan for regular maintenance. Migrating to a hardened image is not enough; teams must automate the regular updating of their base image.
Wisdom Ekpotu is a DevOps engineer and technical writer focused on building infrastructure with cloud-native technologies.
