Skip to main content
Version: Next

Project Copacetic: Directly patch container image vulnerabilities

copa is a CLI tool written in Go and based on buildkit. It can apply targeted updates from supported scanner reports such as Trivy, or comprehensive package updates without a report.

Why?​

We needed the ability to patch containers quickly without going upstream for a full rebuild. As the window between vulnerability disclosure and active exploitation continues to narrow, there is a growing operational need to patch critical security vulnerabilities in container images so they can be quickly redeployed into production. The need is especially acute when those vulnerabilities are:

  • inherited from base images several levels deep and waiting on updated releases to percolate through the supply chain is not an option
  • found in 3rd party app images you don't maintain with update cadences that don't meet your security SLAs.

In addition to filling the operational gap not met by left-shift security practices and tools, the ability of copa to patch a container without requiring a rebuild of the container image provides other benefits:

  • Allows users other than the image publishers to also patch container images, such as DevSecOps engineers.
  • Reduces the storage and transmission costs of redistributing patched images by only creating an additional patch layer, instead of rebuilding the entire image which usually results in different layer hashes that break layer caching.
  • Reduces the turnaround time for patching a container image by not having to wait for base image updates and being a faster operation than a full image rebuild.
  • Reduces the complexity of patching the image from running a rebuild pipeline to running a single tool on the image.

How?​

The copa tool is an extensible engine that:

  1. Determines the update scope from a supported vulnerability report or, for a comprehensive update, from package metadata in the target image.
  2. Obtains and processes the updates using the appropriate package-manager tooling or Chisel release definitions. New adapters can be written to support more package managers and report formats.
  3. Applies the resulting update binaries to the container image using buildkit.

This approach is motivated by the core principles of making direct container patching broadly applicable and accessible:

  • Copa supports patching existing container images.
    • Devs don't need to build their images using specific tools or modify them in some way just to support container patching.
  • Copa supports containers without package managers including distroless containers
    • Copa does not support Chainguard's wolfi-based images
  • Copa works with the existing vulnerability scanning and mitigation ecosystems.
    • Image publishers don't need to create new workflows for container patching since Copa supports patching container images using the security update packages already being published today.
    • Consumers do not need to migrate to a new and potentially more limited support ecosystem for custom distros or change their container vulnerability scanning pipelines to include remediation, since Copa can be integrated seamlessly as an extra step to patch containers based on those scanning reports.
  • Copa reduces the technical expertise needed and waiting on dependencies needed to patch an image.
    • For OS package vulnerabilities, no specialized knowledge about a specific image is needed to be patch it as Copa relies on the vulnerability remediation knowledge already embedded in the reports produced by popular container scanning tools today.

For more details, refer to the copa design documentation.