Troubleshooting

Bulk Patching: Images Not Being Skipped

When using bulk image patching with --push and -r, Copa checks vulnerability reports to avoid unnecessary re-patching. If images are not being skipped as expected:

Check Reports Directory

• The skip feature requires the -r flag
• Without this flag, Copa will always patch (no skip detection)
• Verify the directory path is correct and accessible

Check Report Files

• Report files must contain an ArtifactName field (standard in Trivy JSON output)
• Copa reads the ArtifactName from inside each JSON report to match it to images
• You can name report files anything you want (e.g., nginx-report.json, alpine.json)
• If no matching ArtifactName is found, Copa proceeds with patching (fail-open)

Verify Report Contents

• Reports must be in a format Copa can parse (Trivy JSON, native format, etc.)
• Ensure reports were generated for the correct patched image tags
• Empty or invalid reports trigger fail-open behavior (proceeds with patching)

Check Scanner Configuration

• Use the --scanner flag to specify the report format (default: trivy)
• Supported scanners: trivy, native, custom plugins
• The scanner must match the format of your report files

Registry Access

• Skip detection queries the registry for existing patched tags
• Ensure Copa has read access to the target registry
• Registry errors trigger fail-open behavior (proceeds with patching)

Scan Workflow

# 1. Patch images
copa patch --config bulk.yaml --push

# 2. Scan patched images (not source images!)
# Name files however you want - Copa reads ArtifactName from the JSON
trivy image registry.io/nginx:1.25.3-patched -f json -o reports/nginx-patched.json

# 3. Run with skip detection
copa patch --config bulk.yaml --push -r ./reports

Cross-Registry Workflows

• If you patch images from one registry and push to a different registry, specify target.registry in your config
• Example: Source image from quay.io/opstree/redis:v8.2.1, patched image pushed to ghcr.io/myorg/redis:v8.2.1-patched
• Configuration:

target:
  registry: "ghcr.io/myorg"  # Target registry for patched images

images:
  - name: "redis"
    image: "quay.io/opstree/redis"  # Source registry
    tags:
      strategy: "list"
      list: ["v8.2.1"]

• With target.registry set, Copa queries the correct registry for existing patched tags and matches reports correctly

Getting downloaded package ... version ... lower than required ... for update error when trying to patch an image

This error means that the package manager is trying to install a version of the package that is lower than the version that was required from the scanner report. This can happen for a few reasons:

• Package repositories are not updated to the latest version of the package. For example, sometimes there is a lag between when a CVE is detected by Trivy using Red Hat vulnerability database and when it is available in the package repositories for CentOS.

To diagnose this, look at the Trivy report and find the applicable vulnerability database (found under DataSource -> URL). Search for the corresponding package receiving the error in the database to check if it is available.

• Scanner reports are not up to date. Make sure to run the scanner with the latest vulnerability database. If you are using Trivy, it is recommended to pull the latest version of the Trivy DB, and not rely on cached or stale versions.

To verify the package version discrepancies, you can compare the package version provided by the package repositories and the scanner reports. Follow the Trivy documentation on how to find the security advisory data sources, and then compare the package version in the scanner report with the applicable security advisory, and applicable package repository.

If you are continuing to see this and the package repositories and vulnerability databases are not updated, you can either:

• use the --ignore-errors flag or filter the applicable vulnerability in the scanner.

• update all packages without any scanner reports. This can be done by not providing a scanner report to Copa, and Copa will update all packages to the latest version available in the package repositories.

Copa and Trivy throw errors when Oracle Linux is passed in

Copa supports patching Oracle Linux in two ways:

With a vulnerability scan, --ignore-errors must be passed in. This will patch all CVEs aside from false positives reported by Trivy:

copa patch -r /oracle-7.9-vulns.json -i docker.io/library/oraclelinux:7.9 --ignore-errors

Without a vulnerability scan, Copa will update all packages in the image:

copa patch -i docker.io/library/oraclelinux:7.9

Oracle reports CVEs in a way that causes Trivy to report false positives that Copa will be unable to patch. To patch the entire image, use the Copa --ignore-errors flag or omit the vulnerability scan report to upgrade all outdated packages. See this GitHub issue for more information.

Filtering Vulnerabilities

You might want to filter/ignore some of the vulnerabilities while patching. To do so, you need to first filter those undesired vulnerabilities from your scanner output.

For Trivy, vulnerabilities can be filtered by the following 2 ways:

Rego Policy

An example rego file which demonstrates how to ignore certain Vulnerability IDs or Package Names:

$ cat trivy_ignore.rego

package trivy

import data.lib.trivy

default ignore = false


# Ignore the following Vulnerability IDs
ignore_vulnerability_ids := {
   "CVE-2018-14618"
}
# Ignore the following Package Names
ignore_pkgs := {"bash", "vim"}


# For ignoring vulnID
ignore {
   input.VulnerabilityID == ignore_vulnerability_ids[_]
}
# For ignoring pkgName
ignore {
	input.PkgName == ignore_pkgs[_]
}

After adding the above rego file, run the image scan with the --ignore-policy flag followed by the file name to ignore them while scanning:

trivy image --ignore-policy trivy_ignore.rego ruby:2.4.0

In the above example, the vulnerability "CVE-2018-14618" and the packages "bash" & "vim" are ignored while scanning, and hence patching the image.

Ignore File

Use a .trivyignore file to list all the vulnerabilities you want to ignore.

Example:

$ cat .trivyignore

# Accept the risk
CVE-2018-14618

In the above example, the vulnerability CVE-2018-14618 is ignored while scanning, and hence while patching the image.

For a more detailed explanation on how to ignore certain vulnerabilities with Trivy, please refer to the official documentation here.