feat: langmgr and python #1091

sozercan · 2025-06-04T22:25:42Z

Signed-off-by: Sertac Ozercan sozercan@gmail.com
Part of #147

Hold merge after current release

Describe the changes in this pull request using active verbs such as Add, Remove, Replace ...

Adds experimental app-level patching support with Python language manager

This PR introduces scaffolding for experimental support for patching application-level dependencies (libraries/packages) in addition to OS packages, with initial Python support.

Key Features

🔬 Experimental Feature

Requires COPA_EXPERIMENTAL=1 environment variable
Subject to breaking changes in future releases

📦 Package Type Filtering

New --pkg-types flag with values: os, library, os,library
Defaults to os (maintains backward compatibility)
Enables separate patching policies for OS vs application dependencies

🎯 Patch Level Control

New --library-patch-level flag with values: patch, minor, major
Defaults to patch for conservative updates
Implements preference ordering: patch > minor > major for safest upgrades

🐍 Python Support

Support for pip-managed Python packages. Other package managers are not considered for this initial PR.
Special handling for certifi (always latest version)

Example Usage

export COPA_EXPERIMENTAL=1
export IMAGE=mcr.microsoft.com/azure-cli:2.50.0 

# Scan for package vulnerabilities
trivy image --vuln-type os,library --ignore-unfixed -f json -o scan.json $IMAGE

# Patch only libraries with patch level updates only
copa patch -i $IMAGE -r scan.json --pkg-types library --library-patch-level patch

# Mixed OS and library patching with major level updates allowed
copa patch -i $IMAGE -r scan.json --pkg-types os,library --library-patch-level major

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

codecov · 2025-06-05T00:38:36Z

Codecov Report

❌ Patch coverage is 50.41021% with 544 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.45%. Comparing base (3974d9c) to head (b5bceba).

Files with missing lines	Patch %	Lines
pkg/langmgr/python.go	36.01%	180 Missing and 3 partials ⚠️
pkg/patch/single.go	23.52%	77 Missing and 1 partial ⚠️
pkg/patch/manifest.go	0.00%	67 Missing ⚠️
pkg/patch/core.go	0.00%	65 Missing ⚠️
pkg/report/trivy.go	78.21%	42 Missing and 14 partials ⚠️
pkg/patch/cmd.go	15.78%	46 Missing and 2 partials ⚠️
pkg/utils/utils.go	53.84%	12 Missing ⚠️
pkg/langmgr/langmgr.go	88.77%	8 Missing and 3 partials ⚠️
pkg/patch/patch.go	25.00%	7 Missing and 2 partials ⚠️
pkg/types/v1alpha2/convert.go	89.28%	4 Missing and 2 partials ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1091      +/-   ##
==========================================
+ Coverage   48.80%   49.45%   +0.64%     
==========================================
  Files          35       38       +3     
  Lines        4143     5136     +993     
==========================================
+ Hits         2022     2540     +518     
- Misses       1985     2433     +448     
- Partials      136      163      +27

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

…trieval functionality Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

…und for buildx v0.25.0 bug Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

…nutes Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

Copilot

Pull Request Overview

This PR introduces experimental app-level patching support for Python packages, extending Copa's patching capabilities beyond OS packages to include application-level dependencies. The feature is gated behind the COPA_EXPERIMENTAL=1 environment variable and introduces new flags for package type filtering and library patch level control.

Adds scaffolding for experimental app-level patching with initial Python support
Introduces package type filtering (--pkg-types) to separate OS and library patching
Implements patch level control (--library-patch-level) for conservative version upgrades

Reviewed Changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
website/sidebars.js	Adds navigation entry for app-level patching documentation
website/docs/app-level-patching.md	Comprehensive documentation for the new app-level patching feature
pkg/types/unversioned/types.go	Updates manifest structure to separate OS and language updates
pkg/report/trivy.go	Implements version selection logic and Python package parsing
pkg/langmgr/python.go	Core Python package manager implementation
pkg/patch/cmd.go	Adds experimental CLI flags for package types and patch levels
pkg/patch/core.go	Integrates language managers into the core patching workflow

pkg/langmgr/python.go

pkg/report/trivy.go

pkg/patch/single.go

pkg/patch/patch.go

pkg/langmgr/python.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sertaç Özercan <852750+sozercan@users.noreply.github.com>

robert-cronin · 2025-08-08T05:39:56Z

website/docs/app-level-patching.md

@@ -0,0 +1,176 @@
+---


should we also update website/docs/scanner-plugins.md since the interface UpdatedManifest will change?

i missed this, good point! introduced v1alpha2 to support app patching flow, kept v1alpha1 for now

pkg/langmgr/python.go

robert-cronin · 2025-08-10T23:43:38Z

pkg/langmgr/python.go

+			version := strings.TrimSpace(parts[1])
+			installedPkgs[pkgName] = version
+		} else {
+			// Handle other formats that might be encountered in pip freeze output


is this a TODO or will this be a limitation of the python library patching?

@robert-cronin let's do a follow up on this when/if we encounter any that's not ==

ashnamehrotra · 2025-08-11T20:52:11Z

pkg/langmgr/langmgr.go

+type UpdateMap map[string]*UpdatePackageInfo
+
+func GetValidatedUpdatesMap(
+	updates unversioned.LangUpdatePackages,


since the functionality for validating updates and version comparing is similar with what we do for pkgmgr, maybe we could combine this into a helper folder to share the code?

can we do a follow up for this one

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

pkg/vex/openvex.go

ashnamehrotra · 2025-09-03T21:30:18Z

pkg/types/v1alpha2/types.go

+	OSUpdates   UpdatePackages     `json:"osupdates"`
+	LangUpdates LangUpdatePackages `json:"langupdates"`


can both OSUpdates and LangUpdates be of type UpdatePackages?

I wanted to keep these separately so we can define a boundary between OS and Lang updates

pkg/pkgmgr/pkgmgr.go

pkg/report/trivy.go

robert-cronin · 2025-09-08T10:09:38Z

pkg/patch/manifest.go

 	idxBytes, desc, err := resolver.Combine(ctx, srcRefs, annotations, false)
 	if err != nil {
 		return fmt.Errorf("failed to combine sources into manifest list: %w", err)
 	}

+	// Workaround for buildx v0.25.0 bug: manually inject index-level annotations into manifest bytes
+	// TODO: fixed in buildx v0.26.0+ so remove after upgrading


I think were close to solving this TODO, just flagging it here

can we open a new pr for this

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

sozercan added 14 commits June 3, 2025 23:03

feat: langmgr and python

05549a9

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

rename

02305b8

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

pin

1889aaf

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

validate

3b59147

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

pip freeze all

44fe8e3

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

lint

b8320dc

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

handle multiple langs

b825081

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

pkg-types

f85a523

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

experimental flag

5a26993

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

patch version pref

bb9751d

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

address fail if no os vulns

6a93691

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

handle ignore errors with multple pkgs

5f9ae1d

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

single layer

c75aafe

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

pick highest

e8a2b9b

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

github-project-automation bot moved this to 🆕 New in Copacetic Workboard Jun 4, 2025

github-project-automation bot added this to Copacetic Workboard Jun 4, 2025

Merge branch 'main' into lang-python2

a5951c4

sozercan added 12 commits June 5, 2025 04:18

test image

c8f4720

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

lint

ccd7ee1

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

library requires scanner report

709d6ff

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

tests

0a63715

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

lint

dcbd79a

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

lint

a44a48b

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

library-patch-level

4daea82

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

certifi exception

c35c12c

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

e2e

5cd4751

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

updates

ad9fc5b

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

docs

9ab7b9e

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

linty lint

4e2bf31

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

robert-cronin and others added 5 commits August 6, 2025 02:22

fix linting issues

162f4a5

Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

feat: increase timeout for multiplatform tests and add annotations re…

8a18ff2

…trieval functionality Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

feat: inject index-level annotations into manifest bytes as a workaro…

5ed4ed7

…und for buildx v0.25.0 bug Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

fix: increase timeout for multiplatform plugin tests from 40 to 50 mi…

2baaa40

…nutes Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>

ci: free disk space

6ccbeb7

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

sozercan marked this pull request as ready for review August 6, 2025 19:29

sozercan requested review from ashnamehrotra, Copilot and robert-cronin August 6, 2025 19:29

Copilot AI reviewed Aug 6, 2025

View reviewed changes

Update pkg/report/trivy.go

872e07d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sertaç Özercan <852750+sozercan@users.noreply.github.com>

robert-cronin reviewed Aug 10, 2025

View reviewed changes

ashnamehrotra reviewed Aug 11, 2025

View reviewed changes

sozercan added 5 commits August 22, 2025 21:19

add v1alpha2 and filter python-pkgs

5308dd6

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

fix test

dcec71d

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

Merge branch 'main' into lang-python2

dea6dcf

lint

d2aadf2

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

Merge branch 'main' into lang-python2

a23ad52

sozercan requested review from robert-cronin and ashnamehrotra September 2, 2025 20:08

fix test

b5bceba

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

sozercan force-pushed the lang-python2 branch from b4706cd to b5bceba Compare September 2, 2025 20:09

ashnamehrotra reviewed Sep 3, 2025

View reviewed changes

robert-cronin reviewed Sep 8, 2025

View reviewed changes

sozercan added 5 commits September 9, 2025 22:58

python vex output

b8cfa93

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

revert pkgmgr changes

98cb8e9

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

Merge branch 'main' into lang-python2

44e2e5f

lint

2d06410

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

remove free disk space and move to large workers

ca7aedf

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

sozercan force-pushed the lang-python2 branch from fd420f3 to ca7aedf Compare September 9, 2025 23:44

		OSUpdates UpdatePackages `json:"osupdates"`
		LangUpdates LangUpdatePackages `json:"langupdates"`

feat: langmgr and python #1091

Are you sure you want to change the base?

feat: langmgr and python #1091

Conversation

sozercan commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Adds experimental app-level patching support with Python language manager

Key Features

Example Usage

Uh oh!

codecov bot commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sozercan commented Jun 4, 2025 •

edited

Loading

codecov bot commented Jun 5, 2025 •

edited

Loading