Fix/runner packaging #2507
Fix/runner packaging #2507
Conversation
Other small fixes to improve performance of backends.
This reverts commit 4a19497.
There was a problem hiding this comment.
Pull Request Overview
This PR focuses on fixing runner packaging by updating package policy declarations, improving environment variable handling, and enhancing archive content processing. The changes aim to improve the robustness and compatibility of the Metaflow packaging system.
- Updates package policy syntax from
METAFLOW_PACKAGE = 1toMETAFLOW_PACKAGE_POLICY = "include" - Enhances environment variable merging logic with support for override semantics
- Improves archive member handling and extraction processes
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test_included_modules/my_decorators.py | Updates package policy declaration syntax |
| test/core/tests/runtime_dag.py | Adds "code" to excluded properties list for task comparison |
| metaflow/runner/subprocess_manager.py | Refactors environment variable handling with improved merging logic |
| metaflow/runner/click_api.py | Adds flow mutator initialization and reformats tuple assignments |
| metaflow/plugins/pypi/conda_decorator.py | Updates environment variable handling to support override syntax |
| metaflow/packaging_sys/v1.py | Adds error handling for module loading and fixes content marker path |
| metaflow/packaging_sys/tar_backend.py | Adds member name extraction and updates method signatures |
| metaflow/packaging_sys/backend.py | Adds abstract methods for member handling and updates interfaces |
| metaflow/packaging_sys/init.py | Refactors archive content processing and environment variable handling |
| metaflow/package/init.py | Updates module selection criteria and extraction method calls |
| metaflow/metaflow_environment.py | Improves environment variable export logic with override support |
| metaflow/extension_support/init.py | Adds conditional check for package generation |
| metaflow/client/core.py | Adds BytesIO seek operations for proper stream positioning |
| mod, | ||
| True, # This is a Metaflow module (see filter below) | ||
| ) | ||
| except: |
There was a problem hiding this comment.
Using a bare 'except:' clause catches all exceptions including system-exiting exceptions like KeyboardInterrupt and SystemExit. Consider catching specific exceptions or using 'except Exception:' instead.
| except: | |
| except (AttributeError, TypeError, FileNotFoundError): |
Copilot uses AI. Check for mistakes.
| ): | ||
| return True | ||
| return False | ||
| except: |
There was a problem hiding this comment.
Using a bare 'except:' clause catches all exceptions including system-exiting exceptions like KeyboardInterrupt and SystemExit. Consider catching specific exceptions or using 'except Exception:' instead.
| except: | |
| except Exception: |
Copilot uses AI. Check for mistakes.
Debug messages Fix using runner with new packaging Remove un-necssary import Fix issues with location of package marker Fix runtime dag test Fixes Netflix#2512 and broken test Fixes with archive extractons Fix HUGE inneficiency with old packages. Other small fixes to improve performance of backends. Change METAFLOW_PACKAGE -> METAFLOW_PACKAGE_POLICY Fix handling of METAFLOW_EXTRACTED_ROOT TO REVERT: add more debugging to figure out test issues Revert "TO REVERT: add more debugging to figure out test issues" This reverts commit 4a19497. [commit-squasher] squashed from 12 commits starting from 384e3bf [fix/runner_packaging] onto ed5b88f [valay/test-packaging-changes]
|
Merging due to no one opposing it :). |
No description provided.