[WIP] Feat/user decorators v2 #2463

romain-intel · 2025-06-18T09:55:40Z

No description provided.

This also includes packaging for local runs

…ompute plugins

Two types of decorators are supported: - a "pure" Python decorator that allows you to execute something before and after a step. You also have the possibility of entirely skipping a step (which is a fairly requested feature) - mutators (Flow level or Step level). These mutators allow you to modify the flow/steps dynamically. This is particularly useful when put together with configs. Still needs quite a few tests but it should be workable. A few features to try out: - You should be able to add mutators on a base flow spec class and have it work in a derived class - the add_decorator function on the MutableStep should take strings, actual MF decorators and user decorators - the add_decorator function on the MutableFlow should take strings and actual flow level decorators. - --with should work with any step level decorator (user defined including step mutators): - the name is the FQN for the class. It supports passing arguments as well - you can also use a shorter name provided that name is available *somewhere* where the flowspec is defined (either itself or one of its base classes) - mix and match decorators should work - the UserStepDecorator class has a init function that can take arguments if you want to build something a bit more fancy - the mutators also take arguments - config values can be used for any of those arguments (and of course can be used inside any of those functions)

- inserted_by is now a list - decorators are now auto registered on import allowing use even if not imported in the file; the name of the decorator is also automatically calculated as the shortest identifying one - `package info` now shows which decorators are injected by which portions of code - added compatibility with pylint for user_step_decorator Still known: - configs in arguments to user decorators - dash in config

Other bug fixes to also make things more natural and just plain old stupid bugs

…*kwargs normally

…or_specs

…sue with auto import of modules.

metaflow/plugins/cards/card_decorator.py

saikonen · 2025-07-11T10:32:37Z

metaflow/plugins/kubernetes/kubernetes_decorator.py

@@ -177,8 +179,6 @@ class KubernetesDecorator(StepDecorator):
    target_platform = KUBERNETES_CONDA_ARCH or "linux-64"

    def init(self):
-        super(KubernetesDecorator, self).init()


removed as unnecessary? same question for all the decorators init() changes

why removed? It is necessary. You need to call the base class' init because that is what resolves the attributes.

wait, I removed it? Ok, let me check

Ah right -- I now use this other function called external_init

…s_v2

The packaging PR #2463 introduced the `extract_args_kwargs_from_decorator_spec` to parse the decospecs added from the command line. This function has a subtle bug where it returns from the for loop instead of the end of the function.

When doing something like : ``` from metaflow import project .... mymutableflow.add_decorator( project, deco_kwargs={"name":"abc"}) ``` Metaflow breaks with the error : ``` File ".../metaflow/flowspec.py", line 312, in _process_config_decorators deco.pre_mutate(mutable_flow) File "mydeco.py", line 67, in pre_mutate mutable_flow.add_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 409, in add_decorator _add_flow_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 353, in _add_flow_decorator d for d in self._flow_cls._flow_decorators if d.name == flow_deco.name ^^^^^^ AttributeError: 'str' object has no attribute 'name' ``` This is because we dont parse the _flow_cls._flow_decorators in the right way (as a dict instead of list)

When doing something like : ``` from metaflow import project .... mymutableflow.add_decorator( project, deco_kwargs={"name":"abc"}) ``` Metaflow breaks with the error : ``` File ".../metaflow/flowspec.py", line 312, in _process_config_decorators deco.pre_mutate(mutable_flow) File "mydeco.py", line 67, in pre_mutate mutable_flow.add_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 409, in add_decorator _add_flow_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 353, in _add_flow_decorator d for d in self._flow_cls._flow_decorators if d.name == flow_deco.name ^^^^^^ AttributeError: 'str' object has no attribute 'name' ``` This is because we dont handle the _flow_cls._flow_decorators in the right way (as a dict instead of list)

When doing something like : ``` from metaflow import project .... mymutableflow.add_decorator( project, deco_kwargs={"name":"abc"}) ``` Metaflow breaks with the error : ``` File ".../metaflow/flowspec.py", line 312, in _process_config_decorators deco.pre_mutate(mutable_flow) File "mydeco.py", line 67, in pre_mutate mutable_flow.add_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 409, in add_decorator _add_flow_decorator( File ".../metaflow/user_decorators/mutable_flow.py", line 353, in _add_flow_decorator d for d in self._flow_cls._flow_decorators if d.name == flow_deco.name ^^^^^^ AttributeError: 'str' object has no attribute 'name' ``` This is because we dont parse the _flow_cls._flow_decorators in the right way (as a dict instead of list)

`deconame` is not declared and the code would have crashed on that code path. replaced it with the correct variable name

- Same flavor of bug fix as Netflix#2513

- Same flavor of bug fix as #2513 Reproduce by doing something like : ```python from metaflow import project .... mymutableflow.add_decorator( project, deco_kwargs={"name":"abc"}, duplicates=3) ```

romain-intel changed the base branch from master to feat/packaging-v2-again June 18, 2025 17:01

romain-intel force-pushed the feat/user_decorators_v2 branch from 719dcd3 to 5342612 Compare June 19, 2025 10:23

romain-intel added 2 commits July 3, 2025 02:33

New packaging support

255f4c8

This also includes packaging for local runs

Address comments

68ddbcd

romain-intel force-pushed the feat/packaging-v2-again branch from 301b0d5 to 68ddbcd Compare July 3, 2025 09:33

romain-intel force-pushed the feat/user_decorators_v2 branch from 2f97ddd to a57a7f1 Compare July 3, 2025 11:02

romain-intel added 2 commits July 6, 2025 16:48

Address comments: make async packaging optional and also changes to c…

697fc95

…ompute plugins

Fixup blob call

97bd763

romain-intel force-pushed the feat/user_decorators_v2 branch from a57a7f1 to 6fb4fbf Compare July 7, 2025 09:15

Hopefully make it compatible with other compute backends

6bd2387

romain-intel force-pushed the feat/user_decorators_v2 branch from 1436b0c to 38486fc Compare July 7, 2025 16:15

romain-intel added 16 commits July 8, 2025 13:53

Fix default metadata and local packaging

c225d81

Typo on inserted_by

55971c8

Fix using configs in user decorators

b74e9a9

Allow decorators with arguments and decospecs

eb11ba5

Other bug fixes to also make things more natural and just plain old stupid bugs

Fix packaging metadata

0872f5c

Fix init() for FlowMutators

e7cc5b4

Fix printing of WrapClass

d50e4e3

Fix flow decorator level config args

47b47a6

Updated show command; fixed an issue with flow mutators

fea79fd

Fix auto including of decorator modules

f6b5cb3

Fix init method for step/flow decorators -- they now take *args and *…

b9dd888

…*kwargs normally

WIP

d630546

Improve add/remove decorator for step decorators

7ea80b7

Made MutableFlow and MutableStep consistent; fixed issue with decorat…

5f1e2bd

…or_specs

romain-intel force-pushed the feat/user_decorators_v2 branch from 38486fc to 5f1e2bd Compare July 9, 2025 00:00

romain-intel added 2 commits July 9, 2025 15:02

Address more comments; made FlowMutatorMeta more consistent. Fixed is…

435342b

…sue with auto import of modules.

Rah

107ea9a

romain-intel added 6 commits July 9, 2025 16:08

Infinite loop fix

00757c9

Fix step decorator validity

4359202

More bug fixes

8dfefa1

Make runner use packaged metaflow

2759269

Revert unneeded change

49193bf

Fix wrapped function

5fe11fe

savingoyal marked this pull request as ready for review July 11, 2025 03:02

saikonen force-pushed the feat/packaging-v2-again branch from c225d81 to a59c07f Compare July 11, 2025 08:36

Base automatically changed from feat/packaging-v2-again to master July 11, 2025 08:37

Merge branch 'master' into feat/user_decorators_v2

23a48e2

saikonen reviewed Jul 11, 2025

View reviewed changes

romain-intel added 5 commits July 11, 2025 15:52

Fixup merge and fix tests

88ebbab

Merge remote-tracking branch 'origin/master' into feat/user_decorator…

4bd74be

…s_v2

Fix returning wrapped function from end step; fix parallel

a5c0294

Still trying to get runner to work nicely inside installed metaflow

fb93fb6

Woops

43269c1

savingoyal merged commit 85fb152 into master Jul 12, 2025
29 of 30 checks passed

savingoyal deleted the feat/user_decorators_v2 branch July 12, 2025 14:27

valayDave added a commit to valayDave/metaflow that referenced this pull request Jul 17, 2025

[bug fix] fix subtle bug from Netflix#2463

315379c

valayDave mentioned this pull request Jul 17, 2025

[bug fix] fix subtle bug from netflix/metaflow#2463 #2498

Merged

valayDave added a commit that referenced this pull request Aug 11, 2025

[bug fix] fix subtle bug from #2463

1e5f9a3

`deconame` is not declared and the code would have crashed on that code path. replaced it with the correct variable name

savingoyal pushed a commit that referenced this pull request Aug 11, 2025

[bug fix] fix subtle bug from #2463 (#2540)

30f9e54

`deconame` is not declared and the code would have crashed on that code path. replaced it with the correct variable name

valayDave added a commit to valayDave/metaflow that referenced this pull request Aug 11, 2025

[bug fix] fix subtle bug from Netflix#2463

a527e75

- Same flavor of bug fix as Netflix#2513

romain-intel mentioned this pull request Aug 21, 2025

WIP: Add the ability to package more libraries #2269

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Feat/user decorators v2 #2463

[WIP] Feat/user decorators v2 #2463

Uh oh!

romain-intel commented Jun 18, 2025

Uh oh!

Uh oh!

saikonen Jul 11, 2025

Uh oh!

romain-intel Jul 11, 2025

Uh oh!

romain-intel Jul 11, 2025

Uh oh!

romain-intel Jul 11, 2025

Uh oh!

Uh oh!

Uh oh!

[WIP] Feat/user decorators v2 #2463

[WIP] Feat/user decorators v2 #2463

Uh oh!

Conversation

romain-intel commented Jun 18, 2025

Uh oh!

Uh oh!

saikonen Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

romain-intel Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

romain-intel Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

romain-intel Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!