Skip to content

Metaflow + PyTorch: '_ClassNamespace' object is not iterable #2512

@timoffex

Description

@timoffex

An error is thrown when running a flow with PyTorch imported:

Traceback (most recent call last):
  File ".../site-packages/metaflow/cli.py", line 649, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  ...
  File ".../site-packages/metaflow/package/__init__.py", line 87, in __init__
    self._mfcontent = MetaflowCodeContentV1(criteria=_module_selector)
  File ".../site-packages/metaflow/packaging_sys/v1.py", line 64, in __init__
    self._modules = {
  File ".../site-packages/metaflow/packaging_sys/v1.py", line 67, in <dictcomp>
    set(
TypeError: '_ClassNamespace' object is not iterable

To reproduce, try the helloworld.py example with import torch at the top (using the latest versions torch==2.7.1 and metaflow==2.16.5).

PyTorch adds a custom ModuleType-derived class to sys.modules here, which defines __getattr__ to always return a _ClassNamespace object. Unfortunately, that means that getattr(<that module>, "__path__") returns something unexpected.

IMO PyTorch is at fault here, but PyTorch has had this code for a while (maybe 11 months) so it's likely that many Metaflow users cannot use the most recent versions with PyTorch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions