Skip to content

Conversation

albertvillanova
Copy link
Member

@albertvillanova albertvillanova commented Aug 6, 2025

This PR fixes a bug in the LocalPythonExecutor where ast.GeneratorExp nodes were incorrectly returning lists instead of proper Python generators.

Previously, generator expressions like (x for x in range(10)) would be evaluated eagerly and return a list, which breaks the expected lazy evaluation semantics of Python generators and could cause memory issues with large sequences.

Changes made:

  • Fix evaluate_generatorexp function: Modified the implementation to return an actual Python generator object instead of a list
  • Maintaine lazy evaluation: Generator expressions now properly implement lazy evaluation as expected in Python
  • Preserve functionality: All existing logic for handling comprehension generators, conditions, and variable scoping remains intact

Tests:

  • Add regression tests to verify generator expressions return proper generator objects

This fix ensures that the LocalPythonExecutor correctly handles generator expressions according to Python's standard behavior, maintaining both performance and semantic correctness.

@albertvillanova albertvillanova marked this pull request as ready for review August 6, 2025 16:36
@albertvillanova albertvillanova merged commit f7b4a33 into huggingface:main Aug 6, 2025
3 checks passed
@albertvillanova albertvillanova deleted the fix-generator-exp branch August 6, 2025 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant