[Typing][B-01] Add type annotations for `python/paddle/io/reader.py` #65587

enkilee · 2024-07-01T02:15:00Z

PR Category

User Experience

PR Types

Improvements

Description

[Type Hints] 为公开 API 标注类型提示信息 #65008
B-01

paddle-bot · 2024-07-01T02:15:05Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

megemini · 2024-07-01T08:34:11Z

python/paddle/io/reader.py

@@ -382,25 +397,37 @@ class DataLoader:
        please see :code:`paddle.io.IterableDataset`
    """

+    return_list: bool
+    collate_fn: default_collate_fn | None


需要是个 Callable ～可以参考以下 default_collate_fn 的输入输出类型，貌似还挺复杂的 ... ...

恩，我试着改改。

megemini · 2024-07-01T08:39:53Z

python/paddle/io/reader.py

+    collate_fn: default_collate_fn | None
+    use_buffer_reader: bool
+    prefetch_factor: int
+    worker_init_fn: Callable[..., Any] | None


参考了一下测试用例：

class TestDynamicDataLoaderIterInitFuncSplit(unittest.TestCase): def test_main(self): place = base.CPUPlace() with base.dygraph.guard(place): dataset = RangeIterableDataset(0, 10) def worker_spliter(worker_id): worker_info = get_worker_info() dataset = worker_info.dataset start = dataset.start end = dataset.end num_per_worker = int( math.ceil((end - start) / float(worker_info.num_workers)) ) worker_id = worker_info.id dataset.start = start + worker_id * num_per_worker dataset.end = min(dataset.start + num_per_worker, end) dataloader = DataLoader( dataset, places=place, num_workers=1, batch_size=1, drop_last=True, worker_init_fn=worker_spliter, ) rets = [] for d in dataloader: rets.append(d.numpy()[0][0]) assert tuple(sorted(rets)) == tuple(range(0, 10))

这里貌似可以是 Callable[[int], None]

megemini · 2024-07-01T08:41:13Z

python/paddle/io/reader.py

+    prefetch_factor: int
+    worker_init_fn: Callable[..., Any] | None
+    dataset: Dataset
+    feed_list: list[Tensor] | tuple[Tensor]


用 Sequence 可否？

收到，谢谢

megemini · 2024-07-01T08:43:19Z

python/paddle/io/reader.py

+    worker_init_fn: Callable[..., Any] | None
+    dataset: Dataset
+    feed_list: list[Tensor] | tuple[Tensor]
+    places: list[Place] | tuple[Place] | list[str]


用 paddle._typing 的 PlaceLike ？ Sequence[PlaceLike]

Paddle 的 Place 与 CPUPlace 没有继承关系，所以不能直接用 Place ～

p.s. 我也是最近才发现的 🤣

收到，谢谢

megemini · 2024-07-02T10:46:39Z

python/paddle/io/reader.py

+    import numpy.typing as npt
+
+    from paddle import Tensor
+    from python.paddle._typing.device_like import PlaceLike


Suggested change

from python.paddle._typing.device_like import PlaceLike

from python.paddle._typing import PlaceLike

megemini · 2024-07-02T10:49:13Z

python/paddle/io/reader.py

@@ -382,25 +404,37 @@ class DataLoader:
        please see :code:`paddle.io.IterableDataset`
    """

+    return_list: bool
+    collate_fn: Callable[[_Collate_Fn_State], None]


这个确实有点麻烦，我试了一下：

from __future__ import annotations import numbers import numpy as np import numpy.typing as npt from typing import Protocol, overload, Sequence, Mapping, Any, TYPE_CHECKING, AnyStr, TypeVar import paddle if TYPE_CHECKING: from paddle import Tensor KT = TypeVar('KT') VT = TypeVar('VT') @overload def default_collate_fn(batch: Sequence[npt.NDArray[Any]] | Sequence[numbers.Number]) -> npt.NDArray[Any]: ... @overload def default_collate_fn(batch: Sequence[Tensor]) -> Tensor: ... @overload def default_collate_fn(batch: Sequence[AnyStr]) -> AnyStr: ... @overload def default_collate_fn(batch: Sequence[Mapping[KT, VT]]) -> Mapping[KT, VT]: ... @overload def default_collate_fn(batch: Sequence[Sequence[VT]]) -> Sequence[VT]: ... def default_collate_fn(batch): sample = batch[0] if isinstance(sample, np.ndarray): batch = np.stack(batch, axis=0) return batch elif isinstance(sample, paddle.Tensor): return paddle.stack(batch, axis=0) elif isinstance(sample, numbers.Number): batch = np.array(batch) return batch elif isinstance(sample, (str, bytes)): return batch elif isinstance(sample, Mapping): return { key: default_collate_fn([d[key] for d in batch]) for key in sample } elif isinstance(sample, Sequence): sample_fields_num = len(sample) if not all(len(sample) == sample_fields_num for sample in iter(batch)): raise RuntimeError( "fields number not same among samples in a batch" ) return [default_collate_fn(fields) for fields in zip(*batch)] raise TypeError( "batch data con only contains: tensor, numpy.ndarray, " f"dict, list, number, but got {type(sample)}" ) class _Collate_Fn(Protocol): @overload def __call__(self, batch: Sequence[npt.NDArray[Any]] | Sequence[numbers.Number]) -> npt.NDArray[Any]: ... @overload def __call__(self, batch: Sequence[Tensor]) -> Tensor: ... @overload def __call__(self, batch: Sequence[AnyStr]) -> AnyStr: ... @overload def __call__(self, batch: Sequence[Mapping[KT, VT]]) -> Mapping[KT, VT]: ... @overload def __call__(self, batch: Sequence[Sequence[VT]]) -> Sequence[VT]: ... fn: _Collate_Fn = default_collate_fn

@SigureMo 帮忙看看这个 _Collate_Fn 的 Protocol 看看是否可行？

这是把default_collate_fn在这里又重载了？得这么重复么。

目测没啥问题，检查如果没问题就没问题，只有一些风格的小问题

_Collate_Fn -> _CollateFn

KT -> _K，VT -> _V，K 和 V 和 T 一样是常用的泛型变量

关于 _CollateFn 类型要用 Protocol 重写一遍的问题，没有解决方案，前段时间刚吐槽过一次，但就是得写两遍重复的代码（paddle.jit.save 那边就是这样的）

这是把default_collate_fn在这里又重载了？得这么重复么。

抱歉，刚看到这里有留言 ... ...

我这里是举例，根据 default_collate_fn 可以推导出 _CollateFn 的写法～

只保留 _CollateFn 就可以了～不需要把 default_collate_fn 再写一边～～～ 🤣🤣🤣

megemini · 2024-07-02T10:49:58Z

python/paddle/io/reader.py

+    worker_init_fn: Callable[[int], None]
+    dataset: Dataset
+    feed_list: Sequence[Tensor] | None
+    places: Sequence[PlaceLike] | list[str] | None


Suggested change

places: Sequence[PlaceLike] | list[str] | None

places: Sequence[PlaceLike] | None

应该可以不需要 list[str] 了吧～ PlaceLike 包括 str 了～

megemini · 2024-07-02T10:51:12Z

python/paddle/io/reader.py

另外，DataLoader 的

def __len__(self): def __iter__(self): def __call__(self):

需要也标注一下～不然在 for 循环等地方可能无法确定类型～

SigureMo · 2024-07-03T01:59:31Z

python/paddle/io/reader.py

+
+    from paddle import Tensor
+    from python.paddle._typing.device_like import PlaceLike
+    from python.paddle.io.dataloader.dataset import Dataset


哪来的 python.？

SigureMo · 2024-07-03T02:12:35Z

python/paddle/io/reader.py

@@ -382,25 +404,37 @@ class DataLoader:
        please see :code:`paddle.io.IterableDataset`
    """

+    return_list: bool
+    collate_fn: Callable[[_Collate_Fn_State], None]


目测没啥问题，检查如果没问题就没问题，只有一些风格的小问题

_Collate_Fn -> _CollateFn

KT -> _K，VT -> _V，K 和 V 和 T 一样是常用的泛型变量

关于 _CollateFn 类型要用 Protocol 重写一遍的问题，没有解决方案，前段时间刚吐槽过一次，但就是得写两遍重复的代码（paddle.jit.save 那边就是这样的）

ooooo-create · 2024-07-07T17:21:26Z

python/paddle/io/reader.py

+        batch_size: int = 1,
+        shuffle: bool = False,
+        drop_last: bool = False,
+        collate_fn: Callable[[_CollateFn], None] = None,


（这里是不是 _CollateFn | None 呢

~~collate_fn: Callable[[_CollateFn], None] | None = None, ？~~

collate_fn: _CollateFn | None = None

megemini · 2024-07-08T05:38:38Z

python/paddle/io/reader.py

+@overload
+def default_collate_fn(
+    batch: Sequence[npt.NDArray[Any]] | Sequence[numbers.Number],
+) -> npt.NDArray[Any]:
+    ...
+
+
+@overload
+def default_collate_fn(batch: Sequence[Tensor]) -> Tensor:
+    ...
+
+
+@overload
+def default_collate_fn(batch: Sequence[AnyStr]) -> AnyStr:
+    ...
+
+
+@overload
+def default_collate_fn(batch: Sequence[Mapping[_K, _V]]) -> Mapping[_K, _V]:
+    ...
+
+
+@overload
+def default_collate_fn(batch: Sequence[Sequence[_V]]) -> Sequence[_V]:
+    ...
+
+
+def default_collate_fn(batch):
+    sample = batch[0]
+    if isinstance(sample, np.ndarray):
+        batch = np.stack(batch, axis=0)
+        return batch
+    elif isinstance(sample, paddle.Tensor):
+        return paddle.stack(batch, axis=0)
+    elif isinstance(sample, numbers.Number):
+        batch = np.array(batch)
+        return batch
+    elif isinstance(sample, (str, bytes)):
+        return batch
+    elif isinstance(sample, Mapping):
+        return {
+            key: default_collate_fn([d[key] for d in batch]) for key in sample
+        }
+    elif isinstance(sample, Sequence):
+        sample_fields_num = len(sample)
+        if not all(len(sample) == sample_fields_num for sample in iter(batch)):
+            raise RuntimeError(
+                "fields number not same among samples in a batch"
+            )
+        return [default_collate_fn(fields) for fields in zip(*batch)]
+
+    raise TypeError(
+        "batch data con only contains: tensor, numpy.ndarray, "
+        f"dict, list, number, but got {type(sample)}"
+    )


不用这里再写一边，如果有需要，在这个函数的文件那里写就行～ 🤣

Update

这个函数在 python/paddle/io/dataloader/collate.py ～不是公开 API ，先不标注了～

megemini · 2024-07-08T05:39:02Z

python/paddle/io/reader.py

+        ...
+
+
+fn: _CollateFn = default_collate_fn


这个也不用～保留上面的 _CollateFn 就行～

python/paddle/io/reader.py

megemini

其他的暂时没发现啥问题了～辛苦～ 🤟🤟🤟

python/paddle/io/reader.py

SigureMo

LGTMeow

…addlePaddle#65587) --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>

fix

a1e1ec1

paddle-bot bot added the contributor External developers label Jul 1, 2024

luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Jul 1, 2024

luotao1 assigned luotao1 and SigureMo Jul 1, 2024

megemini mentioned this pull request Jul 1, 2024

[Type Hints] 为公开 API 标注类型提示信息 #65008

Closed

megemini reviewed Jul 1, 2024

View reviewed changes

enkilee added 2 commits July 2, 2024 08:52

fix

75e6f60

fix

cd81b6e

megemini reviewed Jul 2, 2024

View reviewed changes

SigureMo reviewed Jul 3, 2024

View reviewed changes

fix

8d8a701

ooooo-create reviewed Jul 7, 2024

View reviewed changes

megemini reviewed Jul 8, 2024

View reviewed changes

python/paddle/io/reader.py Outdated Show resolved Hide resolved

fix

dcc366b

megemini reviewed Jul 8, 2024

View reviewed changes

python/paddle/io/reader.py Outdated Show resolved Hide resolved

enkilee and others added 5 commits July 9, 2024 08:42

fix

a41ea2d

move imports, avoid to use union in return

bf01b50

hidden protocol at runtime

aee7716

Merge branch 'develop' into typing-b01

9fc7625

ignore generic

9183244

SigureMo approved these changes Jul 13, 2024

View reviewed changes

SigureMo merged commit 475deec into PaddlePaddle:develop Jul 13, 2024

lixcli pushed a commit to lixcli/Paddle that referenced this pull request Jul 22, 2024

[Typing][B-01] Add type annotations for python/paddle/io/reader.py (P…

407030f

…addlePaddle#65587) --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>

enkilee deleted the typing-b01 branch January 8, 2025 08:04

	from python.paddle._typing.device_like import PlaceLike
	from python.paddle._typing import PlaceLike

	places: Sequence[PlaceLike] \| list[str] \| None
	places: Sequence[PlaceLike] \| None

[Typing][B-01] Add type annotations for python/paddle/io/reader.py #65587

[Typing][B-01] Add type annotations for python/paddle/io/reader.py #65587

Uh oh!

Conversation

enkilee commented Jul 1, 2024

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Jul 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

megemini Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

megemini Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Update

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

megemini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SigureMo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

[Typing][B-01] Add type annotations for `python/paddle/io/reader.py` #65587

[Typing][B-01] Add type annotations for `python/paddle/io/reader.py` #65587

megemini Jul 8, 2024 •

edited

Loading

megemini Jul 8, 2024 •

edited

Loading