-
-
Notifications
You must be signed in to change notification settings - Fork 3k
Closed
Labels
Description
Bug Report
There has been a sizeable slowdown (33x) in areas of our codebase in recent versions of mypy
(since 0.981); using the latest version I was finally able to track down and profile the culprits to create a minimal example which seems to be related to Union
typechecking.
To Reproduce
from __future__ import annotations
from typing import Dict, Union, Mapping, Protocol, Any, Optional, List
import numpy as np
class _TensorLike(Protocol):
"""Protocol defining what methods and operations a Generic Tensor can perform."""
def __mul__(self: _TensorLike, other: Any) -> _TensorLike:
"""Multiplies the self tensor with the other tensor and returns the result."""
...
def __sub__(self: _TensorLike, other: Any) -> _TensorLike:
"""Subtracts the other tensor from the self tensor and returns the result."""
...
def squeeze(self: _TensorLike, axis: Optional[Any] = None) -> _TensorLike:
"""Returns a tensor with all the dimensions of input of size 1 removed."""
...
_Weights = Mapping[str, _TensorLike]
class Test:
_other_worker_shares: List[int]
def _encode_secret(self, secret: Union[_TensorLike, np.ndarray]) -> np.ndarray:
return np.ones(5)
def _reconstruct_secret(self, shares: List[Union[np.ndarray, int]]) -> np.ndarray:
return np.ones(5)
def _encode_and_reconstruct_state_dict(
self, secret_state_dict: Union[_Weights, Mapping[str, np.ndarray]]
) -> Dict[str, np.ndarray]:
"""Encodes and reconstructs state dict from own and worker shares.
Encrypts `secret_state_dict` using `own_shares` then reconstructs it using
`self._other_worker_shares`.
"""
encrypted_state_dict = [
self._encode_secret(v) for _, v in secret_state_dict.items()
]
reconstructed = [
self._reconstruct_secret([*self._other_worker_shares, value])
for value in encrypted_state_dict
]
x = list(secret_state_dict)
z = zip(x, reconstructed)
d = dict(z)
return d
Expected Behavior
Timings between versions are substantially different:
0.971
= 3.042s1.1.1
= 1m44.30s
Per-line timings give:
7 106.4
8 10.9
11 2.0
12 1.9
15 1.4
16 1.0
18 10.4
19 4.8
20 2.1
22 58.1
24 46.5
25 2.0
28 186.1
31 83.3
36 2.0
41 78880690.2
44 416.6
48 8945003.5
49 322.2
50 458.1
51 1.3
Given this (fairly) minimal example in which no typechecking errors are found I would expect there to not be such a slowdown.
Is there a paradigm shift or issue that would explain this and how can we go about making the typechecker happier and faster?
Actual Behavior
Substantial slowdown after 0.971
Your Environment
- Mypy version used: 0.971/1.1.1
- Mypy command-line flags:
--line-checking-stats
for 1.1.1 run - Mypy configuration options from
mypy.ini
(and other config files): None - Python version used: 3.8.16
- numpy: 1.23.5 (though same behaviour in 1.24.2)
sileht