BUG: MemoryError when using "+=" operator with scipy.sparse.lil_matrix #18202
Description
Describe your issue.
When using the "+=" operator to add a sparse matrix to a lil_matrix in scipy.sparse, a MemoryError is raised when the matrix size is larger than a certain limit.
The issue seems to arise because the iadd method of lil_matrix creates a dense numpy array using np.zeros(self.shape, dtype=self.dtype). This behavior is different from the iadd method of csr_matrix and coo_matrix, which directly manipulate the sparse data structures without resorting to dense arrays.
This code works fine with other sparse matrix formats such as csr_matrix and coo_matrix, but raises a MemoryError when using lil_matrix if N is large. However, adding the matrices with the "+" operator instead of "+=" works fine even for lil_matrix.
Reproducing Code Example
import numpy as np
import scipy.sparse as sp
N = 10**7
A = sp.diags(np.ones(N))
L = sp.lil_matrix((N,N))
L += A # raises MemoryErrorError message
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-5-defc1c077c92> in <module>
6
7 L = sp.lil_matrix((N,N))
----> 8 L += A # raises MemoryError
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_lil.py in __iadd__(self, other)
127
128 def __iadd__(self,other):
--> 129 self[:,:] = self + other
130 return self
131
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_lil.py in __setitem__(self, key, x)
329 return self._set_intXint(key[0], key[1], x)
330 # Everything else takes the normal path.
--> 331 IndexMixin.__setitem__(self, key, x)
332
333 def _mul_scalar(self, other):
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_index.py in __setitem__(self, key, x)
135 x = x.tocoo(copy=True)
136 x.sum_duplicates()
--> 137 self._set_arrayXarray_sparse(i, j, x)
138 else:
139 # Make x and i into the same shape
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_lil.py in _set_arrayXarray_sparse(self, row, col, x)
315 return
316 # Fall back to densifying x
--> 317 x = np.asarray(x.toarray(), dtype=self.dtype)
318 x, _ = _broadcast_arrays(x, row)
319 self._set_arrayXarray(row, col, x)
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_coo.py in toarray(self, order, out)
320 def toarray(self, order=None, out=None):
321 """See the docstring for `spmatrix.toarray`."""
--> 322 B = self._process_toarray_args(order, out)
323 fortran = int(B.flags.f_contiguous)
324 if not fortran and not B.flags.c_contiguous:
/usr/local/lib/python3.9/dist-packages/scipy/sparse/_base.py in _process_toarray_args(self, order, out)
1296 return out
1297 else:
-> 1298 return np.zeros(self.shape, dtype=self.dtype, order=order)
1299
1300
MemoryError: Unable to allocate 728. TiB for an array with shape (10000000, 10000000) and data type float64SciPy/NumPy/Python version and system information
1.10.1 1.22.4 sys.version_info(major=3, minor=9, micro=16, releaselevel='final', serial=0)
Build Dependencies:
blas:
detection method: cmake
found: true
include directory: unknown
lib directory: unknown
name: OpenBLAS
openblas configuration: unknown
pc file directory: unknown
version: 0.3.18
lapack:
detection method: cmake
found: true
include directory: unknown
lib directory: unknown
name: OpenBLAS
openblas configuration: unknown
pc file directory: unknown
version: 0.3.18
Compilers:
c:
commands: cc
linker: ld.bfd
name: gcc
version: 10.2.1
c++:
commands: c++
linker: ld.bfd
name: gcc
version: 10.2.1
cython:
commands: cython
linker: cython
name: cython
version: 0.29.33
fortran:
commands: gfortran
linker: ld.bfd
name: gcc
version: 10.2.1
pythran:
include directory: /tmp/pip-build-env-yjjniym9/overlay/lib/python3.9/site-packages/pythran
version: 0.12.1
Machine Information:
build:
cpu: x86_64
endian: little
family: x86_64
system: linux
cross-compiled: false
host:
cpu: x86_64
endian: little
family: x86_64
system: linux
Python Information:
path: /opt/python/cp39-cp39/bin/python
version: '3.9'