CSV files with too many values in a row cause errors #440
Description
Original title: csv.DictReader can have None as key
In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value.
sqlite_utils.utils.rows_from_file cannot handle that:
url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")
with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
db["fab2018"].insert_all(reader, pk="Id")Result:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
chunk = list(chunk)
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
if any("[" in key or "]" in key for key in record.keys()):
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable
Code:
sqlite-utils/sqlite_utils/db.py
Line 3454 in 59be60c
sqlite-utils insert from command line is not affected by this issue.