This repository was archived by the owner on Feb 10, 2021. It is now read-only.
This repository was archived by the owner on Feb 10, 2021. It is now read-only.
Cannot load a pickle file containing an array #108
Description
Tested with python 3.5
Here is a small snippet reproducing the problem:
import pickle
import array
from hdfs3 import HDFileSystem
hdfs = HDFileSystem(host='localhost', port=8020)
a = array.array('d', [1, 2, 3, 4])
# Dump works and the pickle is valid:
# (when retrieved locally using hadoop CLI, pickle can load it)
with hdfs.open("/user/aabadie/test.pkl", "wb") as f:
pickle.dump(a, f)
# But loading via hdfs file object fails:
with hdfs.open("/user/aabadie/test.pkl", "rb") as f:
print(pickle.load(f))Here is the error:
---------------------------------------------------------------------------
EOFError Traceback (most recent call last)
<ipython-input-17-d986117cc344> in <module>()
1 with hdfs.open("/user/aabadie/test.pkl", "rb") as f:
----> 2 print(pickle.load(f))
3
4
5
EOFError: Ran out of input
I get the same result if I try to pickle a numpy array insteand of a Python array.