Multiple torch.load in one file · Issue #45136 · pytorch/pytorch

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account module: serialization triaged

torch.load fails to load multiple content in one file.

To Reproduce

with open('test.pkl', 'wb') as f:
    torch.save(4, f)
    torch.save(5, f)
with open('test.pkl', 'rb') as f:
    assert torch.load(f) == 4
    assert torch.load(f) == 5

Crashes with the following error

~/anaconda3/envs/env38/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    583                     return torch.jit.load(opened_file)
    584                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 585         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
~/anaconda3/envs/env38/lib/python3.8/site-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    753             "functionality.".format(type(f)))
--> 755     magic_number = pickle_module.load(f, **pickle_load_args)
    756     if magic_number != MAGIC_NUMBER:
    757         raise RuntimeError("Invalid magic number; corrupt file?")
UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.
Environment
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A
OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (Debian 8.3.0-6) 8.3.0
Clang version: Could not collect
CMake version: version 3.13.4
Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 9.2.148
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.6.0
[pip] torch-cluster==1.5.4
[pip] torch-geometric==1.4.3
[pip] torch-scatter==2.0.5
[pip] torch-sparse==0.6.7
[pip] torch-spline-conv==1.2.0
[pip] torchvision==0.7.0
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2020.0                      166  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.0.15           py38ha843d7b_0  
[conda] mkl_random                1.1.0            py38h962f231_0  
[conda] numpy                     1.18.1           py38h4f9e942_0  
[conda] numpy-base                1.18.1           py38hde5b4d6_1  
[conda] pytorch                   1.6.0               py3.8_cpu_0  [cpuonly]  pytorch
[conda] torch-cluster             1.5.4                    pypi_0    pypi
[conda] torch-geometric           1.4.3                    pypi_0    pypi
[conda] torch-scatter             2.0.5                    pypi_0    pypi
[conda] torch-sparse              0.6.7                    pypi_0    pypi
[conda] torch-spline-conv         1.2.0                    pypi_0    pypi
[conda] torchvision               0.7.0                  py38_cpu  [cpuonly]  pytorch
cc @ezyang @gchanan @zou3519
  module: serialization
  Issues related to serialization (e.g., via pickle, or otherwise) of PyTorch objects
 label
    Sep 22, 2020
with open('test.pkl', 'wb') as f:
    torch.save(4, f, _use_new_zipfile_serialization=False)
    torch.save(5, f, _use_new_zipfile_serialization=False)
with open('test.pkl', 'rb') as f:
    assert torch.load(f) == 4
    assert torch.load(f) == 5
  triaged
  This issue has been looked at a team member, and triaged and prioritized into an appropriate module
 label
    Sep 29, 2020
          I took a look at this. The reason multiple-loads does not work with zipfile is that the zipfile reader does not advance the position in the file (f in the example) to the end of the zip file it just read. It's possible to get this behavior working, but unfortunately it's non-trivial and likely inefficient due to the way miniz` is writing out files in the archive (i.e. file sizes aren't populated in their header, rather in a trailing data descriptor struct, so traversing to the end of the archive will likely involve scanning every byte in the file.
@mariogeiger can you instead use a different way of commingling save files? For example you could wrap the multiple saves up into another zip file using the Python zipfile library
          @mariogeiger you can use io.BytesIO to avoid writing temp files, e.g.:
import torch
import io
import zipfile
to_write = [4, 5]
with zipfile.ZipFile('foo.zip', 'w') as zf:
  for i, data in enumerate(to_write):
    f = io.BytesIO()
    torch.save(data, f)
    zf.writestr(f'{i}.pkl', f.getbuffer())
with zipfile.ZipFile('foo.zip', 'r') as zf:
  for file, expected in zip(zf.namelist(), to_write):
      with zf.open(file, 'r') as f:
          assert torch.load(io.BytesIO(f.read())) == expected
          Thanks but unfortunately the error is in pickle with me . What does this
error means exactly as I searched a lot for this error . Does that mean the
data may be corrupted ?
On Sat, Apr 3, 2021, 9:21 AM Mario Geiger ***@***.***> wrote:
 I stopped using torch.save / load. I use pickle instead.
 You are receiving this because you commented.
 Reply to this email directly, view it on GitHub
 <#45136 (comment)>,
 or unsubscribe
 <https://github.com/notifications/unsubscribe-auth/AHBVOVD6YR5UMQSC7HXLWODTG26V5ANCNFSM4RV7M62Q>
    module: serialization
  Issues related to serialization (e.g., via pickle, or otherwise) of PyTorch objects
    triaged
  This issue has been looked at a team member, and triaged and prioritized into an appropriate module