Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset can't be pickled #673

Open
zhiltsov-max opened this issue Feb 18, 2022 · 0 comments
Open

Dataset can't be pickled #673

zhiltsov-max opened this issue Feb 18, 2022 · 0 comments
Labels
ENHANCE Enhancement of existing features

Comments

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Feb 18, 2022

Pickling can be useful for dumping and loading and for multiprocessing. It is required for PyTorch, because it heavily relies on multiprocessing in their Dataset. Currently, in many cases Datumaro will fail to pickle, mostly because of lambdas and local declarations.

> pickle.dumps(dataset)
E AttributeError: Can't pickle local object 'Environment.__init__.<locals>._filter.<locals>.<lambda>'

There are some places where such errors arise.

@zhiltsov-max zhiltsov-max added the ENHANCE Enhancement of existing features label Feb 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ENHANCE Enhancement of existing features
Projects
None yet
Development

No branches or pull requests

1 participant