Store and retrieve object sets in ipython!
Create different object sets in ipython for your different experiments or projects! Better than that pesky native %store as it uses dill as backend (instead of pickle) and allows you to better manage your objects dividing them in sets.
It may or may not work with jupyter (it should if you are using the ipython kernel, but testing is still due).
It works with the pre-packaged (c)python3 pickle, but I highly reccomend installing dill, as it offers a much wider object type support (including functions, keras models, and just about anything). IPy-storage will use dill as the default serialization protocol if it is installed.
With the source downloaded, just open the ipython in the root and execute %load_ext ipystore.storage_magic. It is recommended to include it in the ipython initialization to avoid doing this for every new interpreter instance. Installation will be much easier when a setup.py file is done (soon™).
Your different object sets will be divided in folders that will be inside a base directory. The objects will be saved in those folders with their variable names in dill/pickle serialization format, depending on which backend you have installed. Credit is due to a slightly modified pickleshare source code that we are using.
There are two main magics on ipy-storage: %sets and %setstore. The first does operations with sets, while the latter does operations with the open set.
You may create or open a new set with %sets -o <set_name> <base_dir(optional)>. The default base directory defaults to ./ipy_db and will be placed on the current ipython working dir when a new set is created. You may change the current base directory at any time with %sets -b <base_dir>. IF you need to know your current base directory, use %sets -b (no parameters). %sets -d <set_name> <base_dir(optional)> delete the informed set and ALL of its contents. Please note that %sets -d(no parameters) deletes the CURRENT set.
Once a set is open, you may store and retrieve files from a set with %setstore -s <objs_to_store> and %setload -l <objs_to_load>. You may store/load multiple objects at time, e.g. %setstore -[s,l] apple,orange,bars. You can also use %setstore -la to load all objects inside a set at once, and %setstore -sa to update all objects in your database that are present in your namespace.
You can use %sets <base_dir> to list all sets in the appointed base directory (defaults to ipy_db), and %setstore(no parameters) to list all objects inside the current set.
I hope to turn this in a more complete helper for Ipython data storage, including the hability to load files from different sets without having to change sets constantly, show stored objects type and size, and more.