Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save ShapedAsNT with JLD2 #62

Open
Cornelius-G opened this issue Jun 15, 2022 · 1 comment
Open

Save ShapedAsNT with JLD2 #62

Cornelius-G opened this issue Jun 15, 2022 · 1 comment

Comments

@Cornelius-G
Copy link

Currently, it's not possible to save ShapedAsNT objects with JLD2.
Minimal example:

using ValueShapes
using JLD2

x = (a=1, b=2,)
vs = valshape(x)

x_shpd = ShapedAsNT(collect(x),vs)
jldsave("test.jld2", x = x_shpd)

This gives the following stacktrace.txt

@waldie11 and I tried solving that by implementing JLD2.writeas and Base.convert but didn't succeed.

We think the problem is somehow related to NamedTupleShape being deconstructed into its components in "JLD2/src/data/writing_datatypes.jl" finally resulting in the error in the function h5convert. This only happens if NamedTupleShape is ShapedAsNT.
Even though we tried for quite some time, we couldn't find a fix.

We noticed that the problem is the same when trying to store a type with unknown size like this:

T = NTuple{N, Int} where N
jldsave("test.jld2", a = T)

(for which we are not sure if something like that should even be possible)

@waldie11
Copy link

waldie11 commented Jun 17, 2022

It seems to boil down to the missing support in JLD2 of flexible types in structs.

using ValueShapes
using JLD2
jldsave("test.jld2",x=NamedTupleShape{(:a,:b)})
jldsave("test.jld2",x=NamedTupleShape)
struct test{X<:(NTuple{N,Integer} where N)}
    a::X
end
jldsave("test.jld2", a=test)

struct test_workaround
    a::(NTuple{N,Integer} where N)
end
jldsave("test.jld2", a=test_workaround)

NamedTupleShapes frequently dispatches by NamedTupleShapes{names}, where N is left undefined.

Suggestion:

  1. Hope for fix in JLD2, or
  2. reduce struct
--- a/src/named_tuple_shape.jl
+++ b/src/named_tuple_shape.jl
@@ -37,14 +37,14 @@ all(x -> x == 4.2, view(flatview(data), 1, :))
 """
-struct NamedTupleShape{names,AT<:(NTuple{N,ValueAccessor} where N),VT} <: AbstractValueShape
-    _accessors::NamedTuple{names,AT}
+struct NamedTupleShape{names,VT} <: AbstractValueShape
+    _accessors::NamedTuple{names,<:NTuple{N,ValueAccessor}} where N

For our project, the latter has a long rat tail reaching into BAT.jl. However I didn't see any direct usage of the length N information or the type AT anywhere. Will check if Zygote is affected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants