Empty arrays are sometimes serialized as null, and sometimes as []

[aatifsyed]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

$ ./lotus --version
lotus version 1.25.2+calibnet+git.f51f83bfe

Repro Steps

N/A

Describe the Bug

Background

In Go, [] slices have three states, which are serialized differently:

1. Uninitialized (never assigned), serialized as null
2. Initialized, but empty, serialized as []
3. Initialized, but non-empty, serialized as [<item>]

var unrelated bool = true
var s []int
if unrelated {
	s = []int{}
}
ser, _ := json.Marshal(s)
fmt.Printf("%s\n", ser) // `null` or `[]`

playground link

Motivation

Specification of JSON types in the Common Node API FIP
The current draft of the spec represents slices as null | [].

However, the fact that unrelated in the example above affects serialization output causes:

• Issues for nodes like forest that want to reproduce Lotus' RPC output.
  • To exactly match lotus, we'd need to track initialization stack and program logic that Lotus does, which is basically non-deterministic from the outside.
• Issues for clients when switching nodes.

Fix

Lotus should always serialize slices as JSON Arrays, and the Common Node API should specify serialization as such.
AIUI this is a longstanding quirk of go that has existing solutions:

• https://freshman.tech/snippets/go/nil-vs-empty-slices/
• proposal: encoding/json: nilasempty to encode nil-slices as [] golang/go#37711 (comment)
• proposal: encoding/json: nilasempty to encode nil-slices as [] golang/go#37711 (comment)
• https://github.com/pkierski/niltoempty

Logging Information

N/A

[Stebalien]

So, pending the go changes, we have a few options:

1. Mark all slices as "omitempty". This won't really fix the problem, but it'll mean we avoid serializing these fields if they're empty (which is somewhat better).
2. Replace slice types with some generic type APISlice[T] [T] where we define a custom json marshaler that never marshals to nil. Unfortunately, that's a very invasive change. I definitely don't want that type to leak into the rest of the system.
3. Find every case where we return nil in the API and make sure we return an actual empty slice.
4. Use a new serialization library.
5. Do some reflection magic in our API to replace nil slices. This'll likely have a fair amount of overhead.