User defined kind handle to upgrade durable collection shapes

[mhofman]

What is the Problem Being Solved?

Originally virtual collections had no customizable behavior, so when we introduced durability, durable collections used a kind handle internally defined by liveslots.
Then we introduced keyShape and valueShape to virtual collections. However because there are no user handles to manage or redefined these durable collections during upgrade, the user code has no way to change these shapes and upgrade the collection, unlike for a durable object's state (see #7407).

In #7337 (comment) it was brought that an upgradability mechanism would apply to durable collections as well, but without an admin handle to the collection, there is no path for the user code to exercise it.

Description of the Design

Allow user land to create a durable collection providing it own kind handle.
On upgrade these durable collection instances would have to be explicitly redefined during buildRootObject.
We may want to enforce that usage of collection shapes requires an explicit kind handle.
Unlike values which can be lazy migrated, key shapes might require a "compatibility" check to enforce that the new shape is a superset of the previous shape (similar to the original description of #7337

Security Considerations

None?

Scaling Considerations

How do we transition current durable collections with implicit kind handles and shapes defined? It may be possible to iterate and move regular collections, but not weak ones. Maybe we could introduce a wrapper that defines a new collection, and wraps access so that it checks the old collection instance if the entry isn't found in the new collection, and lazily performs migration on access?

Test Plan

TBD

[warner]

Security Considerations

Yeah the whole question of how we express the authority to upgrade a durable collection is a security consideration :)

The short answer would be that we don't allow upgrade of pre-existing collections at all. Only collections created with a new API (e.g. makeScalarBigMapStore(label, { collectionHandle })) would be eligible for upgrade. collectionHandle would replace the old durable: true option, and could come with { version, upgrader } properties for subsequent upgrade.

But that might make owners of existing code pretty sad.

[mhofman]

The only thing I can think of to make these collections upgradable is to endow a power to the vat that during buildRootObject lets you recover the implicit handle of the initial collection. Once upgraded, they are no longer implicit, and their handle cannot be recovered through this mechanism.

My initial idea was to wrap the original collection and lazily upgrade, but really that's inefficient because there is no way to know when all elements have been upgraded, so any collection read will require a second lookup if the entry is missing in the wrapper.