Add initial docs/MessagePassing.md

docs/MessagePassing.md

@@ -0,0 +1,85 @@
+# How Objects Come Home
+
+`snmalloc` returns freed objects back to the front-end allocator whence they came.
+Each allocator has two message structures to consider:
+
+1. An _inbox_, called its message queue,
+   a singly-linked list of (batches of) objects,
+   which contains messages that (might be) for this allocator.
+
+2. An _outbox_, a hash table of singly linked lists of (batches of) objects,
+   keyed on the identity of the allocator responsible for those objects' slabs.
+
+Let us follow the path of an object from `dealloc` back to its home slab's free list.
+Of particular interest is the staged repurposing of the linkage pointers within free objects.
+
+1. `LocalAlloc::dealloc` probes the page map and detects that
+   the `RemoteAllocator` owning the home slab is not the current `LocalAlloc`.
+
+2. `LocalAlloc::dealloc` invokes `RemoteDeallocCache::reserve_space`,
+   which checks whether the current allocator's `RemoteDeallocCache` is over quota.
+
+3. Regardless of the outcome, we call `RemoteDeallocCache::dealloc`,
+   either in `LocalAlloc::dealloc` itself or in `LocalAlloc::dealloc_remote_slow`.
+
+   At this point, the object is converted to a `freelist::Object::T<>`,
+   as if it were about to be inserted into a slab freelist.
+   Within this type,
+   the top word will hold a forward pointer to another free object in the slab,
+   and, if mitigations are enabled,
+   the next word will hold an obfuscated backward address for consistency checks.
+
+4. If free object batching is enabled, the identity of the remote slab
+   (specifically, the address of its associated SlabMetadata structure)
+   will be used to hunt for an "open ring" of free objects in the same slab
+   within this `RemoteDeallocCache`.
+   If one is found, this new object is attached
+   and `RemoteDeallocCache::dealloc` is done.
+
+   Otherwise, one will be closed and flushed by being `forward`-ed
+   to the outbox list associated with the objects' slab's allocator.
+   In so doing, the first object in the ring is converted to a `RemoteMessage`,
+   which (approximately) preserves the `freelist::Object::T<>` nature
+   of the first two words
+   and adds another `freelist::Object::T<>` linkage (pair) afterwards.
+   The first linkage pair is in fact abused
+   to store both a _relative_ pointer to the next object
+   and the number of messages in the ring.
+   The (newly coerced) second pair is used to thread this object
+   onto the outbox chain (and, ultimately, to the recipient allocator's inbox).
+
+5. Assuming, for exposition, that we would now be over quota,
+   we enter `LocalAlloc::dealloc_remote_slow` which calls `RemoteDeallocCache::post` 
+   (indirectly, via a few wrappers).
+   This latter function is responsible for
+
+   1. closing all open rings (if batching is enabled) and
+   2. calling `RemoteAllocator::enqueue`,
+      moving each list to the message queue of the allocator at the front.
+
+   In this distributing, the list that would be associated with the current allocator
+   is instead sharded out to other lists and distribution is repeated.
+   (However, the current allocator will never be the owner of any message in this list;
+   the repeated distribution is to ensure that objects return home in finite steps.)
+   The sharding is done in such a way as to guarantee termination of this loop.
+
+   After `RemoteAllocator::enqueue`,
+   the `freelist::Object::T<>` linkages are used by the `RemoteAllocator` message queue.
+
+6. Eventually, a remote allocator invokes `CoreAlloc::handle_message_queue`,
+   which chains to `CoreAlloc::handle_message_queue_inner` when incoming messages exist.
+   This invokes `CoreAlloc::handle_dealloc_remote` for each message in the queue.
+
+7. `CoreAlloc::handle_dealloc_remote` replicates the test performed in `LocalAlloc::dealloc`
+   to determine whether an object is owned by the invoking allocator or another.
+   In the former case, the object is returned to the appropriate slab;
+   in the latter case, it is pushed into the invoking allocator's remote cache,
+   akin to the handling in `LocalAlloc::dealloc_remote_slow`
+   (with the slight tweak that draining the local cache, if necessary,
+   will be deferred until the entire message queue has been processed).
+
+8. After potentially many hops between allocators,
+   the object will arrive home and will be returned to its slab.
+
+   When that happens, the `freelist::Object::T<>` linkages are used by the slab
+   for its free list(s).