SubscribeForNotifications() gives out-of-bounds changes indexes

[jrlaff]

Hi,

Using Realm's CollectionExtensions.SubscribeForNotifications() to watch for changes in a query results collection, I often receive a change notification with a changes ChangeSet that contains indexes that are out of the bounds of the collection. It happens quite regularly, but I haven't been able to pinpoint a precise scenario that reproduces the issue.

More precisely, changes contains, for example, 5 NewModifiedIndices (0, 1, 2, 3, 4) but sender only contains 4 items. It does not contain any other index (InsertedIndices and DeletedIndices are empty).

No need to say that accessing the sender using those indices triggers an exception.

How am I supposed to handle this?

Any help appreciated. Thanks.

[papafe]

Hi @jrlaff, this sounds like a bug. I understand that you didn't manage to find precisely when it happens, but if you manage to do so it will be really helpful for us to understand what is going on. Even if it does not happen all the time, having a project with which we can reproduce the error will definitely help.

[nirinchev]

Do you get the out of bound indexes before or after you open the transaction? I.e. if you check the size of the collection before you call realm.Write, is it less than the larger index in the collection?

Generally speaking, it's not recommended to open transactions inside a notification callback because that may result in unexpected behavior. To try and explain the issue, every time a Realm file changes, an internal version is created and notifications are emitted by calculating the changes between the previous and the current version. Calculation of changes is done in the background though, so it's possible to have more updates that occur while we're delivering notifications. The notification delivery mechanism ensures that within the notification callback you're seeing the version that we've calculated notifications for. Let's walk this through step by step:

1. Your realm is at version 1.
2. You make some changes bringing it to version 2.
3. Notifications are calculated for the delta 1 -> 2 and the notification callback is emitted.
4. Even if in the background, a write happened bringing the Realm to version 3, the sender in the callback is pinned at version 2 to make sure you won't see nonsensical data, such as getting modification notifications for deleted objects.

Now, if you do a realm.Write inside the notification callback, you're forcing the collection to advance to the latest version as you can't write to an outdated version. You're also forcing notifications to be delivered for the changes between 2 and 3. So what happens when you do realm.Write in the callback is:

1. Another callback will be invoked before realm.Write returns with the changes from 2 to 3, which may do its own Write and trigger yet another callback if there were background changes.
2. When you go back to the original callback, sender will now be at version 3, but the changes parameter will describe the changes in version 2. This means that if the collection was further modified in version 3, you're bound to see unexpected results, such as indexes being out of range.

If you absolutely must use write transactions inside a callback, the recommendation would be to preprocess all the values that you need before opening a write transaction. It's hard to give you an example without knowing your use case or seeing your code, but something like:

query.SubscribeForNotifications((sender, changes, error) =>
{
    // bad, don't do this
    foreach (var index in changes.ModifiedIndices)
    {
        var obj = sender[index];
        realm.Write(() =>
        {
            obj.Foo = "bar";
        });
    }

    // Instead do this
    // Get all objects ahead of time to avoid advancing the collection to the next version
    var modifiedObjs = changes.ModifiedIndices.Select(i => sender[i]).ToArray();

    // Do only one write at the end of the notification block
    realm.Write(() =>
    {
        foreach (var obj in modifiedObjs)
        {
            // Check the object hasn't been deleted when we advanced the version
            if (!obj.IsValid)
            {
                continue;
            }

            // Don't modify an object if it already has the correct value. Otherwise
            // you risk creating an endless feedback loop.
            if (obj.Foo != "bar")
            {
                obj.Foo = "bar";
            }
        }
    });
});

[jrlaff]

Thanks @nirinchev, your answer definitely helps getting a sharper picture of how it works.

I indeed changed my callback body to preprocess the changes and get the objects in an array. This got me rid of the exceptions I had. I also opened one single transaction to process the sub-changes I have to do, after reading the guidelines that encourage to open a single transaction around a loop instead of multiple smaller transactions.

One more question regarding my need to store previous versions of objects so that I can compare what has changed in each object and find out which object got deleted: is there any standard way to get a copy of a realm object that is detached, and thus can be safely stored for later use (that use being comparing to the next version to see what has changed)? I have so far implemented dummy field-by-field copy methods, but that looks so wrong and I don't really know what to do with nested objects.

Thank you very much.

[nirinchev]

Generally, copying things field-by-field is the best way to go about it as it gives you the most flexibility in terms of what you need to copy. There are some amazing 3rd party libraries, such as Automapper that automate a lot the boilerplate code there. For example, if you know that you only care about 2-3 fields, you can configure automapper to produce detached objects with only these fields populated.

Alternatively, you can freeze an object using the .Freeze() extension method. It will keep a reference to an object at a particular version and you can read it even after the original object has been deleted. One caveat here is that you have to be careful not to keep those references forever, otherwise the filesize of the Realm may grow significantly. This is caused by the fact Realm will currently keep all intermediate versions between the oldest pinned (frozen) version and the current one. So if you pinned an object at version 1, then did 1000 transactions alternatingly changing a boolean property, Realm would need to keep 1000 versions around, even though the end state is the same as the frozen state. This is something we're looking to optimize, but don't have a timeframe for it yet.

[jrlaff]

Thank you @nirinchev. I am looking at Automapper and it seems very handy.

I would have marked your messages as answer but it can only be done on direct replies to the main thread...

[nirinchev]

No worries, the important part is that we solve your problem 😄