How to reload IDataProtection keys? Please help 😞

[dario-l]

We are using IDataProtection to protect our confidential data at rest in database. Our application (NET 8) is running on two machines on Linux. When the first instance starts first time it is creating new key. A moment later the second instance starts and create own key too. We use PersistKeysToDbContext<SecurityDbContext>. When SecurityDbContext OnSavedChanges occurs then app sends notification to other app to reload key ring.

Reloading method is like this:

public void ReloadKeys()
{
    lock(_locker)
    {
        // Reset cache
        CancellationTokenSource.CreateLinkedTokenSource(_keyManager.GetCacheExpirationToken()).Cancel();
        
        // Reload keys
        var keys = _keyManager.GetAllKeys();
        
        // This should reset key ring cache - at least that's what we think
        CancellationTokenSource.CreateLinkedTokenSource(_keyManager.GetCacheExpirationToken()).Cancel();
        
        using var _ = _logger.BeginScope(new { ReloadedKeys = string.Join(", ", keys.Select(x => x.KeyId.ToString())) });
        _logger.LogInformation("Reload unknown Data Protection keys");
    }
}

To Unprotect we use the following method:

for (var i = 1; i <= maxRetryCount; i++)
{
    try
    {
        return _protector.Unprotect(value);
    }
    catch (CryptographicException)
    {
        ReloadKeys();

        if (i >= maxRetryCount)
        {
            throw;
        }
    }
}

In both cases keys are not reloaded. In logs we see that ReloadedKeys contains all keys created by the first and second instance of our application but Unprotect still throws CryptographicException with message that key not found even though the GetAllKeys method shows that the given key is loaded.

Maybe our reloading code is wrong. So,,,,

How to properly reload keys so that the key ring can see and use them?

Please help...

PS
Restarting instance (which can not see reloaded keys) helps but that is not a solution at all. 😞

[dario-l]

@amcasey Please, help 🆘

[amcasey]

I haven't seen this pattern before and would be surprised to learn it worked:

CancellationTokenSource.CreateLinkedTokenSource(_keyManager.GetCacheExpirationToken()).Cancel();

Generally speaking, you need to control a CancellationTokenSource in order to be able to cancel a CancellationToken.

You should be able to validate in a toy console app (or just in the debugger in the real app).

Having said that, resetting the cancellation token does seem like it could work - the next Protect or Unprotect call should refresh the IKeyRingProvider. One way to forcibly invalidate the cache is to create a new key. You could also try revoking all keys before a given time and pass a time well in the past (e.g. Jan 1, 1900).

It's definitely true that startup races are a problem for Data Protection. We've made some improvements in 9.0, but there's no perfect solution. The fundamental problem is that a given app instance doesn't know how many app instances there are, so it has no way to know whether consensus (i.e. about which is the default key) has been achieved.

There were already some mitigations in place in 8.0, the main one being a grace period during which unknown keys would trigger a refresh. In practice, that retry seems to be pretty effective.

Another approach you could take would be to start one instance first, let it run until it has created and published a data protection key, and then start the other instances. Obviously, this isn't always possible.

[dario-l]

@amcasey Thank you so much for help.

We will try to use tips from you.

Additionally we will try to invoke private method TriggerAndResetCacheExpirationToken hoping that all those solutions will work together 😄

[dario-l]

My (propably unsafe) method for refreshing cached key ring which is working 😃
At least for NET8 😄

When Protect/Unprotect throws CryptographicException (which most of the time is because key not found) then ReloadKeys is invoked.

public void ReloadKeys()
{
    try
    {
        lock(_locker)
        {
            // NET8
            var method = typeof(XmlKeyManager).GetTypeInfo().GetDeclaredMethod("TriggerAndResetCacheExpirationToken");

            // Reset internal cache
            method?.Invoke(_keyManager, new object?[] { null, null });

            // Reload keys
            var keys = _keyManager.GetAllKeys();

            // Reset internal cache
            method?.Invoke(_keyManager, new object?[] { null, null });

            using var _ = _logger.BeginScope(new { ReloadedKeys = string.Join(", ", keys.Select(x => x.KeyId.ToString())) });
            _logger.LogInformation("Reload unknown Data Protection keys");
        }
    }
    catch (Exception ex)
    {
        _logger.LogWarning(ex, "Failed to reload Data Protection keys");
    }
}

[amcasey]

Why do you need to reset the cache token twice?

[dario-l]

Simple. I do not know which moment is valid: before or after reading keys.

The most important for me: it works 🙂👍