Incorrect push gateway /notify unread counts

[stefanceriu]

We've been receiving more and more reports that application badges are incorrect, showing increasingly larger numbers no matter what actions the users take.

I have debugged this on my own device and can confirm that the count is coming in the push notification payload from the backend. I tried manually marking all rooms as read, leaving rooms, new sessions and everything else I could think of, nothing budges it downwards.

Most recent conversation around this happened starting here where Ben found that

I've got a 5 stuck on my EI and looking at event_push_summary I've got 5 rows for 5 different rooms with notif_count=1 (no rows in event_push_actions for those 5 room ids). If I'm driving the admin api right, I'm not in 4 of those rooms....

We known that correctly computing unread counts in an E2EE environment is not fully possible backend side and plan on tackling that client side but in the meantime whatever counts we do have should work predictably.

Main tracking ticket Element X side: element-hq/element-x-ios#2066

Other potentially related tickets:

• Room notification count not always decreasing when redacting messages #15698
• Room sync: the notifications count is not updated when a push rule is created at room level #1115

[benbz]

Most recent conversation around this happened starting here where Ben found that

I've got a 5 stuck on my EI and looking at event_push_summary I've got 5 rows for 5 different rooms with notif_count=1 (no rows in event_push_actions for those 5 room ids). If I'm driving the admin api right, I'm not in 4 of those rooms....

The 4 rooms I was not in appeared to be rejected invites. I could not reproduce a stuck badge on EI by rejecting test invites from other accounts I control, though I still gained rows in event_push_summary with notif_count=1 for those rejected invites. I didn't get further as to investigating what was different between my test rejected invites and the prior 4 that did contribute towards a stuck badge

[clokep]

If we think this is new, maybe a regression from #16756?

[ara4n]

it could well be new - people really starting complaining most about it over the last month or so.

[jacotec]

Same here, all users are massively complaining about totally wrong unread badges, me included. Definitively related to updating Synapse from .98 to .101.

Also the wrong batch is synced to all devices which points also to the server side.

Hope this will be fixed more sooner than later, it's pretty bad to not see if there are new messages anymore.

[wrjlewis]

Related PR

[morguldir]

Can confirm it’s new, also easily reproducable with invites, each one bumps the notification count by one, even if you accept/decline/leave.

It’s happening in Fluffychat iOS as well, and i think it’s more noticable on iOS since the client can’t just ignore the server counts

[jacotec]

@wrjlewis Do you think the PR you've mentioned will fix it?

[wrjlewis]

@wrjlewis Do you think the PR you've mentioned will fix it?

Sorry no, I've striked out that comment, we haven't fixed this yet, but investigating

[jacotec]

@wrjlewis Do you think the PR you've mentioned will fix it?

Sorry no, I've striked out that comment, we haven't fixed this yet, but investigating

Oh, what a pity :-(

Was there a database update, or can I go back to 1.098 without issues? The impact of this bug is unfortunately massive.

[wrjlewis]

@wrjlewis Do you think the PR you've mentioned will fix it?

Sorry no, I've striked out that comment, we haven't fixed this yet, but investigating

Oh, what a pity :-(

Was there a database update, or can I go back to 1.098 without issues? The impact of this bug is unfortunately massive.

There hasn't been any schema migrations since v1.96.0 so you can roll back safely 👍.

For future reference, you can check here to see which versions are compatible when rolling back.

We're continuing to investigate the badge count issue and want to fix this asap

[anoadragon453]

#16954 should help with @benbz's case of having notifications for rooms that he's left. However @morguldir reported in TWIM that they're still seeing more notification counts than expected:

Haven’t tried the PR, but the count goes up for me even if i accept the invite, and it goes up by 2 when i first receive it 🤔

[qbit]

#16954 should help with @benbz's case of having notifications for rooms that he's left. However @morguldir reported in TWIM that they're still seeing more notification counts than expected:

Haven’t tried the PR, but the count goes up for me even if i accept the invite, and it goes up by 2 when i first receive it 🤔

This cleared a number of stuck notifications for me \o/. Still have one coming from something..

[morguldir]

went to develop on my own HS to try #16954 and it went down to 167 -> 145 on main account and 46 -> 27 on my personal account.

Also, applying v1.99.0 moved it from 145 -> 7 and 27 -> 16 🤔

[anoadragon453]

#16954 is a fix that partially helps the issue that we've identified (with @morguldir's help!) that is caused by #16756 (included since v1.100.0). That fix is currently on release-v1.102 and went out in v1.102.0rc1. That fix is only present on develop currently, and was not included in v1.102.0rc1.

Given that #16954 is only a partial fix, and users are still seeing buggy notification counts, I think it best to revert #16756 (and its performance gains), as well as the partial fix, on the release-v1.102 branch. This would give us working notifications again for v1.102.0.

We can aim to reintroduce #16756 in a later release once the known bugs have been worked out.

[jacotec]

Sounds like a plan! Downgrading to 1.98 fixed it immediately for me (this issue was a real showstopper for me and my users), I'm happy to get back on track once this is solved.

[itsthejb]

Hi all,

Have also been seeing this issue, so also reverted to matrixdotorg/synapse:v1.98.0. Works just fine. I'll wait out the full fix and then go back to latest

[anoadragon453]

Synapse v1.102.0 has been released with #16756 reverted. Notification counts should be fixed.

Thanks to everyone who helped track down the issue! We'll have a fixed version of #16756 out in a later release.