KeyError on recent background updates

[matrixbot]

This issue has been migrated from #16054.

---

Description

We're seeing a KeyError on the recent updates introduced by matrix-org/synapse@11e15d7:

KeyError: 'current_state_events_stream_ordering_idx'
  File "synapse/storage/background_updates.py", line 405, in run_background_updates
    result = await self.do_next_background_update(sleep)
  File "synapse/storage/background_updates.py", line 528, in do_next_background_update
    update_info = self._background_update_handlers[self._current_background_update]

Specifically appears to be missing in BackgroundUpdates._background_update_handlers, but adding some logging indicates it is being registered as a background update:

{"message":"Registering background index update: current_state_events_stream_ordering_idx","namespace":"synapse.storage.background_updates","level":"info","time":1690998470.67,"request":"main","server_name":"beeper.com"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.9,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.98,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}

If I run locally with sqlite there is no error, but I can't see any codepath that would change the behaviour for postgres 🤔

[schildbach]

This database corruption issue could be a root cause: matrix-org/synapse#13476