Network Architecture

[jayakrishnanmm]

What is the network architecture behind Speechmatics ASR ? Is it encoder-decoder based ?
Any relevant research papers for reading ?

[jamied157]

Thanks for your question! I can't give too much detail as that's a core part of our product but suffice to say we a few components that cover a few different architectures (CNNs, Transformers, RNNs).

A particular focus we've had over the last few years has been on Self-Supervised learning and scaling that up with a lot of diverse data:
https://www.speechmatics.com/company/articles-and-news/boosting-sample-efficiency-through-self-supervised-learning

Hope this helps, if you're interested in papers on self-supervised learning, then I might be able to point you to some interesting ones!

[jayakrishnanmm]

Thank you @jamied157 . Yes, pls point me to some SSL papers.

[jamied157]

Sure! Here are a few:

• Wav2Vec 2.0: https://arxiv.org/abs/2006.11477
• BEST-RQ (which google use in their universal speech model) https://proceedings.mlr.press/v162/chiu22a
• WavLM https://arxiv.org/abs/2110.13900

There's a bunch of interesting work here and it all is related and builds on one other. Are you trying to train your own speech recognition system?

[jayakrishnanmm]

Thank you for sharing the pointers. I am a speech recognition researcher having experience in building ASR systems using DNN-HMM (Kaldi) and E2E technologies. Recently I benchmarked Whisper ASR , and in the process happened to see Speechmatics also, which gave surprising results on some test sets. So was just curious about what goes behind this performance. I am interested to know about various n/w architectures and training methods that can give best performance in ASR. Thanks again for your links on SSL, which I am going to explore next. Do you know any open source implementations for this ?

[jamied157]

Wav2Vec(2) is in the fairseq repo: https://github.com/facebookresearch/fairseq/tree/main
WavLM is here: https://github.com/microsoft/unilm/tree/master/wavlm

I can't see any implementations of BEST-RQ but there might be some about.

Great to hear that our results were good! Out of interest, what languages were you testing on? Are you just interested in doing research or do you want to use ASR as part of some application?