ITN runtime. #2001

duj12 · 2023-09-07T12:29:42Z

The fst-based ITN model is compiled from WeTextProcessing,
I made a version of mine, which solve some known corner cases. You can download it from ModelScope

For now, decoder_main and api_main(Both with libtorch backend) is validated after ITN is added.
Other runtime may still have some issue.

For decoder_main, the running command and result is like:

dujing@dujing-pc:/mnt/d/work/code/wenet-itn/runtime/libtorch$ build_debug/bin/decoder_main --model_path ../resource/ASR/final.zip --unit_path ../resource/ASR/units.txt --wav_path ../resource/WAV/number.wav --fst_path ../resource/ASR/TLG.fst --dict_path ../resource/ASR/words.txt --itn_model_dir ../resource/ASR
test 2016年03月08日2018年05月09日呃132 个1358

For api_main, the running command and result is like:

dujing@dujing-pc:/mnt/d/work/code/wenet-itn/runtime/libtorch$ build_debug/bin/api_main --model_dir ../resource/ASR --wav_path ../resource/WAV/number.wav
I0907 20:08:05.275250  8845 wenet_api.cc:48] Reading torch model
I0907 20:08:05.299885  8845 torch_asr_model.cc:35] Num intra-op threads: 1
I0907 20:08:06.364199  8845 torch_asr_model.cc:73] Torch Model Info:
I0907 20:08:06.364248  8845 torch_asr_model.cc:74]      subsampling_rate 4
I0907 20:08:06.364253  8845 torch_asr_model.cc:75]      right context 6
I0907 20:08:06.364255  8845 torch_asr_model.cc:76]      sos 11999
I0907 20:08:06.364257  8845 torch_asr_model.cc:77]      eos 11999
I0907 20:08:06.364260  8845 torch_asr_model.cc:78]      is bidirectional decoder 1
I0907 20:08:06.387041  8845 fst.h:820] FstImpl::ReadHeader: source: ../resource/ASR/TLG.fst, fst_type: vector, arc_type: standard, version: 2, flags: 0
I0907 20:08:12.140612  8845 wenet_api.cc:98] Reading ITN fst
I0907 20:08:12.142271  8845 fst.h:820] FstImpl::ReadHeader: source: ../resource/ASR/zh_itn_tagger.fst, fst_type: vector, arc_type: standard, version: 2, flags: 0
I0907 20:08:12.163174  8845 fst.h:820] FstImpl::ReadHeader: source: ../resource/ASR/zh_itn_verbalizer.fst, fst_type: vector, arc_type: standard, version: 2, flags: 0
I0907 20:08:12.235846  8845 asr_decoder.cc:104] Required 67 get 67
I0907 20:08:12.935179  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:16.780755  8845 asr_decoder.cc:200] Partial CTC result 二零一
I0907 20:08:16.780877  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:24.570278  8845 asr_decoder.cc:200] Partial CTC result 二零一六年
I0907 20:08:24.570405  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.565124  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月
I0907 20:08:29.565235  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.638185  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日
I0907 20:08:29.638310  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.677501  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日
I0907 20:08:29.677613  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.714907  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年
I0907 20:08:29.715016  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.738862  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年
I0907 20:08:29.738973  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.834138  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年五月九
I0907 20:08:29.834255  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.867908  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年五月九日
I0907 20:08:29.868029  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.898561  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年五月九日
I0907 20:08:29.898604  8845 ctc_endpoint.cc:42] Endpointing rule rule2 activated: true,1160,7040
I0907 20:08:29.898610  8845 asr_decoder.cc:125] Endpoint is detected at 707
I0907 20:08:29.898674  8845 asr_decoder.cc:104] Required 64 get 64
I0907 20:08:29.939894  8845 asr_decoder.cc:200] Partial CTC result 二零一六年三月八日二零一八年五月九日
...
...

I0907 20:08:35.052362  8845 queue.h:633] AutoQueue: using LIFO discipline
I0907 20:08:35.052731  8845 asr_decoder.cc:200] Partial CTC result 2016年03月08日2018年05月09日132 个1358
I0907 20:08:35.905037  8845 asr_decoder.cc:84] Rescoring cost latency: 1341ms.
I0907 20:08:35.905140  8845 api_main.cc:42] 1 {
  "nbest" : [{
      "sentence" : "2016年03月08日2018年05月09日呃132 个1358 "
    }],
  "type" : "final_result"
}

In this ITN runtime code, It's worth noting that the version of openfst is upgraded from 1.6.5 to 1.7.2.
So when building TLG.fst, you'd better use the same version of openfst(At least for now, the official verison of openfst in KALDI is 1.7.2(https://github.com/kaldi-asr/kaldi/blob/master/tools/Makefile#L10)).
And make sure the version of pynini in WeTextProcessing is proper(For now, the itn model compile from WeTextProcessing==0.1.1 can run successfully.)
On this version, the N-gram Language model(I have not verify the TLG.fst released by WeNet yet, but I suppose it will work) and ITN model can both work.

robin1001 · 2023-09-08T01:22:30Z

Great job! Please fix the lint problem.

xingchensong · 2023-09-08T01:47:28Z

感谢！晚会看一下

runtime/core/api/wenet_api.cc

runtime/core/bin/CMakeLists.txt

runtime/core/decoder/asr_decoder.h

runtime/core/post_processor/post_processor.cc

runtime/core/post_processor/post_processor.h

xingchensong · 2023-09-08T10:40:04Z

我个人建议，能不能用sub project的方式把wetext的runtime部分引入进来呢？现在相当于把wetext的runtime的代码copy过来了，如果wetext有修改runtime，这边还得同步，@robin1001 觉得呢

xingchensong · 2023-09-08T10:42:45Z

把wetext当成一个subproject引入可以参考我之前的实现： main...xcsong-wetextprocessing
当时没合入主要是规则还不稳定，以及openfst的版本冲突等原因

robin1001 · 2023-09-08T11:19:38Z

我个人建议，能不能用sub project的方式把wetext的runtime部分引入进来呢？现在相当于把wetext的runtime的代码copy过来了，如果wetext有修改runtime，这边还得同步，@robin1001 觉得呢

对，更建议用这种方式。以第三方库的形式集成进来。

robin1001 · 2023-09-08T11:24:28Z

openfst 我们可以直接升级，这块应该没有什么风险。

… both have utils and utils/string.h

duj12 · 2023-09-13T10:18:39Z

把wetext当成一个subproject引入可以参考我之前的实现： main...xcsong-wetextprocessing 当时没合入主要是规则还不稳定，以及openfst的版本冲突等原因

今天试了一下，目前以第三方库引入之后，构建过程中，由于wenet和wetext都包含一个utils/string.h，且在引用这个头文件时都是用了“#include utils/string.h"去导入，导致两个库同时构建时，就会出现其中一个库找不到其名称空间下的utils/string.h中的方法的问题（https://github.com/duj12/wenet/blob/itn-wetext-sub/runtime/core/cmake/wetextprocessing.cmake#L25 这里会遇到构建.o时找不到wetext中的string.h的问题）。这是个fatal的issue，不改动wenet或wetext源码就无法解决。
另外就是由于utils在两个库中都是各自的子模块，导致构建时，必须有一个utils要单独拉出来构建成一个新的其他名字的lib，这样构建就会很不方便且优雅(https://github.com/duj12/wenet/blob/itn-wetext-sub/runtime/core/cmake/wetextprocessing.cmake#L10)。

关于这个问题，你和震东可以想想有啥好的办法解决，最简单的办法就是把wetext中的utils/string.h改个名字。

duj12 · 2023-09-13T10:24:24Z

我个人建议，能不能用sub project的方式把wetext的runtime部分引入进来呢？现在相当于把wetext的runtime的代码copy过来了，如果wetext有修改runtime，这边还得同步，@robin1001 觉得呢

@xingchensong 目前这个部分的代码是我之前很早就拷贝的，有些函数方法名称还没有同步到最新的大写字母开头的那个版本上。然后命名空间改了一下，直接复用了utils模块，同时改了一下代码以便通过cpp-lint和clang-format的检测。

robin1001 · 2023-09-14T01:39:23Z

@xingchensong 帮忙看一下

xingchensong · 2023-09-15T10:11:58Z

sorry，miss掉了才看到

xingchensong · 2023-09-16T03:39:09Z

wetext 已完成文件更名：wenet-e2e/WeTextProcessing#111

use wetext subproject.

duj12 · 2023-09-16T09:29:42Z

wetext 已完成文件更名：wenet-e2e/WeTextProcessing#111

@xingchensong subproject的方式集成已经完成了，现在的编译方式就是，只要ITN选项打开，就会直接把wetext_processor编到wenet库里面。没有加额外的宏。
然后decoder_main中使用时，输入参数中如果加上ITN模型路径且两个特定fst文件存在，结果就会执行ITN。
api中使用时，如果模型路径下面包含了两个特定名称的fst文件，结果就会执行ITN。

xingchensong · 2023-09-18T00:17:22Z

great ，感谢!!!

xingchensong · 2023-10-11T09:37:33Z

runtime/core/decoder/asr_decoder.h

@@ -140,7 +140,7 @@ class AsrDecoder {
  std::shared_ptr<PostProcessor> post_processor_;
  std::shared_ptr<ContextGraph> context_graph_;

-  std::shared_ptr<fst::Fst<fst::StdArc>> fst_ = nullptr;
+  std::shared_ptr<fst::VectorFst<fst::StdArc>> fst_ = nullptr;
  // output symbol table
  std::shared_ptr<fst::SymbolTable> symbol_table_;
  // e2e unit symbol table


update: 原来1.6.5的openfst使用fst::Fst读取，为了wetext更新到1.7.2的openfst后，需要改成fst::VectorFst

潜在问题纪录：recipe中的make_tlg.sh中构图所使用的fstcompile，不能是wenet自带的 https://github.com/kkm000/openfst/archive/refs/tags/win/1.7.2.1.tar.gz （为了适配windows做了一些修改），而需要官方原版openfst，http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.7.2.tar.gz

该问题仅针对构图过程存在，构图使用原版1.7.2, 推理时使用kkm000版1.7.2，是可以正常推理的

TODO: 需要在recipe中添加相关的提醒 OR 用patch方式修复一下kkm000的openfst？

@duj12 @pengzhendong @robin1001 any ideas?

原版和 kkm 区别是？打什么样的 patch？

fstcompile的时候会报这个错。

google了一下找到一个解释 wincentbalin/compile-static-openfst#1

原本的策略是linux和win用不同的openfst（win用kkm，linux用mjansche）后来在这个pr 里，统一用kkm了b59ca15

Potential problem log: fstcompile used in the composition of the make_tlg.sh in the recipe cannot be the https://github.com/kkm000/openfst/archive/refs/tags/win/1.7.2.1.tar.gz that comes with Wenet (some modifications have been made to adapt to Windows), And you need the official original openfst, http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.7.2.tar.gz

This problem only exists for the composition process, the original version 1.7.2 is used for composition, and the kkm000 version 1.7.2 is used for inference, which can be reasoned normally

I tried to use http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.7.2.tar.gz, but it's not work, I had an error fst

maiphong0411 · 2024-03-05T04:16:42Z

and I had a warning

roney123 · 2024-03-20T06:51:28Z

@duj12 大佬，这样加ITN的话，word_pieces的字、时间戳和ITN之后的是不是不一致了

duj12 added 3 commits September 7, 2023 19:51

wenet_api and decoder_main support ITN.

c6a5f71

remove resource dir

e4849ce

turn websocket cmake on

e0b10a3

robin1001 requested a review from xingchensong September 8, 2023 01:22

duj12 added 4 commits September 8, 2023 11:54

fix cpplint

019499f

fix clang-format

c7fa8d4

fix clang-format

719ba9f

fix clang-format

a36f0e8

xingchensong reviewed Sep 8, 2023

View reviewed changes

dujing added 2 commits September 13, 2023 12:03

fix CRLF.

de7872b

try to add wetext as subproject, but failed. Because wetext and wenet…

3364b11

… both have utils and utils/string.h

xingchensong mentioned this pull request Sep 15, 2023

feat(runtime): support wenet-text-processing cmake build being a sub-… wenet-e2e/WeTextProcessing#111

Merged

dujing and others added 2 commits September 16, 2023 17:20

add wetext subproject, intergrate ITN.

30f6b2b

Merge pull request #1 from duj12/itn-wetext-sub

06d3dea

use wetext subproject.

dujing added 2 commits September 16, 2023 17:58

add fst file exist check.

60e87b0

include file.h in params.h

f27a7f9

xingchensong approved these changes Sep 18, 2023

View reviewed changes

xingchensong merged commit e8f451e into wenet-e2e:main Sep 18, 2023
6 checks passed

xingchensong mentioned this pull request Sep 29, 2023

Setup.py fatal error: 'processor/wetext_processor.h' file not found #2032

Closed

xingchensong reviewed Oct 11, 2023

View reviewed changes

xingchensong mentioned this pull request Nov 20, 2023

Compose结果为空 wenet-e2e/WeTextProcessing#164

Closed

xingchensong mentioned this pull request Dec 23, 2023

跑aishell例子加stage7的LM报错问题 #2254

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ITN runtime. #2001

ITN runtime. #2001

duj12 commented Sep 7, 2023 •

edited

Loading

robin1001 commented Sep 8, 2023

xingchensong commented Sep 8, 2023

xingchensong commented Sep 8, 2023

xingchensong commented Sep 8, 2023

robin1001 commented Sep 8, 2023

robin1001 commented Sep 8, 2023

duj12 commented Sep 13, 2023

duj12 commented Sep 13, 2023

robin1001 commented Sep 14, 2023

xingchensong commented Sep 15, 2023

xingchensong commented Sep 16, 2023

duj12 commented Sep 16, 2023 •

edited

Loading

xingchensong commented Sep 18, 2023

xingchensong Oct 11, 2023

xingchensong Oct 11, 2023 •

edited

Loading

xingchensong Oct 11, 2023

xingchensong Oct 11, 2023

robin1001 Oct 11, 2023

xingchensong Oct 12, 2023

xingchensong Oct 12, 2023 •

edited

Loading

maiphong0411 Mar 5, 2024 •

edited

Loading

maiphong0411 commented Mar 5, 2024

roney123 commented Mar 20, 2024

ITN runtime. #2001

ITN runtime. #2001

Conversation

duj12 commented Sep 7, 2023 • edited Loading

robin1001 commented Sep 8, 2023

xingchensong commented Sep 8, 2023

xingchensong commented Sep 8, 2023

xingchensong commented Sep 8, 2023

robin1001 commented Sep 8, 2023

robin1001 commented Sep 8, 2023

duj12 commented Sep 13, 2023

duj12 commented Sep 13, 2023

robin1001 commented Sep 14, 2023

xingchensong commented Sep 15, 2023

xingchensong commented Sep 16, 2023

duj12 commented Sep 16, 2023 • edited Loading

xingchensong commented Sep 18, 2023

xingchensong Oct 11, 2023

Choose a reason for hiding this comment

xingchensong Oct 11, 2023 • edited Loading

Choose a reason for hiding this comment

xingchensong Oct 11, 2023

Choose a reason for hiding this comment

xingchensong Oct 11, 2023

Choose a reason for hiding this comment

robin1001 Oct 11, 2023

Choose a reason for hiding this comment

xingchensong Oct 12, 2023

Choose a reason for hiding this comment

xingchensong Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

maiphong0411 Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

maiphong0411 commented Mar 5, 2024

roney123 commented Mar 20, 2024

duj12 commented Sep 7, 2023 •

edited

Loading

duj12 commented Sep 16, 2023 •

edited

Loading

xingchensong Oct 11, 2023 •

edited

Loading

xingchensong Oct 12, 2023 •

edited

Loading

maiphong0411 Mar 5, 2024 •

edited

Loading