求助：examples/aishell/asr0/ 的 run.sh 运行到 stage 5 时报错 #1170

dabaoyasu · 2021-12-20T02:41:12Z

dabaoyasu
Dec 20, 2021

请求帮助，examples/aishell/asr0/ 的 run.sh 运行到 stage 5 时报如下错误，会是什么原因？

(/users/xunhongsuhuanjie/deepspeech2/tools/venv) xunhongsuhuanjie@suntek-PR4768GW-SX:~/deepspeech2/PaddleSpeech/examples/aishell/asr0$ bash run.sh --stage 5 --stop_stage 5
checkpoint name deepspeech2

'[' 3 '!=' 3 ']'
++ echo 0
++ awk -F , '{print NF}'
ngpu=1
echo 'using 1 gpus...'
using 1 gpus...
config_path=conf/deepspeech2.yaml
jit_model_export_path=exp/deepspeech2/checkpoints/avg_1.jit
model_type=offline
bash local/download_lm_ch.sh
'[' 0 -ne 0 ']'
python3 -u /users/xunhongsuhuanjie/deepspeech2/PaddleSpeech/paddlespeech/s2t/exps/deepspeech2/bin/test_export.py --ngpu 1 --config conf/deepspeech2.yaml --result_file exp/deepspeech2/checkpoints/avg_1.jit.rsl --export_path exp/deepspeech2/checkpoints/avg_1.jit --model_type offline
----------- test_export.py Arguments -----------
benchmark_batch_size: None
benchmark_max_step: None
checkpoint_path: None
conf: None
config: conf/deepspeech2.yaml
dump_config: None
enable_auto_log: False
export_path: exp/deepspeech2/checkpoints/avg_1.jit
model_type: offline
ngpu: 1
opts: None
output: None
profiler_options: None
result_file: exp/deepspeech2/checkpoints/avg_1.jit.rsl
seed: None

model_type:offline
collator:
augmentation_config: conf/augmentation.json
batch_size: 64
delta_delta: False
dither: 1.0
feat_dim: None
keep_transcription_text: False
max_freq: None
mean_std_filepath: data/mean_std.json
n_fft: None
num_workers: 2
random_seed: 0
shuffle_method: batch_shuffle
sortagrad: True
spectrum_type: linear
spm_model_prefix: None
stride_ms: 10.0
target_dB: -20
target_sample_rate: 16000
unit_type: char
use_dB_normalization: True
vocab_filepath: data/lang_char/vocab.txt
window_ms: 20.0
data:
dev_manifest: data/manifest.dev
manifest:
max_input_len: 27.0
max_output_input_ratio: inf
max_output_len: inf
min_input_len: 0.0
min_output_input_ratio: 0.0
min_output_len: 0.0
test_manifest: data/manifest.test
train_manifest: data/manifest.train
decoding:
alpha: 1.9
batch_size: 128
beam_size: 300
beta: 5.0
cutoff_prob: 0.99
cutoff_top_n: 40
decoding_method: ctc_beam_search
error_rate_type: cer
lang_model_path: data/lm/zh_giga.no_cna_cmn.prune01244.klm
num_proc_bsearch: 10
model:
blank_id: 0
ctc_grad_norm_type: instance
num_conv_layers: 2
num_rnn_layers: 3
rnn_layer_size: 1024
share_rnn_weights: False
use_gru: True
training:
accum_grad: 1
checkpoint:
kbest_n: 50
latest_n: 5
global_grad_clip: 3.0
log_interval: 100
lr: 0.002
lr_decay: 0.83
n_epoch: 80
weight_decay: 1e-06
2021-12-20 10:36:23.131 | INFO | paddlespeech.s2t.utils.utility:all_version:45 - Deps Module Version:[('python', '3.7.11 (default, Jul 27 2021, 14:32:16) \n[GCC 7.5.0]'),
('paddle', '2.2.1'),
('paddle_commit', 'a5cf2e305b744e3ebd2f2210341f88d349d4ec5e'),
('soundfile', '0.10.3')]
2021-12-20 10:36:23.132 | INFO | paddlespeech.s2t.training.trainer:init:112 - Rank: 0/1
2021-12-20 10:36:23.132 | WARNING | paddlespeech.s2t.frontend.featurizer.text_featurizer:init:60 - TextFeaturizer: not have vocab file.
2021-12-20 10:36:23.300 | INFO | paddlespeech.s2t.frontend.augmentor.augmentation:init:123 - Augmentation: []
2021-12-20 10:36:23.327 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:228 - BLANK id: 0
W1220 10:36:23.333899 26061 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 10.2, Runtime API Version: 10.2
2021-12-20 10:36:23.328 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:229 - UNK id: 1
2021-12-20 10:36:23.328 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:230 - EOS id: 4300
2021-12-20 10:36:23.329 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:231 - SOS id: 4300
2021-12-20 10:36:23.329 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:232 - SPACE id: -1
2021-12-20 10:36:23.330 | INFO | paddlespeech.s2t.frontend.featurizer.text_featurizer:_load_vocabulary_from_file:233 - MASKCTC id: -1
2021-12-20 10:36:23.330 | INFO | paddlespeech.s2t.exps.deepspeech2.model:setup_dataloader:261 - Setup test Dataloader!
W1220 10:36:23.336942 26061 device_context.cc:465] device: 0, cuDNN Version: 8.2.
2021-12-20 10:36:25.104 | INFO | paddlespeech.s2t.modules.loss:init:41 - CTCLoss Loss reduction: sum, div-bs: True
2021-12-20 10:36:25.104 | INFO | paddlespeech.s2t.modules.loss:init:42 - CTCLoss Grad Norm Type: instance
2021-12-20 10:36:25.105 | INFO | paddlespeech.s2t.modules.loss:init:73 - CTCLoss() kwargs:{'norm_by_times': True}, not support: {'norm_by_batchsize': False, 'norm_by_total_logits_len': False}
2021-12-20 10:36:25.105 | INFO | paddlespeech.s2t.exps.deepspeech2.model:setup_model:172 - DeepSpeech2Model(
(encoder): CRNNEncoder(
(conv): ConvStack(
(conv_in): ConvBn(
(conv): Conv2D(1, 32, kernel_size=[41, 11], stride=[2, 3], padding=(20, 5), data_format=NCHW)
(bn): BatchNorm2D(num_features=32, momentum=0.9, epsilon=1e-05)
)
(conv_stack): LayerList(
(0): ConvBn(
(conv): Conv2D(32, 32, kernel_size=[21, 11], stride=[2, 1], padding=(10, 5), data_format=NCHW)
(bn): BatchNorm2D(num_features=32, momentum=0.9, epsilon=1e-05)
)
)
)
(rnn): RNNStack(
(rnn_stacks): LayerList(
(0): BiGRUWithBN(
(fw_fc): Linear(in_features=1312, out_features=3072, dtype=float32)
(fw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(bw_fc): Linear(in_features=1312, out_features=3072, dtype=float32)
(bw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(fw_cell): GRUCell()
(bw_cell): GRUCell()
(fw_rnn): RNN(
(cell): GRUCell()
)
(bw_rnn): RNN(
(cell): GRUCell()
)
)
(1): BiGRUWithBN(
(fw_fc): Linear(in_features=2048, out_features=3072, dtype=float32)
(fw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(bw_fc): Linear(in_features=2048, out_features=3072, dtype=float32)
(bw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(fw_cell): GRUCell()
(bw_cell): GRUCell()
(fw_rnn): RNN(
(cell): GRUCell()
)
(bw_rnn): RNN(
(cell): GRUCell()
)
)
(2): BiGRUWithBN(
(fw_fc): Linear(in_features=2048, out_features=3072, dtype=float32)
(fw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(bw_fc): Linear(in_features=2048, out_features=3072, dtype=float32)
(bw_bn): BatchNorm1D(num_features=3072, momentum=0.9, epsilon=1e-05, data_format=NLC)
(fw_cell): GRUCell()
(bw_cell): GRUCell()
(fw_rnn): RNN(
(cell): GRUCell()
)
(bw_rnn): RNN(
(cell): GRUCell()
)
)
)
)
)
(decoder): CTCDecoder(
(dropout): Dropout(p=0.0, axis=None, mode=upscale_in_train)
(ctc_lo): Linear(in_features=2048, out_features=4301, dtype=float32)
(criterion): CTCLoss(
(loss): CTCLoss()
)
)
)
2021-12-20 10:36:25.106 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_in.conv.weight | [32, 1, 41, 11] | 14432 | True
2021-12-20 10:36:25.106 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_in.bn.weight | [32] | 32 | True
2021-12-20 10:36:25.107 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_in.bn.bias | [32] | 32 | True
2021-12-20 10:36:25.107 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_in.bn._mean | [32] | 32 | False
2021-12-20 10:36:25.107 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_in.bn._variance | [32] | 32 | False
2021-12-20 10:36:25.108 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_stack.0.conv.weight | [32, 32, 21, 11] | 236544 | True
2021-12-20 10:36:25.108 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_stack.0.bn.weight | [32] | 32 | True
2021-12-20 10:36:25.108 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_stack.0.bn.bias | [32] | 32 | True
2021-12-20 10:36:25.109 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_stack.0.bn._mean | [32] | 32 | False
2021-12-20 10:36:25.109 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.conv.conv_stack.0.bn._variance | [32] | 32 | False
2021-12-20 10:36:25.109 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_fc.weight | [1312, 3072] | 4030464 | True
W1220 10:36:25.130453 26061 analysis_predictor.cc:798] The one-time configuration of analysis predictor failed, which may be due to native predictor called first and its configurations taken effect.
--- Running analysis [ir_graph_build_pass]
--- Running analysis [ir_graph_clean_pass]
--- Running analysis [ir_analysis_pass]
--- Running IR pass [is_test_pass]
--- Running IR pass [simplify_with_basic_ops_pass]
--- Running IR pass [conv_affine_channel_fuse_pass]
--- Running IR pass [conv_eltwiseadd_affine_channel_fuse_pass]
--- Running IR pass [conv_bn_fuse_pass]
I1220 10:36:25.349269 26061 fuse_pass_base.cc:57] --- detected 2 subgraphs
--- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
--- Running IR pass [embedding_eltwise_layernorm_fuse_pass]
--- Running IR pass [multihead_matmul_fuse_pass_v2]
--- Running IR pass [squeeze2_matmul_fuse_pass]
--- Running IR pass [reshape2_matmul_fuse_pass]
--- Running IR pass [flatten2_matmul_fuse_pass]
--- Running IR pass [map_matmul_v2_to_mul_pass]
I1220 10:36:25.361861 26061 fuse_pass_base.cc:57] --- detected 7 subgraphs
--- Running IR pass [map_matmul_v2_to_matmul_pass]
--- Running IR pass [map_matmul_to_mul_pass]
--- Running IR pass [fc_fuse_pass]
I1220 10:36:25.366539 26061 fuse_pass_base.cc:57] --- detected 1 subgraphs
--- Running IR pass [fc_elementwise_layernorm_fuse_pass]
--- Running IR pass [conv_elementwise_add_act_fuse_pass]
--- Running IR pass [conv_elementwise_add2_act_fuse_pass]
--- Running IR pass [conv_elementwise_add_fuse_pass]
--- Running IR pass [transpose_flatten_concat_fuse_pass]
--- Running IR pass [runtime_context_cache_pass]
--- Running analysis [ir_params_sync_among_devices_pass]
I1220 10:36:25.388357 26061 ir_params_sync_among_devices_pass.cc:45] Sync params from CPU to GPU
--- Running analysis [adjust_cudnn_workspace_size_pass]
--- Running analysis [inference_op_replace_pass]
--- Running analysis [memory_optimize_pass]
I1220 10:36:25.498596 26061 memory_optimize_pass.cc:216] Cluster name : cast_0.tmp_0 size: 4
I1220 10:36:25.498631 26061 memory_optimize_pass.cc:216] Cluster name : audio size: 644
I1220 10:36:25.498636 26061 memory_optimize_pass.cc:216] Cluster name : tmp_80 size: 17204
I1220 10:36:25.498639 26061 memory_optimize_pass.cc:216] Cluster name : batch_norm_2.tmp_2 size: 12288
I1220 10:36:25.498643 26061 memory_optimize_pass.cc:216] Cluster name : linear_20.tmp_1 size: 17204
I1220 10:36:25.498647 26061 memory_optimize_pass.cc:216] Cluster name : linear_15.tmp_0 size: 12288
I1220 10:36:25.498651 26061 memory_optimize_pass.cc:216] Cluster name : cast_1.tmp_0 size: 8
I1220 10:36:25.498656 26061 memory_optimize_pass.cc:216] Cluster name : audio_len size: 8
I1220 10:36:25.498659 26061 memory_optimize_pass.cc:216] Cluster name : batch_norm_5.tmp_2 size: 12288
--- Running analysis [ir_graph_to_program_pass]
I1220 10:36:25.643460 26061 analysis_predictor.cc:717] ======= optimize end =======
I1220 10:36:25.647202 26061 naive_executor.cc:98] --- skip [feed], feed -> audio_len
I1220 10:36:25.647218 26061 naive_executor.cc:98] --- skip [feed], feed -> audio
I1220 10:36:25.650734 26061 naive_executor.cc:98] --- skip [softmax_0.tmp_0], fetch -> fetch
I1220 10:36:25.650749 26061 naive_executor.cc:98] --- skip [tmp_11], fetch -> fetch
2021-12-20 10:36:25.110 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.110 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.111 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.111 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.111 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_fc.weight | [1312, 3072] | 4030464 | True
2021-12-20 10:36:25.112 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.112 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.112 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.113 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.113 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.113 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.fw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.114 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.114 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.0.bw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.114 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_fc.weight | [2048, 3072] | 6291456 | True
2021-12-20 10:36:25.115 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.115 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.116 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.116 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.116 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_fc.weight | [2048, 3072] | 6291456 | True
2021-12-20 10:36:25.117 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.117 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.117 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.118 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.118 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.118 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.fw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.119 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.119 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.1.bw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.119 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_fc.weight | [2048, 3072] | 6291456 | True
2021-12-20 10:36:25.120 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.120 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.121 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.122 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.122 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_fc.weight | [2048, 3072] | 6291456 | True
2021-12-20 10:36:25.122 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_bn.weight | [3072] | 3072 | True
2021-12-20 10:36:25.123 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_bn.bias | [3072] | 3072 | True
2021-12-20 10:36:25.123 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_bn._mean | [3072] | 3072 | False
2021-12-20 10:36:25.124 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_bn._variance | [3072] | 3072 | False
2021-12-20 10:36:25.125 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.125 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.fw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.126 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_cell.weight_hh | [3072, 1024] | 3145728 | True
2021-12-20 10:36:25.127 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - encoder.rnn.rnn_stacks.2.bw_cell.bias_hh | [3072] | 3072 | True
2021-12-20 10:36:25.127 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - decoder.ctc_lo.weight | [2048, 4301] | 8808448 | True
2021-12-20 10:36:25.128 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:57 - decoder.ctc_lo.bias | [4301] | 4301 | True
2021-12-20 10:36:25.128 | INFO | paddlespeech.s2t.utils.layer_tools:print_params:60 - Total parameters: 54.0, 58.42M elements.
2021-12-20 10:36:25.129 | INFO | paddlespeech.s2t.exps.deepspeech2.model:setup_model:175 - Setup model!
2021-12-20 10:36:25.651 | INFO | paddlespeech.s2t.exps.deepspeech2.model:test:411 - Test Total Examples: 7176

C++ Traceback (most recent call last):

No stack trace in paddle, may be caused by external reasons.

Error Message Summary:

FatalError: Segmentation fault is detected by the operating system.
[TimeInfo: *** Aborted at 1639967787 (unix time) try "date -d @1639967787" if you are using GNU date ***]
[SignalInfo: *** SIGSEGV (@0x0) received by PID 26061 (TID 0x7efe5888d700) from PID 0 ***]

./local/test_export.sh: line 28: 26061 Segmentation fault (core dumped) python3 -u ${BIN_DIR}/test_export.py --ngpu ${ngpu} --config ${config_path} --result_file ${jit_model_export_path}.rsl --export_path ${jit_model_export_path} --model_type ${model_type}

'[' 139 -ne 0 ']'
echo 'Failed in evaluation!'
Failed in evaluation!
exit 1

Answered by Jackwaterveg

Dec 20, 2021

请问有使用预训练模型吗？wget https://paddlespeech.bj.bcebos.com/s2t/aishell/asr0/ds2.model.tar.gz

View full answer

Jackwaterveg · 2021-12-20T03:08:00Z

Jackwaterveg
Dec 20, 2021
Collaborator

请问有使用预训练模型吗？wget https://paddlespeech.bj.bcebos.com/s2t/aishell/asr0/ds2.model.tar.gz

9 replies

dabaoyasu Dec 20, 2021
Author

以下命令安装的paddle：
python -m pip install paddlepaddle-gpu==2.2.1 -i https://mirror.baidu.com/pypi/simple

Jackwaterveg Dec 20, 2021
Collaborator

其他还可以试试是不是显卡的问题，设置CUDA_VISIBLE_DEVICES=,只是用cpu来推理看看。
如果还是不行，那么要解决这个问题需要系统环境对齐一下。我这里的环境测试是可以跑的，我使用的是docker来安装paddlespeech。使用的docker镜像是 registry.baidubce.com/paddlepaddle/paddle:2.1.1-gpu-cuda10.2-cudnn7，系统为ubuntu16.04。请问你那里使用的是什么版本的系统？

dabaoyasu Dec 20, 2021
Author

我用的也是Ubuntu 16.04 LTS，但没有用docker。
Linux suntek-PR4768GW-SX 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

NCCL version 2.10.3+cuda10.2
cuDNN Version: 8.2

import paddle
paddle.utils.run_check()
Running verify PaddlePaddle program ...
W1220 18:44:30.625463 38584 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 10.2, Runtime API Version: 10.2
W1220 18:44:30.628871 38584 device_context.cc:465] device: 0, cuDNN Version: 8.2.
PaddlePaddle works well on 1 GPU.
W1220 18:44:33.162925 38584 parallel_executor.cc:617] Cannot enable P2P access from 0 to 4
W1220 18:44:33.162984 38584 parallel_executor.cc:617] Cannot enable P2P access from 0 to 5
W1220 18:44:33.162999 38584 parallel_executor.cc:617] Cannot enable P2P access from 0 to 6
W1220 18:44:33.163014 38584 parallel_executor.cc:617] Cannot enable P2P access from 0 to 7
W1220 18:44:34.098471 38584 parallel_executor.cc:617] Cannot enable P2P access from 1 to 4
W1220 18:44:34.098517 38584 parallel_executor.cc:617] Cannot enable P2P access from 1 to 5
W1220 18:44:34.098526 38584 parallel_executor.cc:617] Cannot enable P2P access from 1 to 6
W1220 18:44:34.098532 38584 parallel_executor.cc:617] Cannot enable P2P access from 1 to 7
W1220 18:44:35.071377 38584 parallel_executor.cc:617] Cannot enable P2P access from 2 to 4
W1220 18:44:35.071422 38584 parallel_executor.cc:617] Cannot enable P2P access from 2 to 5
W1220 18:44:35.071430 38584 parallel_executor.cc:617] Cannot enable P2P access from 2 to 6
W1220 18:44:35.071437 38584 parallel_executor.cc:617] Cannot enable P2P access from 2 to 7
W1220 18:44:36.019466 38584 parallel_executor.cc:617] Cannot enable P2P access from 3 to 4
W1220 18:44:36.019511 38584 parallel_executor.cc:617] Cannot enable P2P access from 3 to 5
W1220 18:44:36.019520 38584 parallel_executor.cc:617] Cannot enable P2P access from 3 to 6
W1220 18:44:36.019527 38584 parallel_executor.cc:617] Cannot enable P2P access from 3 to 7
W1220 18:44:36.019541 38584 parallel_executor.cc:617] Cannot enable P2P access from 4 to 0
W1220 18:44:36.019564 38584 parallel_executor.cc:617] Cannot enable P2P access from 4 to 1
W1220 18:44:36.019589 38584 parallel_executor.cc:617] Cannot enable P2P access from 4 to 2
W1220 18:44:36.019598 38584 parallel_executor.cc:617] Cannot enable P2P access from 4 to 3
W1220 18:44:37.509740 38584 parallel_executor.cc:617] Cannot enable P2P access from 5 to 0
W1220 18:44:37.509786 38584 parallel_executor.cc:617] Cannot enable P2P access from 5 to 1
W1220 18:44:37.509799 38584 parallel_executor.cc:617] Cannot enable P2P access from 5 to 2
W1220 18:44:37.509810 38584 parallel_executor.cc:617] Cannot enable P2P access from 5 to 3
W1220 18:44:38.361558 38584 parallel_executor.cc:617] Cannot enable P2P access from 6 to 0
W1220 18:44:38.361603 38584 parallel_executor.cc:617] Cannot enable P2P access from 6 to 1
W1220 18:44:38.361610 38584 parallel_executor.cc:617] Cannot enable P2P access from 6 to 2
W1220 18:44:38.361618 38584 parallel_executor.cc:617] Cannot enable P2P access from 6 to 3
W1220 18:44:39.124289 38584 parallel_executor.cc:617] Cannot enable P2P access from 7 to 0
W1220 18:44:39.124332 38584 parallel_executor.cc:617] Cannot enable P2P access from 7 to 1
W1220 18:44:39.124341 38584 parallel_executor.cc:617] Cannot enable P2P access from 7 to 2
W1220 18:44:39.124348 38584 parallel_executor.cc:617] Cannot enable P2P access from 7 to 3
NCCL version 2.10.3+cuda10.2
W1220 18:44:47.481825 38584 fuse_all_reduce_op_pass.cc:76] Find all_reduce operators: 2. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 2.
PaddlePaddle works well on 8 GPUs.
PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.

dabaoyasu Dec 21, 2021
Author

用CPU执行是正常的，没有报错。

Jackwaterveg Dec 21, 2021
Collaborator

我猜测可能和cudnn的版本有关系，因为10.2搭配的都是cudnn7.2，对于使用8.2，需要一段时间验证可行性。
静态图的推理结果和动态图的推理结果，也就是stage3的结果是完全一致的。如果很想使用这个静态图的推理功能，可以尝试使用docker安装：
https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/install_cn.md#%E9%80%89%E6%8B%A91-%E4%BD%BF%E7%94%A8docker%E5%AE%B9%E5%99%A8%E6%8E%A8%E8%8D%90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

求助：examples/aishell/asr0/ 的 run.sh 运行到 stage 5 时报错 #1170

{{title}}

Replies: 1 comment 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

求助：examples/aishell/asr0/ 的 run.sh 运行到 stage 5 时报错 #1170

dabaoyasu Dec 20, 2021

C++ Traceback (most recent call last):

Error Message Summary:

Replies: 1 comment · 9 replies

Jackwaterveg Dec 20, 2021 Collaborator

dabaoyasu Dec 20, 2021 Author

Jackwaterveg Dec 20, 2021 Collaborator

dabaoyasu Dec 20, 2021 Author

dabaoyasu Dec 21, 2021 Author

Jackwaterveg Dec 21, 2021 Collaborator

dabaoyasu
Dec 20, 2021

Replies: 1 comment 9 replies

Jackwaterveg
Dec 20, 2021
Collaborator

dabaoyasu Dec 20, 2021
Author

Jackwaterveg Dec 20, 2021
Collaborator

dabaoyasu Dec 20, 2021
Author

dabaoyasu Dec 21, 2021
Author

Jackwaterveg Dec 21, 2021
Collaborator