Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training stops after few epochs #120

Open
amit0018 opened this issue Apr 2, 2021 · 1 comment
Open

Training stops after few epochs #120

amit0018 opened this issue Apr 2, 2021 · 1 comment

Comments

@amit0018
Copy link

amit0018 commented Apr 2, 2021 •
edited
Loading

Here is the training file

#!/bin/sh

script_dir=dirname $0
main_dir=$script_dir/../
data_dir=$main_dir/data
working_dir=$main_dir/model

device=0,1
#independent variables (toolkit locations)
#. $main_dir/../vars

#language-dependent variables (source and target language)
. $main_dir/vars

CUDA_VISIBLE_DEVICES=$device python $nematus_home/nematus/train.py
--model $working_dir/model
--datasets $data_dir/korean-english-park.train.bpe.$src $data_dir/korean-english-park.train.bpe.$trg
--valid_datasets $data_dir/korean-english-park.dev.bpe.$src $data_dir/korean-english-park.dev.bpe.$trg
--dictionaries $data_dir/korean-english-park.train.bpe.ko.json $data_dir/korean-english-park.train.bpe.en.json
--valid_script $script_dir/validate.sh
--reload latest_checkpoint
--dim_word 512
--dim 1024
--lrate 0.0005
--optimizer adam
--maxlen 200
--batch_size 80
--valid_batch_size 40
--validFreq 10000
--dispFreq 1000
--saveFreq 30000
--sampleFreq 10000
--tie_decoder_embeddings
--layer_normalisation
--dec_base_recurrence_transition_depth 2
--enc_recurrence_transition_depth 2
--rnn_dropout_hidden 0.5
--rnn_dropout_embedding 0.5
--rnn_dropout_source 0.3
--rnn_dropout_target 0.3
--label_smoothing 0.2
--token_batch_size 1000
--valid_token_batch_size 1000
--patience 10

validate.sh file mentioned below:

#!/bin/sh

script_dir=dirname $0
main_dir=$script_dir/../
data_dir=$main_dir/data
working_dir=$main_dir/model

device=0,1

#language-dependent variables (source and target language)
. $main_dir/vars

dev_prefix=korean-english-park.dev
dev=$dev_prefix.bpe.$src
ref=$dev_prefix.$trg
prefix=$working_dir/model

CUDA_VISIBLE_DEVICES=$device python $nematus_home/nematus/translate.py
-m $prefix
-i $data_dir/$dev
-o $working_dir/$dev.output.dev
-k 5
-n

bash $script_dir/postprocess.sh < $working_dir/$dev.output.dev > $working_dir/$dev.output.postprocessed.dev

BEST=cat ${prefix}_best_bleu || echo 0
$nematus_home/data/multi-bleu-detok.perl /scratch/amitkumar.rs.cse17.iitbhu/ko-en/parallel/$ref < $working_dir/$dev.output.postprocessed.dev >> ${prefix}_bleu_scores
BLEU=$nematus_home/data/multi-bleu-detok.perl /scratch/amitkumar.rs.cse17.iitbhu/ko-en/parallel/$ref < $working_dir/$dev.output.postprocessed.dev | cut -f 3 -d ' ' | cut -f 1 -d ','
BETTER=echo "$BLEU > $BEST" | bc

echo "BLEU = $BLEU"

echo "BLEU = $BLEU"

if [ "$BETTER" = "1" ]; then
echo "new best; saving"
echo $BLEU &gt; ${prefix}_best_bleu
cp ${prefix}.json ${prefix}.best_bleu.json
fi

Following errors in training:

2021-04-02 07:11:20.787601: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
INFO: Namespace(adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, batch_size=80, beam_freq=10000, beam_size=12, clip_c=1.0, datasets=['scripts/..//data/korean-english-park.train.bpe.ko', 'scripts/..//data/korean-english-park.train.bpe.en'], decay_c=0.0, dictionaries=['scripts/..//data/korean-english-park.train.bpe.ko.json', 'scripts/..//data/korean-english-park.train.bpe.en.json'], dim_per_factor=[512], disp_freq=1000, embedding_size=512, exponential_smoothing=0.0, factors=1, finish_after=10000000, gradient_aggregation_steps=1, keep_train_set_in_memory=False, label_smoothing=0.2, layer_normalization_type='layernorm', learning_rate=0.0005, learning_schedule='constant', loss_function='cross-entropy', map_decay_c=0.0, max_epochs=5000, max_len_a=1.5, max_len_b=5, max_sentences_of_sampling=0, max_sentences_per_device=0, max_tokens_per_device=0, maxibatch_size=20, maxlen=200, model_type='rnn', model_version=0.2, mrt_alpha=0.005, mrt_loss='SENTENCEBLEU n=4', mrt_ml_mix=0, mrt_reference=False, n_best=False, normalization_alpha=0.0, optimizer='adam', output_hidden_activation='tanh', patience=10, plateau_steps=0, preprocess_script=None, print_per_token_pro=False, prior_model=None, reload='latest_checkpoint', reload_training_progress=True, rnn_dec_base_transition_depth=2, rnn_dec_deep_context=False, rnn_dec_depth=1, rnn_dec_high_transition_depth=1, rnn_dropout_embedding=0.5, rnn_dropout_hidden=0.5, rnn_dropout_source=0.3, rnn_dropout_target=0.3, rnn_enc_depth=1, rnn_enc_transition_depth=2, rnn_layer_normalization=True, rnn_lexical_model=False, rnn_use_dropout=True, sample_freq=10000, sample_way='beam_search', samplesN=100, sampling_temperature=1.0, save_freq=30000, saveto='scripts/..//model/model', shuffle_each_epoch=True, softmax_mixture_size=1, sort_by_length=True, source_dataset='scripts/..//data/korean-english-park.train.bpe.ko', source_dicts=['scripts/..//data/korean-english-park.train.bpe.ko.json'], source_vocab_sizes=[9910], state_size=1024, summary_dir=None, summary_freq=0, target_dataset='scripts/..//data/korean-english-park.train.bpe.en', target_dict='scripts/..//data/korean-english-park.train.bpe.en.json', target_embedding_size=512, target_vocab_size=5953, theano_compat=False, tie_decoder_embeddings=True, tie_encoder_decoder_embeddings=False, token_batch_size=1000, transformer_dec_depth=6, transformer_drophead=0.0, transformer_dropout_attn=0.1, transformer_dropout_embeddings=0.1, transformer_dropout_relu=0.1, transformer_dropout_residual=0.1, transformer_enc_depth=6, transformer_ffn_hidden_size=2048, transformer_num_heads=8, translation_maxlen=200, translation_strategy='beam_search', valid_batch_size=40, valid_bleu_source_dataset='scripts/..//data/korean-english-park.dev.bpe.ko', valid_datasets=['scripts/..//data/korean-english-park.dev.bpe.ko', 'scripts/..//data/korean-english-park.dev.bpe.en'], valid_freq=10000, valid_script='scripts/validate.sh', valid_source_dataset='scripts/..//data/korean-english-park.dev.bpe.ko', valid_target_dataset='scripts/..//data/korean-english-park.dev.bpe.en', valid_token_batch_size=1000, warmup_steps=8000)
2021-04-02 07:11:41.096118: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-02 07:11:41.108886: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2400000000 Hz
2021-04-02 07:11:41.110970: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5612befedba0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-02 07:11:41.111051: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-04-02 07:11:41.116171: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-04-02 07:11:42.267591: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5612bf0af6e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-02 07:11:42.267741: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2021-04-02 07:11:42.270571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:5e:00.0 name: Tesla V100-PCIE-16GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-04-02 07:11:42.270674: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-02 07:11:42.276043: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-04-02 07:11:42.279407: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-04-02 07:11:42.280411: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-04-02 07:11:42.283420: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-04-02 07:11:42.285053: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-04-02 07:11:47.565824: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-04-02 07:11:47.571036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-02 07:11:47.571172: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-02 07:11:49.279069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-02 07:11:49.279154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2021-04-02 07:11:49.279172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2021-04-02 07:11:49.281268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14729 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:5e:00.0, compute capability: 7.0)
2021-04-02 07:11:49.286117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:5e:00.0 name: Tesla V100-PCIE-16GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-04-02 07:11:49.286209: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-02 07:11:49.286244: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-04-02 07:11:49.286269: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-04-02 07:11:49.286293: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-04-02 07:11:49.286316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-04-02 07:11:49.286340: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-04-02 07:11:49.286364: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-04-02 07:11:49.288785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-02 07:11:49.288839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-02 07:11:49.288879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2021-04-02 07:11:49.288899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2021-04-02 07:11:49.291537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/device:GPU:0 with 14729 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:5e:00.0, compute capability: 7.0)
INFO: Building model...
WARNING: From /home/amitkumar.rs.cse17.iitbhu/nematus/nematus/rnn_model.py:59: dropout (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
WARNING: From /home/amitkumar.rs.cse17.iitbhu/envNem1/lib/python3.5/site-packages/tensorflow-2.3.1-py3.5-linux-x86_64.egg/tensorflow/python/keras/legacy_tf_layers/core.py:271: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
2021-04-02 07:11:51.977754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:5e:00.0 name: Tesla V100-PCIE-16GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 15.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-04-02 07:11:51.977826: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-02 07:11:51.977870: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-04-02 07:11:51.977889: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-04-02 07:11:51.977907: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-04-02 07:11:51.977925: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-04-02 07:11:51.977942: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-04-02 07:11:51.977965: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-04-02 07:11:51.979691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
INFO: Initializing model parameters from scratch...
INFO: Done
INFO: Reading data...
INFO: Done
INFO: Initial uidx=0
INFO: Starting epoch 0
2021-04-02 07:12:23.860174: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
INFO: [2021-04-02 07:33:59] Epoch: 0 Update: 1000 Loss/word: 6.966987346043379 Words/sec: 696.856504668986 Sents/sec: 21.713141805022023
INFO: [2021-04-02 07:55:22] Epoch: 0 Update: 2000 Loss/word: 6.580592949380649 Words/sec: 703.13946539418 Sents/sec: 22.02518408136046
INFO: [2021-04-02 08:16:18] Epoch: 0 Update: 3000 Loss/word: 6.428336913131417 Words/sec: 722.7127947165809 Sents/sec: 22.798869988266855
INFO: Starting epoch 1
INFO: [2021-04-02 08:37:23] Epoch: 1 Update: 4000 Loss/word: 6.330259053899345 Words/sec: 716.8921478972981 Sents/sec: 22.429987003584493
INFO: [2021-04-02 08:58:25] Epoch: 1 Update: 5000 Loss/word: 6.2662326674608595 Words/sec: 717.9178678249805 Sents/sec: 22.6803643125988
INFO: [2021-04-02 09:19:13] Epoch: 1 Update: 6000 Loss/word: 6.226474244214676 Words/sec: 726.10896397206 Sents/sec: 22.971330964205684
INFO: Starting epoch 2
INFO: [2021-04-02 09:40:21] Epoch: 2 Update: 7000 Loss/word: 6.17927944352242 Words/sec: 714.946995990232 Sents/sec: 22.401948114913043
INFO: [2021-04-02 10:01:17] Epoch: 2 Update: 8000 Loss/word: 6.147852558704621 Words/sec: 719.9293130745198 Sents/sec: 22.550224623889665
INFO: [2021-04-02 10:22:19] Epoch: 2 Update: 9000 Loss/word: 6.124515647550154 Words/sec: 718.4753338888931 Sents/sec: 22.38133746941181
INFO: Starting epoch 3
INFO: [2021-04-02 10:43:16] Epoch: 3 Update: 10000 Loss/word: 6.093494990647863 Words/sec: 722.7513458365172 Sents/sec: 22.890552982759697
INFO: SOURCE: T@@ M@@ Z@@ 닷@@ 컴@@ 은 듀@@ 브@@ 로가 네@@ 버@@ 더@@ 주 라@@ 스@@ 베이@@ 거@@ 스에 있는 자택에서 사망했다고 보도했다 .
INFO: TARGET: Du@@ B@@ row died at his home in Las Vegas , Nevada , according to T@@ M@@ Z@@ .@@ com@@ .
INFO: SAMPLE: American G@@ reported C@@ D wasn 't Roger C@@ bad C@@ y@@ d@@ y@@ d , Paris and h@@ ol@@ ly behind your rushed safely today .
INFO: SOURCE: 추@@ 후 에 요@@ 금이 적용@@ 될 수도 있지만 현@@ 재@@ 는 무@@ 료@@ 다 .
INFO: TARGET: initially , there will be no f@@ ee to apply for travel author@@ iz@@ ation , although one may be imposed later .
INFO: SAMPLE: after car@@ c@@ ers often be used pa@@ rent , but a gi@@ gh@@ ty ency has been thought to vote oe @-@ on test !
INFO: SOURCE: 일부 펀@@ 드@@ 매@@ 니@@ 저는 " 기업 이@@ 윤@@ 이 증가@@ 해 주식@@ 시장이 활@@ 성@@ 화@@ 됐다 " 고 전했다 .
INFO: TARGET: fund man@@ ag@@ ers said strong corporate profit growth was supporting stocks .
INFO: SAMPLE: more than $ 3.@@ 5 bailout attacked C@@ m@@ succeed for more barrels a rise , " 29@@ ,000 yen re@@ str@@ ong@@ s to investors .
INFO: SOURCE: 이 바이러@@ 스는 서@@ 쪽으로 빠르게 퍼@@ 지고 있으며 , 관리들은 이것이 곧 서부 해안@@ 에@@ 까지 이를 것으로 우려하고 있다 .
INFO: TARGET: the virus has been spreading quickly w@@ est@@ ward , with officials f@@ ear@@ ing it will soon reach the West Coast .
INFO: SAMPLE: camps are cleared to sea winds and often @-@ anger gl@@ om@@ ing to Iraq as " coast of Hospital always . documentary loc@@ als politicians northwest the route . term of increasing exposure because
INFO: SOURCE: 잭슨 측의 한 변호사는 유@@ 효@@ 한 계약@@ 은 없@@ 었으며 돈@@ 은 대@@ 가@@ 성 없이 주@@ 어진 것이라고 말했습니다 .
INFO: TARGET: a lawyer for Jackson says there was never a v@@ ali@@ d agreement , and that the money was given fre@@ ely .
INFO: SAMPLE: what was not free , but he says to pr@@ os@@ ful citing something they have enti@@ Senator child .
INFO: SOURCE: 총격@@ 범@@ 과 그의 여자@@ 친구 사이 의 말@@ 다@@ 툼 때문에 이번 공격이 발생@@ 했다는 , 확인@@ 되지 않은 보도가 있었다 .
INFO: TARGET: un@@ confirmed reports say the attack followed an ar@@ gu@@ ment between the gunman and his girlfriend .
INFO: SAMPLE: police had left the militants with his wife , cos@@ the@@ gh@@ ics and two were monitoring .
INFO: SOURCE: 왕@@ 복@@ 선은 이틀@@ 간 우주 정거@@ 장을 조사한 뒤 호주 남@@ 동부 3@@ 36@@ km 위에서 작업을 종@@ 료@@ 할 예정이다 .
INFO: TARGET: the shuttle 's two @-@ day ch@@ as@@ e of the space station ended about 2@@ 10 miles above southeastern Australia .
INFO: SAMPLE: captain R@@ on@@ submitted Pat@@ g@@ el will were vi@@ fi@@ ory , as Ch@@ ang@@ am@@ land international space station will maintain to Tibet . Be@@ at@@ ari televised congratul@@ ate the space station
INFO: SOURCE: 한국은 또한 전통적인 경제 논@@ 리에 명@@ 백@@ 히 거@@ 슬@@ 리는 이런 변화를 이@@ 룩@@ 해@@ 내@@ 길 바라고 있다 .
INFO: TARGET: Korea also hopes to accompl@@ ish the shift in apparent de@@ fi@@ ance of con@@ v@@ enti@@ onal economic lo@@ gi@@ c .
INFO: SAMPLE: Korea also ay which for so balance in@@ land .
INFO: SOURCE: 크레@@ 디는 " 체포 당시 15@@ 달 된 아들을 안@@ 고 있었다 " 고 말했다 .
INFO: TARGET: " I was holding my baby at the time , " said McC@@ ready of her 15 @-@ month @-@ old son .
INFO: SAMPLE: ing walked him in London , " P@@ u@@ k@@ crowd said .
INFO: SOURCE: 이슬람 반군@@ 과의 이번 분쟁@@ 은 199@@ 0@@ 년 내@@ 전 이후 레바논@@ 에서 발생한 최악의 유혈@@ 충돌로 기록@@ 되고 있다 .
INFO: TARGET: the sometimes @-@ fi@@ er@@ ce battles mark the worst internal violence since the end of Lebanon 's civil war in 1990 .
INFO: SAMPLE: Su@@ am@@ there still show Robert A@@ March in the biggest looks at northern Iraq in reasons every year for political fighting momentum ;
INFO: SOURCE: T@@ M@@ Z@@ 닷@@ 컴@@ 은 듀@@ 브@@ 로가 네@@ 버@@ 더@@ 주 라@@ 스@@ 베이@@ 거@@ 스에 있는 자택에서 사망했다고 보도했다 .
INFO: TARGET: Du@@ B@@ row died at his home in Las Vegas , Nevada , according to T@@ M@@ Z@@ .@@ com@@ .
INFO: SAMPLE 0: the A@@ T@@ A said he was killed in a hospital in the Los Angeles capital . Cost/Len/Avg -34.647682189941406/76/-0.45589055513080795
INFO: SAMPLE 1: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles capital . Cost/Len/Avg -34.67599105834961/80/-0.43344988822937014
INFO: SAMPLE 2: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles area . Cost/Len/Avg -34.80378341674805/77/-0.4519971872304941
INFO: SAMPLE 3: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of C@@ as@@ e . Cost/Len/Avg -43.47795104980469/91/-0.4777796818659856
INFO: SAMPLE 4: the A@@ T@@ A said he was killed in a hospital in the Los Angeles city of C@@ as@@ e . Cost/Len/Avg -43.573909759521484/87/-0.5008495374657642
INFO: SAMPLE 5: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of B@@ ri@@ ot@@ a . Cost/Len/Avg -45.60481262207031/96/-0.4750501314798991
INFO: SAMPLE 6: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of P@@ h@@ ol@@ o . Cost/Len/Avg -45.89029312133789/95/-0.48305571706671463
INFO: SAMPLE 7: the A@@ T@@ A said he was killed in a hospital in the Los Angeles city of P@@ h@@ ol@@ o . Cost/Len/Avg -45.95040512084961/91/-0.5049495068225232
INFO: SAMPLE 8: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of P@@ h@@ il@@ d . Cost/Len/Avg -46.11641311645508/95/-0.4854359275416324
INFO: SAMPLE 9: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of B@@ ri@@ ot@@ a , the Associated Press reported . Cost/Len/Avg -50.92622756958008/128/-0.39786115288734436
INFO: SAMPLE 10: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of B@@ ri@@ ot@@ a , the AP reported . Cost/Len/Avg -51.131935119628906/114/-0.44852574666341144
INFO: SAMPLE 11: the A@@ T@@ A was killed in a Los Angeles hospital in the Los Angeles city of B@@ ri@@ ot@@ a , the official said . Cost/Len/Avg -51.474334716796875/116/-0.4437442647999731
INFO: SOURCE: 추@@ 후 에 요@@ 금이 적용@@ 될 수도 있지만 현@@ 재@@ 는 무@@ 료@@ 다 .
INFO: TARGET: initially , there will be no f@@ ee to apply for travel author@@ iz@@ ation , although one may be imposed later .
INFO: SAMPLE 0: but it is expected to be expected to be held . Cost/Len/Avg -23.384572982788086/47/-0.4975441060167678
INFO: SAMPLE 1: but it is expected to be expected to be released . Cost/Len/Avg -23.43704605102539/51/-0.4595499225691253
INFO: SAMPLE 2: but it is expected to be expected to be held , but it 's not going to be . Cost/Len/Avg -37.967037200927734/80/-0.47458796501159667
INFO: SAMPLE 3: but it is expected to be expected to be held , but it 's not enough to be . Cost/Len/Avg -38.5172119140625/81/-0.47552113474151236
INFO: SAMPLE 4: but it is expected to be expected to be held , but it 's not going to be able . Cost/Len/Avg -40.010398864746094/85/-0.4707105748793658
INFO: SAMPLE 5: but it is expected to be expected to be held , but it will not be able to be . Cost/Len/Avg -40.27976989746094/79/-0.5098705050311511
INFO: SAMPLE 6: but it is expected to be expected to be held , but it will not be able to be able . Cost/Len/Avg -42.25888442993164/84/-0.5030819574991862
INFO: SAMPLE 7: but it is expected to be expected to be held , but it 's not going to be able to be . Cost/Len/Avg -43.747947692871094/91/-0.4807466779436384
INFO: SAMPLE 8: but it is expected to be expected to be held , but it 's not going to be able to go . Cost/Len/Avg -44.41551208496094/91/-0.48808255038418613
INFO: SAMPLE 9: but it is expected to be expected to be held , but it will not be able to be able to be . Cost/Len/Avg -46.169952392578125/90/-0.5129994710286458
INFO: SAMPLE 10: but it is expected to be expected to be held , but it 's not going to be able to be able . Cost/Len/Avg -46.30955505371094/96/-0.4823911984761556
INFO: SAMPLE 11: but it is expected to be expected to be held , but it will not be able to be able to be able . Cost/Len/Avg -48.63726806640625/95/-0.5119712428042763
INFO: SOURCE: 일부 펀@@ 드@@ 매@@ 니@@ 저는 " 기업 이@@ 윤@@ 이 증가@@ 해 주식@@ 시장이 활@@ 성@@ 화@@ 됐다 " 고 전했다 .
INFO: TARGET: fund man@@ ag@@ ers said strong corporate profit growth was supporting stocks .
INFO: SAMPLE 0: according to the company , the company said , " it 's a lot of growth . Cost/Len/Avg -37.93861770629883/82/-0.4626660695890101
INFO: SAMPLE 1: according to the company , the company said , " it 's a lot of recession . Cost/Len/Avg -37.9777946472168/85/-0.4467975840849035
INFO: SAMPLE 2: according to the company , the company said , " it 's a lot of economic growth . Cost/Len/Avg -38.13514709472656/91/-0.4190675504915007
INFO: SAMPLE 3: according to the company , the company said , " it 's a lot of interest rates . Cost/Len/Avg -38.815006256103516/90/-0.43127784729003904
INFO: SAMPLE 4: according to the company , the company said , " it 's a lot of recession . " Cost/Len/Avg -38.95130157470703/92/-0.4233837127685547
INFO: SAMPLE 5: according to the company , the company said , " it 's a lot of economic growth . " Cost/Len/Avg -39.129940032958984/98/-0.3992851023771325
INFO: SAMPLE 6: according to the company , the company said , " it 's a lot of interest rates . " Cost/Len/Avg -39.95598602294922/97/-0.41191738167988884
INFO: SAMPLE 7: according to the company , the company said , " it 's a lot of recession , " he said . Cost/Len/Avg -40.88528060913086/102/-0.4008360844032437
INFO: SAMPLE 8: according to the company , the company said , " it 's a lot of economic growth , " he said . Cost/Len/Avg -41.4897346496582/108/-0.38416420971905746
INFO: SAMPLE 9: according to the company , the company said , " it 's a lot of interest rates , " he said . Cost/Len/Avg -41.738468170166016/107/-0.3900791417772525
INFO: SAMPLE 10: according to the company , the company said , " it 's a lot of the economy , " he said . Cost/Len/Avg -42.56211471557617/104/-0.4092511030343863
INFO: SAMPLE 11: according to the company , the company said , " it 's a lot of economic growth , " the company said . Cost/Len/Avg -43.87964630126953/117/-0.37503971197666264
INFO: SOURCE: 이 바이러@@ 스는 서@@ 쪽으로 빠르게 퍼@@ 지고 있으며 , 관리들은 이것이 곧 서부 해안@@ 에@@ 까지 이를 것으로 우려하고 있다 .
INFO: TARGET: the virus has been spreading quickly w@@ est@@ ward , with officials f@@ ear@@ ing it will soon reach the West Coast .
INFO: SAMPLE 0: the virus is expected to be in the region . Cost/Len/Avg -21.322120666503906/44/-0.4845936515114524
INFO: SAMPLE 1: the virus is expected to be in the region , officials said . Cost/Len/Avg -23.85358238220215/61/-0.39104233413446143
INFO: SAMPLE 2: the virus is expected to be in the area , officials said . Cost/Len/Avg -24.124916076660156/59/-0.40889688265525687
INFO: SAMPLE 3: the virus is expected to be in the region , but officials say they are still in the region . Cost/Len/Avg -40.866912841796875/93/-0.4394291703419019
INFO: SAMPLE 4: the virus is expected to be in the region , but officials said it would be in the region . Cost/Len/Avg -41.33380126953125/91/-0.45421759636847525
INFO: SAMPLE 5: the virus is expected to be in the region , but officials say they are still in the area . Cost/Len/Avg -41.385223388671875/91/-0.45478267460078986
INFO: SAMPLE 6: the virus is expected to be in the region , but officials say they are still in the border . Cost/Len/Avg -41.601173400878906/93/-0.44732444517074094
INFO: SAMPLE 7: the virus is expected to be in the region , but officials said it would be in the area . Cost/Len/Avg -41.86735916137695/89/-0.470419765858168
INFO: SAMPLE 8: the virus is expected to be in the region , but officials say they are still in the town . Cost/Len/Avg -41.90886306762695/91/-0.4605369567871094
INFO: SAMPLE 9: the virus is expected to be in the region , but officials said it would be in the border . Cost/Len/Avg -41.95485305786133/91/-0.4610423412951794
INFO: SAMPLE 10: the virus is expected to be in the region , but officials say they are still in the city . Cost/Len/Avg -42.01584243774414/91/-0.4617125542609246
INFO: SAMPLE 11: the virus is expected to be in the region , but officials say they are still in the town of the region . Cost/Len/Avg -46.920806884765625/105/-0.44686482747395834
INFO: SOURCE: 잭슨 측의 한 변호사는 유@@ 효@@ 한 계약@@ 은 없@@ 었으며 돈@@ 은 대@@ 가@@ 성 없이 주@@ 어진 것이라고 말했습니다 .
INFO: TARGET: a lawyer for Jackson says there was never a v@@ ali@@ d agreement , and that the money was given fre@@ ely .
INFO: SAMPLE 0: the judge says it 's not clear that it 's not clear that it 's not enough . Cost/Len/Avg -35.635536193847656/91/-0.3915992988334907
INFO: SAMPLE 1: the judge says it 's not clear that it 's not clear that it 's not enough to be . Cost/Len/Avg -39.26071548461914/97/-0.40474964417133136
INFO: SAMPLE 2: the judge says it 's not clear that it was not clear that it 's not enough to be . Cost/Len/Avg -39.52196502685547/93/-0.4249673658801663
INFO: SAMPLE 3: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not enough . Cost/Len/Avg -45.02193832397461/117/-0.38480289165790266
INFO: SAMPLE 4: the judge says it 's not clear that it was not clear that it 's not clear that it 's not enough . Cost/Len/Avg -45.23143768310547/113/-0.40027820958500415
INFO: SAMPLE 5: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not a lot . Cost/Len/Avg -47.7430419921875/116/-0.41157794820851296
INFO: SAMPLE 6: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not enough to be . Cost/Len/Avg -48.963233947753906/123/-0.3980750727459667
INFO: SAMPLE 7: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not a little bit . Cost/Len/Avg -49.09718704223633/123/-0.3991641222946043
INFO: SAMPLE 8: the judge says it 's not clear that it was not clear that it 's not clear that it 's not enough to be . Cost/Len/Avg -49.15306091308594/119/-0.41305093204273896
INFO: SAMPLE 9: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not enough to be . " Cost/Len/Avg -52.79254150390625/130/-0.40609647310697117
INFO: SAMPLE 10: the judge says it 's not clear that it was not clear that it 's not clear that it 's not enough to be . " Cost/Len/Avg -53.00259017944336/126/-0.42065547761462985
INFO: SAMPLE 11: the judge says it 's not clear that it 's not clear that it 's not clear that it 's not a little bit . " Cost/Len/Avg -53.03470993041992/130/-0.40795930715707635
INFO: SOURCE: 총격@@ 범@@ 과 그의 여자@@ 친구 사이 의 말@@ 다@@ 툼 때문에 이번 공격이 발생@@ 했다는 , 확인@@ 되지 않은 보도가 있었다 .
INFO: TARGET: un@@ confirmed reports say the attack followed an ar@@ gu@@ ment between the gunman and his girlfriend .
INFO: SAMPLE 0: the incident was found in the attack . Cost/Len/Avg -16.111059188842773/39/-0.4131040817651993
INFO: SAMPLE 1: the incident was found in the attack , but it was not reported . Cost/Len/Avg -25.738344192504883/65/-0.39597452603853667
INFO: SAMPLE 2: the incident was found in the attack , but it was not immediately reported . Cost/Len/Avg -26.580915451049805/77/-0.34520669416947797
INFO: SAMPLE 3: the incident was found in the attack , but it was not immediately known . Cost/Len/Avg -27.21120262145996/74/-0.36771895434405355
INFO: SAMPLE 4: the incident was found in the attack , but the incident was not immediately known . Cost/Len/Avg -28.442596435546875/84/-0.3386023385184152
INFO: SAMPLE 5: the incident was found in the attack , but it was no evidence of the attack . Cost/Len/Avg -29.79374885559082/78/-0.3819711391742413
INFO: SAMPLE 6: the incident was found in the attack , but it was not immediately known as the attack . Cost/Len/Avg -32.138160705566406/88/-0.3652063716541637
INFO: SAMPLE 7: the incident was found in the attack , but it was not clear that the attack was not known . Cost/Len/Avg -34.64654541015625/92/-0.37659288489300274
INFO: SAMPLE 8: the incident was found in the attack , but it was not clear that the incident was not known . Cost/Len/Avg -34.85503005981445/94/-0.37079819212568566
INFO: SAMPLE 9: the incident was found in the attack , but it was not immediately clear that he was not known . Cost/Len/Avg -35.30630874633789/96/-0.3677740494410197
INFO: SAMPLE 10: the incident was found in the attack , but it was not immediately clear that the attack was not reported . Cost/Len/Avg -35.64750289916992/107/-0.33315423270252265
INFO: SAMPLE 11: the incident was found in the attack , but it was not clear that the attack was not immediately known . Cost/Len/Avg -35.765377044677734/104/-0.34389785619882435
INFO: SOURCE: 왕@@ 복@@ 선은 이틀@@ 간 우주 정거@@ 장을 조사한 뒤 호주 남@@ 동부 3@@ 36@@ km 위에서 작업을 종@@ 료@@ 할 예정이다 .
INFO: TARGET: the shuttle 's two @-@ day ch@@ as@@ e of the space station ended about 2@@ 10 miles above southeastern Australia .
INFO: SAMPLE 0: London , England ( CNN ) Prince Harry is expected to be in the first time after the first time of the space station after the first time of the space station . Cost/Len/Avg -52.85684585571289/160/-0.33035528659820557
/home/amitkumar.rs.cse17.iitbhu/nematus/nematus/train.py:290: RuntimeWarning: divide by zero encountered in true_divide
i, sample, cost, len(sample), cost/len(sample))
INFO: SAMPLE 1: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 2: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 3: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 4: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 5: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 6: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 7: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 8: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 9: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 10: Cost/Len/Avg -10000000.0/0/-inf
INFO: SAMPLE 11: Cost/Len/Avg -10000000.0/0/-inf
INFO: SOURCE: 한국은 또한 전통적인 경제 논@@ 리에 명@@ 백@@ 히 거@@ 슬@@ 리는 이런 변화를 이@@ 룩@@ 해@@ 내@@ 길 바라고 있다 .
INFO: TARGET: Korea also hopes to accompl@@ ish the shift in apparent de@@ fi@@ ance of con@@ v@@ enti@@ onal economic lo@@ gi@@ c .
INFO: SAMPLE 0: South Korea also said it will also be a new economic crisis . Cost/Len/Avg -24.42115020751953/62/-0.3938895194761215
INFO: SAMPLE 1: South Korea also said that Korea is also likely to be a new economic crisis . Cost/Len/Avg -29.65618896484375/78/-0.3802075508313301
INFO: SAMPLE 2: South Korea also said that Korea is also likely to be a new economy . Cost/Len/Avg -29.85259246826172/70/-0.4264656066894531
INFO: SAMPLE 3: South Korea also said that Korea is also likely to make a new economic crisis . Cost/Len/Avg -29.87238121032715/80/-0.3734047651290894
INFO: SAMPLE 4: South Korea also said that Korea is also likely to be a major economic crisis . Cost/Len/Avg -29.930070877075195/80/-0.37412588596343993
INFO: SAMPLE 5: South Korea also said that Korea is also likely to take a new economic crisis . Cost/Len/Avg -30.075361251831055/80/-0.3759420156478882
INFO: SAMPLE 6: South Korea also said that Korea is also likely to get a new economic crisis . Cost/Len/Avg -30.59653091430664/79/-0.3872978596747676
INFO: SAMPLE 7: South Korea also said that Korea is also likely to have a new economic crisis . Cost/Len/Avg -30.841949462890625/80/-0.3855243682861328
INFO: SAMPLE 8: South Korea also said that Korea is also likely to be a new economic crisis in the world . Cost/Len/Avg -35.9841194152832/91/-0.3954298836844308
INFO: SAMPLE 9: South Korea also said that Korea is also likely to make a new economic crisis in the world . Cost/Len/Avg -36.09120178222656/93/-0.3880774385185652
INFO: SAMPLE 10: South Korea also said that Korea is also likely to be a new economic crisis in the country . Cost/Len/Avg -37.242855072021484/93/-0.40046080722603744
INFO: SAMPLE 11: South Korea also said that Korea is also likely to make a new economic crisis in the country . Cost/Len/Avg -37.32313919067383/95/-0.392875149375514
INFO: SOURCE: 크레@@ 디는 " 체포 당시 15@@ 달 된 아들을 안@@ 고 있었다 " 고 말했다 .
INFO: TARGET: " I was holding my baby at the time , " said McC@@ ready of her 15 @-@ month @-@ old son .
INFO: SAMPLE 0: he said he had been arrested last month . Cost/Len/Avg -18.11133575439453/42/-0.43122227986653644
INFO: SAMPLE 1: he said he had been arrested last year . Cost/Len/Avg -18.825273513793945/41/-0.45915301253155966
INFO: SAMPLE 2: he said he was taken to his wife . Cost/Len/Avg -19.07192039489746/35/-0.5449120112827845
INFO: SAMPLE 3: he said he had been arrested last month , " he said . Cost/Len/Avg -22.439355850219727/59/-0.3803280652579615
INFO: SAMPLE 4: he said he was taken to his wife , " he said . Cost/Len/Avg -22.90827751159668/52/-0.4405437982999362
INFO: SAMPLE 5: he said he had been arrested last month , " she said . Cost/Len/Avg -23.00881576538086/60/-0.38348026275634767
INFO: SAMPLE 6: he said he was taken to his wife , " she said . Cost/Len/Avg -23.43367576599121/53/-0.44214482577341907
INFO: SAMPLE 7: he said he had been arrested last month , " he said . " Cost/Len/Avg -28.204511642456055/66/-0.4273410854917584
INFO: SAMPLE 8: he said he was taken to his wife , " he said . " Cost/Len/Avg -28.393362045288086/59/-0.4812434244964082
INFO: SAMPLE 9: he said he had been arrested last month , " he said . " it was a child . Cost/Len/Avg -36.910091400146484/83/-0.44469989638730706
INFO: SAMPLE 10: he said he had been arrested last month , " he said . " it was a child . " Cost/Len/Avg -38.843902587890625/90/-0.43159891764322916
INFO: SAMPLE 11: he said he had been arrested last month , " he said . " it was not a child . Cost/Len/Avg -39.1064338684082/87/-0.44949923986676094
INFO: SOURCE: 이슬람 반군@@ 과의 이번 분쟁@@ 은 199@@ 0@@ 년 내@@ 전 이후 레바논@@ 에서 발생한 최악의 유혈@@ 충돌로 기록@@ 되고 있다 .
INFO: TARGET: the sometimes @-@ fi@@ er@@ ce battles mark the worst internal violence since the end of Lebanon 's civil war in 1990 .
INFO: SAMPLE 0: the Islamic militant group has been held in the northern city of southern Lebanon . Cost/Len/Avg -30.197078704833984/84/-0.35948903220040457
INFO: SAMPLE 1: the Islamic militant group has been held in the northern city of northern Lebanon . Cost/Len/Avg -30.308401107788086/84/-0.3608142989022391
INFO: SAMPLE 2: the Islamic militant group has been held in the northern city of northern Iraq . Cost/Len/Avg -30.530426025390625/81/-0.37691883981963736
INFO: SAMPLE 3: the Islamic militant group has been held in the northern city of northern Iraq in the region . Cost/Len/Avg -37.04274368286133/95/-0.38992361771432976
INFO: SAMPLE 4: the Islamic militant group has been held in the northern city of northern Lebanon , which has been in the region . Cost/Len/Avg -44.6102180480957/115/-0.3879149395486583
INFO: SAMPLE 5: the Islamic militant group has been held in the northern city of southern Lebanon , which has been in the region . Cost/Len/Avg -44.716522216796875/115/-0.38883932362432067
INFO: SAMPLE 6: the Islamic militant group has been held in the northern city of northern Iraq , which has been in the region . Cost/Len/Avg -44.74925994873047/112/-0.39954696382795063
INFO: SAMPLE 7: the Islamic militant group has been held in the northern city of northern Lebanon , which has been the worst in the region . Cost/Len/Avg -48.818878173828125/125/-0.390551025390625
INFO: SAMPLE 8: the Islamic militant group has been held in the northern city of southern Lebanon , which has been the worst in the region . Cost/Len/Avg -48.91313171386719/125/-0.3913050537109375
INFO: SAMPLE 9: the Islamic militant group has been held in the northern city of northern Iraq , which has been the worst in the region . Cost/Len/Avg -48.95859909057617/122/-0.4012999925457063
INFO: SAMPLE 10: the Islamic militant group has been held in the northern city of northern Lebanon , which has been the worst in the country . Cost/Len/Avg -49.13996124267578/126/-0.38999969240218874
INFO: SAMPLE 11: the Islamic militant group has been held in the northern city of southern Lebanon , which has been the worst in the country . Cost/Len/Avg -49.23438262939453/126/-0.3907490684872582
INFO: Seen 8
INFO: Seen 20
INFO: Seen 33
INFO: Seen 48
INFO: Seen 63
INFO: Seen 79
INFO: Seen 96
INFO: Seen 114
INFO: Seen 132
INFO: Seen 151
INFO: Seen 171
INFO: Seen 191
INFO: Seen 212
INFO: Seen 234
INFO: Seen 256
INFO: Seen 279
INFO: Seen 304
INFO: Seen 329
INFO: Seen 355
INFO: Seen 382
INFO: Seen 410
INFO: Seen 439
INFO: Seen 470
INFO: Seen 502
INFO: Seen 535
INFO: Seen 570
INFO: Seen 608
INFO: Seen 649
INFO: Seen 694
INFO: Seen 746
INFO: Seen 800
INFO: Seen 811
INFO: Seen 828
INFO: Seen 848
INFO: Seen 871
INFO: Seen 898
INFO: Seen 930
INFO: Seen 970
INFO: Seen 1000
INFO: Validation cross entropy (AVG/SUM/N_SENTS/N_TOKENS): 187.86940924072266 187869.40924072266 1000 33910
INFO: scripts/..//model/tmpv6367j94/model is not in all_model_checkpoint_paths. Manually adding it.
INFO: Starting external validation.
INFO: NOTE: Length of translations is capped to 200
INFO: Translated 40 sents
INFO: Translated 80 sents
INFO: Translated 120 sents
INFO: Translated 160 sents
INFO: Translated 200 sents
INFO: Translated 240 sents
INFO: Translated 280 sents
INFO: Translated 320 sents
INFO: Translated 360 sents
INFO: Translated 400 sents
INFO: Translated 440 sents
INFO: Translated 480 sents
INFO: Translated 520 sents
INFO: Translated 560 sents
INFO: Translated 600 sents
INFO: Translated 640 sents
INFO: Translated 680 sents
INFO: Translated 720 sents
INFO: Translated 760 sents
INFO: Translated 800 sents
INFO: Translated 840 sents
INFO: Translated 880 sents
INFO: Translated 920 sents
INFO: Translated 960 sents
INFO: Translated 1000 sents
INFO: Translated 1000 sents in 23.049481868743896 sec. Speed 43.38492317070448 sents/sec
Traceback (most recent call last):
File "/home/amitkumar.rs.cse17.iitbhu/nematus/nematus/train.py", line 522, in
train(config, sess)
File "/home/amitkumar.rs.cse17.iitbhu/nematus/nematus/train.py", line 322, in train
score = validate_with_script(sess, beam_search_sampler)
File "/home/amitkumar.rs.cse17.iitbhu/nematus/nematus/train.py", line 426, in validate_with_script
stderr=subprocess.PIPE)
File "/opt/ohpc/pub/apps/anaconda3/envs/py35/lib/python3.5/subprocess.py", line 676, in init
restore_signals, start_new_session)
File "/opt/ohpc/pub/apps/anaconda3/envs/py35/lib/python3.5/subprocess.py", line 1289, in _execute_child
raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
@rsennrich
Copy link
Collaborator

rsennrich commented Apr 7, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rsennrich @amit0018