How much disk space is required for training deepspeech model

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 10:43am]

Hi all,

I am continuing training from the pre-trained model using frozen graph
parameter provided. I am using common voice data for training. I am not
using GPU. When I start my training, it is throwing me a memory error
with the below command.

./DeepSpeech.py slash --n_hidden 2048 slash --initialize_from_frozen_model
models/output_graph.pb slash --checkpoint_dir fine_tuning_checkpoints
slash --epoch 3 slash --train_files
cv_corpus_v1/cv-valid-train.csv,cv_corpus_v1/cv-other-train.csv
slash --dev_files cv_corpus_v1/cv-valid-dev.csv slash --test_files
cv_corpus_v1/cv-valid-test.csv

How much disk space is required to do this process?

Thank you.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/how-much-disk-space-is-required-for-training-deepspeech-model]

[JRMeyer]

>>> lissyx
[February 22, 2018, 11:40am]

What is this 'memory error' ? We cannot diagnose an error that you do
not give us ...

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 11:56am]

./DeepSpeech.py slash --n_hidden 2048 slash --initialize_from_frozen_model
models/output_graph.pb slash --checkpoint_dir fine_tuning_checkpoints
slash --epoch 3 slash --train_files
cv_corpus_v1/cv-valid-train.csv,cv_corpus_v1/cv-other-train.csv
slash --dev_files cv_corpus_v1/cv-valid-dev.csv slash --test_files
cv_corpus_v1/cv-valid-test.csv

W Parameter slash --validation_step needs to be slash >0 for early stopping to
work slash
I Initializing from frozen model: models/output_graph.pb slash
Traceback (most recent call last): slash
File './DeepSpeech.py', line 1861, in slash
tf.app.run() slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py',
line 124, in run slash
slash _sys.exit(main(argv)) slash
File './DeepSpeech.py', line 1818, in main slash
train() slash
File './DeepSpeech.py', line 1592, in train slash
session.run(init_from_frozen_model_op, feed_dict=feed_dict) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 539, in run slash
run_metadata=run_metadata) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 1013, in run slash
run_metadata=run_metadata) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 1104, in run slash
raise six.reraise( slash *original_exc_info) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 1089, in run slash
return self. slash _sess.run( slash *args, slash DEEPSPEECH.cdx deepspeech.commands DEEPSPEECH.pages DEEPSPEECH.warc.gz discourse.mozilla.org html-to-markdown.sh shell-conver-html-to-split-posts.sh sorted-deepspeech-posts slash *kwargs) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 1153, in run slash
feed_dict, options) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py',
line 1179, in slash _call_hook_before_run slash
request = hook.before_run(run_context) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py',
line 434, in before_run slash
slash 'graph.pbtxt slash ') slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/framework/graph_io.py',
line 69, in write_graph slash
text_format.MessageToString(graph_def)) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 163, in MessageToString slash
printer.PrintMessage(message) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 352, in PrintMessage slash
self.PrintField(field, element) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 383, in PrintField slash
self.PrintFieldValue(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 419, in PrintFieldValue slash
self. slash _PrintMessageFieldValue(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 404, in slash _PrintMessageFieldValue slash
self.PrintMessage(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 349, in PrintMessage slash
self.PrintField(field, entry_submsg) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 383, in PrintField slash
self.PrintFieldValue(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 419, in PrintFieldValue slash
self. slash _PrintMessageFieldValue(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 404, in slash _PrintMessageFieldValue slash
self.PrintMessage(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 354, in PrintMessage slash
self.PrintField(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 383, in PrintField slash
self.PrintFieldValue(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 419, in PrintFieldValue slash
self. slash _PrintMessageFieldValue(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 404, in slash _PrintMessageFieldValue slash
self.PrintMessage(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 354, in PrintMessage slash
self.PrintField(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 383, in PrintField slash
self.PrintFieldValue(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 419, in PrintFieldValue slash
self. slash _PrintMessageFieldValue(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 404, in slash _PrintMessageFieldValue slash
self.PrintMessage(value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 354, in PrintMessage slash
self.PrintField(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 383, in PrintField slash
self.PrintFieldValue(field, value) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_format.py',
line 437, in PrintFieldValue slash
out.write(text_encoding.CEscape(out_value, out_as_utf8)) slash
File
'/home/megha/git-lfs-2.3.4/deepspeech-venv/local/lib/python2.7/site-packages/google/protobuf/text_encoding.py',
line 79, in CEscape slash
return ''.join( slash _cescape_byte_to_str slash [Ord© slash ] for c in text) slash
MemoryError

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 12:02pm]

This is hardly readable, can you ensure you are using proper code
formatting ? Some valuable information might also be lost in
reformatting if you are not using proper code format.

Can you give information on your system's specs ?

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 12:10pm]

slash $ cat /proc/cpuinfo

processor : 0 slash
vendor_id : GenuineIntel slash
cpu family : 6 slash
model : 78 slash
model name : Intel® Core™ i7-6500U CPU 2.50GHz slash
stepping : 3 slash
microcode : 0x8a slash
cpu MHz : 471.789 slash
cache size : 4096 KB slash
physical id : 0 slash
siblings : 4 slash
core id : 0 slash
cpu cores : 2 slash
apicid : 0 slash
initial apicid : 0 slash
fpu : yes slash
fpu_exception : yes slash
cpuid level : 22 slash
wp : yes slash
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm
abm 3dnowprefetch intel_pt tpr_shadow vnmi flexpriority ept vpid
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap
clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp
hwp_notify hwp_act_window hwp_epp slash
bugs : slash
bogomips : 5184.00 slash
clflush size : 64 slash
cache_alignment : 64 slash
address sizes : 39 bits physical, 48 bits virtual slash
power management:

processor : 1 slash
vendor_id : GenuineIntel slash
cpu family : 6 slash
model : 78 slash
model name : Intel® Core™ i7-6500U CPU 2.50GHz slash
stepping : 3 slash
microcode : 0x8a slash
cpu MHz : 425.134 slash
cache size : 4096 KB slash
physical id : 0 slash
siblings : 4 slash
core id : 1 slash
cpu cores : 2 slash
apicid : 2 slash
initial apicid : 2 slash
fpu : yes slash
fpu_exception : yes slash
cpuid level : 22 slash
wp : yes slash
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm
abm 3dnowprefetch intel_pt tpr_shadow vnmi flexpriority ept vpid
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap
clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp
hwp_notify hwp_act_window hwp_epp slash
bugs : slash
bogomips : 5184.00 slash
clflush size : 64 slash
cache_alignment : 64 slash
address sizes : 39 bits physical, 48 bits virtual slash
power management:

processor : 2 slash
vendor_id : GenuineIntel slash
cpu family : 6 slash
model : 78 slash
model name : Intel® Core™ i7-6500U CPU 2.50GHz slash
stepping : 3 slash
microcode : 0x8a slash
cpu MHz : 413.391 slash
cache size : 4096 KB slash
physical id : 0 slash
siblings : 4 slash
core id : 0 slash
cpu cores : 2 slash
apicid : 1 slash
initial apicid : 1 slash
fpu : yes slash
fpu_exception : yes slash
cpuid level : 22 slash
wp : yes slash
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm
abm 3dnowprefetch intel_pt tpr_shadow vnmi flexpriority ept vpid
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap
clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp
hwp_notify hwp_act_window hwp_epp slash
bugs : slash
bogomips : 5184.00 slash
clflush size : 64 slash
cache_alignment : 64 slash
address sizes : 39 bits physical, 48 bits virtual slash
power management:

processor : 3 slash
vendor_id : GenuineIntel slash
cpu family : 6 slash
model : 78 slash
model name : Intel® Core™ i7-6500U CPU 2.50GHz slash
stepping : 3 slash
microcode : 0x8a slash
cpu MHz : 399.902 slash
cache size : 4096 KB slash
physical id : 0 slash
siblings : 4 slash
core id : 1 slash
cpu cores : 2 slash
apicid : 3 slash
initial apicid : 3 slash
fpu : yes slash
fpu_exception : yes slash
cpuid level : 22 slash
wp : yes slash
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm
abm 3dnowprefetch intel_pt tpr_shadow vnmi flexpriority ept vpid
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap
clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp
hwp_notify hwp_act_window hwp_epp slash
bugs : slash
bogomips : 5184.00 slash
clflush size : 64 slash
cache_alignment : 64 slash
address sizes : 39 bits physical, 48 bits virtual slash
power management:

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 12:15pm]

Formatting, please. And you have not given specs regarding your storage
nor RAM.

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 12:29pm]

# dmidecode 3.0

Getting SMBIOS data from sysfs. slash
SMBIOS 2.8 present.

Handle 0x0026, DMI type 16, 23 bytes slash
Physical Memory Array slash
Location: System Board Or Motherboard slash
Use: System Memory slash
Error Correction Type: None slash
Maximum Capacity: 16 GB slash
Error Information Handle: No Error slash
Number Of Devices: 2

Handle 0x0027, DMI type 17, 40 bytes slash
Memory Device slash
Array Handle: 0x0026 slash
Error Information Handle: No Error slash
Total Width: Unknown slash
Data Width: Unknown slash
Size: No Module Installed slash
Form Factor: Unknown slash
Set: None slash
Locator: ChannelA-DIMM0 slash
Bank Locator: BANK 0 slash
Type: Unknown slash
Type Detail: None slash
Speed: Unknown slash
Manufacturer: Not Specified slash
Serial Number: Not Specified slash
Asset Tag: Not Specified slash
Part Number: Not Specified slash
Rank: Unknown slash
Configured Clock Speed: Unknown slash
Minimum Voltage: Unknown slash
Maximum Voltage: Unknown slash
Configured Voltage: Unknown

Handle 0x0028, DMI type 17, 40 bytes slash
Memory Device slash
Array Handle: 0x0026 slash
Error Information Handle: No Error slash
Total Width: Unknown slash
Data Width: Unknown slash
Size: No Module Installed slash
Form Factor: Unknown slash
Set: None slash
Locator: ChannelA-DIMM1 slash
Bank Locator: BANK 1 slash
Type: Unknown slash
Type Detail: None slash
Speed: Unknown slash
Manufacturer: Not Specified slash
Serial Number: Not Specified slash
Asset Tag: Not Specified slash
Part Number: Not Specified slash
Rank: Unknown slash
Configured Clock Speed: Unknown slash
Minimum Voltage: Unknown slash
Maximum Voltage: Unknown slash
Configured Voltage: Unknown

Handle 0x0029, DMI type 17, 40 bytes slash
Memory Device slash
Array Handle: 0x0026 slash
Error Information Handle: No Error slash
Total Width: 64 bits slash
Data Width: 64 bits slash
Size: 8192 MB slash
Form Factor: SODIMM slash
Set: None slash
Locator: ChannelB-DIMM0 slash
Bank Locator: BANK 2 slash
Type: DDR3 slash
Type Detail: Synchronous slash
Speed: 1600 MHz slash
Manufacturer: Samsung slash
Serial Number: 23156510 slash
Asset Tag: 9876543210 slash
Part Number: M471B1G73EB0-YK0 slash
Rank: 2 slash
Configured Clock Speed: 1600 MHz slash
Minimum Voltage: 1.25 V slash
Maximum Voltage: 1.35 V slash
Configured Voltage: 1.35 V

Handle 0x002A, DMI type 17, 40 bytes slash
Memory Device slash
Array Handle: 0x0026 slash
Error Information Handle: No Error slash
Total Width: Unknown slash
Data Width: Unknown slash
Size: No Module Installed slash
Form Factor: Unknown slash
Set: None slash
Locator: ChannelB-DIMM1 slash
Bank Locator: BANK 3 slash
Type: Unknown slash
Type Detail: None slash
Speed: Unknown slash
Manufacturer: Not Specified slash
Serial Number: Not Specified slash
Asset Tag: Not Specified slash
Part Number: Not Specified slash
Rank: Unknown slash
Configured Clock Speed: Unknown slash
Minimum Voltage: Unknown slash
Maximum Voltage: Unknown slash
Configured Voltage: Unknown

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 12:33pm]

8GB is the amount of (GPU) memory required for training on a GPU. I
don't think you are running out of disk space, but out of memory. How
much swap do you have configured?

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 1:50pm]

slash $swapon -s

Filename Type Size Used Priority
/dev/sda4 partition 499708 310104 -1

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 3:28pm]

So, 8GB of memory and 500M of swap? That's likely not enough, I fear.
Try to reduce the size of your dataset: --limit_train x (and same for
dev and test sets).

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 4:01pm]

Okay. I will try that. Thank you for your response. But just for
knowledge, if I don't limit the dataset size then what should be my
minimum disk space. can you tell me?

As I am planning to buy a new laptop, I can have this information in
mind Because I am planning to dedicate this laptop just for Speech
recognition project.

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 4:03pm]

There are too many variables here in your problem, it depends on the
dataset, mostly, I guess. Also, your first message is still incorrectly
formatted, so it's possible that valuable information to understand the
stacktrace are missing.

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 4:13pm]

Also, I don't think you can expect to train on full Common Voice dataset
(hundreds of hours of audio) on a single laptop without a GPU: training
that amount of data on four dedicated TITANX GPUs would already take one
full week.

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 4:17pm]

Yes, I am also planning to get a laptop with GPU. This is the reason I
am upgrading to next laptop. So few specifications of hardware can help
we look towards it.

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 4:19pm]

Also, [ slash erogol](
spotted that it is actually when reading the model that it is failing.
Obviously not enough memory. Heap consumption on loading the protobuf
model is peaking at slash ~4.5GiB of memory. That's just to read the file.

I would
strongly advise against running serious training on a laptop: their
GPUs are not as powerful as desktop ones, the heat constraints are
really going to be a big issue. You might kill your hardware.

[Archived Post]

[JRMeyer]

>>> meghagowda5193
[February 22, 2018, 4:25pm]

Okay, thank you
![:slight_smile:](
If I use limit parameter, how does it take the dataset? once the limit I
specified is taken, the next set will be automatically taken?. Is there
any thread on this topic where I can go through.

[Archived Post]

[JRMeyer]

>>> lissyx
[February 22, 2018, 4:43pm]

As much as I can remember, the limit is about the overall set of data.

[Archived Post]