Getting bad_alloc Error on N-1 Strided using Certain Block Sizes #251
Comments
|
So here are some preliminary findings: On smog: aprun -n 16 ./fs_test.smog.x -target plfs://users/dbonnie/plfs_n1/out.%s -io mpi -time 5 -barriers aopen -shift -type 2 -sync -size 1 -nodb -strided 1 A single node writing 1 byte blocks will cause this with only a 5 second write time when using a glib backend (because glib is MUCH faster than posix). On rrz: mpirun -n 16 ./fs_test.localhost.x -strided 1 -target plfs:/users/dbonnie/plfs_n1/out.%s -io mpi -time 30 -barriers aopen -shift -type 2 -sync -size 1 -nodb Causes a failure in 30 seconds (because PanFS even with glib is much slower than smog's Lustre). I have the sneaking feeling this is an index bug or something because these runs create millions and millions of index entries. I have confirmed that this also occurs with compress_contiguous set to false in the plfsrc. My plfsrc files contain very similar entries between the two systems: rrz:
smog:
So first thought is this may be in older releases as well but it's hard to test. I think Alfred or I will try to confirm that soon with a longer test suite for 2.3. |
|
Also, the errors change between runs which makes me think memory corruption in some form or another although bad_alloc seems to be quite common. |
|
I've narrowed this down a bit more -- I'm almost positive the problem lies in the indexes when they get to a certain size. So far the following causes the error: 128 procs (16 nodes), 70000 1-byte writes each proc. Each index is 3.3 MB, total memory usage per node for indexes should be about 3.4 GB per node(each has 16 GB)...so this probably shouldn't fail. 16 procs (2 nodes), 400000 1-byte writes each proc. Each index is 28 MB, total memory usage for indexes should be 3.5 GB per node (each has 16 GB)...so this probably shouldn't fail either. 1 proc, 25000000 1 byte writes. Total index is 1.2 GB. So clearly something fishy is going on here. I'm going to test at larger scales and with an older version of PLFS to see if I can narrow this down a bit better. |
|
Updates: 512 procs (64 nodes), 15000 1 byte writes each. Each index is 704 KB, total usage per node is ~2.75 GB. All of the above are from RRZ. Smog is a bit different but fails at similar counts per node (though memory usage for indexes at 1088 procs on Smog results in ~12 GB...way more than what fails on RRZ). The strided access is what's causing every write to be a separate index entry but it shouldn't be crashing where it is. |
|
Single node on RRZ: 8 procs, 1000000 objects each, roughly the same memory usage in aggregate to crash. |
|
Confirmed that the exact same workload fails on PLFS 2.3...about 46 MB of index per proc, so loading it on a single node uses around 2.9 GB of RAM. There's 16 GB on a node on RRZ. |
|
Im just curious how our regression failed to find this in 2.3 but did for On Mon, Jun 3, 2013 at 5:24 PM, David Bonnie notifications@github.comwrote:
|
|
2.4 is faster than 2.3. The test was right on the edge of failing with 2.3 in terms of object counts after 300 seconds of writing. |
|
That's pretty awesome actually. Why faster? On Jun 3, 2013, at 5:30 PM, David Bonnie notifications@github.com wrote:
|
|
Various little things all adding up I think. I might have jumped the gun a bit on 2.3 failing - I had linked things weirdly for the my custom MPI and I think I hooked into that accidentally for my 2.3 test. I'll update when I figure out what I managed to muck up. :) |
|
Okay - fixed my install and I've gotten 2.3 to fail with OpenMPI 1.6.4 on RRZ in the exact same manner as 2.4rc2. |
|
Watching a run on a single node I was able to see what's happening.
Here's what top shows as the memory is rising on one proc: 11459 dbonnie 20 0 134m 13m 4636 R 100.2 0.1 2:45.79 fs_test.localho That proc will rise to just over 5.6 GB, then the following happens: 11525 dbonnie 20 0 1581m 1.4g 4860 R 100.6 9.1 3:47.22 fs_test.localho The proc that was using 5+ GB of memory starts to drop in memory usage slightly but every other proc's memory usage skyrockets to the point of a crash. |
|
How did this pass regression in 2.3 then? No longer sounds like race caused by increased performance. On Jun 4, 2013, at 2:40 PM, David Bonnie notifications@github.com wrote:
|
|
The workload I established isn't time-based so it hits it regardless of speed. |
|
How big are the droppping files on the backend after the write chuck On Tue, Jun 04, 2013 at 11:40:26AM -0700, David Bonnie wrote:
|
|
Here's the contents of the hostdir: -rw-r--r-- 1 dbonnie dbonnie 0 Jun 4 12:32 dropping.data.1370370765.643511.rrz004.localdomain.0 I wouldn't imagine this is enough data to cause an issue unless the index entries expand drastically in-memory. |
|
On Tue, Jun 04, 2013 at 12:29:28PM -0700, David Bonnie wrote:
Yeah, I don't see how you get to 5GB from that. and if you complete chuck |
|
It's definitely all hitting the backend before doing the open call (I did a write-only test and the results match up with the above backend listing). |
|
On Tue, Jun 04, 2013 at 12:43:50PM -0700, David Bonnie wrote:
what happens if you generate a global index after a write chuck |
|
Flattening the index results in a 367 MB global index. Trying to read it now. |
|
Looks like the same behavior on read. Is there a way to use the global index when doing an MPI read or is that handled automatically? |
|
On Tue, Jun 04, 2013 at 02:25:41PM -0700, David Bonnie wrote:
It should look for the global index first and use it if it is found. you could try writing a simple plfs API program that does a e.g. something like main() { chuck |
|
So trying to use the "plfs_flatten_close" MPI-IO hint results in an instant crash on close in fs_test due to a seg fault. The backtrace: Mostly just putting that here for future reference. Still digging. |
|
The "flatten" flag for fs_test is only valid if you use I/O type of PLFS. This is only used if the "io" is 'plfs'. If non-zero it tells PLFS toflatten the index file for better performance."flatten" : [ 1 ],I'm not sure about any hint to pass. I don't see anything in the code or template files that mentions using such a hint. So, you could try to change the io type to plfs and then turn the flatten flag on and see if that works. Brett |
|
[ignore this]Actually, using fs_test "--flatten 1" results in a crash too, even with a fairly small workload.[/ignore this] EDIT: I found the flatten flag by looking in ad_plfs -- so it should work for MPI-IO, no? |
|
Using io of "plfs"? I didn't look at the code closely, but if you use io that is not "plfs" you should just get an exit from the application without any I/O happening and a note that you can use flatten with an io that is not "plfs". Brett |
|
Using either plfs or mpi the -flatten 1 flag crashes with a seg fault. Using the mpi-io hint "plfs_flatten_close" clearly starts the flattening process but then crashes with a seg fault as well. |
|
I'm assuming you've checked that workload=n-1 in the plfsrc. Flatten John On Wed, Jun 5, 2013 at 11:27 AM, David Bonnie notifications@github.comwrote:
|
|
OK. So, it's a bug for fs_test to let you use the --flatten flag with io of "mpi". Do you see this line printed? "INFO RANK[%d]: Flattening plfs index where rank is zero (0)? Only rank zero is allowed to call plfs_flatten_index. Look at the flatten_file function in fs_test.c. Put some debug statements in there to make sure the target path is correct to determine if fs_test is messing up or plfs_flatten_index is messing up. Thanks, |
|
Yeah it prints that for rank 0 only (though with the mpi-io hint I believe it's trying to flatten on all ranks). This is to an N-1 mount, single backend, nothing crazy in terms of parameters. I'll keep digging. |
|
If the hint is trying to flatten on all ranks, then that must be in the ad_* code because fs_test's flatten_file will only call plfs_flatten_index if it is rank 0 or if it's N-N. Brett |
|
A note on memory usage (running plfs_map) for reading an index from two procs writing 12,000,000 objects of 1-byte each, strided. Each index dropping is ~550 MB. Someone correct me if I'm wrong, but shouldn't I expect to see approximately 550 MB * [2 nodes] in terms of memory usage when reading the index with plfs_map? Instead, I see nearly 10 GB. Reading back the index with "--uniform 0" to only read a single rank also results in a huge amount of memory usage in relation to the size of the index (4.6 GB). |
|
Sounds like a gigantic memory leak. John On Jun 5, 2013, at 3:57 PM, David Bonnie notifications@github.com wrote:
|
|
yeah. if it happens in plfs_map that's cool because it chuck On Wed, Jun 05, 2013 at 01:07:17PM -0700, John Bent wrote:
|
|
I'm tracking down where the memory is being allocated. I don't think it's a leak (as in allocated but not freed) but more in how the memory is handled for the global index creation. |
|
Just an update on this - to reduce memory usage there really just needs to be a rewrite of the global index creation (I don't believe there is really a bug here, just higher memory usage than I initially expected). Perhaps a shared-memory approach on a per-node basis? |
|
Have you been able to quantify the memory allocation? earlier you said "Flattening the index results in a 367 MB global index" chuck On Mon, Jun 10, 2013 at 10:18:34AM -0700, David Bonnie wrote:
|
|
The in-memory map results in most of the memory pressure (over 90% of the in-use memory is from the global index data structures). The on-disk size is roughly tripled in-memory per-process so that coupled with the various buffers and mmap'd files causes memory exhaustion with the cases I highlighted above. A lot of the memory usage when using plfs_map initially was caused by a massive string construction for the output. I removed that and memory usage fell dramatically (though was still higher than I initially expected) to the quantities above. |
|
What is the ratio of in-memory size to on-disk size? John On Jun 10, 2013, at 12:32 PM, David Bonnie <notifications@github.commailto:notifications@github.com> wrote: The in-memory map results in most of the memory pressure (over 90% of the in-use memory is from the global index data structures). The on-disk size is roughly tripled in-memory per-process so that coupled with the various buffers and mmap'd files causes memory exhaustion with the cases I highlighted above. A lot of the memory usage when using plfs_map initially was caused by a massive string construction for the output. I removed that and memory usage fell dramatically (though was still higher than I initially expected) to the quantities above. — |
|
In my limited tests it's 2-3 times larger in-memory than on-disk. |
In running a sweep of N-1 strided performance using fs_test and plfs version 2.4rc2 on rrz, I got bad_alloc errors when opening for read. This happened using a 32k block size. I got some errors and some successes using other block sizes.
Here is the command line I used:
mpirun -n 32 /users/atorrez/Testing-ext/fs_test/fs_test/fs_test.rrz.x -strided 1 -target plfs:/var/tmp/plfs.atorrez1/atorrez/out.%s -io mpi -time 300 -barriers aopen -shift -type 2 -sync -size 32768
Dave Bonnie is seeing the same errors on smog and rrz using smaller block sizes and shorter -time parameters.
The text was updated successfully, but these errors were encountered: