mdb_v8 is sometimes unable to load core files generated on Linux

[kelapure]

I have tried multiple times to get the mdb debugger working with core dumps generated on linux to no avail.

The app in question uses node 0.12.7.The node.js process runs in a Cloud Foundry container. The debugging was done on a SmartOS ec2 box.

The latest mdb_V8 module released is 1.1.2 from November 2015. I even tried building from the master and loading it to no avail. Can someone please illustrate how to do this with a simple instructions.

root@ip-10-152-178-106.ec2.internal:/root/dmp# mdb node-v0.12.7-linux-x64/bin/node node.0.core

mdb: warning: librtld_db failed to initialize; shared library information will not be available

> ::load /root/dmp/mdb_v8_amd64.so
mdb_v8 version: 1.1.1 (release, from 28cedf2)
mdb: failed to determine V8 version: no mapping for address

> ::jsstack
native: _ZN2v88internal7Isolate13IsErrorObjectENS0_6HandleINS0_6ObjectEE...
native: _ZN2v88internal2IC19UpdatePolymorphicICENS0_6HandleINS0_4NameEEE...
native: _ZN2v88internal16HStoreNamedField20HasOutOfBoundsAccessEi+0x21
        (1 internal frame elided)

How does one sanity check the inputs to the mdb. Is mdb_v8 module broken for linux cores.

[davepacheco]

Sorry, I replied on nodejs/node-v0.1x-archive#9214 before I saw this. It would probably be better to keep that discussion here. If you could make the core file available, that would be helpful.

[misterdjules]

I recently ran into a similar, and maybe identical, issue when trying to load a core files generated from a node process running the 64 bits binary of node version 4.2.2 on a RHEL 6 Linux distribution with a Linux kernel at version 2.6.32-573.7.1.el6.x86_64.

If we look at the node binary that the process was running when it generated that core file to determine at which address the V8 version symbols are located, we get the following output:

[jgilli@dev ~]$ readelf -Wa node-v4.2.2-linux-x64/bin/node | grep Version | grep -E 'major|minor|patch|build' | c++filt 
   407: 00000000018e4250     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::patch_
   794: 00000000018e4280     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::major_
 15695: 00000000018e4270     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::minor_
 25600: 00000000018e4260     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::build_
 20409: 00000000018e4260     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::build_
 30643: 00000000018e4250     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::patch_
 41183: 00000000018e4280     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::major_
 43869: 00000000018e4270     4 OBJECT  GLOBAL DEFAULT   26 v8::internal::Version::minor_
[jgilli@dev ~]$ 

So we can tell that the addresses of these symbols are between 0x18e4250 and 0x18e4280, which matches what readelf outputs about the .data section:

[jgilli@dev ~]$ readelf -Wa node-v4.2.2-linux-x64/bin/node | grep ' .data '
  [26] .data             PROGBITS        00000000018cd9e0 12cd9e0 0176b8 00  WA  0   0 32
   03     .init_array .fini_array .jcr .data.rel.ro .dynamic .got .got.plt .data .bss 
[jgilli@dev ~]$

The .data section is between addresses 0x18cd9e0 and 0x18e5098 (0x18cd9e0 + 0x176b8), and so all the V8 versions symbols are in this section, which makes sense.

Now we can take a look at the core file and see if these memory regions are present. I can't share the core file itself publicly, but here are the memory areas that readelf -a outputs from this core:

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  NOTE           0x00000000000020d8 0x0000000000000000 0x0000000000000000
                 0x0000000000003f0c 0x0000000000000000  R      1
  LOAD           0x0000000000005fe4 0x0000000000400000 0x0000000000000000
                 0x0000000000d7a000 0x0000000000d7a000  R E    1
  LOAD           0x0000000000d7ffe4 0x000000000137a000 0x0000000000000000
                 0x0000000000019000 0x0000000000019000  RW     1
  LOAD           0x0000000000d98fe4 0x0000000001393000 0x0000000000000000
                 0x000000000000e000 0x000000000000e000  RW     1
  LOAD           0x0000000000da6fe4 0x0000000002f03000 0x0000000000000000
                 0x0000000001966000 0x0000000001966000  RW     1
  LOAD           0x000000000270cfe4 0x0000003cb8400000 0x0000000000000000
                 0x0000000000020000 0x0000000000020000  R E    1
  LOAD           0x000000000272cfe4 0x0000003cb861f000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R      1
  LOAD           0x000000000272dfe4 0x0000003cb8620000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x000000000272efe4 0x0000003cb8621000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x000000000272ffe4 0x0000003cb8800000 0x0000000000000000
                 0x000000000018a000 0x000000000018a000  R E    1
  LOAD           0x00000000028b9fe4 0x0000003cb8b8a000 0x0000000000000000
                 0x0000000000004000 0x0000000000004000  R      1
  LOAD           0x00000000028bdfe4 0x0000003cb8b8e000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x00000000028befe4 0x0000003cb8b8f000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000028c3fe4 0x0000003cb8c00000 0x0000000000000000
                 0x0000000000017000 0x0000000000017000  R E    1
  LOAD           0x00000000028dafe4 0x0000003cb8e17000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R      1
  LOAD           0x00000000028dbfe4 0x0000003cb8e18000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x00000000028dcfe4 0x0000003cb8e19000 0x0000000000000000
                 0x0000000000004000 0x0000000000004000  RW     1
  LOAD           0x00000000028e0fe4 0x0000003cb9000000 0x0000000000000000
                 0x0000000000002000 0x0000000000002000  R E    1
  LOAD           0x00000000028e2fe4 0x0000003cb9202000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R      1
  LOAD           0x00000000028e3fe4 0x0000003cb9203000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x00000000028e4fe4 0x0000003cb9400000 0x0000000000000000
                 0x0000000000007000 0x0000000000007000  R E    1
  LOAD           0x00000000028ebfe4 0x0000003cb9606000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R      1
  LOAD           0x00000000028ecfe4 0x0000003cb9607000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x00000000028edfe4 0x0000003cb9800000 0x0000000000000000
                 0x0000000000083000 0x0000000000083000  R E    1
  LOAD           0x0000000002970fe4 0x0000003cb9a82000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R      1
  LOAD           0x0000000002971fe4 0x0000003cb9a83000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x0000000002972fe4 0x0000003cba800000 0x0000000000000000
                 0x0000000000016000 0x0000000000016000  R E    1
  LOAD           0x0000000002988fe4 0x0000003cbaa15000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x0000000002989fe4 0x000000b5ae500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002a89fe4 0x00000427b9300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002b89fe4 0x000004b451500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002c89fe4 0x0000070d60000000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002d89fe4 0x000008a5a4400000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002e89fe4 0x0000098989300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000002f89fe4 0x00000a12a9000000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003089fe4 0x00000b84f93fa000 0x0000000000000000
                 0x0000000000400000 0x0000000000400000  RW     1
  LOAD           0x0000000003489fe4 0x00000c226f100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003589fe4 0x00000c9c42500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003689fe4 0x00000cc1ce200000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003789fe4 0x00000cc25f500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003889fe4 0x00000cdac8e00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003989fe4 0x00000ce2af700000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003a89fe4 0x00000d075a100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003b89fe4 0x00000de9f0100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003c89fe4 0x00000e4997c00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003d89fe4 0x00000f2a99b00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003e89fe4 0x00000f8e1b300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000003f89fe4 0x00000fb0b6100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004089fe4 0x00000ff752800000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004189fe4 0x000010e2f8500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004289fe4 0x000010e7cd700000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004389fe4 0x0000117c3eb00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004489fe4 0x000016a447000000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004589fe4 0x0000181780f00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004689fe4 0x0000190db7b00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004789fe4 0x0000197afc780000 0x0000000000000000
                 0x0000000000020000 0x0000000000020000  RW     1
  LOAD           0x00000000047a9fe4 0x00001a3220300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000048a9fe4 0x00001b3043e00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000049a9fe4 0x00001b7216800000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004aa9fe4 0x00001b7fc8d00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004ba9fe4 0x00001d67b8900000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004ca9fe4 0x00001f552f600000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004da9fe4 0x00001fe455d00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004ea9fe4 0x00002003dbf00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000004fa9fe4 0x000020106f300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000050a9fe4 0x00002063f6200000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000051a9fe4 0x000020b92a800000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000052a9fe4 0x0000215799300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000053a9fe4 0x000023332fda8000 0x0000000000000000
                 0x0000000000040000 0x0000000000040000  RW     1
  LOAD           0x00000000053e9fe4 0x00002356d4b00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000054e9fe4 0x0000239ead600000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000055e9fe4 0x0000264054b00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000056e9fe4 0x00002660b0100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000057e9fe4 0x0000268b7d900000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000058e9fe4 0x000027af64500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000059e9fe4 0x0000283339600000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005ae9fe4 0x00002a404e400000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005be9fe4 0x00002b6580c00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005ce9fe4 0x00002c12e2300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005de9fe4 0x00002c87d8100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005ee9fe4 0x00002e67d8a00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000005fe9fe4 0x00002ed786000000 0x0000000000000000
                 0x0000000001000000 0x0000000001000000  RW     1
  LOAD           0x0000000006fe9fe4 0x00002ed787000000 0x0000000000000000
                 0x0000000001000000 0x0000000001000000  RW     1
  LOAD           0x0000000007fe9fe4 0x00002edfcd100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000080e9fe4 0x00002f371bb00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000081e9fe4 0x00002f8025a00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000082e9fe4 0x0000312ef9d00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000083e9fe4 0x000032157a100000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000084e9fe4 0x00003249592fd000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R E    1
  LOAD           0x00000000084eafe4 0x00003364af500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000085eafe4 0x000033fe50300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000086eafe4 0x0000340614b00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000087eafe4 0x0000343418c00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000088eafe4 0x0000343d56e00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000089eafe4 0x000035491d300000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000008aeafe4 0x000039153a000000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000008beafe4 0x0000391bd1400000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000008ceafe4 0x00003983cd600000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000008deafe4 0x00003b04aad00000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000008deffe4 0x00003b04aad06000 0x0000000000000000
                 0x0000000000003000 0x0000000000003000  RWE    1
  LOAD           0x0000000008df2fe4 0x00003b04aae00000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000008df7fe4 0x00003b04aae06000 0x0000000000000000
                 0x0000000000002000 0x0000000000002000  RWE    1
  LOAD           0x0000000008df9fe4 0x00003b04aaf00000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000008dfefe4 0x00003b04aaf06000 0x0000000000000000
                 0x0000000000002000 0x0000000000002000  RWE    1
  LOAD           0x0000000008e00fe4 0x00003b04ab000000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000008e05fe4 0x00003b04ab006000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x0000000008efefe4 0x00003b04ab600000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000008f03fe4 0x00003b04ab606000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x0000000008ffcfe4 0x00003b04ab700000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x0000000009001fe4 0x00003b04ab706000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000090fafe4 0x00003b04ab800000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000090fffe4 0x00003b04ab806000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000091f8fe4 0x00003b04ab900000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000091fdfe4 0x00003b04ab906000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000092f6fe4 0x00003b04ac100000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000092fbfe4 0x00003b04ac106000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000093f4fe4 0x00003b04ac300000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000093f9fe4 0x00003b04ac306000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000094f2fe4 0x00003b04ac800000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000094f7fe4 0x00003b04ac806000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000095f0fe4 0x00003b04ad100000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000095f5fe4 0x00003b04ad106000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000096eefe4 0x00003b04ad400000 0x0000000000000000
                 0x0000000000005000 0x0000000000005000  RW     1
  LOAD           0x00000000096f3fe4 0x00003b04ad406000 0x0000000000000000
                 0x00000000000f9000 0x00000000000f9000  RWE    1
  LOAD           0x00000000097ecfe4 0x00003ce604400000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000098ecfe4 0x00003d66b1d00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x00000000099ecfe4 0x00003d8a56f00000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000009aecfe4 0x00003e6823500000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000009becfe4 0x00003f6649200000 0x0000000000000000
                 0x0000000000100000 0x0000000000100000  RW     1
  LOAD           0x0000000009cecfe4 0x00007f61c4000000 0x0000000000000000
                 0x0000000000d3a000 0x0000000000d3a000  RW     1
  LOAD           0x000000000aa26fe4 0x00007f61cc000000 0x0000000000000000
                 0x00000000002ab000 0x00000000002ab000  RW     1
  LOAD           0x000000000acd1fe4 0x00007f61d0000000 0x0000000000000000
                 0x0000000000364000 0x0000000000364000  RW     1
  LOAD           0x000000000b035fe4 0x00007f61d4000000 0x0000000000000000
                 0x00000000002c9000 0x00000000002c9000  RW     1
  LOAD           0x000000000b2fefe4 0x00007f61d8b65000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000bcfefe4 0x00007f61d9566000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000c6fefe4 0x00007f61d9f67000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000d0fefe4 0x00007f61da968000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000dafefe4 0x00007f61db369000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000e4fefe4 0x00007f61dbd6a000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000eefefe4 0x00007f61dc76b000 0x0000000000000000
                 0x0000000000a00000 0x0000000000a00000  RW     1
  LOAD           0x000000000f8fefe4 0x00007f61dd16c000 0x0000000000000000
                 0x0000000000a05000 0x0000000000a05000  RW     1
  LOAD           0x0000000010303fe4 0x00007f61ddb71000 0x0000000000000000
                 0x00000000000eb000 0x00000000000eb000  R E    1
  LOAD           0x00000000103eefe4 0x00007f61dde5b000 0x0000000000000000
                 0x0000000000008000 0x0000000000008000  R      1
  LOAD           0x00000000103f6fe4 0x00007f61dde63000 0x0000000000000000
                 0x0000000000002000 0x0000000000002000  RW     1
  LOAD           0x00000000103f8fe4 0x00007f61dde65000 0x0000000000000000
                 0x0000000000016000 0x0000000000016000  RW     1
  LOAD           0x000000001040efe4 0x00007f61dde84000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  RW     1
  LOAD           0x000000001040ffe4 0x00007ffec143a000 0x0000000000000000
                 0x0000000000015000 0x0000000000015000  RW     1
  LOAD           0x0000000010424fe4 0x00007ffec14ec000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R E    1
  LOAD           0x0000000010425fe4 0xffffffffff600000 0x0000000000000000
                 0x0000000000001000 0x0000000000001000  R E    1

It seems that there's no mapping in this core file that corresponds to a region that would include the .data section with the V8 version data, as there's no mapping that includes addresses between 0x18e4250 and 0x18e4280.

To confirm that this is what's going on, here's a DTrace script that I used to understand what mdb and mdb_v8 were doing when trying to read the V8 version symbols:

#!/usr/sbin/dtrace -Fs

#pragma D option quiet

pid$target:mdb:mdb_lookup_by_name:entry
{
        printf("looking up symbol %s\n", copyinstr(arg0));
}

pid$target:mdb:mdb_lookup_by_name:return
{
        printf("%d\n", arg1); 
}

pid$target:mdb:mdb_readsym:entry 
{
        printf("reading symbol %s\n", copyinstr(arg2));
}

pid$target:mdb:mdb_readsym:return
{
        printf("%d\n", arg1);
}

pid$target:mdb:mdb_tgt_readsym:entry /arg4 == -1/
{
        printf("reading symbol %s with MDB_TGT_OBJ_EVERY\n", copyinstr(arg5));
}

pid$target:mdb:pt_lookup_by_name_thr:entry
{
        printf("looking for name %s\n", copyinstr(arg2));
}

pid$target:mdb:pt_lookup_by_name_thr:return
{
        printf("%d\n", arg1);
}

pid$target:mdb:pt_vread:entry
{
        printf("size: %d, addr: %d\n", arg2, arg3); 
}

pid$target:mdb:pt_vread:return
{
        printf("%d\n", arg1); 
}

pid$target:libproc.so.1:Paddr2mptr:entry
{
        printf("addr: %d\n", arg1);
}

pid$target:libproc.so.1:Paddr2mptr:return
{
        printf("%d\n", arg1);
}

pid$target:libproc.so.1:Pxlookup_by_name:entry
{
        printf("symbol name: %s\n", copyinstr(arg3));
}

pid$target:libproc.so.1:Pxlookup_by_name:return
{
    printf("%d\n", arg1);
}

When running the following command, where core is the core file which mdb_v8 fails to load:

$ echo "::load /home/jgilli/mdb_v8/build/amd64/mdb_v8.so" | mdb node-v4.2.2-linux-x64/bin/node core

we get the following output from the DTrace script mentioned above:

[root@dev /home/jgilli]# ./mdb_v8.d -p $(pgrep mdb)
CPU FUNCTION                                 
  2  -> mdb_readsym                           reading symbol _ZN2v88internal7Version6major_E
  2    -> mdb_tgt_readsym                     reading symbol _ZN2v88internal7Version6major_E with MDB_TGT_OBJ_EVERY
  2      -> pt_lookup_by_name_thr             looking for name _ZN2v88internal7Version6major_E
  2        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  2          -> Paddr2mptr                    addr: 0
  2          <- Paddr2mptr                    0
  2        <- Pxlookup_by_name                4294967295
  2        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  2          -> Paddr2mptr                    addr: 0
  2          <- Paddr2mptr                    0
  2        <- Pxlookup_by_name                4294967295
  2      <- pt_lookup_by_name_thr             0
  2      -> pt_vread                          size: 4, addr: 26100352
  2        -> Paddr2mptr                      addr: 26100352
  2        <- Paddr2mptr                      0
  2      <- pt_vread                          -1
  2    <- mdb_readsym                         -1
  2    -> mdb_lookup_by_name                  looking up symbol _ZN2v88internal7Version6major_E
  2      -> pt_lookup_by_name_thr             looking for name _ZN2v88internal7Version6major_E
  2        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  2          -> Paddr2mptr                    addr: 0
  2          <- Paddr2mptr                    0
  2        <- Pxlookup_by_name                4294967295
  2        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  2          -> Paddr2mptr                    addr: 0
  2          <- Paddr2mptr                    0
  2        <- Pxlookup_by_name                4294967295
  2      <- pt_lookup_by_name_thr             0
  2    <- mdb_lookup_by_name                  0
  2    -> Paddr2mptr                          addr: 26100352
  2    <- Paddr2mptr                          0
  2    -> Paddr2mptr                          addr: 26100352
  2    <- Paddr2mptr                          0

We can see that mdb fails to read from address 26100352, which is 0x18e4280 and corresponds to the address we found reading from the node binary with readelf to find the address of the V8 major version symbol.

Now, let's generate a core file from a node process running node v4.2.2 (same version we used previously) on a very similar operating system, a RHEL 6 Linux system running a Linux kernel at version 2.6.32-431.29.2.el6.x86_64.

Using readelf -a on this core file, we can see that it includes memory regions that span the addresses for the V8 versioning data:

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  NOTE           0x00000000000012a0 0x0000000000000000 0x0000000000000000
                 0x0000000000001338 0x0000000000000000         0
  LOAD           0x0000000000003000 0x0000000000400000 0x0000000000000000
                 0x0000000000001000 0x00000000012cd000  R E    1000
  LOAD           0x0000000000004000 0x00000000018cc000 0x0000000000000000
                 0x000000000001a000 0x000000000001a000  RW     1000
  LOAD           0x000000000001e000 0x00000000018e6000 0x0000000000000000
                 0x0000000000011000 0x0000000000011000  RW     1000

When using mdb_v8 to inspect this core file we get:

[jgilli@dev ~]$ echo "::load /home/jgilli/mdb_v8/build/amd64/mdb_v8.so" | mdb node-v4.2.2-linux-x64/bin/node ~/core-rhel6-4.2.2-x64 
mdb: warning: librtld_db failed to initialize; shared library information will not be available
mdb_v8 version: 1.1.2 (dev)
V8 version: 4.5.103.35
Autoconfigured V8 support from target
C++ symbol demangling enabled

which means that the V8 versioning info was found. The DTrace script mentioned above outputs the following (truncated):

[root@dev /home/jgilli]# ./mdb_v8.d -p $(pgrep mdb)
CPU FUNCTION                                 
  3  -> mdb_readsym                           reading symbol _ZN2v88internal7Version6major_E
  3    -> mdb_tgt_readsym                     reading symbol _ZN2v88internal7Version6major_E with MDB_TGT_OBJ_EVERY
  3      -> pt_lookup_by_name_thr             looking for name _ZN2v88internal7Version6major_E
  3        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  3        <- Pxlookup_by_name                4294967295
  3        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  3        <- Pxlookup_by_name                4294967295
  3        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  3        <- Pxlookup_by_name                4294967295
  3        -> Pxlookup_by_name                symbol name: _ZN2v88internal7Version6major_E
  3        <- Pxlookup_by_name                4294967295
  3      <- pt_lookup_by_name_thr             0
  3      -> pt_vread                          size: 4, addr: 26100352
  3        -> Paddr2mptr                      addr: 26100352
  3        <- Paddr2mptr                      5550224
  3      <- pt_vread                          4
  3    <- mdb_readsym   

which shows mdb reading from the same address (26100352, or 0x18e4280) and actually succeeding at reading 4 bytes from that address.

So it seems that it might be possible that for two identical binaries, on very similar platforms, some core files do not contain the .data segments needed to find the symbols that need to be read by mdb_v8 to properly inspect the core file.

I'm not sure what could be causing this. We talked about the possibility that /proc/pid/coredump_filter was set, on the system where the core file generated was missing some segment, to a value that would prevent some pages to be written to disk when dumping core, but I've been unable to verify that theory so far.

We also talked about having mdb or mdb_v8 attempt to read symbols that are missing from the core file from the binary itself, where they are always present. I haven't tried to implement that yet.

[kelapure]

I don't see any Javascript frames after parsing of the coredump:

mdb ./node/bin/node ./core.16

> ::load mdb_v8/build/amd64/mdb_v8.so
mdb_v8 version: 1.1.2 (dev)
V8 version: 3.28.71.19
Autoconfigured V8 support from target
> ::jsstack
native: uv__io_poll+0x19c
native: uv_run+0x166
native: _ZN4node5StartEiPPc+0x181
        (1 internal frame elided)

You will find all the artifacts used for debugging and reproducing the defect node-dump-artifacts

I built the mdb_v8 on a Joyent Instance (nodejs 15.1.2).

[yunong]

@kelapure This doesn't mean mdb_v8 isn't working. It could be because your Node application isn't doing anything. If there's nothing active, then all you'll see on the stack is just libuv polling as this is expected behaviour. Have you tried to take a core dump of a Node process hot on CPU? Take a core dump of a simple Node process that runs an infinite for loop, and see if you can see JS stacks.

[kelapure]

@yunong ah interesting ... I will run an infinite loop app and then take a core dump when it is stuck. Thanks for the tip. More forthcoming ....

[kelapure]

@yunong you are right. I recreated an infinite loop and took a core dump of the node.js process. I can now can see the JS frames in addition to the native frames. I built mdb_v8.so from master

[root@6de24ae5-f178-4d11-aa0f-c027c81e9f84 ~/defect]# mdb node/bin/node core.16
mdb: warning: librtld_db failed to initialize; shared library information will not be available

> ::load mdb_v8/build/amd64/mdb_v8.so
mdb_v8 version: 1.1.2 (dev)
V8 version: 3.28.71.19
Autoconfigured V8 support from target

> ::jsstack
js:     <anonymous> (as <anon>)
js:     func2
js:     func1
js:     main
js:     <anonymous> (as <anon>)
        (1 internal frame elided)
js:     callbacks
        (1 internal frame elided)
js:     param
js:     pass
        (1 internal frame elided)
js:     <anonymous> (as Router._dispatch)
js:     router
js:     next
        (1 internal frame elided)
js:     multipart
js:     <anonymous> (as <anon>)
        (1 internal frame elided)
js:     <anonymous> (as <anon>)
js:     emit
js:     <anonymous> (as <anon>)
js:     _tickCallback
        (1 internal frame elided)
        (1 internal frame elided)
native: _ZN2v88internal9Execution4CallEPNS0_7IsolateENS0_6HandleINS0_6Ob...
native: _ZN2v88Function4CallENS_6HandleINS_5ValueEEEiPS3_+0xb7
native: _ZN4node9AsyncWrap12MakeCallbackEN2v86HandleINS1_8FunctionEEEiPN...
native: _ZN4node19StreamWrapCallbacks6DoReadEP11uv_stream_slPK8uv_buf_t1...
native: uv__read+0x22f
native: uv__stream_io+0x230
native: uv__io_poll+0x3ad
native: uv_run+0x166
native: _ZN4node5StartEiPPc+0x181

> ::jsstack -v
js:     <anonymous> (as <anon>)
          file: /home/vcap/app/server.js
          posn: line 44
          this: cfa7c950bf1 (<unknown>)

             39
             40 function func1() { func2(); }
             41
             42 function func2()
             43 {
             44         (function () {
             45                 for (;;)
             46                         ;
             47         })();
             48 }
             49
             50
             51 app.listen(process.env.PORT || config.port);
             52 console.log("Listening on port: " + (process.env.PORT || config.port));
....
....

This defect is now closed. The artifacts used to validate the working debug session are here
http://node-dumps.cfapps.pez.pivotal.io/

[kelapure]

This issue cannot be recreated with the latest mdb_v8.so built on the Joyent box.

For completeness, I was taking core dumps of the node.js runtime within Cloud Foundry setup by the Node.js buildpack. For detail see http://cloud.rohitkelapure.com/2016/04/debugging-nodejs-high-cpu-crash-and.html

[misterdjules]

The error message that was mentioned in the first comment:

> ::load /root/dmp/mdb_v8_amd64.so
mdb_v8 version: 1.1.1 (release, from 28cedf2)
mdb: failed to determine V8 version: no mapping for address

indicates that mdb_v8 was unable to find the V8 versioning symbols in the core file, which is required to inspect it properly. Later comments mention that mdb_v8 was able to inspect a Linux core file (the error message mentioned above is not present). This means that the versioning information, along with other required data, was present.

I would think that these different comments describe different results from different runs of mdb_v8 that load different core files. So the problem seems to be, as I described in my comment, that sometimes Linux core files are generated with some required information missing, and not that versions of mdb_v8 older than the latest version are unable to load some core files generated on Linux.

It seems to me we still don't know what causes this, and that this problem is not fixed. Thus, I think we should:

1. reopen this issue (which is what I'm doing now)
2. gather as much information as possible on the core file that mdb_v8 was unable to load, and how it was generated. @kelapure do you still have that core file around? If so, could you please share it with us by posting a URL we could use to download it? Is there a setup that allows you to reproduce this problem?
3. consider the approach of loading symbols that are missing from the core file from the binary itself

[robertkowalski]

@misterdjules i can confirm this is still an issue with core dumps from ubuntu created by node 8

i used mdb_v8_amd64.so v1.3

[misterdjules]

@robertkowalski Hey Robert, great to hear from you again!

Can you paste the whole output that you got from mdb_v8 in your case? I'd just want to make sure that the error you encountered is the same as the one mention in the original comment for this issue.

mdb_v8 currently does not support inspecting core dumps generated from node processes using Node.js at versions > 6.x regardless of on which platform those core dumps were generated, so I'm thinking that you may have encountered different errors.

[robertkowalski]

> ::load /root/mdbv8/mdb_v8_amd64.so
mdb_v8 version: 1.3.0 (release, from 28f2bfa)
mdb: failed to determine V8 version: operation not supported by target

ha, you are right, its even a different message

mdb_v8 currently does not support inspecting core dumps generated from node processes using Node.js at versions > 6.x

uh noes, yes i read that later in the issue tracker here. core dump tooling for node doesn't look well right now :/

[misterdjules]

@robertkowalski https://github.com/nodejs/llnode supports Node.js at versions > 8.x, but it doesn't provide the same functionality as mdb_v8.