NetBSD Problem Report #55533

From www@netbsd.org  Sun Aug  2 19:57:54 2020
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 295DB1A9217
	for <gnats-bugs@gnats.NetBSD.org>; Sun,  2 Aug 2020 19:57:54 +0000 (UTC)
Message-Id: <20200802195752.A6D191A923A@mollari.NetBSD.org>
Date: Sun,  2 Aug 2020 19:57:52 +0000 (UTC)
From: coypu@sdf.org
Reply-To: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Subject: nodejs fails with "out of memory" on aarch64 (with reproducer)
X-Send-Pr-Version: www-1.0

>Number:         55533
>Category:       kern
>Synopsis:       nodejs fails with "out of memory" on aarch64 (with reproducer)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 02 20:00:00 +0000 2020
>Last-Modified:  Fri Aug 14 17:20:01 +0000 2020
>Originator:     coypu
>Release:        NetBSD-9.x, NetBSD-9.99.47,..
>Organization:
>Environment:
NetBSD 9.99.47
>Description:

>How-To-Repeat:
arm64$ echo "global.gc();" > test.js

arm64$ gdb /usr/pkg/bin/node -q
Reading symbols from /usr/pkg/bin/node...
(gdb) handle SIGILL nostop noprint pass
Signal        Stop	Print	Pass to program	Description
SIGILL        No	No	Yes		Illegal instruction
(gdb) set pagination off
(gdb) break _exit
Breakpoint 1 at 0x200574d34
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>run --expose-gc test.js
>end
(gdb) run --expose-gc test.js
Starting program: /usr/pkg/bin/node --expose-gc test.js
[New LWP 2 of process 4614]
[New LWP 3 of process 4614]
[New LWP 4 of process 4614]
[New LWP 5 of process 4614]
[New LWP 6 of process 4614]
[New LWP 7 of process 4614]

Thread 1 "" hit Breakpoint 1, 0x0000f8149830148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 4483]
[New LWP 3 of process 4483]
[New LWP 4 of process 4483]
[New LWP 5 of process 4483]
[New LWP 6 of process 4483]
[New LWP 7 of process 4483]

Thread 1 "" hit Breakpoint 1, 0x0000f806fdd6148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 4261]
[New LWP 3 of process 4261]
[New LWP 4 of process 4261]
[New LWP 5 of process 4261]
[New LWP 6 of process 4261]
[New LWP 7 of process 4261]

Thread 1 "" hit Breakpoint 1, 0x0000f4dffd23148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 4784]
[New LWP 3 of process 4784]
[New LWP 4 of process 4784]
[New LWP 5 of process 4784]
[New LWP 6 of process 4784]
[New LWP 7 of process 4784]

Thread 1 "" hit Breakpoint 1, 0x0000f3ed9c53148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 4785]
[New LWP 3 of process 4785]
[New LWP 4 of process 4785]
[New LWP 5 of process 4785]
[New LWP 6 of process 4785]
[New LWP 7 of process 4785]

Thread 1 "" hit Breakpoint 1, 0x0000ffb4ec9b148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 766]
[New LWP 3 of process 766]
[New LWP 4 of process 766]
[New LWP 5 of process 766]
[New LWP 6 of process 766]
[New LWP 7 of process 766]

Thread 1 "" hit Breakpoint 1, 0x0000fffb7c9e148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 3949]
[New LWP 3 of process 3949]
[New LWP 4 of process 3949]
[New LWP 5 of process 3949]
[New LWP 6 of process 3949]
[New LWP 7 of process 3949]

Thread 1 "" hit Breakpoint 1, 0x0000fd736c9b148c in _exit () from /usr/lib/libc.so.12
[New LWP 2 of process 4631]
[New LWP 3 of process 4631]
[New LWP 4 of process 4631]
[New LWP 5 of process 4631]
[New LWP 6 of process 4631]

<--- Last few GCs --->


<--- JS stacktrace --->


#
# Fatal process OOM in insufficient memory to create an Isolate
#


Thread 1 "" received signal SIGTRAP, Trace/breakpoint trap.
0x0000000200fc97f4 in v8::base::OS::Abort() ()
(gdb) pmap
Undefined command: "pmap".  Try "help".
(gdb) bt
#0  0x0000000200fc97f4 in v8::base::OS::Abort() ()
#1  0x00000002008f6510 in v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) ()
#2  0x00000002008f6678 in v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) ()
#3  0x0000000200a7dd8c in v8::internal::Heap::ReserveSpace(std::vector<v8::internal::Heap::Chunk, std::allocator<v8::internal::Heap::Chunk> >*, std::vector<unsigned long, std::allocator<unsigned long> >*) ()
#4  0x0000000200d985b8 in v8::internal::DeserializerAllocator::ReserveSpace() ()
#5  0x0000000200da58e0 in v8::internal::StartupDeserializer::DeserializeInto(v8::internal::Isolate*) ()
#6  0x0000000200a2063c in v8::internal::Isolate::Init(v8::internal::ReadOnlyDeserializer*, v8::internal::StartupDeserializer*) ()
#7  0x0000000200da4650 in v8::internal::Snapshot::Initialize(v8::internal::Isolate*) [clone .part.176] ()
#8  0x000000020091ffec in v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) ()
#9  0x000000020062742c in node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<unsigned long, std::allocator<unsigned long> > const*) ()
#10 0x00000002005cc3e8 in node::Start(int, char**) ()
#11 0x000000020057903c in ___start ()
#12 0x0000fffff80b0af0 in _rtld_start () from /usr/libexec/ld.elf_so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)



Ktracing the parent gdb yields:

  4631      1 node     RET   mmap 264386832678912/0xf0755b3ec000
  4631      1 node     CALL  mmap(0x306c0000,0x7f000,PROT_NONE,0x1042<PRIVATE,NORESERVE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4631      1 node     RET   mmap 812384256/0x306c0000
  4631      1 node     CALL  mmap(0,0x1000,PROT_READ|PROT_WRITE,0x1002<PRIVATE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4631      1 node     RET   mmap 264386832674816/0xf0755b3eb000
  4631      1 node     CALL  mmap(0x26f00000,0x7f000,PROT_NONE,0x1042<PRIVATE,NORESERVE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4631      1 node     RET   mmap -1 errno 12 Cannot allocate memory
  4631      1 node     CALL  mmap(0x26f00000,0x7f000,PROT_NONE,0x1042<PRIVATE,NORESERVE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4631      1 node     RET   mmap -1 errno 12 Cannot allocate memory
  4473      1 gdb      CALL  mmap(0,0x1eb000,PROT_READ,0x2<PRIVATE,FILE,ALIGN=NONE>,0xe,0,0x16a9000)
  4473      1 gdb      RET   mmap 270583654772736/0xf6182a9ff000
  4473      1 gdb      CALL  mmap(0xf6182ae60000,0x1000,PROT_READ|PROT_WRITE,0x1002<PRIVATE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4473      1 gdb      RET   mmap 270583654768640/0xf6182a9fe000
  4473      1 gdb      CALL  mmap(0xf6182ae72000,0x1000,PROT_READ|PROT_WRITE,0x1002<PRIVATE,ANONYMOUS,ALIGN=NONE>,0xffffffff,0,0)
  4473      1 gdb      RET   mmap 270583654768640/0xf6182a9fe000



pmap of the node process:


newt@arm64 ~ [SIGUSR1|0]> pmap (pgrep node)
0000000025539000     28K                     [ anon ]
0000000025540000      4K read/write          [ anon ]
0000000025541000      4K                     [ anon ]
0000000025542000    244K read/write          [ anon ]
000000002557F000      4K                     [ anon ]
0000000025580000 130788K                     [ anon ]
00000000306C0000    256K read/write          [ anon ]
000000003DE00000    256K read/write          [ anon ]
0000000040E40000    256K read/write          [ anon ]
0000000044F80000    256K read/write          [ anon ]
0000000053440000    256K read/write          [ anon ]
000000005AE40000    148K read                [ anon ]
0000000200100000     64K read/exec          -?- 
0000000200110000  15040K read/exec          -?- 
0000000200FC0000     64K read/exec          -?- 
0000000200FD0000  10000K read/exec          -?- 
00000002019A4000     72K read/write         -?- 
00000002019B6000    108K read/write          [ anon ]
0000F07557DC0000     64K                     [ anon ]
0000F07557DD0000   8128K read/write          [ anon ]
0000F075585C0000     64K read/write          [ anon ]
0000F075585D0000     64K                     [ anon ]
0000F075585E0000   8128K read/write          [ anon ]
0000F07558DD0000     64K read/write          [ anon ]
0000F07558DE0000     64K                     [ anon ]
0000F07558DF0000   8128K read/write          [ anon ]
0000F075595E0000     64K read/write          [ anon ]
0000F075595F0000     64K                     [ anon ]
0000F07559600000   8128K read/write          [ anon ]
0000F07559DF0000    128K read/write          [ anon ]
0000F07559E10000   1984K read/write          [ anon ]
0000F0755A3F0000     64K                     [ anon ]
0000F0755A400000   8128K read/write          [ anon ]
0000F0755ABF0000    128K read/write          [ anon ]
0000F0755AC10000    960K read/write          [ anon ]
0000F0755AD00000   3072K read/write          [ anon ]
0000F0755B000000     64K read/write          [ anon ]
0000F0755B010000   1984K read/write          [ anon ]
0000F0755B3EB000      4K read/write          [ anon ]
0000F0755B3EC000    296K read/write          [ anon ]
0000F0755B436000   1016K read/write          [ anon ]
0000F0755B534000     72K read/write          [ anon ]
0000F0755B546000      4K read/write          [ anon ]
0000F0755B547000    164K read/write          [ anon ]
0000F0755B570000     24K read/exec          -?- 
0000F0755B576000     64K                    -?- 
0000F0755B586000      4K read/write         -?- 
0000F0755B587000     36K read/write          [ anon ]
0000F0755B590000     96K read/exec          -?- 
0000F0755B5A8000     60K                    -?- 
0000F0755B5B7000      4K read/write         -?- 
0000F0755B5B8000     32K read/write          [ anon ]
0000F0755B5C0000    384K read/exec          -?- 
0000F0755B620000     64K read/exec          -?- 
0000F0755B630000    960K read/exec          -?- 
0000F0755B720000     64K read/exec          -?- 
0000F0755B730000    272K read/exec          -?- 
0000F0755B774000     60K                    -?- 
0000F0755B783000     68K read/write         -?- 
0000F0755B794000    112K read/write          [ anon ]
0000F0755B7B0000   1856K read/write          [ anon ]
0000F0755B980000     64K read/write          [ anon ]
0000F0755B990000     64K read/write          [ anon ]
0000F0755B9A0000     84K read/write          [ anon ]
0000F0755B9B5000     44K read/write          [ anon ]
0000F0755B9C0000     72K read/exec          -?- 
0000F0755B9D2000     64K                    -?- 
0000F0755B9E2000      4K read/write         -?- 
0000F0755B9E3000      4K read/write          [ anon ]
0000F0755B9E4000     20K read/write          [ anon ]
0000F0755B9E9000     28K read/write          [ anon ]
0000F0755B9F0000     72K read/exec          -?- 
0000F0755BA02000     64K                    -?- 
0000F0755BA12000      4K read/write         -?- 
0000F0755BA13000     52K read/write          [ anon ]
0000F0755BA20000    192K read/exec          -?- 
0000F0755BA50000     60K                    -?- 
0000F0755BA5F000      4K read/write         -?- 
0000F0755BA60000    768K read/exec          -?- 
0000F0755BB20000     64K read/exec          -?- 
0000F0755BB30000    728K read/exec          -?- 
0000F0755BBE6000     60K                    -?- 
0000F0755BBF5000     52K read/write         -?- 
0000F0755BC02000     16K read/write          [ anon ]
0000F0755BC06000     40K read/write          [ anon ]
0000F0755BC10000     28K read/exec          -?- 
0000F0755BC17000     64K                    -?- 
0000F0755BC27000      4K read/write         -?- 
0000F0755BC28000     12K read/write          [ anon ]
0000F0755BC2B000     20K read/write          [ anon ]
0000F0755BC30000  28120K read/exec          -?- 
0000F0755D7A6000     60K                    -?- 
0000F0755D7B5000     64K read/write         -?- 
0000F0755D7C5000     44K read/write          [ anon ]
0000F0755D7D0000   1860K read/exec          -?- 
0000F0755D9A1000     60K                    -?- 
0000F0755D9B0000     76K read/write         -?- 
0000F0755D9C3000     52K read/write          [ anon ]
0000F0755D9D0000   2924K read/exec          -?- 
0000F0755DCAB000     60K                    -?- 
0000F0755DCBA000     72K read/write         -?- 
0000F0755DCCC000     16K read/write          [ anon ]
0000F0755DCD0000    520K read/exec          -?- 
0000F0755DD52000     60K                    -?- 
0000F0755DD61000     52K read/write         -?- 
0000F0755DD6E000      8K read/write          [ anon ]
0000F0755DD70000   2564K read/exec          -?- 
0000F0755DFF1000     64K                    -?- 
0000F0755E001000    192K read/write         -?- 
0000F0755E031000     12K read/write          [ anon ]
0000F0755E034000     48K read/write          [ anon ]
0000F0755E040000    148K read/exec          -?- 
0000F0755E065000     60K                    -?- 
0000F0755E074000     12K read/write         -?- 
0000F0755E077000     36K read/write          [ anon ]
0000F0755E080000     84K read/exec          -?- 
0000F0755E095000     60K                    -?- 
0000F0755E0A4000      4K read/write         -?- 
0000F0755E0A5000     44K read/write          [ anon ]
0000F0755E0B0000    168K read/exec          -?- 
0000F0755E0DA000     60K                    -?- 
0000F0755E0E9000      8K read/write         -?- 
0000F0755E0EB000     20K read/write          [ anon ]
fish: âap (pgrep node)âerminated by signal SIGBUS (Misaligned address error)
>Fix:

>Release-Note:

>Audit-Trail:
From: coypu@sdf.org
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/55533: nodejs fails with "out of memory" on aarch64
 (with reproducer)
Date: Mon, 3 Aug 2020 22:25:39 +0000

 From mlelstv, this fails with ENOMEM on aarch64 but not amd64.
 (Allocating an already in use address)

 #include <stdio.h>
 #include <err.h>
 #include <sys/mman.h>

 int main()
 {
         void *p;

         p = (void *) mmap;

         p = mmap(p,0x7f000,PROT_NONE,MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, 0xffffffff, 0);
         if (p == MAP_FAILED)
                 err(1,"mmap");

         printf("mmap = %p\n", p);

         return 0;
 }

From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/55533: nodejs fails with "out of memory" on aarch64
Date: Wed, 5 Aug 2020 23:08:37 +0200

 The bug is not aarch64 specific and also occurs on other architectures
 including amd64.

 If mmap is called with a non-zero hint (and not MAP_FIXED) and
 collides with an existing allocation, it only succeeds if
 the mapping fits into the first gap.

 The first gap on amd64 seems to be a bit larger so that the mapping
 used by nodejs still fits.


 Greetings,
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

From: "Maya Rashish" <maya@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55533 CVS commit: pkgsrc/lang/nodejs
Date: Wed, 5 Aug 2020 21:49:18 +0000

 Module Name:	pkgsrc
 Committed By:	maya
 Date:		Wed Aug  5 21:49:18 UTC 2020

 Modified Files:
 	pkgsrc/lang/nodejs: Makefile distinfo
 	pkgsrc/lang/nodejs/patches:
 	    patch-deps_v8_src_base_platform_platform-posix.cc

 Log Message:
 nodejs: workaround issue for netbsd/aarch64 in PR port-arm/55533

 NetBSD mmap might fail depending on the choice of hint addr given, so don't
 give a hint at all.

 bump PKGREVISION.


 To generate a diff of this commit:
 cvs rdiff -u -r1.188 -r1.189 pkgsrc/lang/nodejs/Makefile
 cvs rdiff -u -r1.175 -r1.176 pkgsrc/lang/nodejs/distinfo
 cvs rdiff -u -r1.5 -r1.6 \
     pkgsrc/lang/nodejs/patches/patch-deps_v8_src_base_platform_platform-posix.cc

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: coypu@sdf.org
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/55533: nodejs fails with "out of memory" on aarch64
Date: Thu, 6 Aug 2020 00:49:35 +0000

 Sorry to not include it in the commit message.
 The patch is aarch64 specific because aarch64 is currently broken and
 can only get better, I'd need a lot more testing to feel at ease about
 changing amd64 which doesn't seem to fail often.

From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/55533: nodejs fails with "out of memory" on aarch64
 (with reproducer)
Date: Sat, 8 Aug 2020 14:08:20 +0200

 The mmap failure is pretty forward.

 The memory map is a list of adjacent memory regions, each with a part
 that is allocated (start..end) followed by a part that is free
 (end..end+gap).

 mmap() searches the list until it finds a gap that is large enough and
 where the requested alignment can be fullfilled.

 When a zero hint is used, the search starts from the beginning of the
 list and everything is fine.

 But with a non-zero hint, the search starts with the region that
 includes the given address. This is technically wrong, since the
 beginning of the list is never searched. But if the hint is
 somewhere in the heap, the search will reach the end of the heap
 and find space there.

 This also works when we do top down allocation. The search goes
 backwards and the "end" of the heap is its lowest address.

 Things go wrong if you use a hint that is not in the heap, e.g.
 a stack address for bottom up allocation, or a program address
 for top down allocation.

 To be correct, the search would need to wrap around until it
 reaches the start position to really search the full list.



 In our code we use an additional red-black-tree to locate
 a region for a given address faster than the original
 linear search. The tree even maintains a "max gap" value
 for subtrees, so we could just walk the tree and prune
 subtrees when searching for free space.

 But we don't do that. We just search downwards, using
 a simple heuristic. This speeds up the search when
 we need go to the end of the heap, but otherwise may
 miss gaps that are large enough. In that case we fall
 back to the linear search.

 Instead of wrapping around the linear search, we could
 also do a complete tree walk. The linear search wouldn't
 be necessary then.


 Greetings,
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

Responsible-Changed-From-To: port-arm-maintainer->kern-bug-people
Responsible-Changed-By: skrll@NetBSD.org
Responsible-Changed-When: Tue, 11 Aug 2020 07:06:30 +0000
Responsible-Changed-Why:
Bug is MI


From: "Benny Siegert" <bsiegert@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55533 CVS commit: [pkgsrc-2020Q2] pkgsrc/lang/nodejs
Date: Fri, 14 Aug 2020 17:18:38 +0000

 Module Name:	pkgsrc
 Committed By:	bsiegert
 Date:		Fri Aug 14 17:18:38 UTC 2020

 Modified Files:
 	pkgsrc/lang/nodejs [pkgsrc-2020Q2]: Makefile distinfo
 	pkgsrc/lang/nodejs/patches [pkgsrc-2020Q2]:
 	    patch-deps_v8_src_base_platform_platform-posix.cc

 Log Message:
 Pullup ticket #6296 - requested by maya
 lang/nodejs: aarch64 bugfix, PR port-arm/55533

 (via patch)

 ---
    Module Name:    pkgsrc
    Committed By:   maya
    Date:           Wed Aug  5 21:49:18 UTC 2020

    Modified Files:
 	   pkgsrc/lang/nodejs: Makefile distinfo
 	   pkgsrc/lang/nodejs/patches:
 	       patch-deps_v8_src_base_platform_platform-posix.cc

    Log Message:
    nodejs: workaround issue for netbsd/aarch64 in PR port-arm/55533

    NetBSD mmap might fail depending on the choice of hint addr given, so don't
    give a hint at all.

    bump PKGREVISION.


 To generate a diff of this commit:
 cvs rdiff -u -r1.185 -r1.185.2.1 pkgsrc/lang/nodejs/Makefile
 cvs rdiff -u -r1.172 -r1.172.2.1 pkgsrc/lang/nodejs/distinfo
 cvs rdiff -u -r1.5 -r1.5.18.1 \
     pkgsrc/lang/nodejs/patches/patch-deps_v8_src_base_platform_platform-posix.cc

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.