NetBSD Problem Report #54395

From www@netbsd.org  Mon Jul 22 04:00:43 2019
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 234617A184
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 22 Jul 2019 04:00:43 +0000 (UTC)
Message-Id: <20190722040041.C85CB7A1D3@mollari.NetBSD.org>
Date: Mon, 22 Jul 2019 04:00:41 +0000 (UTC)
From: rokuyama.rk@gmail.com
Reply-To: rokuyama.rk@gmail.com
To: gnats-bugs@NetBSD.org
Subject: earmv7hf binaries trigger kernel panic on aarch64
X-Send-Pr-Version: www-1.0

>Number:         54395
>Category:       kern
>Synopsis:       earmv7hf binaries trigger kernel panic on aarch64
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jul 22 04:05:00 +0000 2019
>Closed-Date:    Fri Nov 01 09:41:11 +0000 2019
>Last-Modified:  Fri Nov 01 09:41:11 +0000 2019
>Originator:     Rin Okuyama
>Release:        8.99.51
>Organization:
Department of Physics, Meiji University
>Environment:
NetBSD  8.99.51 NetBSD 8.99.51 (GENERIC64_UVMHIST) #0: Mon Jul 22 12:12:23 JST 2019  rin@latipes:/build/src/sys/arch/evbarm/compile/GENERIC64_UVMHIST evbarm
>Description:
KASSERT fires when running earmv7hf binary on aarch64:

# file /emul/netbsd32/bin/sh
/emul/netbsd32/bin/sh: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /libexec/ld.elf_so, for NetBSD 8.99.50, compiled for: earmv7hf, not stripped
# chroot /emul/netbsd32 su -
panic: kernel diagnostic assertion "!topdown || hint <= orig_hint" failed: file "../../../../uvm/uvm_map.c", line 2164 hint: fbefc000, orig_hint: f2c2f000
cpu1: Begin traceback...
trace fp ffffffc044447950
fp ffffffc044447970 vpanic() at ffffffc0004a4508 netbsd:vpanic+0x198
fp ffffffc0444479d0 kern_assert() at ffffffc0005e1344 netbsd:kern_assert+0x5c
fp ffffffc044447a60 uvm_map_findspace() at ffffffc0004139a4 netbsd:uvm_map_findspace+0x714
fp ffffffc044447b20 uvm_map_prepare() at ffffffc000414528 netbsd:uvm_map_prepare+0x3f0
fp ffffffc044447bd0 uvm_map() at ffffffc000418eac netbsd:uvm_map+0x64
fp ffffffc044447c60 uvm_mmap.part.0() at ffffffc00041c1a4 netbsd:uvm_mmap.part.0+0x174
fp ffffffc044447cd0 sys_mmap() at ffffffc00041ca58 netbsd:sys_mmap+0x378
fp ffffffc044447d80 netbsd32_mmap() at ffffffc0001b30c8 netbsd:netbsd32_mmap+0x40
fp ffffffc044447de0 netbsd32_syscall() at ffffffc00008020c netbsd:netbsd32_syscall+-0x127ff4
tf ffffffc044447ed0 el0_trap() at ffffffc0000744c0 netbsd:el0_trap
---- trapframe 0xffffffc044447ed0 (304 bytes) ----
    pc=00000000f2b33560,   spsr=0000000060080010 (AArch32)
   esr=00000000460000c5,    far=00000000f2b25f4c
    r0=0000000000000000,     r1=0000000000002000
    r2=0000000000000000,     r3=000000000d001002
    r4=0000000000000000,     r5=0000000000002000
    r6=0000000000000000,     r7=00000000fff21084
    r8=00000000f2b41543,     r9=0000000001000000
   r10=0000000000000000,    r11=00000000fff20ed4
   r12=00000000f2b3355c, sp=r13=00000000fff20ea8
lr=r14=00000000f2a7be9c, pc=r15=00000000f2b33560
------------------------------------------------
cpu1: End traceback...
Stopped in pid 8.1 (su) at      netbsd:cpu_Debugger+0x4:        ret

Kernel timestamps are omitted here for readability. uvmhist reads:

db{1}> show kernhist
...
1563765238.656282 uvm_map_prepare#695@1: called!
1563765238.656283 uvm_map_prepare#695@1: (map=0xffff0000bed65a10, start=0xf2c2f000, size=8192, flags=0x80010)
1563765238.656283 uvm_map_prepare#695@1:   uobj/offset 0/-1
1563765238.656284 uvm_map_findspace#665@1: called!
1563765238.656285 uvm_map_findspace#665@1: (map=0xffff0000bed65a10, hint=0xf2c2f000, len=8192, flags=0x80010)
1563765238.656286 uvm_map_lookup_entry#3431@1: called!
1563765238.656287 uvm_map_lookup_entry#3431@1: (map=0xffff0000bed65a10,addr=0xf2c2f000,ent=0xffffffc044447b18)
1563765238.656289 uvm_map_lookup_entry#3431@1: <- failed!
1563765238.656290 uvm_map_findspace#665@1: <- got it!  (result=0xfbefc000)
db{1}>

The failure began after this commit:

http://cvsweb.netbsd.org/bsdweb.cgi/src/external/bsd/jemalloc/dist/src/pages.c#rev1.5

by which jemalloc for earmv7hf became to mmap 8KB-aligned anonymous
meomory (Note that pagesizes are 4KB and 8KB for aarch64 and earmv7hf,
respectively).

Dirty printf debug reveals the scenario below:

(1) An earmv7hf binary requires 8KB-aligned anonymous memory mapped at
any address.

(2) Since native pagesie is 4KB, uvm_map() requires address lower than
0xf2c2f000, which is 4KB-aligned but not 8KB-aligned.

(3) uvm_findspace() tries to find entry of hint = 0xf2c2f000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1919

(4) Since entry is not available, and UVM_FLAG_FIXED is not specified,
uvm_map_space_avail() is called for hint = 0xf2c2f000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1942

(5) Since 8KB alignment is requires but 0xf2c2f000 (= *start, start =
&hint in uvm_findspace()) is not aligned, uvm_map_space_avail()
truncate it to 0xf2c2e000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1809

Then, fails to find a space and returns 0 to uvm_findspace().

(6) uvm_findspace() consider
"Still there is a chance to fit if hint > entry->end.":

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1950

However, this assumption of hint > entry->end is no longer valid,
since hint is truncated in step (5).

(7) Then, hint is updated based on a wrong assumption:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1973

This results in KASSERT falure for hint <= orig_hint:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#2163
>How-To-Repeat:
Execute some 32bit binaries on aarch64 kernel.

"cd /usr/tests && atf-run | atf-report" in chroot environment
definitely causes panic.
>Fix:
As described above, this is not MD, but MI bug. uvm_findspace() assumes
that hint is not modified by uvm_map_space_avail(). But this is not true
when alignment is required but hint is not aligned properly.

To fix this, align hint at the beginning of uvm_findspace(). Then,
inconsistent does not occur:

http://www.netbsd.org/~rin/uvm_map_20190722.patch

With this patch, I can run earmv7hf userland on aarch64 kernel more than
a week without problems.

>Release-Note:

>Audit-Trail:
From: Valery Ushakov <uwe@stderr.spb.ru>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/54395: earmv7hf binaries trigger kernel panic on aarch64
Date: Tue, 1 Oct 2019 15:36:58 +0300

 On Mon, Jul 22, 2019 at 04:05:00 +0000, rokuyama.rk@gmail.com wrote:

 > >Fix:
 > As described above, this is not MD, but MI bug. uvm_findspace() assumes
 > that hint is not modified by uvm_map_space_avail(). But this is not true
 > when alignment is required but hint is not aligned properly.
 > 
 > To fix this, align hint at the beginning of uvm_findspace(). Then,
 > inconsistent does not occur:
 > 
 > http://www.netbsd.org/~rin/uvm_map_20190722.patch

 +#define ALIGN_VA(va, align, topdown) do { \
 +	if ((align) != 0 && ((va) & ((align) - 1)) != 0) { \
 +		if (topdown) \
 +			(va) &= ~((align) - 1); \

 Use roundown2() here?

 +		else \
 +			(va) = roundup(va, align); \

 Use roundup2() here instead, which avoids multiplication?


 We don't seem to have a convenience macro for testing power of 2
 alignment to use in the test though.

 -uwe

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org
Cc: Valery Ushakov <uwe@stderr.spb.ru>, Joerg Sonnenberger <joerg@bec.de>
Subject: Re: kern/54395: earmv7hf binaries trigger kernel panic on aarch64
Date: Wed, 2 Oct 2019 17:59:22 +0900

 On 2019/10/01 21:40, Valery Ushakov wrote:
 > +#define ALIGN_VA(va, align, topdown) do { \
 > +	if ((align) != 0 && ((va) & ((align) - 1)) != 0) { \
 > +		if (topdown) \
 > +			(va) &= ~((align) - 1); \
 >   
 > Use roundown2() here?
 >   
 > +		else \
 > +			(va) = roundup(va, align); \
 >   
 > Use roundup2() here instead, which avoids multiplication?

 Thank you for your comments! Yes, we can use round{down,up}2() here.

 > We don't seem to have a convenience macro for testing power of 2
 > alignment to use in the test though.

 We have powerof2(9) macro in sys/param.h:

 https://nxr.netbsd.org/xref/src/sys/sys/param.h#427
      427 #define	powerof2(x)	((((x)-1)&(x))==0)

 ALIGN_VA() macro in the patch is used in uvm_map_space_avail() and
 uvm_map_findspace(). For the latter, we already have an equivalent
 check:

 https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1850
     1850 struct vm_map_entry *
     1851 uvm_map_findspace(struct vm_map *map, vaddr_t hint, vsize_t length,
     1852     vaddr_t *result /* OUT */, struct uvm_object *uobj, voff_t uoffset,
     1853     vsize_t align, int flags)
     ....
     1864 	KASSERT((flags & UVM_FLAG_COLORMATCH) != 0 || (align & (align - 1)) == 0);

 I rewrote this with powerof2(9). For the former, I added KASSERT.

 I updated the patch accordingly. Also, VA_ALIGN() macro was turned
 into inline function va_align() (Thanks Joerg for his comment):

 http://www.netbsd.org/~rin/uvm_map_20191002.patch

 I will commit the revised patch and send a pullup request in the
 weekend, if there's no objections.

 Thanks,
 rin

From: Joerg Sonnenberger <joerg@bec.de>
To: Rin Okuyama <rokuyama.rk@gmail.com>
Cc: gnats-bugs@netbsd.org, Valery Ushakov <uwe@stderr.spb.ru>
Subject: Re: kern/54395: earmv7hf binaries trigger kernel panic on aarch64
Date: Wed, 2 Oct 2019 12:27:23 +0200

 On Wed, Oct 02, 2019 at 05:59:22PM +0900, Rin Okuyama wrote:
 > Thank you for your comments! Yes, we can use round{down,up}2() here.

 Looks good. There is one final concern that just hit me. Do we have to
 care about overflow in the topdown case? Would it be useful to assert
 that?

 Joerg

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: kern/54395: earmv7hf binaries trigger kernel panic on aarch64
Date: Sun, 27 Oct 2019 18:05:22 +0900

 I updated the patch:

 	http://www.netbsd.org/~rin/uvm_map_20191027.patch

 See discussion in tech-kern@ for more details:

 	http://mail-index.netbsd.org/tech-kern/2019/10/27/msg025629.html

From: "Rin Okuyama" <rin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/54395 CVS commit: src/sys/uvm
Date: Fri, 1 Nov 2019 08:26:18 +0000

 Module Name:	src
 Committed By:	rin
 Date:		Fri Nov  1 08:26:18 UTC 2019

 Modified Files:
 	src/sys/uvm: uvm_map.c

 Log Message:
 PR kern/54395

 - Align hint for virtual address at the beginning of uvm_map() if
   required. Otherwise, it will be rounded up/down in an unexpected
   way by uvm_map_space_avail(), which results in assertion failure.

   Fix kernel panic when executing earm binary (8KB pages) on aarch64
   (4KB pages), which relies on mmap(2) with MAP_ALIGNED flag.

 - Use inline functions/macros consistently.

 - Add some more KASSERT's.

 For more details, see the PR as well as discussion on port-kern:
 http://mail-index.netbsd.org/tech-kern/2019/10/27/msg025629.html


 To generate a diff of this commit:
 cvs rdiff -u -r1.364 -r1.365 src/sys/uvm/uvm_map.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->pending-pullups
State-Changed-By: rin@NetBSD.org
State-Changed-When: Fri, 01 Nov 2019 08:33:09 +0000
State-Changed-Why:
[pullup-9 #388]
https://releng.netbsd.org/cgi-bin/req-9.cgi?show=388


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/54395 CVS commit: [netbsd-9] src/sys/uvm
Date: Fri, 1 Nov 2019 09:36:33 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Fri Nov  1 09:36:32 UTC 2019

 Modified Files:
 	src/sys/uvm [netbsd-9]: uvm_map.c

 Log Message:
 Pull up following revision(s) (requested by rin in ticket #388):

 	sys/uvm/uvm_map.c: revision 1.365

 PR kern/54395

 - Align hint for virtual address at the beginning of uvm_map() if
    required. Otherwise, it will be rounded up/down in an unexpected
    way by uvm_map_space_avail(), which results in assertion failure.
    Fix kernel panic when executing earm binary (8KB pages) on aarch64
    (4KB pages), which relies on mmap(2) with MAP_ALIGNED flag.
 - Use inline functions/macros consistently.
 - Add some more KASSERT's.

 For more details, see the PR as well as discussion on port-kern:
 http://mail-index.netbsd.org/tech-kern/2019/10/27/msg025629.html


 To generate a diff of this commit:
 cvs rdiff -u -r1.362 -r1.362.2.1 src/sys/uvm/uvm_map.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: rin@NetBSD.org
State-Changed-When: Fri, 01 Nov 2019 09:41:11 +0000
State-Changed-Why:
Thanks Martin for pulling this up!


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.