NetBSD Problem Report #50469

From khorben@kwarx.station.defora.lan  Wed Nov 25 00:28:09 2015
Return-Path: <khorben@kwarx.station.defora.lan>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 22132A654B
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 25 Nov 2015 00:28:09 +0000 (UTC)
Message-Id: <20151125002958.1B06B1B9B1@kwarx.station.defora.lan>
Date: Wed, 25 Nov 2015 01:29:58 +0100 (CET)
From: Pierre Pronchery <khorben@defora.org>
To: gnats-bugs@NetBSD.org
Subject: PaX ASLR breaks netbsd32 emulation
X-Send-Pr-Version: 3.95

>Number:         50469
>Category:       kern
>Synopsis:       PaX ASLR breaks netbsd32 emulation
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Nov 25 00:30:00 +0000 2015
>Closed-Date:    Thu Aug 17 19:00:19 +0000 2017
>Last-Modified:  Thu Aug 17 19:00:19 +0000 2017
>Originator:     Pierre Pronchery
>Release:        NetBSD 7.0
>Organization:
The DeforaOS Project
>Environment:
System: NetBSD kwarx.station.defora.lan 7.0 NetBSD 7.0 (GENERIC) amd64
Architecture: x86_64
Machine: amd64
>Description:
NetBSD fails to execute 32-bit binaries when using netbsd32 emulation.
The mmap(2) syscall seems to be the culprint, and the kernel logs the
following message:

netbsd32_mmap: retval out of range: 0xfffffdb0f4aa9000
(without end-of-line character, I might add)
>How-To-Repeat:
$ tar xzf $OBJDIR/releasedir/i386/binary/sets/base.tgz
$ ./bin/sh 
Cannot map anonymous memoryCannot allocate memory: Cannot allocate memory
$ ktruss ./bin/sh
Cannot map anonymous memoryCannot allocate memory: Cannot allocate memory
 13411      1 ktruss   fcntl(0x4, 0x3, 0)          = 4194305
 13411      1 ktruss   emul(netbsd)
 13411      1 ktruss   fcntl(0x4, 0x4, 0x400001)   = 0
 13411      1 sh       execve("./bin/sh", 0x7f7fffb3c7a8, 0x7f7fffb3c7b8) JUSTRETURN
 13411      1 sh       emul(netbsd32)
 13411      1 sh       netbsd32_mmap(0, 0x8000, 0x3, 0x1002, 0xffffffffffffffff, 0, 0, 0) Err#12 ENOMEM
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6b0, 0x1b) = 27
       "Cannot map anonymous memory"
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6b8, 0x16) = 22
       "Cannot allocate memory"
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6ac, 0x19) = 25
       ": Cannot allocate memory\n"
 13411      1 sh       netbsd32_exit(0x1)
>Fix:
The following commit works around the problem, but is wrong and should
not be applied as-is. With this change, netbsd32 emulation works again,
but mmap(2) no longer returns randomized values - thus breaking ASLR.

commit a2ac44cb29bb9c65cc7436bd5dc74eba7db54034
Author: Pierre Pronchery <khorben@EdgeBSD.org>
Date:   Tue Oct 27 00:39:12 2015 +0100

    Tentative fix for netbsd32_mmap() with PAX_ASLR

diff --git a/sys/uvm/uvm_mmap.c b/sys/uvm/uvm_mmap.c
index cbb45be..ed4684d 100644
--- a/sys/uvm/uvm_mmap.c
+++ b/sys/uvm/uvm_mmap.c
@@ -342,6 +342,10 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args *uap, register_t *retval)
 	size += pageoff;			/* add offset */
 	size = (vsize_t)round_page(size);	/* round up */

+#ifdef PAX_ASLR
+	pax_aslr(l, &addr, orig_addr, flags);
+#endif /* PAX_ASLR */
+
 	/*
 	 * now check (MAP_FIXED) or get (!MAP_FIXED) the "addr"
 	 */
@@ -417,10 +421,6 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args *uap, register_t *retval)
 	pax_mprotect(l, &prot, &maxprot);
 #endif /* PAX_MPROTECT */

-#ifdef PAX_ASLR
-	pax_aslr(l, &addr, orig_addr, flags);
-#endif /* PAX_ASLR */
-
 	/*
 	 * now let kernel internal function uvm_mmap do the work.
 	 */

>Release-Note:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/50469: PaX ASLR breaks netbsd32 emulation
Date: Wed, 25 Nov 2015 09:19:47 +0100

 On Wed, Nov 25, 2015 at 12:30:00AM +0000, Pierre Pronchery wrote:
 >  	size += pageoff;			/* add offset */
 >  	size = (vsize_t)round_page(size);	/* round up */
 >  
 > +#ifdef PAX_ASLR
 > +	pax_aslr(l, &addr, orig_addr, flags);
 > +#endif /* PAX_ASLR */

 You need to explain this a bit - the PAX_ASLR code is obfuscated (for
 example nxr can not even find the pax_aslr() definition).

 If you move the ASLR up this far, you may break addr == 0 handling further
 down?

 A better fix probably is to

  a) make the pax_aslr() call emulation specific

 or

  b) fix pax_aslr() to respect the the emulation specifics/VA settings of
     the passed lwp (including VM_MAP_TOPDOWN and the limits)


 Martin

From: Pierre Pronchery <khorben@defora.org>
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: Christos Zoulas <christos@zoulas.com>
Subject: Re: kern/50469: PaX ASLR breaks netbsd32 emulation
Date: Sun, 20 Mar 2016 01:13:10 +0100

 This is a multi-part message in MIME format.
 --------------010207040706070009060308
 Content-Type: text/plain; charset=windows-1252
 Content-Transfer-Encoding: 8bit

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 			Hi Martin, (Christos)

 On 11/25/15 09:20, Martin Husemann wrote:
 >> size += pageoff;			/* add offset */ size =
 >> (vsize_t)round_page(size);	/* round up */
 >> 
 >> +#ifdef PAX_ASLR +	pax_aslr(l, &addr, orig_addr, flags); +#endif
 >> /* PAX_ASLR */
 > 
 > You need to explain this a bit - the PAX_ASLR code is obfuscated
 > (for example nxr can not even find the pax_aslr() definition).

 It isn't obfuscated (see sys/kern/kern_pax.c).

 > If you move the ASLR up this far, you may break addr == 0 handling
 > further down?

 Yes, as I mentioned, the commit attached was wrong.

 > A better fix probably is to
 > 
 > a) make the pax_aslr() call emulation specific
 > 
 > or
 > 
 > b) fix pax_aslr() to respect the the emulation specifics/VA
 > settings of the passed lwp (including VM_MAP_TOPDOWN and the
 > limits)

 I finally managed to do enough of this to fix 32-bits emulation.

 I am attaching a patch here, and adding the underlying log below.
 (or see
 http://git.edgebsd.org/gitweb/?p=edgebsd-src.git;a=log;h=refs/heads/khor
 ben/pax-aslr-mmap)

 Ok to commit?

 commit f5e48327700df06086286b11e37ca54b3c76f8a1
 Author: Pierre Pronchery <khorben@EdgeBSD.org>
 Date:   Sun Mar 20 00:51:24 2016 +0100

     Let PaX ASLR know about the current emulation

     This effectively fixes PaX ASLR with 32-bits emulation on 64-bits
     platforms. Without this knowledge, the offset applied for 32-bits
     programs was really meant for a 64-bits address space - thus
     shifting the address up to 12 bits, with a success rate of about
     1/4096. This offset is calculated once in the lifetime of the
     process, which therefore behaved normally when able to start.

 commit f691057c2fd010f497ce112a68739f3592a1dce3
 Author: Pierre Pronchery <khorben@EdgeBSD.org>
 Date:   Sun Mar 20 00:57:55 2016 +0100

     Remove pax_adjust() as it is not used anywhere

 commit bfb172bc14dfa13b1a99c7fed949b6f8c5c1d65d
 Author: Pierre Pronchery <khorben@EdgeBSD.org>
 Date:   Sun Mar 20 00:58:36 2016 +0100

     Detect failures in PaX ASLR earlier

 commit a061a231df2265dfcfb0828b630f4f1f0ae5854f
 Author: Pierre Pronchery <khorben@EdgeBSD.org>
 Date:   Sun Mar 20 01:01:21 2016 +0100

     Avoid a few casts in debugging code

 commit e032672d5f9b5d2370e853c5a64c76aae1ba5ca8
 Author: Pierre Pronchery <khorben@EdgeBSD.org>
 Date:   Sun Mar 20 01:01:35 2016 +0100

     Typo

 - -- 
 khorben
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1

 iQIcBAEBAgAGBQJW7esWAAoJEDA4y9uYhpcD2H8QAJ2qufPLu3tYICnZbQ+JW5gZ
 cePrMJJVtJZyElhW6eNITK1W4f86feb/RBjcsAC3NliE87Tez/gZevjKBhVyQ+Z3
 ez81Sx9Zngx+t0k3gcEXU3bnryK8U+zrzNfz7cUNYGBC8KSeRUobJcnsDPpqtunW
 24IixcQWTNZQtp6d9r7t9Q3OSdpZV0Q0QLgZrmPi6Bu6cU30IdEnrKNAs9Pym9vR
 8RJXpPFoP+W9I2Yc6IncCUnXCiqvMSHx+2e7aXjFJ24jzE3vQpSEMnXgF1juIM8u
 Z8vRA4edNAOiIPMJnGytby2P/ouMbpvU7Faz4RqYkWQCBIhyWVkqt9uBaC67C6qX
 Tz7+bAObxp7r4Z3Ex0NffWjiACn/snIp21rHrE39M87Gd+MLsny8scr7rZBWezlX
 P6iVDlsCbvWs+vUU9B5zTMGjrVk2MB5lIxqdYuTkohhxwH09HEEdqVjkzNt2MBtc
 ECS0cV1VmmzNJj925UOV+/GQJ4zzTx+uGQOtKkQ3UG06GyVNXgBA8IAif35Qau7z
 ETWTijmdTKj67LBBg01kwZ9b+mE8Og8Zrn2ogHODfCVY43Vr/s6n++yFj6TGh81d
 t4UxiSnKStRWyfam/gg2fz0m9Nj6XuFeBNfBMjhII07DkwFKHfPg6gLoE+eCP5nI
 PB+eMiDZiRvRezaz+FMl
 =tReU
 -----END PGP SIGNATURE-----

 --------------010207040706070009060308
 Content-Type: text/x-patch;
  name="patch-pax_aslr_emul32.diff"
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment;
  filename="patch-pax_aslr_emul32.diff"

 diff --git a/sys/compat/netbsd32/netbsd32_netbsd.c b/sys/compat/netbsd32/=
 netbsd32_netbsd.c
 index d9b44741..1a18205 100644
 --- a/sys/compat/netbsd32/netbsd32_netbsd.c
 +++ b/sys/compat/netbsd32/netbsd32_netbsd.c
 @@ -1491,6 +1491,22 @@ netbsd32_setrlimit(struct lwp *l, const struct net=
 bsd32_setrlimit_args *uap, reg
  	return (dosetrlimit(l, l->l_proc, which, &alim));
  }
 =20
 +static int
 +netbsd32_mmap_recover(struct lwp *l, u_long addr, long len)
 +{
 +	struct sys_munmap_args ua;
 +	register_t retval =3D 0;
 +	int error;
 +
 +	SCARG(&ua, addr) =3D (void *)addr;
 +	SCARG(&ua, len) =3D len;
 +	error =3D sys_munmap(l, &ua, &retval);
 +	if (error)
 +		printf("netbsd32_mmap_recover: error %d (addr=3D0x%lx, len=3D0x%lx)\n"=
 ,
 +				error, addr, len, error);
 +	return EOVERFLOW;
 +}
 +
  int
  netbsd32_mmap(struct lwp *l, const struct netbsd32_mmap_args *uap, regis=
 ter_t *retval)
  {
 @@ -1530,9 +1546,12 @@ netbsd32_mmap(struct lwp *l, const struct netbsd32=
 _mmap_args *uap, register_t *r
  #endif
  	error =3D sys_mmap(l, &ua, retval);
  	if ((u_long)*retval > (u_long)UINT_MAX) {
 +#ifdef DEBUG_MMAP
  		printf("netbsd32_mmap: retval out of range: 0x%lx\n",
  		    (u_long)*retval);
 -		/* Should try to recover and return an error here. */
 +#endif
 +		/* try to recover and return an error */
 +		error =3D netbsd32_mmap_recover(l, *retval, SCARG(&ua, len));
  	}
  	return (error);
  }
 diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c
 index befccae..8c2a138 100644
 --- a/sys/kern/kern_exec.c
 +++ b/sys/kern/kern_exec.c
 @@ -1124,7 +1124,7 @@ execve_runproc(struct lwp *l, struct execve_data * =
 restrict data,
  	vm->vm_minsaddr =3D (void *)epp->ep_minsaddr;
 =20
  #ifdef PAX_ASLR
 -	pax_aslr_init(l, vm);
 +	pax_aslr_init(l, vm, epp->ep_flags);
  #endif /* PAX_ASLR */
 =20
  	/* Now map address space. */
 diff --git a/sys/kern/kern_pax.c b/sys/kern/kern_pax.c
 index 796b908..589f7c1 100644
 --- a/sys/kern/kern_pax.c
 +++ b/sys/kern/kern_pax.c
 @@ -300,13 +300,17 @@ pax_aslr_active(struct lwp *l)
  }
 =20
  void
 -pax_aslr_init(struct lwp *l, struct vmspace *vm)
 +pax_aslr_init(struct lwp *l, struct vmspace *vm, u_int ep_flags)
  {
  	if (!pax_aslr_active(l))
  		return;
 =20
 -	vm->vm_aslr_delta_mmap =3D PAX_ASLR_DELTA(cprng_fast32(),
 -	    PAX_ASLR_DELTA_MMAP_LSB, PAX_ASLR_DELTA_MMAP_LEN);
 +	if (ep_flags & EXEC_32)
 +		vm->vm_aslr_delta_mmap =3D PAX_ASLR_DELTA(cprng_fast32(),
 +			PAX_ASLR_DELTA_MMAP_LSB, PAX_ASLR_DELTA_MMAP_LEN / 2);
 +	else
 +		vm->vm_aslr_delta_mmap =3D PAX_ASLR_DELTA(cprng_fast32(),
 +			PAX_ASLR_DELTA_MMAP_LSB, PAX_ASLR_DELTA_MMAP_LEN);
  }
 =20
  void
 @@ -317,21 +321,23 @@ pax_aslr(struct lwp *l, vaddr_t *addr, vaddr_t orig=
 _addr, int f)
 =20
  	if (!(f & MAP_FIXED) && ((orig_addr =3D=3D 0) || !(f & MAP_ANON))) {
  #ifdef PAX_ASLR_DEBUG
 -		uprintf("applying to 0x%lx orig_addr=3D0x%lx f=3D%x\n",
 -		    (unsigned long)*addr, (unsigned long)orig_addr, f);
 +		uprintf("applying to 0x%" PRIxVADDR
 +				" orig_addr=3D0x%" PRIxVADDR " f=3D0x%x\n",
 +		    *addr, orig_addr, f);
  #endif
  		if (!(l->l_proc->p_vmspace->vm_map.flags & VM_MAP_TOPDOWN))
  			*addr +=3D l->l_proc->p_vmspace->vm_aslr_delta_mmap;
  		else
  			*addr -=3D l->l_proc->p_vmspace->vm_aslr_delta_mmap;
  #ifdef PAX_ASLR_DEBUG
 -		uprintf("result 0x%lx\n", *addr);
 +		uprintf("result 0x%" PRIxVADDR "\n", *addr);
  #endif
  	}
  #ifdef PAX_ASLR_DEBUG
  	else
 -	    uprintf("not applying to 0x%lx orig_addr=3D0x%lx f=3D%x\n",
 -		(unsigned long)*addr, (unsigned long)orig_addr, f);
 +	    uprintf("not applying to 0x%" PRIxVADDR
 +			    " orig_addr=3D0x%" PRIxVADDR " f=3D0x%x\n",
 +		*addr, orig_addr, f);
  #endif
  }
 =20
 @@ -343,7 +349,8 @@ pax_aslr_stack(struct lwp *l, struct exec_package *ep=
 p, u_long *max_stack_size)
  		    PAX_ASLR_DELTA_STACK_LSB,
  		    PAX_ASLR_DELTA_STACK_LEN);
  #ifdef PAX_ASLR_DEBUG
 -		uprintf("stack 0x%lx d=3D0x%lx 0x%lx\n",
 +		uprintf("stack 0x%" PRIxVADDR " d=3D0x%lx"
 +				" 0x%" PRIxVADDR "\n",
  		    epp->ep_minsaddr, d, epp->ep_minsaddr - d);
  #endif
  		epp->ep_minsaddr -=3D d;
 diff --git a/sys/sys/pax.h b/sys/sys/pax.h
 index 5ae228d..c80e583 100644
 --- a/sys/sys/pax.h
 +++ b/sys/sys/pax.h
 @@ -46,7 +46,6 @@ struct vmspace;
  #endif /* PAX_ASLR */
 =20
  void pax_init(void);
 -void pax_adjust(struct lwp *, uint32_t);
 =20
  void pax_mprotect(struct lwp *, vm_prot_t *, vm_prot_t *);
  int pax_segvguard(struct lwp *, struct vnode *, const char *, bool);
 @@ -54,7 +53,7 @@ int pax_segvguard(struct lwp *, struct vnode *, const c=
 har *, bool);
  #define	PAX_ASLR_DELTA(delta, lsb, len)	\
      (((delta) & ((1UL << (len)) - 1)) << (lsb))
  bool pax_aslr_active(struct lwp *);
 -void pax_aslr_init(struct lwp *, struct vmspace *);
 +void pax_aslr_init(struct lwp *, struct vmspace *, u_int);
  void pax_aslr_stack(struct lwp *, struct exec_package *, u_long *);
  void pax_aslr(struct lwp *, vaddr_t *, vaddr_t, int);
 =20
 diff --git a/sys/sys/proc.h b/sys/sys/proc.h
 index dd34d8e..b24ec61 100644
 --- a/sys/sys/proc.h
 +++ b/sys/sys/proc.h
 @@ -180,7 +180,7 @@ struct emul {
  };
 =20
  /*
 - * Emulation miscelaneous flags
 + * Emulation miscellaneous flags
   */
  #define	EMUL_HAS_SYS___syscall	0x001	/* Has SYS___syscall */
 =20
 diff --git a/sys/uvm/uvm_mmap.c b/sys/uvm/uvm_mmap.c
 index cbb45be..8043f0f 100644
 --- a/sys/uvm/uvm_mmap.c
 +++ b/sys/uvm/uvm_mmap.c
 @@ -352,10 +352,12 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args =
 *uap, register_t *retval)
  		if (addr & PAGE_MASK)
  			return (EINVAL);
 =20
 +#ifndef PAX_ASLR
  		error =3D range_test(addr, size, true);
  		if (error) {
  			return error;
  		}
 +#endif
 =20
  	} else if (addr =3D=3D 0 || !(flags & MAP_TRYFIXED)) {
 =20
 @@ -419,6 +421,11 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args *=
 uap, register_t *retval)
 =20
  #ifdef PAX_ASLR
  	pax_aslr(l, &addr, orig_addr, flags);
 +
 +	error =3D range_test(addr, size, true);
 +	if (error) {
 +		goto out;
 +	}
  #endif /* PAX_ASLR */
 =20
  	/*

 --------------010207040706070009060308--

From: Martin Husemann <martin@duskware.de>
To: Pierre Pronchery <khorben@defora.org>
Cc: gnats-bugs@NetBSD.org, Christos Zoulas <christos@zoulas.com>
Subject: Re: kern/50469: PaX ASLR breaks netbsd32 emulation
Date: Sun, 20 Mar 2016 13:08:36 +0100

 On Sun, Mar 20, 2016 at 01:13:10AM +0100, Pierre Pronchery wrote:
 > I finally managed to do enough of this to fix 32-bits emulation.

 I still think this is wrong. The code should not hardcode UINT_MAX
 as upper limit and fail-and-recover sounds like a bogus aproach
 for this anyway.

 Why not create proper VA limits for aslr and set them up MD and emulation
 specific?

 Martin

From: Pierre Pronchery <khorben@defora.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@NetBSD.org, Christos Zoulas <christos@zoulas.com>
Subject: Re: kern/50469: PaX ASLR breaks netbsd32 emulation
Date: Sun, 20 Mar 2016 14:48:19 +0100

 On 03/20/16 13:08, Martin Husemann wrote:
 > On Sun, Mar 20, 2016 at 01:13:10AM +0100, Pierre Pronchery wrote:
 >> I finally managed to do enough of this to fix 32-bits emulation.
 > 
 > I still think this is wrong. The code should not hardcode UINT_MAX
 > as upper limit and fail-and-recover sounds like a bogus aproach
 > for this anyway.

 This has nothing to do with ASLR but with 32-bits emulation. The check
 with UINT_MAX is performed regardless of the presence of ASLR. The same
 check can also be found in emul_sunos32 and emul_svr4_32 (I have a
 separate patch for both, in the same branch).

 The recovery code is independent from the actual fix (as can be seen
 from my commit messages). I wrote and added this part here because:
 - there were comments mentioning it should be done;
 - this code path was hit in the case of ASLR because the offset it
   applies did not fit in the address range.

 Ironically in the case of ASLR it was not possible to recover, since
 sys_munmap() checks if the address to unmap is in range; thus leaking an
 entire memory area in the process...

 > Why not create proper VA limits for aslr and set them up MD and emulation
 > specific?

 ASLR has no specific VA limit that I can see.

 -- 
 khorben

From: Martin Husemann <martin@duskware.de>
To: Pierre Pronchery <khorben@defora.org>
Cc: gnats-bugs@NetBSD.org, Christos Zoulas <christos@zoulas.com>
Subject: Re: kern/50469: PaX ASLR breaks netbsd32 emulation
Date: Sun, 20 Mar 2016 17:52:44 +0100

 On Sun, Mar 20, 2016 at 02:48:19PM +0100, Pierre Pronchery wrote:
 > ASLR has no specific VA limit that I can see.

 But the vmspace it works on has!

 Martin

State-Changed-From-To: open->pending-pullups
State-Changed-By: khorben@NetBSD.org
State-Changed-When: Mon, 28 Mar 2016 21:58:10 +0000
State-Changed-Why:
Awaiting pullup-7 #1149


State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Thu, 17 Aug 2017 19:00:19 +0000
State-Changed-Why:
The original pullup ticket was refused, and there were numerous other
changes which were not pulled up. The changes are on netbsd-8, I think
it's unrealistic to get them pulled up to previous releases.


>Unformatted:
 netbsd-7 from beginning of October 2015
 $NetBSD: uvm_mmap.c,v 1.148.4.2 2015/01/11 06:27:40 snj Exp $

Home	PR Database Search