NetBSD Problem Report #43294

From martin@aprisoft.de  Wed May 12 09:49:06 2010
Return-Path: <martin@aprisoft.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id DC61B63B8FE
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 12 May 2010 09:49:05 +0000 (UTC)
Message-Id: <20100512083737.8036FAF5816@emmas.aprisoft.de>
Date: Wed, 12 May 2010 10:37:37 +0200 (CEST)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@gnats.NetBSD.org
Subject: locking against myself in ioctl setting up a gre interface
X-Send-Pr-Version: 3.95

>Number:         43294
>Category:       kern
>Synopsis:       locking against myself in ioctl setting up a gre interface
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed May 12 09:50:01 +0000 2010
>Closed-Date:    Tue Feb 04 17:16:34 +0000 2014
>Last-Modified:  Tue Feb 04 17:16:34 +0000 2014
>Originator:     Martin Husemann
>Release:        NetBSD 5.99.29
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD nelly.aprisoft.de 5.99.29 NetBSD 5.99.29 (NELLY.MP) #127: Mon May 10 14:49:18 CEST 2010 martin@emmas.aprisoft.de:/nelly/usr/src/sys/arch/sparc64/compile/NELLY.MP sparc64
Architecture: sparc64
Machine: sparc64
>Description:

On a LOCKDEBUG kernel run the following command (ip numbers are arbitrary):

ifconfig gre0 create
ifconfig gre0 tunnel 10.1.1.1 10.1.2.1
ifconfig gre0 192.168.170.1/30

Watch a LOCKDEBUG panic:

Mutex error: lockdebug_wantlock: locking against myself

lock address : 0x000000000d209f40 type     :     sleep/adaptive
initialized  : 0x0000000001276be4                              
shared holds :                  0 exclusive:                  1
shares wanted:                  0 exclusive:                  1
current cpu  :                  1 last held:                  1
current lwp  : 0x000000000e6337e0 last held: 0x000000000e6337e0
last locked  : 0x00000000011ea884 unlocked : 0x00000000011ea300
owner field  : 0x000000000e6337e0 wait/spin:                0/0

Turnstile chain at 0x18b96d0.
=> No active turnstile for this lock.

panic: LOCKDEBUG
Stopped in pid 34.1 (ifconfig) at       netbsd:cpu_Debugger+0x4:        nop
db{1}> bt                                                                  
lockdebug_abort1(18d4220, 7, 17022e0, 174cb20, 1, 44732c8) at netbsd:lockdebug_a
bort1+0x7c                                                                     
mutex_enter(d209f40, eb972e8, e, 18da000, 2, 3) at netbsd:mutex_enter+0x204
sosetlock(44d7b10, 18432d8, 564, 177bc00, 2, 3) at netbsd:sosetlock+0x58   
rip_usrreq(38, 0, 2f, 0, e6337e0, e6337e0) at netbsd:rip_usrreq+0x12c   
rip_usrreq_wrapper(44d7b10, 0, 0, 2f, 0, e6337e0) at netbsd:rip_usrreq_wrapper+0
x20                                                                            
socreate(15f5fc8, e6cb2d0, 3, 2f, e6337e0, 0) at netbsd:socreate+0x11c
fsocreate(2, 0, 3, 2f, e6337e0, e6cb5cc) at netbsd:fsocreate+0x64     
gre_ioctl(0, c0706984, 4473200, 7, 0, 44732a8) at netbsd:gre_ioctl+0x718
in_ifinit(4475000, 4473200, 0, 0, 0, 44732c8) at netbsd:in_ifinit+0x100 
in_control(37, 8040691a, e6cbc80, 4475000, 1, e6337e0) at netbsd:in_control+0xbc
0                                                                              
udp_usrreq_wrapper(3ff3d40, b, 8040691a, e6cbc80, 4475000, e6337e0) at netbsd:ud
p_usrreq_wrapper+0x20                                                          
compat_ifioctl(3ff3d40, 8040691a, 8040691a, e6cbc80, e6337e0, badcafe) at netbsd
:compat_ifioctl+0xfc                                                           
ifioctl(2d, 8040691a, e6cbc80, e6337e0, 18b6800, e6337e0) at netbsd:ifioctl+0x1c
c                                                                              
soo_ioctl(3ff3d40, 8040691a, e6cbc80, e6c8000, 100a600, 308) at netbsd:soo_ioctl
+0x2dc                                                                         
sys_ioctl(0, e6cbdc0, e6cbe00, badcafe, badcafe, badcafe) at netbsd:sys_ioctl+0x
bc                                                                             
syscall_plain(e6cbed0, e6cbf58, 40943488, 0, 40943488, 800) at netbsd:syscall_pl
ain+0x14c                                                                      
..

Last locked field points to in_control+0xae4


>How-To-Repeat:
see above

>Fix:
pass a "already locked" flag?

>Release-Note:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/43294: locking against myself in ioctl setting up a gre interface
Date: Wed, 12 May 2010 12:01:15 +0200

 Which is:

 (gdb) list *(in_control+0xae4)
 0x11ea884 is in in_control (../../../../netinet/in.c:504).
 499                     break;
 500     
 501             case SIOCAIFADDR:
 502                     maskIsNew = 0;
 503                     hostIsNew = 1;
 504                     mutex_enter(softnet_lock);
 505                     if (ia->ia_addr.sin_family != AF_INET)
 506                             ;
 507                     else if (ifra->ifra_addr.sin_len == 0) {
 508                             ifra->ifra_addr = ia->ia_addr;

 Martin

From: Andrew Doran <ad@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: kern/43294: locking against myself in ioctl setting up a gre
	interface
Date: Wed, 12 May 2010 10:04:49 +0000

 > martin
 > this locking panic you ran into
 > someone hammered a locking fix into the socket code without getting it
 +reviewed. if i had time i would back it out, contact the comitter and post it
 +to tech-net.
 > softnet_lock was deliberately NOT taken for interface ioctl operations.
 <martin> andrew: it is now kern/43294, please comment there
 <martin> why wasn't it taken?
 > because interface ioctls do stuff like allocate memory with M_WAITOK and so
 +on.
 > it needs more complex serialization
 <martin> can of worms, I see
 > the fallback position currently is kernel_lock+splnet.
 > but there are still race conditions with that.

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/43294: locking against myself in ioctl setting up a gre
	interface
Date: Thu, 13 May 2010 06:21:37 +0000

 On Wed, May 12, 2010 at 09:50:01AM +0000, martin@NetBSD.org wrote:
  > ifconfig gre0 create
  > ifconfig gre0 tunnel 10.1.1.1 10.1.2.1
  > ifconfig gre0 192.168.170.1/30

 If the fix we know about resolves this problem, does that mean we can
 close PR 16704?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Masaru OKI <oki@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/43294 CVS commit: src/sys/netinet
Date: Sat, 15 May 2010 05:02:46 +0000

 Module Name:	src
 Committed By:	oki
 Date:		Sat May 15 05:02:46 UTC 2010

 Modified Files:
 	src/sys/netinet: in.c

 Log Message:
 Backout rev.1.137.  It causes troubles, see PR kern/43294.
 We needs more discussion/a more general solution.


 To generate a diff of this commit:
 cvs rdiff -u -r1.137 -r1.138 src/sys/netinet/in.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: dyoung@NetBSD.org
State-Changed-When: Thu, 20 Oct 2011 01:31:37 +0000
State-Changed-Why:
I've fixed this (I think) in -current, give it a try?

Dave


State-Changed-From-To: feedback->open
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sun, 06 Nov 2011 19:19:43 +0000
State-Changed-Why:
gre clearly still has serious problems


From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/43294: locking against myself in ioctl setting up a gre interface
Date: Sun, 6 Nov 2011 20:18:18 +0100

 It is not possible to reporduce the panic any more, because the commands
 given in the PR now fail:

 # ifconfig gre0 create
 # ifconfig gre0 tunnel 10.1.1.1 10.1.2.1
 # ifconfig gre0 192.168.170.1/30
 ifconfig: SIOCAIFADDR: Can't assign requested address

 But if I then do:

 # ifconfig gre0 destroy

 I get a kernel panic:

 Mutex error: kmem_free: allocation contains active lock

 lock address : 0x00000000122fb740 type     :     sleep/adaptive
 initialized  : 0x0000000001194b04                              
 shared holds :                  0 exclusive:                  0
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0x0000000012331400 last held: 000000000000000000
 last locked  : 0x0000000001195f58 unlocked*: 0x0000000001195fa0
 owner field  : 000000000000000000 wait/spin:                0/0

 Turnstile chain at 0x1c85cc0.
 => No active turnstile for this lock.

 panic: LOCKDEBUG
 Stopped in pid 380.1 (ifconfig) at      netbsd:cpu_Debugger+0x4:        nop

From: David Young <dyoung@pobox.com>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, martin@NetBSD.org
Subject: Re: kern/43294: locking against myself in ioctl setting up a gre
 interface
Date: Wed, 16 Nov 2011 00:21:51 -0600

 On Sun, Nov 06, 2011 at 07:20:05PM +0000, Martin Husemann wrote:
 > The following reply was made to PR kern/43294; it has been noted by GNATS.
 > 
 > From: Martin Husemann <martin@duskware.de>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: kern/43294: locking against myself in ioctl setting up a gre interface
 > Date: Sun, 6 Nov 2011 20:18:18 +0100
 > 
 >  It is not possible to reporduce the panic any more, because the commands
 >  given in the PR now fail:
 >  
 >  # ifconfig gre0 create
 >  # ifconfig gre0 tunnel 10.1.1.1 10.1.2.1
 >  # ifconfig gre0 192.168.170.1/30
 >  ifconfig: SIOCAIFADDR: Can't assign requested address

 EADDRNOTAVAIL probably comes from the socreate() call that occurs when
 the interface is brought UP by assigning an address.  I think the
 problem is that 10.1.1.1 is not assigned to any interface, so gre0
 cannot sobind() it.  If there should be an error at all, it should be at
 'ifconfig gre0 tunnel 10.1.1.1 ...'.

 >  But if I then do:
 >  
 >  # ifconfig gre0 destroy
 >  
 >  I get a kernel panic:
 >  
 >  Mutex error: kmem_free: allocation contains active lock

 I think that I have fixed this with the latest revision of sys/net/if.c.

 Dave

 -- 
 David Young
 dyoung@pobox.com    Urbana, IL    (217) 721-9981

State-Changed-From-To: open->feedback
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sat, 01 Feb 2014 19:00:49 +0000
State-Changed-Why:
need to verify the fix


State-Changed-From-To: feedback->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Tue, 04 Feb 2014 17:16:34 +0000
State-Changed-Why:
fixed


>Unformatted:
 Applies to NetBSD 5.1RC1 as well, fix needs urgent pullup!

Home	PR Database Search