NetBSD Problem Report #46217

From root@forglen.anduin.org.uk  Sat Mar 17 19:13:48 2012
Return-Path: <root@forglen.anduin.org.uk>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	by www.NetBSD.org (Postfix) with ESMTP id 152CE63D96B
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 17 Mar 2012 19:13:48 +0000 (UTC)
Message-Id: <20120317175829.C75CC5683B@forglen.anduin.org.uk>
Date: Sat, 17 Mar 2012 17:58:29 +0000 (GMT)
From: root@forglen.anduin.org.uk
Reply-To: root@forglen.anduin.org.uk
To: gnats-bugs@gnats.NetBSD.org
Subject: NetBSD 6.0_BETA hangs with CARP enabled 
X-Send-Pr-Version: 3.95

>Number:         46217
>Category:       kern
>Synopsis:       Configuring CARP leads to hangs on boot
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bouyer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Mar 17 19:15:01 +0000 2012
>Closed-Date:    Fri Apr 13 00:58:48 +0000 2012
>Last-Modified:  Fri Apr 13 00:58:48 +0000 2012
>Originator:     Dave Tyson
>Release:        NetBSD 6.0_BETA
>Organization:
	Wirral Caving Group
>Environment:


System: NetBSD forglen.anduin.org.uk 6.0_BETA NetBSD 6.0_BETA (GENERIC) #0: Wed Mar 14 12:37:03 GMT 2012 root@forglen.anduin.org.uk:/usr/obj/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:
	Compile and install a kernel with CARP enabled. Configure a
carp interface and reboot machine. Note that the machine locks up during
startup. Cannot be pinged on any interface.
>How-To-Repeat:
         Edit GENERIC config to uncomment CARP. Compile. Create a carp
interface e.g:

cat /etc/ifconfig.carp0                                  
create
vhid 1 192.168.0.220 netmask 255.255.255.0

Boot system:

> boot netbsd.carp
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 6.0_BETA (CARP) #0: Wed Mar 14 14:38:44 GMT 2012
        root@forglen.anduin.org.uk:/usr/obj/sys/arch/i386/compile/CARP
total memory = 1023 MB
avail memory = 992 MB
mainbus0 (root)
cpu0 at mainbus0 apid 0: Intel(R) Pentium(R) 4 CPU 1.60GHz, id 0xf12
ioapic0 at mainbus0 apid 2
acpi0 at mainbus0: Intel ACPICA 20110623
acpibut0 at acpi0 (PBTN, PNP0C0C): ACPI Power Button
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
npx1 at acpi0 (COPR, PNP0C04): io 0xf0-0xff irq 13
pckbc1 at acpi0 (PS2M, PNP0F03) (aux port): irq 12
pckbc2 at acpi0 (PS2K, PNP0303) (kbd port): io 0x60,0x64 irq 1
FDC0 (PNP0700) at acpi0 not configured
UAR1 (PNP0501) at acpi0 not configured
LPT (PNP0400) at acpi0 not configured
FWH (INT0800) at acpi0 not configured
MEM (PNP0C01) at acpi0 not configured
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
pckbd0 at pckbc2 (kbd slot)
pckbc2: using irq 1 for kbd slot
wskbd0 at pckbd0 mux 1
pms0 at pckbc2 (aux slot)
pckbc2: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 0 function 0: vendor 0x8086 product 0x1a30 (rev. 0x04)
agp0 at pchb0: aperture at 0xe0000000, size 0x10000000
ppb0 at pci0 dev 1 function 0: vendor 0x8086 product 0x1a31 (rev. 0x04)
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0: vendor 0x10de product 0x0111 (rev. 0xb2)
wsdisplay0 at vga1 kbdmux 1
drm at vga1 not configured
ppb1 at pci0 dev 30 function 0: vendor 0x8086 product 0x244e (rev. 0x05)
pci2 at ppb1 bus 2
fxp0 at pci2 dev 8 function 0: Intel i82562 Ethernet (rev. 0x03)
fxp0: interrupting at ioapic0 pin 20
fxp0: Ethernet address 00:03:47:e6:31:34
inphy0 at fxp0 phy 1: i82562ET 10/100 media interface, rev. 0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
bktr has not been converted to device_t
bktr0 at pci2 dev 10 function 0
bktr0: interrupting at ioapic0 pin 22
bktr0: Pinnacle/Miro TV, Temic NTSC tuner.
puc0 at pci2 dev 11 function 0: Nanjing QinHeng Electronics 2S (com, com)
com3 at puc0 port 0: interrupting at ioapic0 pin 23
com3: ns16550a, working fifo
com4 at puc0 port 1: interrupting at ioapic0 pin 23
com4: ns16550a, working fifo
ichlpcib0 at pci0 dev 31 function 0: vendor 0x8086 product 0x2440 (rev. 0x05)
ichlpcib0: 24-bit timer
gpio0 at ichlpcib0: 64 pins
fwhrng0 at ichlpcib0: Intel Firmware Hub Random Number Generator
piixide0 at pci0 dev 31 function 1: Intel 82801BA IDE Controller (ICH2) (rev. 
0x05)
piixide0: primary channel ignored (disabled)
piixide0: secondary channel interrupting at ioapic0 pin 15
atabus0 at piixide0 channel 1
uhci0 at pci0 dev 31 function 2: vendor 0x8086 product 0x2442 (rev. 0x05)
uhci0: interrupting at ioapic0 pin 19
usb0 at uhci0: USB revision 1.0
ichsmb0 at pci0 dev 31 function 3: vendor 0x8086 product 0x2443 (rev. 0x05)
ichsmb0: interrupting at ioapic0 pin 17
iic0 at ichsmb0: I2C bus
spdmem0 at iic0 addr 0x51
spdmem0: DDR SDRAM, no parity or ECC, 512MB, 286MHz (PC-2300)
uhci1 at pci0 dev 31 function 4: vendor 0x8086 product 0x2444 (rev. 0x05)
uhci1: interrupting at ioapic0 pin 23
usb1 at uhci1: USB revision 1.0
auich0 at pci0 dev 31 function 5: i82801BA (ICH2) AC-97 Audio
auich0: interrupting at ioapic0 pin 17
auich0: ac97: Analog Devices AD1885 codec; headphone, Analog Devices Phat 
Stereo
auich0: ac97: ext id 0x1<VRA>
isa0 at ichlpcib0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
acpicpu0 at cpu0: ACPI CPU
audio0 at auich0: full duplex, playback, capture, mmap, independent
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
uhub0 at usb0: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1 at usb1: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 1: <HITACHI CDR-8435, , 0010> cdrom removable
wd0 at atabus0 drive 0
wd0: <ST340014A>
wd0: 38146 MB, 77504 cyl, 16 head, 63 sec, 512 bytes/sect x 78125000 sectors
uhub2 at uhub1 port 2: vendor 0x8086 product 0x1122, class 9/0, rev 1.10/0.00, 
addr 2
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
Sat Mar 17 14:46:04 GMT 2012
Starting root file system check:
/dev/rwd0a: file system is clean; not checking
swapctl: setting dump device to /dev/wd0b
swapctl: adding /dev/wd0b as swap device at priority 0
Starting file system checks:
/dev/rwd0e: file system is clean; not checking
/dev/rwd0f: file system is clean; not checking
/dev/rwd0g: file system is clean; not checking
/dev/rwd0h: file system is clean; not checking
Setting tty flags.
Setting sysctl variables:
ddb.onpanic: 1 -> 0
Starting network.
Hostname: forglen.anduin.org.uk
route: writing to routing socket: File exists
IPv6 mode: host
Configuring network interfaces: fxp0 carp0.
Adding interface aliases:.
add net default: gateway 192.168.0.1
Building databases: dev, utmp, utmpx.
Keyboard encoding -> uk
Starting syslogd.
Mounting all filesystems...
Clearing temporary files.
Updating fontconfig cache:

<<<<break sent to interrupt hang>>>>

fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c026e8a4 cs 8 eflags 282 cr2 bb80b000 ilevel 8
Stopped in pid 0.2 (system) at  netbsd:breakpoint+0x4:  popl    %ebp
db{0}> bt
breakpoint(c0c44f80,3f8,5,c0c45940,0,c0cd2480,d9876f6c,c056dbce,228,0) at 
netbsd
:breakpoint+0x4
comintr(c20e3280,d99ccc18,ffe7f7ef,bdfffffe,ffffffff,77ffffff,fcfefd7f,ff7fdffc,
fefbfff7,ffffff7f) at netbsd:comintr+0x599
--- switch to interrupt stack ---
Xintr_ioapic_edge10() at netbsd:Xintr_ioapic_edge10+0xb5
--- interrupt ---
x86_stihlt(1,1,7,c20f6d20,c0c45940,c24fa7e0,c20f6d20,c054bf9d,c2405da8,0) at 
net
bsd:x86_stihlt+0x5
acpicpu_cstate_idle_enter(c2405da8,0,d99cccec,c055a64e,c0c45940,c20f4d40,c0c4594
0,d99cccec,c20f6d20,c0c45940) at netbsd:acpicpu_cstate_idle_enter+0x42
acpicpu_cstate_idle(c20f6d20,0,c05466a2,c20f6d20,33ce33d4,cc23cc73,33cc33cc,cc33
cc23,37cc33cc,cc33cc33) at netbsd:acpicpu_cstate_idle+0xa8
idle_loop(c20f6d20,e89000,e92000,0,c0100307,0,0,0,0,0) at 
netbsd:idle_loop+0x17c

db{0}> ps
PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
217      1 3   0        80           c24fa7e0           fc-cache nanoslp
221      1 3   0        80           c24fad20                 sh wait
177      1 2   0         0           c24fa2a0            syslogd
11       1 3   0        80           c24cf020                 sh pipe_rd
10       1 3   0        80           c24cf2c0                 sh wait
9        1 3   0        80           c24cf560                 sh wait
8        1 3   0        80           c24cf800                 sh wait
2        1 3   0        80           c24cfaa0                 sh wait
1        1 3   0        80           c2407d20               init wait
0       43 3   0       200           c24fa540            physiod physiod
0       42 3   0       200           c24cfd40           aiodoned aiodoned
0       41 3   0       200           c23ff800            ioflush syncer
0       40 3   0       200           c2407000           pgdaemon pgdaemon
0       37 3   0       200           c23ff020          cryptoret crypto_w
0       36 3   0       200           c23ffaa0               usb1 usbevt
0       35 3   0       200           c24072a0         usbtask-dr usbtsk
0       34 3   0       200           c2407540         usbtask-hc usbtsk
0       33 3   0       200           c23ff560               usb0 usbevt
0       32 3   0       200           c23ff2c0          atapibus0 sccomp
0       30 3   0       200           c2407a80              unpgc unpgc
0       29 3   0       200           c24077e0        vmem_rehash vmem_rehash
0       20 3   0       200           c23ffd40               iic0 iicintr
0       19 3   0       200           c2366000            atabus0 atath
0       18 3   0       200           c23662a0               pms0 pmsreset
0       17 3   0       200           c2366540               apm0 apmev
0       16 3   0       200           c23667e0             sysmon smtaskq
0       15 3   0       200           c2366a80         pmfsuspend pmfsuspend
0       14 3   0       200           c2366d20           pmfevent pmfevent
0       13 3   0       200           c20fe020         sopendfree sopendfr
0       12 3   0       200           c20fe2c0           nfssilly nfssilly
0       11 3   0       200           c20fe560            cachegc cachegc
0       10 3   0       200           c20fe800              vrele vrele
0        9 3   0       200           c20feaa0             vdrain vdrain
0        8 3   0       200           c20fed40          modunload mod_unld
0        7 3   0       200           c20f6000            xcall/0 xcall
0        6 1   0       200           c20f62a0          softser/0
0        5 3   0       200           c20f6540          softclk/0 carpdel
0        4 1   0       200           c20f67e0          softbio/0
0        3 1   0       200           c20f6a80          softnet/0
0    >   2 7   0       201           c20f6d20             idle/0
0        1 3   0       200           c0ca0660            swapper uvm
db{0}> sync
dumping to dev 0,1 offset 8
dump fatal page fault in supervisor mode
trap type 6 code 2 eip c02b576c cs 8 eflags 10246 cr2 dabc3dc0 ilevel 8
panic: trap
cpu0: Begin traceback...
printf_nolog(c0c0e768,d9876b18,d9876b18,c02b576c,8,10246,dabc3dc0,8,c054bbc4,c0cdd9c6) 
at netbsd:printf_nolog
trap_tss() at netbsd:trap_tss
--- trap via task gate ---
1:
cpu0: End traceback...

dumping to dev 0,1 offset 8
dump fatal page fault in supervisor mode
trap type 6 code 2 eip c02b576c cs 8 eflags 10246 cr2 dabc3dc0 ilevel 8
Skipping crash dump on recursive panic
panic: trap
cpu0: Begin traceback...
printf_nolog(c0c0e768,d9876914,d9876914,c02b576c,8,10246,dabc3dc0,8,c054bbc4,c0cdd9c6) 
at netbsd:printf_nolog
trap_tss() at netbsd:trap_tss
--- trap via task gate ---
1:
cpu0: End traceback...
rebooting...

>Fix:
	not known

>Release-Note:

>Audit-Trail:

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/46217: NetBSD 6.0_BETA hangs with CARP enabled
Date: Sat, 24 Mar 2012 15:08:56 +0100

 On Sat, Mar 17, 2012 at 07:15:01PM +0000, root@forglen.anduin.org.uk wrote:
 > System: NetBSD forglen.anduin.org.uk 6.0_BETA NetBSD 6.0_BETA (GENERIC) #0: Wed Mar 14 12:37:03 GMT 2012 root@forglen.anduin.org.uk:/usr/obj/sys/arch/i386/compile/GENERIC i386
 > Architecture: i386
 > Machine: i386
 > >Description:
 > 	Compile and install a kernel with CARP enabled. Configure a
 > carp interface and reboot machine. Note that the machine locks up during
 > startup. Cannot be pinged on any interface.
 > >How-To-Repeat:
 >          Edit GENERIC config to uncomment CARP. Compile. Create a carp
 > interface e.g:
 > 
 > cat /etc/ifconfig.carp0                                  
 > create
 > vhid 1 192.168.0.220 netmask 255.255.255.0
 > 
 > Boot system:
 > 
 > [...]
 > Updating fontconfig cache:
 > 
 > <<<<break sent to interrupt hang>>>>
 > 
 > fatal breakpoint trap in supervisor mode
 > trap type 1 code 0 eip c026e8a4 cs 8 eflags 282 cr2 bb80b000 ilevel 8
 > Stopped in pid 0.2 (system) at  netbsd:breakpoint+0x4:  popl    %ebp
 > db{0}> bt
 > breakpoint(c0c44f80,3f8,5,c0c45940,0,c0cd2480,d9876f6c,c056dbce,228,0) at 
 > netbsd
 > :breakpoint+0x4
 > comintr(c20e3280,d99ccc18,ffe7f7ef,bdfffffe,ffffffff,77ffffff,fcfefd7f,ff7fdffc,
 > fefbfff7,ffffff7f) at netbsd:comintr+0x599
 > --- switch to interrupt stack ---
 > Xintr_ioapic_edge10() at netbsd:Xintr_ioapic_edge10+0xb5
 > --- interrupt ---
 > x86_stihlt(1,1,7,c20f6d20,c0c45940,c24fa7e0,c20f6d20,c054bf9d,c2405da8,0) at 
 > net
 > bsd:x86_stihlt+0x5
 > acpicpu_cstate_idle_enter(c2405da8,0,d99cccec,c055a64e,c0c45940,c20f4d40,c0c4594
 > 0,d99cccec,c20f6d20,c0c45940) at netbsd:acpicpu_cstate_idle_enter+0x42
 > acpicpu_cstate_idle(c20f6d20,0,c05466a2,c20f6d20,33ce33d4,cc23cc73,33cc33cc,cc33
 > cc23,37cc33cc,cc33cc33) at netbsd:acpicpu_cstate_idle+0xa8
 > idle_loop(c20f6d20,e89000,e92000,0,c0100307,0,0,0,0,0) at 
 > netbsd:idle_loop+0x17c
 > 
 > db{0}> ps
 > PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 > 217      1 3   0        80           c24fa7e0           fc-cache nanoslp

 Some debugging shows that softclock (and so scheduling) is not
 working any more.

 This is caused by ip_carp.c 1.42 -> 1.44; you actually can't sleep in
 the softclock thread because it will never wake up. The attached patch
 (reverting ip_carp.c 1.42 -> 1.44) fixes the problem for me.

 Antti, what is exactly the ABI issue with DELAY() you mention in 
 the commit message ? kern/40505 doesn't give lots of details either.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/46217: NetBSD 6.0_BETA hangs with CARP enabled
Date: Sat, 24 Mar 2012 17:32:21 +0100

 --0eh6TmSyL6TZE2Uz
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 On Sat, Mar 24, 2012 at 03:08:56PM +0100, Manuel Bouyer wrote:
 > 
 > This is caused by ip_carp.c 1.42 -> 1.44; you actually can't sleep in
 > the softclock thread because it will never wake up. The attached patch
 > (reverting ip_carp.c 1.42 -> 1.44) fixes the problem for me.

 I think we can remove the delay completely; please try the attached
 patch instead

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

 --0eh6TmSyL6TZE2Uz
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename=diff

 Index: ip_carp.c
 ===================================================================
 RCS file: /cvsroot/src/sys/netinet/ip_carp.c,v
 retrieving revision 1.47
 diff -u -p -u -r1.47 ip_carp.c
 --- ip_carp.c	19 Nov 2011 22:51:25 -0000	1.47
 +++ ip_carp.c	24 Mar 2012 16:30:22 -0000
 @@ -1146,9 +1146,6 @@ carp_send_arp(struct carp_softc *sc)

  		in = &ifatoia(ifa)->ia_addr.sin_addr;
  		arprequest(sc->sc_carpdev, in, in, CLLADDR(sc->sc_if.if_sadl));
 -
 -		/* XXX: why does this need to delay? */
 -		kpause("carpdel", false, MAX(1, mstohz(1)), NULL);
  	}
  	splx(s);
  }
 @@ -1170,9 +1167,6 @@ carp_send_na(struct carp_softc *sc)
  		in6 = &ifatoia6(ifa)->ia_addr.sin6_addr;
  		nd6_na_output(sc->sc_carpdev, &mcast, in6,
  		    ND_NA_FLAG_OVERRIDE, 1, NULL);
 -
 -		/* XXX: why does this need to delay? */
 -		kpause("carpdel6", false, MAX(1, mstohz(1)), NULL);
  	}
  	splx(s);
  }

 --0eh6TmSyL6TZE2Uz--

From: Dave Tyson <dtyson@wirralcavinggroup.org.uk>
To: gnats-bugs@gnats.netbsd.org
Cc: Manuel Bouyer <bouyer@antioche.eu.org>,
 current-users@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: kern/46217: NetBSD 6.0_BETA hangs with CARP enabled
Date: Sun, 25 Mar 2012 11:54:05 +0100

 I tested the patch and it fixes the hang. A quick test shows the CARP 
 interface responds to pings. I haven't tested failover yet as I need to
 configure up another system.

 Thanks for the fast response. Could you commit it and request a pullup to 
 NetBSD-6

 Cheers,
 Dave

 -- 
 =====================================================================
 Phone: 07805784357
 Open Source O/S: www.netbsd.org
 Caving: http://www.wirralcavinggroup.org.uk
 =====================================================================

From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/46217 CVS commit: src/sys/netinet
Date: Tue, 27 Mar 2012 17:48:18 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Tue Mar 27 17:48:18 UTC 2012

 Modified Files:
 	src/sys/netinet: ip_carp.c

 Log Message:
 Do not sleep in callout context, this will hang the clock soft interrupt.
 Should fix PR kern/46217.


 To generate a diff of this commit:
 cvs rdiff -u -r1.47 -r1.48 src/sys/netinet/ip_carp.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Responsible-Changed-From-To: kern-bug-people->bouyer
Responsible-Changed-By: bouyer@NetBSD.org
Responsible-Changed-When: Tue, 27 Mar 2012 17:54:42 +0000
Responsible-Changed-Why:
I commited a fix.


State-Changed-From-To: open->pending-pullups
State-Changed-By: bouyer@NetBSD.org
State-Changed-When: Tue, 27 Mar 2012 17:54:42 +0000
State-Changed-Why:
pullup-6 #145


From: "Jeff Rizzo" <riz@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/46217 CVS commit: [netbsd-6] src/sys/netinet
Date: Mon, 2 Apr 2012 18:25:35 +0000

 Module Name:	src
 Committed By:	riz
 Date:		Mon Apr  2 18:25:35 UTC 2012

 Modified Files:
 	src/sys/netinet [netbsd-6]: ip_carp.c

 Log Message:
 Pull up following revision(s) (requested by bouyer in ticket #145):
 	sys/netinet/ip_carp.c: revision 1.48
 Do not sleep in callout context, this will hang the clock soft interrupt.
 Should fix PR kern/46217.


 To generate a diff of this commit:
 cvs rdiff -u -r1.47 -r1.47.4.1 src/sys/netinet/ip_carp.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Fri, 13 Apr 2012 00:58:48 +0000
State-Changed-Why:
Fixed and pulled up.


>Unformatted:
 	no

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.