NetBSD Problem Report #41139

From tsutsui@ceres.dti.ne.jp  Sat Apr  4 13:24:07 2009
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id 24EC663B8A5
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  4 Apr 2009 13:24:07 +0000 (UTC)
Message-Id: <200904041323.n34DNxIS016753@mirage.ceres.dti.ne.jp>
Date: Sat, 4 Apr 2009 22:23:59 +0900 (JST)
From: tsutsui@ceres.dti.ne.jp
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@gnats.NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: pipe(2) seems broken on VAX
X-Send-Pr-Version: 3.95

>Number:         41139
>Category:       port-vax
>Synopsis:       pipe(2) seems broken on VAX
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-vax-maintainer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Apr 04 13:25:00 +0000 2009
>Closed-Date:    Sun Apr 12 11:21:47 +0000 2009
>Last-Modified:  Thu Apr 30 18:15:01 +0000 2009
>Originator:     Izumi Tsutsui
>Release:        NetBSD 5.0_RC3
>Organization:
>Environment:
System: NetBSD 5.0_RC3 (GENERIC) #0: Fri Apr  3 08:50:02 UTC 2009  \
builds@b6.netbsd.org:/home/builds/ab/netbsd-5/vax/200904020000Z-obj/\
home/builds/ab/netbsd-5/src/sys/arch/vax/compile/GENERIC vax
Architecture: vax
Machine: vax
Running on simh-vax
>Description:
pipe(2) seems broken on vax.
"tar xzvf /tmp/bast.tgz" causes the following error
---
# tar xzpf base.tgz
tar: Cannot identify format. Searching...
---
as noted in this thread:
http://mail-index.NetBSD.org/port-vax/2009/03/31/msg000641.html

5.0_RC1 and -current GENERIC kernels have the same problem.

>How-To-Repeat:
Install NetBSD/vax 5.0_RC3 onto simh, then try the following command:

---
 :
NetBSD 5.0_RC3 (GENERIC) #0: Fri Apr 3 08:50:02 UTC 2009

Welcome to NetBSD!
 :
Terminal type? [unknown] 
Terminal type is unknown.
We recommend creating a non-root account and using su(1) for root access.
# gzip -c /netbsd > /tmp/test
# gunzip -t < /tmp/test
# gzip -c /netbsd | gunzip -t
gunzip: unknown compression format
# 
---

5.0_RC1 GENERIC says:

---
 :
NetBSD 5.0_RC1 (GENERIC) #0: Wed Jan 28 14:22:28 PST 2009

Welcome to NetBSD!
 :
Terminal type? [unknown] 
Terminal type is unknown.
We recommend creating a non-root account and using su(1) for root access.
# gzip -c /netbsd > /tmp/test
# gunzip -t < /tmp/test
# gzip -c /netbsd | gunzip -t
gunzip: data stream error
# 
---       

>Fix:
Unknown, but GENERIC kernel compiled with "options PIPE_SOCKETPAIR"
works fine.
(so INSTALL kernel doesn't have the problem on extracting binary sets)

---
Izumi Tsutsui

>Release-Note:

>Audit-Trail:
From: "Michael L. Hitch" <mhitch@lightning.msu.montana.edu>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Tue, 7 Apr 2009 14:48:39 -0600 (MDT)

    The problem appears to only occur when the requested transfer size is > 
 PIPE_SIZE, and an additional buffer of BIG_PIPE_SIZE is allocated and 
 used.  The defaults are 16KB and 64KB respectively.  Setting BIG_PIPE_SIZE 
 to a smaller value (I've tested 63KB) seems to cure the problem.  The real 
 root of the problem is that sys_pipe.c uses uiomove() to transfer the 
 data, which uses copyin()/copyout().  On the vax, this is done with the 
 movc3 instruction, which uses a 16-bit length, and will not transfer 
 anything when the length is 0 (which it will be with a 64KB size).

    The simplest fix would be to set BIG_PIPE_SIZE in 
 sys/arch/vax/include/param.h.  There does appear to be an improvment 
 between using 32KB and 63KB in the one test I did.

    A better fix would be to make copyin()/copyout() work with 64KB or 
 larger sizes.  I'm not sure what performance impact that might have 
 though.  The quick test I just did shows that just fixing copyin/copyout 
 to check for 64KB and transfer one byte with movb before doing the movc3 
 seems to show that it was slightly slower than when I was using a 63KB 
 BIG_PIPE_SIZE.

    One other note - this also appears to have been the problem with using 
 scp on the vax to transfer files.  My scp copies no longer get corrupted 
 data.  [Since scp does a pipe to run the remote ssh command and copies the 
 data from the pipe, this makes sense.]

 --
 Michael L. Hitch			mhitch@montana.edu
 Computer Consultant
 Information Technology Center
 Montana State University	Bozeman, MT	USA

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, port-vax-maintainer@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Cc: 
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Tue, 7 Apr 2009 17:41:00 -0400

 On Apr 7,  9:25pm, mhitch@lightning.msu.montana.edu ("Michael L. Hitch") wrote:
 -- Subject: Re: port-vax/41139: pipe(2) seems broken on VAX

 | The following reply was made to PR port-vax/41139; it has been noted by GNATS.
 | 
 | From: "Michael L. Hitch" <mhitch@lightning.msu.montana.edu>
 | To: gnats-bugs@NetBSD.org
 | Cc: 
 | Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
 | Date: Tue, 7 Apr 2009 14:48:39 -0600 (MDT)
 | 
 |     The problem appears to only occur when the requested transfer size is > 
 |  PIPE_SIZE, and an additional buffer of BIG_PIPE_SIZE is allocated and 
 |  used.  The defaults are 16KB and 64KB respectively.  Setting BIG_PIPE_SIZE 
 |  to a smaller value (I've tested 63KB) seems to cure the problem.  The real 
 |  root of the problem is that sys_pipe.c uses uiomove() to transfer the 
 |  data, which uses copyin()/copyout().  On the vax, this is done with the 
 |  movc3 instruction, which uses a 16-bit length, and will not transfer 
 |  anything when the length is 0 (which it will be with a 64KB size).
 |  
 |     The simplest fix would be to set BIG_PIPE_SIZE in 
 |  sys/arch/vax/include/param.h.  There does appear to be an improvment 
 |  between using 32KB and 63KB in the one test I did.
 |  
 |     A better fix would be to make copyin()/copyout() work with 64KB or 
 |  larger sizes.  I'm not sure what performance impact that might have 
 |  though.  The quick test I just did shows that just fixing copyin/copyout 
 |  to check for 64KB and transfer one byte with movb before doing the movc3 
 |  seems to show that it was slightly slower than when I was using a 63KB 
 |  BIG_PIPE_SIZE.
 |  
 |     One other note - this also appears to have been the problem with using 
 |  scp on the vax to transfer files.  My scp copies no longer get corrupted 
 |  data.  [Since scp does a pipe to run the remote ssh command and copies the 
 |  data from the pipe, this makes sense.]


 Can't we do?

 	while (len >= 64K) {
 		movc3 64K-1, ...
 		len -= 64K-1;
 	}
 	movc3 len, ...

 In kcopy, copyin and copyout?

 christos

From: Anders Magnusson <ragge@ludd.ltu.se>
To: Christos Zoulas <christos@zoulas.com>
Cc: gnats-bugs@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org,
        tsutsui@ceres.dti.ne.jp
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Wed, 08 Apr 2009 08:01:21 +0200

 Christos Zoulas wrote:
 > |     One other note - this also appears to have been the problem with using 
 > |  scp on the vax to transfer files.  My scp copies no longer get corrupted 
 > |  data.  [Since scp does a pipe to run the remote ssh command and copies the 
 > |  data from the pipe, this makes sense.]
 > 
 > 
 > Can't we do?
 > 
 > 	while (len >= 64K) {
 > 		movc3 64K-1, ...
 > 		len -= 64K-1;
 > 	}
 > 	movc3 len, ...
 > 
 > In kcopy, copyin and copyout?
 > 
 It may be worth trying, but it also may require significantly more code.
 The movc3 instruction handles overlapping copies, but if the data block is
 larger than 64k then it must be done manually.  This is only a problem for kcopy,
 copy{in,out} should not overlap :-)

 The cvs problem is most likely something else, it existed before the fast pipe code.

 -- Ragge



From: "Michael L. Hitch" <mhitch@lightning.msu.montana.edu>
To: Anders Magnusson <ragge@ludd.ltu.se>
Cc: Christos Zoulas <christos@zoulas.com>, gnats-bugs@NetBSD.org
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Wed, 8 Apr 2009 12:57:29 -0600 (MDT)

 On Wed, 8 Apr 2009, Anders Magnusson wrote:

 > It may be worth trying, but it also may require significantly more code.
 > The movc3 instruction handles overlapping copies, but if the data block is
 > larger than 64k then it must be done manually.  This is only a problem for kcopy,
 > copy{in,out} should not overlap :-)

    It's not all that much code, at least for copy{in,out}:

 ENTRY(copyout, 0)
          movl    8(%ap),%r3
          blss    3f              # kernel space
          movl    4(%ap),%r1
          brb     2f

 ENTRY(copyin, 0)
          movl    4(%ap),%r1
          blss    3f              # kernel space
          movl    8(%ap),%r3
 2:      mfpr    $PR_ESP,%r2
          movab   1f,(%r2)
 4:      tstw    14(%ap)         # check if >= 64K
          bneq    5f
          movc3   12(%ap),(%r1),(%r3)
 1:      mfpr    $PR_ESP,%r2
          clrl    (%r2)
          ret
 5:      movc3   $0xfffc,(%r1),(%r3)
          subl2   $0xfffc,12(%ap)
          brb     4b

 3:      mnegl   $1,%r0
          ret

    I did a 64K-4 byte copy so that aligned copies stay aligned, and also 
 changed one register so that the destination address didn't need to be 
 adjusted between the movc3 instructions.

    I've got the same think in kcopy, but haven't added anything to deal 
 with overlapping copies yet.  At the moment, if there are any overlapping 
 copies >= 64K, it's broken anyway since most of the copy is't going to 
 happen anyway with the old kcopy.

 > The cvs problem is most likely something else, it existed before the fast pipe code.

    Well, I just did a cvs update to anoncvs using ssh and had no problems 
 doing it - which I don't think I had ever seen before.

 --
 Michael L. Hitch			mhitch@montana.edu
 Computer Consultant
 Information Technology Center
 Montana State University	Bozeman, MT	USA

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: port-vax-maintainer@NetBSD.org, mhitch@lightning.msu.montana.edu,
        gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org,
        tsutsui@ceres.dti.ne.jp
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Thu, 9 Apr 2009 20:47:08 +0900

 mhitch@lightning.msu.montana.edu wrote:

 >  On Wed, 8 Apr 2009, Anders Magnusson wrote:
 >  
 >  > It may be worth trying, but it also may require significantly more code.
 >  > The movc3 instruction handles overlapping copies, but if the data block is
 >  > larger than 64k then it must be done manually.  This is only a problem for kcopy,
 >  > copy{in,out} should not overlap :-)
 >  
 >     It's not all that much code, at least for copy{in,out}:
 >  
 >  ENTRY(copyout, 0)
 >           movl    8(%ap),%r3
 >           blss    3f              # kernel space
 >           movl    4(%ap),%r1
 >           brb     2f
 >  
 >  ENTRY(copyin, 0)
 >           movl    4(%ap),%r1
 >           blss    3f              # kernel space
 >           movl    8(%ap),%r3
 >  2:      mfpr    $PR_ESP,%r2
 >           movab   1f,(%r2)
 >  4:      tstw    14(%ap)         # check if >= 64K
 >           bneq    5f
 >           movc3   12(%ap),(%r1),(%r3)
 >  1:      mfpr    $PR_ESP,%r2
 >           clrl    (%r2)
 >           ret
 >  5:      movc3   $0xfffc,(%r1),(%r3)
 >           subl2   $0xfffc,12(%ap)
 >           brb     4b
 >  
 >  3:      mnegl   $1,%r0
 >           ret

 This fixes the pipe problem on 5.0.
 Should we commit it and pull up to 5.0?
 ---
 Izumi Tsutsui

From: Anders Magnusson <ragge@ludd.ltu.se>
To: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Cc: gnats-bugs@NetBSD.org, mhitch@lightning.msu.montana.edu,
        gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: port-vax/41139: pipe(2) seems broken on VAX
Date: Fri, 10 Apr 2009 09:00:23 +0200

 Izumi Tsutsui wrote:
 > mhitch@lightning.msu.montana.edu wrote:
 > 
 >>  On Wed, 8 Apr 2009, Anders Magnusson wrote:
 >>  
 >>  > It may be worth trying, but it also may require significantly more code.
 >>  > The movc3 instruction handles overlapping copies, but if the data block is
 >>  > larger than 64k then it must be done manually.  This is only a problem for kcopy,
 >>  > copy{in,out} should not overlap :-)
 >>  
 >>     It's not all that much code, at least for copy{in,out}:
 >>  
 >>  ENTRY(copyout, 0)
 >>           movl    8(%ap),%r3
 >>           blss    3f              # kernel space
 >>           movl    4(%ap),%r1
 >>           brb     2f
 >>  
 >>  ENTRY(copyin, 0)
 >>           movl    4(%ap),%r1
 >>           blss    3f              # kernel space
 >>           movl    8(%ap),%r3
 >>  2:      mfpr    $PR_ESP,%r2
 >>           movab   1f,(%r2)
 >>  4:      tstw    14(%ap)         # check if >= 64K
 >>           bneq    5f
 >>           movc3   12(%ap),(%r1),(%r3)
 >>  1:      mfpr    $PR_ESP,%r2
 >>           clrl    (%r2)
 >>           ret
 >>  5:      movc3   $0xfffc,(%r1),(%r3)
 >>           subl2   $0xfffc,12(%ap)
 >>           brb     4b
 >>  
 >>  3:      mnegl   $1,%r0
 >>           ret
 > 
 > This fixes the pipe problem on 5.0.
 > Should we commit it and pull up to 5.0?
 >
 Please do.

 For an overlaying kcopy, it might be an idea to look at the memcpy code in libkern.

 -- Ragge

From: Izumi Tsutsui <tsutsui@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/41139 CVS commit: src/sys/arch/vax/vax
Date: Fri, 10 Apr 2009 13:40:38 +0000

 Module Name:	src
 Committed By:	tsutsui
 Date:		Fri Apr 10 13:40:38 UTC 2009

 Modified Files:
 	src/sys/arch/vax/vax: subr.S

 Log Message:
 Apply patch from mhitch@:
 Make copyin(9) and copyout(9) work with 64KB or larger data blocks.
 Fixes broken pipe(2) problem mentioned in PR port-vax/41139. Ok'ed by ragge@.

 Should be pulled up to netbsd-5.


 To generate a diff of this commit:
 cvs rdiff -u -r1.25 -r1.26 src/sys/arch/vax/vax/subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Soren Jacobsen <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/41139 CVS commit: [netbsd-5] src/sys/arch/vax/vax
Date: Sat, 11 Apr 2009 06:32:37 +0000

 Module Name:	src
 Committed By:	snj
 Date:		Sat Apr 11 06:32:37 UTC 2009

 Modified Files:
 	src/sys/arch/vax/vax [netbsd-5]: subr.S

 Log Message:
 Pull up following revision(s) (requested by mhitch in ticket #686):
 	sys/arch/vax/vax/subr.S: revision 1.26
 Apply patch from mhitch@:
 Make copyin(9) and copyout(9) work with 64KB or larger data blocks.
 Fixes broken pipe(2) problem mentioned in PR port-vax/41139. Ok'ed by ragge@.


 To generate a diff of this commit:
 cvs rdiff -u -r1.25 -r1.25.4.1 src/sys/arch/vax/vax/subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Sun, 12 Apr 2009 20:21:47 +0900
State-Changed-Why:
Fixed and pulled up to netbsd-5.


From: Soren Jacobsen <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/41139 CVS commit: [netbsd-3] src/sys/arch/vax/vax
Date: Thu, 23 Apr 2009 02:11:53 +0000

 Module Name:	src
 Committed By:	snj
 Date:		Thu Apr 23 02:11:53 UTC 2009

 Modified Files:
 	src/sys/arch/vax/vax [netbsd-3]: subr.S

 Log Message:
 Pull up following revision(s) (requested by mhitch in ticket #2010):
 	sys/arch/vax/vax/subr.S: revision 1.26
 Apply patch from mhitch@:
 Make copyin(9) and copyout(9) work with 64KB or larger data blocks.
 Fixes broken pipe(2) problem mentioned in PR port-vax/41139. Ok'ed by ragge@.
 Should be pulled up to netbsd-5.


 To generate a diff of this commit:
 cvs rdiff -u -r1.10 -r1.10.14.1 src/sys/arch/vax/vax/subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Manuel Bouyer <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/41139 CVS commit: [netbsd-4] src/sys/arch/vax/vax
Date: Thu, 30 Apr 2009 18:14:15 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Thu Apr 30 18:14:15 UTC 2009

 Modified Files:
 	src/sys/arch/vax/vax [netbsd-4]: subr.S

 Log Message:
 Pull up following revision(s) (requested by mhitch in ticket #1309):
 	sys/arch/vax/vax/subr.S: revision 1.26
 Apply patch from mhitch@:
 Make copyin(9) and copyout(9) work with 64KB or larger data blocks.
 Fixes broken pipe(2) problem mentioned in PR port-vax/41139. Ok'ed by ragge@.
 Should be pulled up to netbsd-5.


 To generate a diff of this commit:
 cvs rdiff -u -r1.13 -r1.13.24.1 src/sys/arch/vax/vax/subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search