NetBSD Problem Report #14246

Received: (qmail 5039 invoked from network); 15 Oct 2001 00:11:00 -0000
Message-Id: <20011015001049.E1316BB84@trimalchio.peripatetic>
Date: Mon, 15 Oct 2001 02:10:49 +0200 (CEST)
From: Andreas Persson <pap.is@home.se>
Reply-To: pap.is@home.se
To: gnats-bugs@gnats.netbsd.org
Subject: NEW_PIPE bug when reading and writing from the same process
X-Send-Pr-Version: 3.95

>Number:         14246
>Category:       kern
>Synopsis:       NEW_PIPE bug when reading and writing from the same process
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    jdolecek
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Oct 15 00:12:00 +0000 2001
>Closed-Date:    Wed Nov 28 21:09:28 +0000 2001
>Last-Modified:  Wed Nov 28 21:09:28 +0000 2001
>Originator:     Andreas Persson
>Release:        NetBSD-current
>Organization:
>Environment:
System: NetBSD sisyfos.peripatetic 1.5Y NetBSD 1.5Y (SISYFOS) #3: Mon Oct 15 00:29:50 CEST 2001 ap@sisyfos.peripatetic:/usr/src/sys/arch/i386/compile/SISYFOS i386
Architecture: i386
Machine: i386
>Description:
When a pipe is used in a blocking state to pass data back and forth to the
same process and the size is >= PIPE_MINDIRECT, the process will sleep
indefinetly with wait channel pipdwt. Worse, when the pipe is non-blocking,
it will still sleep after a few writes (the number varies), though I don't
see how this could happen from reading the code. This may be related to PR
kern/14196.

>How-To-Repeat:
Download and compile:
http://www-106.ibm.com/developerworks/linux/library/l-rt4/pipespeed2.cpp
See also:
http://www-106.ibm.com/developerworks/linux/library/l-rt4/?t=gr,p=Pipes

>Fix:
I don't see why we have to sleep in the first place. Shouldn't uvm_loan handle
protecting the shared pages?
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->jdolecek 
Responsible-Changed-By: jdolecek 
Responsible-Changed-When: Mon Oct 15 11:23:20 PDT 2001 
Responsible-Changed-Why:  
NEW_PIPE is mine. 
State-Changed-From-To: open->feedback 
State-Changed-By: jdolecek 
State-Changed-When: Tue Oct 16 11:55:11 PDT 2001 
State-Changed-Why:  
This is expected behaviour. There is NO guarantee > PIPE_SIZE would 
get through without blocking. Linux blocks on4k+1, we start to wait 
for reader once the write is equal to PAGE_SIZE (4k on i386). Any 
application trying to both write and read in same process 
context to pipe is going to loose for some size of buffer, which 
is system dependant.It's application problem - the author of the 
article even notes this: """ 
A write of more than 4K will block unless a reader empties the 

pipe.""" 

Regarding the nonblock case, would you provide me with the code you 
used to test this? 

From: Jaromir Dolecek <jdolecek@netbsd.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/14246
Date: Sun, 21 Oct 2001 20:18:23 +0200 (CEST)

 Andreas Persson wrote:
 > >Regarding the nonblock case, would you provide me with the code you
 > >used to test this?
 > Sure, its attached below. It fails for something like
 > 	./pipespeed2 1024 65536
 > top shows its stuck in pipdwt.

 OK, I'll look at it.

 > Additionally, it seems like OpenBSD is a bit faster. A person having an

 I can't read the message enough to understand which speed he gets
 on OpenBSD for the case in which you get 330 MB/s. What is the
 difference? Note that for such a small buffers as 4KB, the
 context-switch overhead is one of the most significant factors.
 Thus, a thread-based test might be _much_ faster than a process-based
 one, especially if the threads are userland threads only and hence
 avoid context-switch almost altogether[*]

 Would you send me your pipespeed2.c code you use on NetBSD?
 I'd look if there is any problem in there, and I'd also like to
 try it out on my computer.

 Jaromir

  [*] It's my understanding OpenBSD have intergrated userland threads
      only, without any kernel support).
 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org> http://www.NetBSD.org/Ports/i386/ps2.html
 -=  Those who would give up liberty for a little temporary safety deserve  =-
 -=  neither liberty nor safety, and will lose both.  -- Benjamin Franklin  =-

From: Jaromir Dolecek <jdolecek@netbsd.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/14246
Date: Sun, 21 Oct 2001 20:42:56 +0200 (CEST)

 Andreas Persson wrote:
 > I consistently get around 330 MB/sec with 4096 sized writes. Although
 > he is using a blocking threaded pipespeed (which I can't get to work at
 > all in NetBSD, but I guess this is probably a problem with the thread
 > package).

 BTW, which speed do you get if you use blocking pipespeed instead
 of nonblocking? Note that nonblocking mode disables some optimizations
 which could help performance a little.

 Jaromir 

From: Jaromir Dolecek <jdolecek@netbsd.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/14246
Date: Sun, 21 Oct 2001 22:04:57 +0200 (CEST)

 Andreas Persson wrote:
 > Well, the thing is, pipespeed2.cpp is using only one process as well,
 > just writing and reading the pipe after eachother. The threaded version
 > writes with one thread and reads with the other. Hence the performance
 > should be comparable. From my superficial reading of the OpenBSD code
 > it seems like it always copies things into kernel space (i.e. it doesn't
 > use the direct writes like FreeBSD), so I can't see how OpenBSD could be
 > faster, unless there is a bottleneck elsewhere.

 Note that the actual copy operation is tiny fraction of the whole
 call when the buffer is small. I expect the context switch time
 between pipe writer and pipe reader to be major difference here.
 The context switch between two userland threads is near to zero,
 but full process context switch has some overhead. Thus, a test
 using threads might significantly outperform the test using two
 processes. I expect this to be the difference between NetBSD and
 OtherBSD.

 Of course, test using two threads is not exactly real-world test.
 Two threads can pass data to each other much more efficient way,
 for example directly via pointers. Threads don't commonly use pipes
 to pass data each other in real world.  So, the test using threads
 is only for illustration and not really telling anything useful.
 It's still much better if we perform well in this unrealistic
 scenario, of course :)

 > It was attached in the other message :).

 Ah, OK :) I'll take a look. I'll also try the threaded version.
 Or (preferably), it would be cool if you could try to experiment
 with this and get the threaded version working. You might
 consider trying pth or unroven-pthreads from pkgsrc instead
 mit-pthread. I'm quite busy nowadays and it would be nice
 if someone else would work on this too.

 Jaromir

From: Jaromir Dolecek <jdolecek@NetBSD.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/14246
Date: Mon, 22 Oct 2001 22:26:03 +0200 (CEST)

 Andreas Persson wrote:
 > which obviously isn't correct. After that performance jumped to 543 MB/sec,
 > so it appears that at least the i386 microtime implementation has a
 > performance problem.

 Oops, I wouldn't expect that. Good spot! 

 > From reading the source, I see that its doing some
 > port io, which is extremely slow on a modern intel cpu. Perhaps you could
 > ask in tech-kern, or something similar why its so slow? You're better
 > respected than me :).

 I'll do :)

 I'm not quite convinced it's actually necessary or sensible to keep
 the chtime for pipes. The vfs_update() could be omit altogether,
 especially if it's _that_ expensive.

 > In the mean time I'll try to get another 100 MB/sec out of it ;).

 The 100 MB/sec difference might be caused by using 4KB instead 2KB buffer
 for the OpenBSD test. Perhaps you might try to bump PIPE_MINDIRECT in
 syssrc/sys/sys/pipe.h to (2*PAGE_SIZE) instead of current PAGE_SIZE
 and see if that makes the difference? The bump would make NEWPIPE
 use loaned writes for buffers 2*PAGE_SIZE+, i.e. 8KB on i386 and thus
 4KB buffer should use nonloaned write.

 Jaromir
 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org> http://www.NetBSD.org/Ports/i386/ps2.html
 -=  Those who would give up liberty for a little temporary safety deserve  =-
 -=  neither liberty nor safety, and will lose both.  -- Benjamin Franklin  =-

From: Jaromir Dolecek <jdolecek@netbsd.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/14246
Date: Mon, 22 Oct 2001 22:46:15 +0200 (CEST)

 Andreas Persson wrote:
 > I changed your vfs_timestamp to:
 > #define vfs_timestamp(tv) do {                                          \
 >                 (tv)->tv_sec = 0;                                       \
 >                 (tv)->tv_usec = 0;                                      \
 >         } while (0)

 Can you try to change this to more correct:
 #define vfs_timestamp(tv) do {		\
 		int s = splclock();	\
 		(tv) = time;		\
 		splx(s);		\
 	} while (0)

 and see if this still performs as well as your version?

 Jaromir
 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org> http://www.NetBSD.org/Ports/i386/ps2.html
 -=  Those who would give up liberty for a little temporary safety deserve  =-
 -=  neither liberty nor safety, and will lose both.  -- Benjamin Franklin  =-

From: Jaromir Dolecek <jdolecek@netbsd.org>
To: Andreas Persson <pap.is@home.se>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/14246
Date: Thu, 25 Oct 2001 21:29:30 +0200 (CEST)

 Andreas Persson wrote:
 > Well, I was using the 2k figure from OpenBSD. Using 4k, OpenBSD gets
 > 941 MB/sec, whereas NetBSD gets 839 MB/sec with size 4095. So NetBSD
 > is consistently approximately 100 MB/sec behind. The fact that this
 > number remains fairly constant seems to indicate that its related to
 > overhead, as opposed to a slow copyin or copyout. Of course it could
 > be mainboard/chipset related, too.

 This _might_ be due to microtime being sluggish, too - since
 benchmark uses gettimeofday(2) to measure the time, and gettimeofday(2)
 uses microtime(9) internally.

 I looked at the OpenBSD implementation and they have some optimization
 of microtime(9) on pentium+ class of processors, using rdtsc.
 FreeBSD has some whole subsystem for time counters, and seems to use
 the TSC counter too (if available). 
 So, they both _might_ have faster microtime(9) than NetBSD, making
 some operations appear faster and skewing benchmarks in favour
 of them. We appear to have TSC-based microtime(9) implementation
 on 386 SMP branch. I'll experiment with this a bit, to confirm
 or deny that this would change the reported performance.

 Would you be willing to test with (experimental) tsc-based microtime(9)
 if I'd manage to put this into shape for -current?

 Jaromir
 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org> http://www.NetBSD.org/Ports/i386/ps2.html
 -=  Those who would give up liberty for a little temporary safety deserve  =-
 -=  neither liberty nor safety, and will lose both.  -- Benjamin Franklin  =-

From: Jaromir Dolecek <jdolecek@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: pap.is@home.se
Subject: Re: kern/14246: NEW_PIPE bug when reading and writing from the same
 process
Date: Sun, 25 Nov 2001 19:39:31 +0100 (CET)

 Hi,
 I've got to test the second part of problem ("i/o blocks even through
 the FNONBLOCK is set"). The first part is already fixed (the
 threshold was bumped to 8K), and not error even if it would not :)

 Your program, once compiled, indeed blocked in i/o (I used 8k
 buffer, to get it use something more than the current threshold).

 After a bit of tracking, I found out there seemed to be two bugs
 in your non_block() function, in a way F_GETFL and F_SETFL was
 used. Once I changed it to:

 void
 non_block(int fd)
 {
 #if 1
 	int flags;

 	if ((flags = fcntl(fd, F_GETFL)) == -1)
 		err(EXIT_FAILURE, "could not get file flags");
 	flags |= O_NONBLOCK;
 	if (fcntl(fd, F_SETFL, flags) == -1)
 		err(EXIT_FAILURE, "could not set non blocking mode");
 	fcntl(fd, F_GETFL, &flags);
 #endif
 }

 the program worked just fine - the benchmark run to completion
 and did not block.

 So, it seems it was not NEW_PIPE fault :)

 Once I polish the TSC-based i386 microtime(9), I'll announce it on
 current-users, so you'd be able to use that too.

 Since there seem to be no more issues mentioned in this PR, I'd
 like to close it. Do you agree?

 Jaromir
 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org> http://www.NetBSD.org/Ports/i386/ps2.html
 -=  Those who would give up liberty for a little temporary safety deserve  =-
 -=  neither liberty nor safety, and will lose both.  -- Benjamin Franklin  =-
State-Changed-From-To: feedback->closed 
State-Changed-By: jdolecek 
State-Changed-When: Wed Nov 28 13:09:03 PST 2001 
State-Changed-Why:  
Fixed. Thanks for report and help! 
>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.