NetBSD Problem Report #55808

From www@netbsd.org  Mon Nov 16 20:29:47 2020
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 477811A921F
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 16 Nov 2020 20:29:47 +0000 (UTC)
Message-Id: <20201116202946.267EA1A923C@mollari.NetBSD.org>
Date: Mon, 16 Nov 2020 20:29:46 +0000 (UTC)
From: roland.illig@gmx.de
Reply-To: roland.illig@gmx.de
To: gnats-bugs@NetBSD.org
Subject: fread is inefficient in unbuffered mode
X-Send-Pr-Version: www-1.0

>Number:         55808
>Category:       lib
>Synopsis:       fread is inefficient in unbuffered mode
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jdolecek
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 16 20:30:00 +0000 2020
>Closed-Date:    Wed May 05 19:15:00 +0000 2021
>Last-Modified:  Wed May 05 19:15:00 +0000 2021
>Originator:     Roland Illig
>Release:        8.0
>Organization:
none
>Environment:
NetBSD nb8.roland-illig.de 8.0 NetBSD 8.0 (GENERIC.PROF) #0: Sun May 17 14:12:24 UTC 2020  rillig@nb8.roland-illig.de:/home/rillig/proj/src8/sys/arch/amd64/compile/GENERIC.PROF amd64
>Description:
cat <<\EOF > fread-demo.c
#include <stdio.h>

int main(void)
{
        char buf[4096];

        setbuf(stdin, NULL);
        fread(buf, 1, 4096, stdin);
        return 0;
}
EOF

gcc --coverage fread-demo.c -o fread-demo

ktrace ./fread-demo < fread-demo

kdump | grep -w "RET.*read" | uniq -c

The output starts with 4096, which means that read(2) is called 4096 times to read a buffer of size 4096.  This is obviously inefficient.

I experienced this when running gcov in src/usr.bin/make.

cd src/usr.bin/make
USETOOLS=no make clean
time env USETOOLS=no USE_COVERAGE=yes make -s all test

This takes about 14 minutes.  Without the call to setbuf in gcov_open, it takes only 50 seconds.

See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97834

https://github.com/NetBSD/src/blob/netbsd-8/lib/libc/stdio/fread.c#L81

The call to __srefill looks suspicious since that function does not get any information about how many bytes it should read.

>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/55808: fread is inefficient in unbuffered mode
Date: Mon, 16 Nov 2020 21:57:23 +0100

 Cool, OpenBSD already did all the work:

 https://github.com/openbsd/src/blob/master/lib/libc/stdio/fread.c

 "If we're unbuffered" ...

Responsible-Changed-From-To: lib-bug-people->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Thu, 28 Jan 2021 20:07:47 +0000
Responsible-Changed-Why:
I'll look into adopting the OpenBSD change.


State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sun, 31 Jan 2021 16:19:27 +0000
State-Changed-Why:
I've committed the optimization from OpenBSD. It seems to fix the problem.
Can you confirm?


From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55808 CVS commit: src/lib/libc/stdio
Date: Sun, 31 Jan 2021 16:18:22 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Sun Jan 31 16:18:22 UTC 2021

 Modified Files:
 	src/lib/libc/stdio: fread.c

 Log Message:
 for unbuffered I/O arrange for the destination buffer to be filled in one
 go, instead of triggering long series of 1 byte read(2)s; this speeds up
 fread() several order of magnitudes for this case, directly proportional
 to the size of the supplied buffer

 change adapted from OpenBSD rev. 1.19

 fixes PR lib/55808 by Roland Illig


 To generate a diff of this commit:
 cvs rdiff -u -r1.23 -r1.24 src/lib/libc/stdio/fread.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: feedback->pending-pullups
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Wed, 03 Feb 2021 21:34:21 +0000
State-Changed-Why:
Requested pullup to netbsd-9


From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/55808 (fread is inefficient in unbuffered mode)
Date: Sun, 7 Feb 2021 18:16:42 +0100

 On 31.01.2021 17:19, jdolecek@NetBSD.org wrote:
 > I've committed the optimization from OpenBSD. It seems to fix the proble=
 m.
 > Can you confirm?

 I updated to NetBSD 9.99.80, and gcov is now fast by default.  Thank you
 for the fix.

 Roland

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55808 CVS commit: [netbsd-9] src/lib/libc/stdio
Date: Sun, 7 Feb 2021 17:41:02 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Sun Feb  7 17:41:02 UTC 2021

 Modified Files:
 	src/lib/libc/stdio [netbsd-9]: fread.c

 Log Message:
 Pull up following revision(s) (requested by jdolecek in ticket #1198):

 	lib/libc/stdio/fread.c: revision 1.24 (via patch)

 for unbuffered I/O arrange for the destination buffer to be filled in one
 go, instead of triggering long series of 1 byte read(2)s; this speeds up
 fread() several order of magnitudes for this case, directly proportional
 to the size of the supplied buffer
 change adapted from OpenBSD rev. 1.19

 fixes PR lib/55808 by Roland Illig


 To generate a diff of this commit:
 cvs rdiff -u -r1.22 -r1.22.34.1 src/lib/libc/stdio/fread.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Wed, 05 May 2021 19:15:00 +0000
State-Changed-Why:
pullps done. thanks for report.


>Unformatted:

Home	PR Database Search