NetBSD Problem Report #56219

From tsutsui@ceres.dti.ne.jp  Sun May 30 11:32:04 2021
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 990871A928B
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 30 May 2021 11:32:04 +0000 (UTC)
Message-Id: <202105301131.14UBVtnj018813@ceres.dti.ne.jp>
Date: Sun, 30 May 2021 20:31:55 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: sysinst ftp(1) gets Interrupted system call on fetching set via http
X-Send-Pr-Version: 3.95

>Number:         56219
>Category:       install
>Synopsis:       sysinst ftp(1) gets Interrupted system call on fetching set via http
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    lukem
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun May 30 11:35:00 +0000 2021
>Closed-Date:    Sat Jul 10 14:30:17 +0000 2021
>Last-Modified:  Mon Sep 12 17:10:06 +0000 2022
>Originator:     Izumi Tsutsui
>Release:        NetBSD 9.2
>Organization:
>Environment:
System: NetBSD lunatic 9.2 NetBSD 9.2 (INSTALL) #0: Wed May 12 13:15:55 UTC 2021
        mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/luna68k/compile/INSTALL
Architecture: m68k
Machine: luna68k (maybe other slow 68030 machines?)
>Description:
During upgrade from NetBSD/luna68k 9.1 to 9.2,
crunched ftp(1) fetching binary sets via http from cdn.NetBSD.org
get Interrupted system call (EINTR) and fails to fetch and extract sets.

Note this is only seen on LUNA-I (M68030 20MHz / 16MB RAM),
and isn't seen on LUNA-II (M68040 25MHz / 64MB RAM).

We cannot choose fetch via ftp(1) due to another problem
(ftp control session timeout on fetching large sets, as per PR/56129)

>How-To-Repeat:
Perform upgrade install to NetBSD to 9.2 (only on slow 68030 machines?).

Sysinst comptails:
---
/binary/sets/kern-GENERIC.tgz

     Hit enter to continue

Requesting http://cdn.NetBSD/org/pub/NetBSD/NetBSD-9.2/luna68k/binary/sets/kern-GENERIC.tgz
 97% | ********************************  |  1248 KiB   83.01 KiB/s    00:00 ETAftp: Reading file: Interrupted system call
---

https://pbs.twimg.com/media/E2nBohKUYAA44Ur.jpg
https://pbs.twimg.com/media/E2nHfHDVkAAXoAo.jpg
https://pbs.twimg.com/media/E2nm5h0UcAc9w7V.jpg
https://pbs.twimg.com/media/E2npjw4VUAYx839.jpg
etc.

Note1: IIRC this problem didn't happen on 9.0 and 9.1 upgrade.
Note2: This also happens on fetching from not only cdn.NetBSD.org but 
       also ftp.netbsd.org, ftp4.jp.netbsd.org, ftp6.jp.netbsd.org, and
       ftp7.jp.netbsd.org etc. so it doesn't look server side problem.

>Fix:
Check all possible EINTR errors and handle it properly?

For fetching via ftp protocol we also fix for PR/56129
(ftp(1) should keep control session during large xfer)
https://pbs.twimg.com/media/E2nuME0UUAAkeo1.jpg

---
Izumi Tsutsui

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->analyzed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Wed, 02 Jun 2021 13:20:42 +0000
State-Changed-Why:
The WITH_SSL implementation handles EINTR, even for non-SSL transfers.
When compiled with !WITH_SSL, the implementation uses stdio, which
may not handle EINTR robustly.
I speculate that the m68k install media is compiled without WITH_SSL.

A potential fix is to always use the ssl.c fetch_*() functions
even without WITH_SSL, and just conditionally remove the ssl-specific
functions in that file. This would deprecate the stdio backend in !WITH_SSL.


From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: lukem@NetBSD.org, tsutsui@ceres.dti.ne.jp
Subject: Re: install/56219 (sysinst ftp(1) gets Interrupted system call on fetching
	 set via http)
Date: Wed, 2 Jun 2021 22:33:26 +0900

 > I speculate that the m68k install media is compiled without WITH_SSL.

 In src/usr.bin/ftp/Makefile, -DWITH_SSL is not defined in SMALLPROG case
 and src/distrib/common/Makefile.crunch defines SMALLPROG?=1 by default,
 so most ports that use crunched ramdisk (i.e. all but x86 and sparc64?)
 are affected.

 ---
 Izuim Tsutsui

Responsible-Changed-From-To: install-manager->lukem
Responsible-Changed-By: lukem@NetBSD.org
Responsible-Changed-When: Wed, 02 Jun 2021 14:40:24 +0000
Responsible-Changed-Why:
I'll take it - I have a fix in progress.


State-Changed-From-To: analyzed->feedback
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Thu, 03 Jun 2021 10:25:11 +0000
State-Changed-Why:
I've switched the SMALLPROG build to also use the fetch_*() I/O methods
instead of falling back to stdio, because stdio isn't robust with
interruptable signals.

Please try a build of ftp version 20210603 or newer.


From: "Luke Mewburn" <lukem@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/56219 CVS commit: src/usr.bin/ftp
Date: Thu, 3 Jun 2021 10:23:33 +0000

 Module Name:	src
 Committed By:	lukem
 Date:		Thu Jun  3 10:23:33 UTC 2021

 Modified Files:
 	src/usr.bin/ftp: Makefile ssl.c ssl.h version.h

 Log Message:
 use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

 Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
 methods from ssl.c, instead of using stdio, as stdio isn't robust
 when using interruptable signals.

 Disable ssl-specific support in the fetch_*() methods if WITH_SSL
 isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

 The resulting SMALLPROG binary is slightly larger than before
 (e.g., 157KiB vs 153KiB on amd64).

 Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

 PR install/56219


 To generate a diff of this commit:
 cvs rdiff -u -r1.38 -r1.39 src/usr.bin/ftp/Makefile
 cvs rdiff -u -r1.9 -r1.10 src/usr.bin/ftp/ssl.c
 cvs rdiff -u -r1.4 -r1.5 src/usr.bin/ftp/ssl.h
 cvs rdiff -u -r1.92 -r1.93 src/usr.bin/ftp/version.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: lukem@netbsd.org, tsutsui@ceres.dti.ne.jp
Subject: Re: install/56219 (sysinst ftp(1) gets Interrupted system call on fetching
	 set via http)
Date: Sun, 13 Jun 2021 04:24:01 +0900

 > I've switched the SMALLPROG build to also use the fetch_*() I/O methods
 > instead of falling back to stdio, because stdio isn't robust with
 > interruptable signals.
 > 
 > Please try a build of ftp version 20210603 or newer.

 NetBSD/x68k 9.2 INSTALL kernel with patched 20210603 ftp(1)
 no longer gets "INterrupted system call" after getting base.tgz
 (where the 9.2 release version failed).

 ---
 Izumi Tsutsui

State-Changed-From-To: feedback->closed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Sat, 12 Jun 2021 22:36:16 +0000
State-Changed-Why:
submitter confirms problem is resolved


State-Changed-From-To: closed->needs-pullups
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Sun, 13 Jun 2021 14:52:40 +0000
State-Changed-Why:
netbsd-9 is also affected.


From: Luke Mewburn <lukem@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: lukem@netbsd.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
	tsutsui@NetBSD.org, tsutsui@ceres.dti.ne.jp
Subject: Re: install/56219 (sysinst ftp(1) gets Interrupted system call on
 fetching set via http)
Date: Mon, 14 Jun 2021 16:01:59 +1000

 On 21-06-13 14:52, tsutsui@NetBSD.org wrote:
   | Synopsis: sysinst ftp(1) gets Interrupted system call on fetching set via http
   | 
   | State-Changed-From-To: closed->needs-pullups
   | State-Changed-By: tsutsui@NetBSD.org
   | State-Changed-When: Sun, 13 Jun 2021 14:52:40 +0000
   | State-Changed-Why:
   | netbsd-9 is also affected.


 I'll request pullups for various other commits before this one, to
 synchronize the code, minimize merge conflicts, and reduce confusion
 about the advertised version in NetBSD 9.x builds.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/56219 CVS commit: [netbsd-9] src/usr.bin/ftp
Date: Mon, 14 Jun 2021 11:57:39 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Mon Jun 14 11:57:39 UTC 2021

 Modified Files:
 	src/usr.bin/ftp [netbsd-9]: Makefile ssl.c ssl.h version.h

 Log Message:
 Pull up following revision(s) (requested by lukem in ticket #1295):

 	usr.bin/ftp/ssl.c: revision 1.10
 	usr.bin/ftp/ssl.h: revision 1.5
 	usr.bin/ftp/version.h: revision 1.93
 	usr.bin/ftp/Makefile: revision 1.39

 use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

 Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
 methods from ssl.c, instead of using stdio, as stdio isn't robust
 when using interruptable signals.

 Disable ssl-specific support in the fetch_*() methods if WITH_SSL
 isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

 The resulting SMALLPROG binary is slightly larger than before
 (e.g., 157KiB vs 153KiB on amd64).

 Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

 PR install/56219


 To generate a diff of this commit:
 cvs rdiff -u -r1.37 -r1.37.12.1 src/usr.bin/ftp/Makefile
 cvs rdiff -u -r1.8.2.1 -r1.8.2.2 src/usr.bin/ftp/ssl.c
 cvs rdiff -u -r1.4 -r1.4.2.1 src/usr.bin/ftp/ssl.h
 cvs rdiff -u -r1.87.18.2 -r1.87.18.3 src/usr.bin/ftp/version.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: needs-pullups->closed
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Sat, 10 Jul 2021 14:30:17 +0000
State-Changed-Why:
Pulled up to netbsd-9 in ticket #1295.


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/56219 CVS commit: [netbsd-8] src/usr.bin/ftp
Date: Mon, 12 Sep 2022 14:46:52 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Mon Sep 12 14:46:51 UTC 2022

 Modified Files:
 	src/usr.bin/ftp [netbsd-8]: Makefile cmds.c complete.c domacro.c
 	    extern.h fetch.c ftp.1 ftp.c ftp_var.h main.c progressbar.c
 	    progressbar.h ssl.c ssl.h util.c version.h

 Log Message:
 Catch up to current, requested by christos in ticket #1763:

 	usr.bin/ftp/Makefile                            up to 1.39
 	usr.bin/ftp/cmds.c                              up to 1.141
 	usr.bin/ftp/complete.c                          up to 1.47
 	usr.bin/ftp/domacro.c                           up to 1.23
 	usr.bin/ftp/extern.h                            up to 1.82
 	usr.bin/ftp/fetch.c                             up to 1.235
 	usr.bin/ftp/ftp.1                               up to 1.147
 	usr.bin/ftp/ftp.c                               up to 1.174
 	usr.bin/ftp/ftp_var.h                           up to 1.86
 	usr.bin/ftp/main.c                              up to 1.128
 	usr.bin/ftp/progressbar.c                       up to 1.24
 	usr.bin/ftp/progressbar.h                       up to 1.9
 	usr.bin/ftp/ssl.c                               up to 1.11
 	usr.bin/ftp/ssl.h                               up to 1.5
 	usr.bin/ftp/util.c                              up to 1.164
 	usr.bin/ftp/version.h                           up to 1.94

 ftp(1): validate address from PASV and LPSV response.
 ftp(1): use raw write(2) instead of fwrite(3) to avoid stream
 corruption because of the progress bar interrupts.
 Fixes for PR 56219 and PR 55857.
 PR 57003: Support relative redirects.


 To generate a diff of this commit:
 cvs rdiff -u -r1.37 -r1.37.2.1 src/usr.bin/ftp/Makefile
 cvs rdiff -u -r1.137 -r1.137.8.1 src/usr.bin/ftp/cmds.c
 cvs rdiff -u -r1.46 -r1.46.38.1 src/usr.bin/ftp/complete.c
 cvs rdiff -u -r1.22 -r1.22.38.1 src/usr.bin/ftp/domacro.c
 cvs rdiff -u -r1.80 -r1.80.24.1 src/usr.bin/ftp/extern.h
 cvs rdiff -u -r1.228 -r1.228.4.1 src/usr.bin/ftp/fetch.c
 cvs rdiff -u -r1.135 -r1.135.8.1 src/usr.bin/ftp/ftp.1
 cvs rdiff -u -r1.167 -r1.167.6.1 src/usr.bin/ftp/ftp.c
 cvs rdiff -u -r1.84 -r1.84.8.1 src/usr.bin/ftp/ftp_var.h
 cvs rdiff -u -r1.123 -r1.123.8.1 src/usr.bin/ftp/main.c
 cvs rdiff -u -r1.22 -r1.22.24.1 src/usr.bin/ftp/progressbar.c
 cvs rdiff -u -r1.8 -r1.8.38.1 src/usr.bin/ftp/progressbar.h
 cvs rdiff -u -r1.5 -r1.5.8.1 src/usr.bin/ftp/ssl.c
 cvs rdiff -u -r1.3 -r1.3.8.1 src/usr.bin/ftp/ssl.h
 cvs rdiff -u -r1.158 -r1.158.22.1 src/usr.bin/ftp/util.c
 cvs rdiff -u -r1.87 -r1.87.8.1 src/usr.bin/ftp/version.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/56219 CVS commit: [netbsd-8] src/usr.bin/ftp
Date: Mon, 12 Sep 2022 17:08:14 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Mon Sep 12 17:08:13 UTC 2022

 Modified Files:
 	src/usr.bin/ftp [netbsd-8]: Makefile cmds.c complete.c domacro.c
 	    extern.h fetch.c ftp.1 ftp.c ftp_var.h main.c progressbar.c
 	    progressbar.h ssl.c ssl.h util.c version.h

 Log Message:
 Catch up to current, requested by christos in ticket #1763:

         usr.bin/ftp/Makefile                            up to 1.39
         usr.bin/ftp/cmds.c                              up to 1.141
         usr.bin/ftp/complete.c                          up to 1.47
         usr.bin/ftp/domacro.c                           up to 1.23
         usr.bin/ftp/extern.h                            up to 1.82
         usr.bin/ftp/fetch.c                             up to 1.235
         usr.bin/ftp/ftp.1                               up to 1.147
         usr.bin/ftp/ftp.c                               up to 1.174
         usr.bin/ftp/ftp_var.h                           up to 1.86
         usr.bin/ftp/main.c                              up to 1.128
         usr.bin/ftp/progressbar.c                       up to 1.24
         usr.bin/ftp/progressbar.h                       up to 1.9
         usr.bin/ftp/ssl.c                               up to 1.12
         usr.bin/ftp/ssl.h                               up to 1.5
         usr.bin/ftp/util.c                              up to 1.164
         usr.bin/ftp/version.h                           up to 1.94

 ftp(1): validate address from PASV and LPSV response.
 ftp(1): use raw write(2) instead of fwrite(3) to avoid stream
 corruption because of the progress bar interrupts.
 Fixes for PR 56219 and PR 55857.
 PR 57003: Support relative redirects.


 To generate a diff of this commit:
 cvs rdiff -u -r1.37.2.2 -r1.37.2.3 src/usr.bin/ftp/Makefile
 cvs rdiff -u -r1.137.8.2 -r1.137.8.3 src/usr.bin/ftp/cmds.c
 cvs rdiff -u -r1.46.38.2 -r1.46.38.3 src/usr.bin/ftp/complete.c
 cvs rdiff -u -r1.22.38.2 -r1.22.38.3 src/usr.bin/ftp/domacro.c
 cvs rdiff -u -r1.80.24.2 -r1.80.24.3 src/usr.bin/ftp/extern.h
 cvs rdiff -u -r1.228.4.2 -r1.228.4.3 src/usr.bin/ftp/fetch.c
 cvs rdiff -u -r1.135.8.2 -r1.135.8.3 src/usr.bin/ftp/ftp.1
 cvs rdiff -u -r1.167.6.2 -r1.167.6.3 src/usr.bin/ftp/ftp.c
 cvs rdiff -u -r1.84.8.2 -r1.84.8.3 src/usr.bin/ftp/ftp_var.h
 cvs rdiff -u -r1.123.8.2 -r1.123.8.3 src/usr.bin/ftp/main.c
 cvs rdiff -u -r1.22.24.2 -r1.22.24.3 src/usr.bin/ftp/progressbar.c
 cvs rdiff -u -r1.8.38.2 -r1.8.38.3 src/usr.bin/ftp/progressbar.h
 cvs rdiff -u -r1.5.8.2 -r1.5.8.3 src/usr.bin/ftp/ssl.c
 cvs rdiff -u -r1.3.8.2 -r1.3.8.3 src/usr.bin/ftp/ssl.h
 cvs rdiff -u -r1.158.22.2 -r1.158.22.3 src/usr.bin/ftp/util.c
 cvs rdiff -u -r1.87.8.2 -r1.87.8.3 src/usr.bin/ftp/version.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search