NetBSD Problem Report #42607
From louis@maat.zabrico.com Mon Jan 11 15:44:39 2010
Return-Path: <louis@maat.zabrico.com>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 7A49763B844
for <gnats-bugs@gnats.NetBSD.org>; Mon, 11 Jan 2010 15:44:39 +0000 (UTC)
Message-Id: <201001111544.o0BFiaSs022173@maat.zabrico.com>
Date: Mon, 11 Jan 2010 10:44:36 -0500 (EST)
From: louis@zabrico.com
Reply-To: louis@zabrico.com
To: gnats-bugs@gnats.NetBSD.org
Subject: pkg_add/ftp hangs when there are dependencies
X-Send-Pr-Version: 3.95
>Number: 42607
>Category: pkg
>Synopsis: pkg_add hangs in ftp when there are dependencies (multiple packages)
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: joerg
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Jan 11 15:45:00 +0000 2010
>Closed-Date: Wed Jun 29 17:50:34 +0000 2016
>Last-Modified: Wed Jun 29 17:50:34 +0000 2016
>Originator: Louis Guillaume
>Release: NetBSD 5.0_STABLE
>Organization:
>Environment:
System: NetBSD xxx.xxx.xxxxxxxx 5.0_STABLE NetBSD 5.0_STABLE (GENERIC) #13: Wed Dec 30 14:39:00 EST 2009 louis@xxx.xxx.xxxxxxxx:/usr/obj/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:
When attempting to install a package with dependencies over ftp; i.e.
setting PKG_PATH=ftp://xxxx and then running "pkg_add" the process
appears to hang. Packages are not installed. Interestingly, it does
work for a single package in the case where there are no dependencies.
>How-To-Repeat:
Assuming that the graphics/tiff and graphics/jpeg are not installed:
# export PKG_PATH=ftp://some.ftp.server/path/to/packages/All
# pkg_add tiff
<hangs> (Tiff requires jpeg)
# pkg_add jpeg
<Success>
# pkg_add tiff
<Success>
I was able to reproduce this on AIX, Linux and NetBSD. It was happening
with the netbsd-5 pkg_install (in October) and also in the latest pkgsrc
pkg_install (also in October).
In this discussion:
http://mail-index.netbsd.org/tech-pkg/2009/10/14/msg004085.html Joerg
Sonnenberger was able to reproduce the problem and provide (offline)
a workaround: comment out the QUIT command in ftp.c, line 1042 of
libfetch.
See also this discussion:
http://mail-index.netbsd.org/netbsd-users/2010/01/04/msg005242.html
>Fix:
A work-around is above that may be the fix, but I don't know if it is
the right solution.
>Release-Note:
>Audit-Trail:
From: Joerg Sonnenberger <joerg@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/42607 CVS commit: pkgsrc/net/libfetch
Date: Mon, 11 Jan 2010 17:23:10 +0000
Module Name: pkgsrc
Committed By: joerg
Date: Mon Jan 11 17:23:10 UTC 2010
Modified Files:
pkgsrc/net/libfetch: Makefile
pkgsrc/net/libfetch/files: ftp.c
Log Message:
libfetch-2.27:
The connection sharing didn't handle the case of active transfers
correctly and tried to close the connection in that case (PR 42607).
Correctly check if there is a transfer going on and just leave the
connection alone in that case.
To generate a diff of this commit:
cvs rdiff -u -r1.33 -r1.34 pkgsrc/net/libfetch/Makefile
cvs rdiff -u -r1.30 -r1.31 pkgsrc/net/libfetch/files/ftp.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sat, 23 Jan 2010 20:07:15 +0000
State-Changed-Why:
Did Joerg's commit fix the problem?
From: Louis Guillaume <louis@zabrico.com>
To: gnats-bugs@NetBSD.org
Cc: dholland@NetBSD.org, pkg-manager@NetBSD.org, pkgsrc-bugs@NetBSD.org,
gnats-admin@NetBSD.org
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies (multiple
packages))
Date: Tue, 02 Feb 2010 17:36:37 -0500
On 1/23/10 3:07 PM, dholland@NetBSD.org wrote:
> Synopsis: pkg_add hangs in ftp when there are dependencies (multiple packages)
>
> State-Changed-From-To: open->feedback
> State-Changed-By: dholland@NetBSD.org
> State-Changed-When: Sat, 23 Jan 2010 20:07:15 +0000
> State-Changed-Why:
> Did Joerg's commit fix the problem?
I've only been able to test on NetBSD so far; and it appears to work.
However the change is only present in the pkgsrc distribution. It works
with pkgsrc-2009Q4 as well. But this has not been committed to
netbsd-current (as of 2 weeks ago) nor to netbsd-5. These latter ones
are probably more important.
A new install of NetBSD should be able to install binary packages by
setting PKG_PATH and just letting it rip. A further obstacle here is
that the user must use /usr/pkg/sbin/pkg_add rather than just "pkg_add"
and that could be confusing, especially as the default PATH has the
/usr/sbin before /usr/pkg/sbin.
If we can get these pullups I think it's good! In fact it's great.
I don't understand why there was little or no noise about this one. Do
most people not use pkg_add over FTP for binary packages??
For me this was a very painful bug and also embarrassing when my
attempts to show new [potential] pkgsrc users how great it is miserably
fail!
So thank you very, very much for fixing this.
Louis
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies
(multiple packages))
Date: Wed, 3 Feb 2010 00:52:26 +0100
On Tue, Feb 02, 2010 at 10:40:04PM +0000, Louis Guillaume wrote:
> However the change is only present in the pkgsrc distribution. It works
> with pkgsrc-2009Q4 as well. But this has not been committed to
> netbsd-current (as of 2 weeks ago) nor to netbsd-5. These latter ones
> are probably more important.
Yes, because I do not want to rush somewhat intrusive changes into the
stable and release branches. netbsd-5 has a pending pullup, the rest will
follow after time.
> I don't understand why there was little or no noise about this one. Do
> most people not use pkg_add over FTP for binary packages??
So far I have only been able to reproduce this locally, never against
ftp.n.o.
Joerg
State-Changed-From-To: feedback->pending-pullups
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Fri, 05 Feb 2010 02:07:15 +0000
State-Changed-Why:
feedback received; patch works; need to wait for the fix to be merged
From: Louis Guillaume <louis@zabrico.com>
To: gnats-bugs@NetBSD.org
Cc: dholland@NetBSD.org, pkg-manager@NetBSD.org, pkgsrc-bugs@NetBSD.org,
gnats-admin@NetBSD.org
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies (multiple
packages))
Date: Sun, 16 May 2010 15:47:48 -0400
It looks like things are not fully OK as yet!
Last night I went to do some updates to NetBSD machines like so:
o Install latest 5.1_RC1 sets (from a local build) on all
systems.
o Did pkg_rolling-replace -vu on the build box.
o Ran pkg_info -aX | gzip -c > $PKG_PATH/pkg_summary.gz
o On the target: Move /usr/sbin/pkg_add out of the way:
mv /usr/sbin/pkg_add /usr/sbin/pkg_add.old
(because it chokes on >1 package as is the reason for
this PR)
o On the target: pkg_chk -bu -P $PKG_PATH
i.e. ftp://build-machine/pub/packages/All
The update begins fine and starts processing all the packages. But then
I start to get ftp connection failures as in "connection refused". Even
on a manually executed "ftp -a build-machine". inetd on the ftp server
starts refusing ftp connections!
To recover, I restarted inetd on the ftp server. And did the pkg_chk -bu
-P.. and repeated several times until all packages were up-to-date.
Unfortunately, since things are acting so strangely, I can no longer
vouch for the goodness of this fix.
Louis
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies
(multiple packages))
Date: Mon, 17 May 2010 04:24:43 +0000
On Sun, May 16, 2010 at 07:50:04PM +0000, Louis Guillaume wrote:
> The update begins fine and starts processing all the packages. But then
> I start to get ftp connection failures as in "connection refused". Even
> on a manually executed "ftp -a build-machine". inetd on the ftp server
> starts refusing ftp connections!
inetd has a rate-limiting (mis?)feature; are you triggering it? (Check
on the ftp server for a message like "max spawn rate" or "spawning too
fast".)
One possibility that the rate-limiting code in the inetd on your ftp
server is broken. That code doesn't get exercised much, and I dimly
recall some problems with it in the Linux inetd fork ages ago which
might or might not have been fixed in NetBSD. The system logs on the
ftp server should be able to show if this is the case, because each
new ftp connection seen by inetd should get logged.
It's also possible that if you're downloading a lot of really small
packages over a really fast network, a fresh connection for each could
generate enough connections to stop inetd. That would surprise me
some, and I would have thought pkg_add would be using only one ftp
connection and keeping it open for all the packages. But if this is
the problem you can increase the limit by changing "nowait" to
"nowait:256" (or some suitably large number, see inetd.conf(5)) on the
ftp lines in inetd.conf.
Otherwise, pkg_add must be doing something ~broken...
--
David A. Holland
dholland@netbsd.org
State-Changed-From-To: pending-pullups->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 23 Jun 2016 18:36:22 +0000
State-Changed-Why:
this was waiting on some unspecified netbsd-5 pullup, but that's long moot.
Last report was that there was an additional problem.
State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 23 Jun 2016 18:37:03 +0000
State-Changed-Why:
On second thought, ask the submitter for feedback: I made a suggestion in
2010. Also: is this problem still happening at all?
Responsible-Changed-From-To: pkg-manager->joerg
Responsible-Changed-By: bsiegert@NetBSD.org
Responsible-Changed-When: Tue, 28 Jun 2016 14:25:29 +0000
Responsible-Changed-Why:
Joerg, did your change ever make it back into the base system?
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies
(multiple packages))
Date: Wed, 29 Jun 2016 15:27:25 +0200
On Tue, Jun 28, 2016 at 02:25:29PM +0000, bsiegert@NetBSD.org wrote:
> Joerg, did your change ever make it back into the base system?
I'm pretty sure the libfetch version at least in netbsd-6 is up-to-date.
FTP should be working fine, modulo the rate limits on the server side.
Joerg
From: Louis Guillaume <louis@zabrico.com>
To: gnats-bugs@NetBSD.org, joerg@NetBSD.org, gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org
Cc:
Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies
(multiple packages))
Date: Wed, 29 Jun 2016 11:17:39 -0400
On 6/29/16 9:30 AM, Joerg Sonnenberger wrote:
> The following reply was made to PR pkg/42607; it has been noted by GNATS.
>
> From: Joerg Sonnenberger <joerg@bec.de>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: pkg/42607 (pkg_add hangs in ftp when there are dependencies
> (multiple packages))
> Date: Wed, 29 Jun 2016 15:27:25 +0200
>
> On Tue, Jun 28, 2016 at 02:25:29PM +0000, bsiegert@NetBSD.org wrote:
> > Joerg, did your change ever make it back into the base system?
>
> I'm pretty sure the libfetch version at least in netbsd-6 is up-to-date.
> FTP should be working fine, modulo the rate limits on the server side.
>
> Joerg
>
>
It looks like the problem has not shown up since pkg_install-20100319
was released (AFAICT).
The base pkg_add on my system says it's 20130131 (netbsd-7) and I do not
have this problem anymore.
Maybe it's still a problem in netbsd-5, but I don't have any more of
those systems.
Louis
State-Changed-From-To: feedback->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Wed, 29 Jun 2016 17:50:34 +0000
State-Changed-Why:
netbsd-5 is EOL, so we don't need to worry about that :-)
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.