NetBSD Problem Report #47525
From tsutsui@ceres.dti.ne.jp Sat Feb 2 22:05:20 2013
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
by www.NetBSD.org (Postfix) with ESMTP id 479B463D7B3
for <gnats-bugs@gnats.NetBSD.org>; Sat, 2 Feb 2013 22:05:20 +0000 (UTC)
Message-Id: <201302022205.r12M5GrH014732@mirage.localdomain>
Date: Sun, 3 Feb 2013 07:05:16 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@gnats.NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: fetch from sourceforge fails if user sets own FTPUSERAGENT
X-Send-Pr-Version: 3.95
>Number: 47525
>Category: pkg
>Synopsis: fetch from sourceforge fails if user sets own FTPUSERAGENT
>Confidential: no
>Severity: non-critical
>Priority: high
>Responsible: morr
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat Feb 02 22:10:01 +0000 2013
>Closed-Date: Thu Feb 07 13:33:59 +0000 2013
>Last-Modified: Thu Feb 07 14:00:08 +0000 2013
>Originator: Izumi Tsutsui
>Release: NetBSD 6.0
>Organization:
>Environment:
System: NetBSD i386 6.0 + pkgsrc HEAD
Architecture: i386
Machine: i386
>Description:
sourceforge download sites seem to check client's useragent value
and change behavior of download pages so if a user sets own FTPUSERAGENT
("Mozilla/5.0" etc.) for ftp(1) fetching from sourceforge fails.
It was once fixed in pkgsrc/mk/fetch/fetch.mk rev 1.57:
http://mail-index.netbsd.org/pkgsrc-changes/2012/12/14/msg081842.html
but in rev 1.58 FTPUSERAGENT variable is passed again on make fetch:
http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/fetch/fetch.mk.diff?r1=1.57&r2=1.58
>How-To-Repeat:
% cd pkgsrc/x11/mlterm ; env FTPUSERAGENT="Mozilla/5.0" make fetch
>Fix:
Use pkgsrc own variable to override ftp(1)'s FTPUSERAGENT in fetch.mk?
(I'm not sure which package has sites "broken without FTPUSERAGENT" though)
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->morr
Responsible-Changed-By: obache@NetBSD.org
Responsible-Changed-When: Mon, 04 Feb 2013 11:37:12 +0000
Responsible-Changed-Why:
Over to changer.
From: Daniel Horecki <morr@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/47525 (fetch from sourceforge fails if user sets own FTPUSERAGENT)
Date: Mon, 04 Feb 2013 15:02:40 +0100
Hi,
Revision 1.57 of pkgsrc/mk/fetch/fetch.mk broke fetching of x11/rxvt-unicode:
Trying 2001:470:987b::a00:13:80 ...
Requesting http://dist.schmorp.de/rxvt-unicode/rxvt-unicode-9.16.tar.bz2
ftp: Error retrieving file `500 Internal Server Error'
fetch: Unable to fetch expected file rxvt-unicode-9.16.tar.bz2
Trying 2001:470:987b::a00:13:80 ...
Requesting http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-9.16.tar.bz2
ftp: Error retrieving file `500 Internal Server Error'
fetch: Unable to fetch expected file rxvt-unicode-9.16.tar.bz2
It looks like webserver doesn't like empty User-Agent. I just allowed to overwrite FTPUSERAGENT to unbreak it.
As FTPUSERAGENT is usually not set, it is working for both cases.
I've checked user test case with default User-Agent of our ftp and it worked without problem:
# env FTPUSERAGENT="NetBSD-ftp/20100605" make fetch
=> Bootstrap dependency digest>=20010302: found digest-20121220
=> Fetching mlterm-3.1.7.tar.gz
=> Total size: 3535404 bytes
Requesting http://downloads.sourceforge.net/sourceforge/mlterm/mlterm-3.1.7.tar.gz
Redirected to http://downloads.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
Requesting http://downloads.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
Redirected to http://freefr.dl.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
Trying 2001:1b48:10f::7:80 ...
Requesting http://freefr.dl.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
100% |*************************************************************************************************************************************| 3452 KiB 210.67 KiB/s 00:00 ETA
3535404 bytes retrieved in 00:16 (210.67 KiB/s)
# make checksum
=> Bootstrap dependency digest>=20010302: found digest-20121220
=> Checksum SHA1 OK for mlterm-3.1.7.tar.gz
=> Checksum RMD160 OK for mlterm-3.1.7.tar.gz
Why this variable was cleaned in first place?
Regards,
--
Daniel Horecki
http://morr.pl http://linux.pl http://netbsd.pl http://netbsd.org
HAIL ERIS!
BOFH since 1999.
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525 (fetch from sourceforge fails if user sets own FTPUSERAGENT)
Date: Mon, 4 Feb 2013 23:26:00 +0900
> It looks like webserver doesn't like empty User-Agent.
> I just allowed to overwrite FTPUSERAGENT to unbreak it.
> As FTPUSERAGENT is usually not set, it is working for both cases.
Then the right fix is to unset FTPUSERAGENT (to make ftp use the default),
not overwriting it. FTPUSERAGENT is a variable for ftp(1), not pkgsrc.
---
Izumi Tsutsui
From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets own
FTPUSERAGENT
Date: Wed, 06 Feb 2013 22:36:49 +0900
I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
but it is the origin of the fault.
morr@'s change may rise new other faults, because it will work well with
x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
If other sites also has same problem (empty user-agent is not welcomed),
package using such site as MASTER_SITES will also fail to fetch,
because usually, FTPUSERAGENT environment variable is empty.
Adding such hack to affected packages is not smart.
My suggestion is "revert to r1.56".
For Tsutsui-san's situation,
* unset FTPUSERAGENT for the case using pkgsrc
or
* put followings to mk.conf (or set equivalent to environment variable)
PKGSRC_SETENV=${SETENV} -i
--
OBATA Akio / obache@NetBSD.org
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: obache@NetBSD.org, tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Wed, 6 Feb 2013 23:01:50 +0900
> I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
> but it is the origin of the fault.
Why? It's valid usage and documented environment in ftp(1) man page.
ftp(1) is not only for pkgsrc.
> morr@'s change may rise new other faults, because it will work well with
> x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
> If other sites also has same problem (empty user-agent is not welcomed),
> package using such site as MASTER_SITES will also fail to fetch,
> because usually, FTPUSERAGENT environment variable is empty.
ftp(1) uses FTPUSERAGENT only if getenv(3) returns !NULL.
http://nxr.netbsd.org/xref/src/usr.bin/ftp/fetch.c?r=1.201#857
---
if ((useragent=getenv("FTPUSERAGENT")) != NULL) {
fetch_printf(fin, "User-Agent: %s\r\n", useragent);
} else {
fetch_printf(fin, "User-Agent: %s/%s\r\n",
FTP_PRODUCT, FTP_VERSION);
}
---
If FTPUSERAGENT is set but empty, ftp(1) uses empty USERAGENT.
If FTPUSERAGENT is unset, ftp(1) uses the default FTP_PRODUCT
and FTP_VERSION (which is "NetBSD-ftp/20100605") and it has worked
as you see with the default settings.
The following patch makes both rxvt-unicode and sourceforge sites happy,
but I wonder if we should rather fix mk/fetch/fetch script directly.
Index: mk/fetch/fetch.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
retrieving revision 1.58
diff -u -p -r1.58 fetch.mk
--- mk/fetch/fetch.mk 20 Jan 2013 13:22:59 -0000 1.58
+++ mk/fetch/fetch.mk 6 Feb 2013 13:54:03 -0000
@@ -239,10 +239,10 @@ _FETCH_AFTER_ARGS.custom= ${FETCH_AFTER_
_FETCH_RESUME_ARGS.custom= ${FETCH_RESUME_ARGS}
_FETCH_OUTPUT_ARGS.custom= ${FETCH_OUTPUT_ARGS}
-_FETCH_CMD= ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
+_FETCH_CMD= unset FTPUSERAGENT && \
+ ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
CP=${TOOLS_CP:Q} \
ECHO=${TOOLS_ECHO:Q} \
- FTPUSERAGENT=${FTPUSERAGENT:Q} \
FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q} \
FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q} \
(btw FTP_VERSION has not been bumped after recent ssl addition)
---
Izumi Tsutsui
From: Daniel Horecki <morr@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Wed, 06 Feb 2013 21:14:52 +0100
Izumi Tsutsui <tsutsui@ceres.dti.ne.jp> writes:
Hi,
> The following reply was made to PR pkg/47525; it has been noted by GNATS.
>
> From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
> To: gnats-bugs@NetBSD.org
> Cc: obache@NetBSD.org, tsutsui@ceres.dti.ne.jp
> Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
> Date: Wed, 6 Feb 2013 23:01:50 +0900
>
> > I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
> > but it is the origin of the fault.
>
> Why? It's valid usage and documented environment in ftp(1) man page.
> ftp(1) is not only for pkgsrc.
>
> > morr@'s change may rise new other faults, because it will work well with
> > x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
> > If other sites also has same problem (empty user-agent is not welcomed),
> > package using such site as MASTER_SITES will also fail to fetch,
> > because usually, FTPUSERAGENT environment variable is empty.
>
> ftp(1) uses FTPUSERAGENT only if getenv(3) returns !NULL.
> http://nxr.netbsd.org/xref/src/usr.bin/ftp/fetch.c?r=1.201#857
> ---
> if ((useragent=getenv("FTPUSERAGENT")) != NULL) {
> fetch_printf(fin, "User-Agent: %s\r\n", useragent);
> } else {
> fetch_printf(fin, "User-Agent: %s/%s\r\n",
> FTP_PRODUCT, FTP_VERSION);
> }
> ---
>
> If FTPUSERAGENT is set but empty, ftp(1) uses empty USERAGENT.
> If FTPUSERAGENT is unset, ftp(1) uses the default FTP_PRODUCT
> and FTP_VERSION (which is "NetBSD-ftp/20100605") and it has worked
> as you see with the default settings.
>
> The following patch makes both rxvt-unicode and sourceforge sites happy,
> but I wonder if we should rather fix mk/fetch/fetch script directly.
>
> Index: mk/fetch/fetch.mk
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
> retrieving revision 1.58
> diff -u -p -r1.58 fetch.mk
> --- mk/fetch/fetch.mk 20 Jan 2013 13:22:59 -0000 1.58
> +++ mk/fetch/fetch.mk 6 Feb 2013 13:54:03 -0000
> @@ -239,10 +239,10 @@ _FETCH_AFTER_ARGS.custom= ${FETCH_AFTER_
> _FETCH_RESUME_ARGS.custom= ${FETCH_RESUME_ARGS}
> _FETCH_OUTPUT_ARGS.custom= ${FETCH_OUTPUT_ARGS}
>
> -_FETCH_CMD= ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
> +_FETCH_CMD= unset FTPUSERAGENT && \
> + ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
> CP=${TOOLS_CP:Q} \
> ECHO=${TOOLS_ECHO:Q} \
> - FTPUSERAGENT=${FTPUSERAGENT:Q} \
> FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q} \
> FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
> FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q} \
>
> (btw FTP_VERSION has not been bumped after recent ssl addition)
> ---
> Izumi Tsutsui
>
I'm affraid that your patch won't work (make doesn't know anything about unset) but
reverting fetch.mk to 1.56 and changing script alone will work:
Index: mk/fetch/fetch
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch,v
retrieving revision 1.14
diff -u -r1.14 fetch
--- mk/fetch/fetch 2 Aug 2009 02:38:52 -0000 1.14
+++ mk/fetch/fetch 6 Feb 2013 20:11:28 -0000
@@ -115,6 +115,8 @@
: ${TOUCH:=touch}
: ${WC:=wc}
+unset FTPUSERAGENT
+
self="${0##*/}"
usage() {
Index: mk/fetch/fetch.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
retrieving revision 1.58
diff -u -r1.58 fetch.mk
--- mk/fetch/fetch.mk 20 Jan 2013 13:22:59 -0000 1.58
+++ mk/fetch/fetch.mk 6 Feb 2013 20:11:28 -0000
@@ -242,7 +242,6 @@
_FETCH_CMD= ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
CP=${TOOLS_CP:Q} \
ECHO=${TOOLS_ECHO:Q} \
- FTPUSERAGENT=${FTPUSERAGENT:Q} \
FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q} \
FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q} \
Could you try that?
I could add guards for this unset, so only it will be used when FETCH_CMD is set to ftp.
Regards,
--
Daniel Horecki
http://morr.pl http://linux.pl http://netbsd.pl http://netbsd.org
HAIL ERIS!
BOFH since 1999.
From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets
ownFTPUSERAGENT
Date: Thu, 07 Feb 2013 09:10:14 +0900
On Wed, 06 Feb 2013 23:05:03 +0900, Izumi Tsutsui <tsutsui@ceres.dti.ne.jp> wrote:
> > I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
> > but it is the origin of the fault.
> Why? It's valid usage and documented environment in ftp(1) man page.
> ftp(1) is not only for pkgsrc.
ftp(1) is not a Moziilla (compatible) user agent, so it's not a valid value.
--
OBATA Akio / obache@NetBSD.org
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Thu, 7 Feb 2013 18:48:01 +0900
> I'm affraid that your patch won't work (make doesn't know anything about unset) but
> reverting fetch.mk to 1.56 and changing script alone will work:
>
> Index: mk/fetch/fetch
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/fetch/fetch,v
> retrieving revision 1.14
> diff -u -r1.14 fetch
> --- mk/fetch/fetch 2 Aug 2009 02:38:52 -0000 1.14
> +++ mk/fetch/fetch 6 Feb 2013 20:11:28 -0000
> @@ -115,6 +115,8 @@
> : ${TOUCH:=touch}
> : ${WC:=wc}
>
> +unset FTPUSERAGENT
> +
> self="${0##*/}"
>
> usage() {
> Index: mk/fetch/fetch.mk
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
> retrieving revision 1.58
> diff -u -r1.58 fetch.mk
> --- mk/fetch/fetch.mk 20 Jan 2013 13:22:59 -0000 1.58
> +++ mk/fetch/fetch.mk 6 Feb 2013 20:11:28 -0000
> @@ -242,7 +242,6 @@
> _FETCH_CMD= ${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q} \
> CP=${TOOLS_CP:Q} \
> ECHO=${TOOLS_ECHO:Q} \
> - FTPUSERAGENT=${FTPUSERAGENT:Q} \
> FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q} \
> FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
> FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q} \
>
>
> Could you try that?
The following tests also work with your diff:
% ( rm -rf distfiles/rxvt-unicode-* && cd x11/rxvt-unicode && env FTPUSERAGENT= make fetch )
% ( rm -rf distfiles/rxvt-unicode-* && cd x11/rxvt-unicode && env FTPUSERAGENT='Mozilla/5.0' make fetch )
% ( rm -rf distfiles/mlterm-* && cd x11/mlterm && env FTPUSERAGENT= make fetch )
% ( rm -rf distfiles/mlterm-* && cd x11/mlterm && env FTPUSERAGENT='Mozilla/5.0' make fetch )
> I could add guards for this unset, so only it will be used when FETCH_CMD is set to ftp.
That's also fine.
---
Izumi Tsutsui
State-Changed-From-To: open->closed
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Thu, 07 Feb 2013 22:33:59 +0900
State-Changed-Why:
obache@ says setting FTPUSERAGENT is invalid.
From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/47525 CVS commit: pkgsrc/mk/fetch
Date: Thu, 7 Feb 2013 13:59:10 +0000
Module Name: pkgsrc
Committed By: obache
Date: Thu Feb 7 13:59:10 UTC 2013
Modified Files:
pkgsrc/mk/fetch: fetch.mk
Log Message:
FTPUSERAGENT is a environment variable for ftp(1), not a make variable.
No need to pass here.
related to PR 47525.
To generate a diff of this commit:
cvs rdiff -u -r1.58 -r1.59 pkgsrc/mk/fetch/fetch.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.