NetBSD Problem Report #47525

From tsutsui@ceres.dti.ne.jp  Sat Feb  2 22:05:20 2013
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	by www.NetBSD.org (Postfix) with ESMTP id 479B463D7B3
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  2 Feb 2013 22:05:20 +0000 (UTC)
Message-Id: <201302022205.r12M5GrH014732@mirage.localdomain>
Date: Sun, 3 Feb 2013 07:05:16 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@gnats.NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: fetch from sourceforge fails if user sets own FTPUSERAGENT
X-Send-Pr-Version: 3.95

>Number:         47525
>Category:       pkg
>Synopsis:       fetch from sourceforge fails if user sets own FTPUSERAGENT
>Confidential:   no
>Severity:       non-critical
>Priority:       high
>Responsible:    morr
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 02 22:10:01 +0000 2013
>Closed-Date:    Thu Feb 07 13:33:59 +0000 2013
>Last-Modified:  Thu Feb 07 14:00:08 +0000 2013
>Originator:     Izumi Tsutsui
>Release:        NetBSD 6.0
>Organization:
>Environment:
System: NetBSD i386 6.0 + pkgsrc HEAD
Architecture: i386
Machine: i386
>Description:
sourceforge download sites seem to check client's useragent value
and change behavior of download pages so if a user sets own FTPUSERAGENT
("Mozilla/5.0" etc.) for ftp(1) fetching from sourceforge fails.

It was once fixed in pkgsrc/mk/fetch/fetch.mk rev 1.57:
http://mail-index.netbsd.org/pkgsrc-changes/2012/12/14/msg081842.html
but in rev 1.58 FTPUSERAGENT variable is passed again on make fetch:
http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/mk/fetch/fetch.mk.diff?r1=1.57&r2=1.58

>How-To-Repeat:
% cd pkgsrc/x11/mlterm ; env FTPUSERAGENT="Mozilla/5.0" make fetch

>Fix:
Use pkgsrc own variable to override ftp(1)'s FTPUSERAGENT in fetch.mk?
(I'm not sure which package has sites "broken without FTPUSERAGENT" though)

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: pkg-manager->morr
Responsible-Changed-By: obache@NetBSD.org
Responsible-Changed-When: Mon, 04 Feb 2013 11:37:12 +0000
Responsible-Changed-Why:
Over to changer.


From: Daniel Horecki <morr@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/47525 (fetch from sourceforge fails if user sets own FTPUSERAGENT)
Date: Mon, 04 Feb 2013 15:02:40 +0100

 Hi,

 Revision 1.57 of  pkgsrc/mk/fetch/fetch.mk broke fetching of x11/rxvt-unicode:

 Trying 2001:470:987b::a00:13:80 ...
 Requesting http://dist.schmorp.de/rxvt-unicode/rxvt-unicode-9.16.tar.bz2
 ftp: Error retrieving file `500 Internal Server Error'
 fetch: Unable to fetch expected file rxvt-unicode-9.16.tar.bz2
 Trying 2001:470:987b::a00:13:80 ...
 Requesting http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-9.16.tar.bz2
 ftp: Error retrieving file `500 Internal Server Error'
 fetch: Unable to fetch expected file rxvt-unicode-9.16.tar.bz2

 It looks like webserver doesn't like empty User-Agent. I just allowed to overwrite FTPUSERAGENT to unbreak it.
 As FTPUSERAGENT is usually not set, it is working for both cases.

 I've checked user test case with default User-Agent of our ftp and it worked without problem:

 # env FTPUSERAGENT="NetBSD-ftp/20100605" make fetch
 => Bootstrap dependency digest>=20010302: found digest-20121220
 => Fetching mlterm-3.1.7.tar.gz
 => Total size: 3535404 bytes
 Requesting http://downloads.sourceforge.net/sourceforge/mlterm/mlterm-3.1.7.tar.gz
 Redirected to http://downloads.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
 Requesting http://downloads.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
 Redirected to http://freefr.dl.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
 Trying 2001:1b48:10f::7:80 ...
 Requesting http://freefr.dl.sourceforge.net/project/mlterm/01release/mlterm-3.1.7/mlterm-3.1.7.tar.gz
 100% |*************************************************************************************************************************************|  3452 KiB  210.67 KiB/s    00:00 ETA
 3535404 bytes retrieved in 00:16 (210.67 KiB/s)
 # make checksum
 => Bootstrap dependency digest>=20010302: found digest-20121220
 => Checksum SHA1 OK for mlterm-3.1.7.tar.gz
 => Checksum RMD160 OK for mlterm-3.1.7.tar.gz

 Why this variable was cleaned in first place?

 Regards,

 -- 
 Daniel Horecki
 http://morr.pl http://linux.pl http://netbsd.pl http://netbsd.org
 HAIL ERIS!
 BOFH since 1999.

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525 (fetch from sourceforge fails if user sets own FTPUSERAGENT)
Date: Mon, 4 Feb 2013 23:26:00 +0900

 >  It looks like webserver doesn't like empty User-Agent.
 >  I just allowed to overwrite FTPUSERAGENT to unbreak it.
 >  As FTPUSERAGENT is usually not set, it is working for both cases.

 Then the right fix is to unset FTPUSERAGENT (to make ftp use the default),
 not overwriting it. FTPUSERAGENT is a variable for ftp(1), not pkgsrc.
 ---
 Izumi Tsutsui

From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets own
 FTPUSERAGENT
Date: Wed, 06 Feb 2013 22:36:49 +0900

 I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
 but it is the origin of the fault.

 morr@'s change may rise new other faults, because it will work well with
 x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
 If other sites also has same problem (empty user-agent is not welcomed),
 package using such site as MASTER_SITES will also fail to fetch,
 because usually, FTPUSERAGENT environment variable is empty.
 Adding such hack to affected packages is not smart.

 My suggestion is "revert to r1.56".

 For Tsutsui-san's situation,
 * unset FTPUSERAGENT for the case using pkgsrc
 or
 * put followings to mk.conf (or set equivalent to environment variable)
    PKGSRC_SETENV=${SETENV} -i

 -- 
 OBATA Akio / obache@NetBSD.org

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: obache@NetBSD.org, tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Wed, 6 Feb 2013 23:01:50 +0900

 >  I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
 >  but it is the origin of the fault.

 Why?  It's valid usage and documented environment in ftp(1) man page.
 ftp(1) is not only for pkgsrc.

 >  morr@'s change may rise new other faults, because it will work well with
 >  x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
 >  If other sites also has same problem (empty user-agent is not welcomed),
 >  package using such site as MASTER_SITES will also fail to fetch,
 >  because usually, FTPUSERAGENT environment variable is empty.

 ftp(1) uses FTPUSERAGENT only if getenv(3) returns !NULL.
 http://nxr.netbsd.org/xref/src/usr.bin/ftp/fetch.c?r=1.201#857
 ---
 		if ((useragent=getenv("FTPUSERAGENT")) != NULL) {
 			fetch_printf(fin, "User-Agent: %s\r\n", useragent);
 		} else {
 			fetch_printf(fin, "User-Agent: %s/%s\r\n",
 			    FTP_PRODUCT, FTP_VERSION);
 		}
 ---

 If FTPUSERAGENT is set but empty, ftp(1) uses empty USERAGENT.
 If FTPUSERAGENT is unset, ftp(1) uses the default FTP_PRODUCT
 and FTP_VERSION (which is "NetBSD-ftp/20100605") and it has worked
 as you see with the default settings.

 The following patch makes both rxvt-unicode and sourceforge sites happy,
 but I wonder if we should rather fix mk/fetch/fetch script directly.

 Index: mk/fetch/fetch.mk
 ===================================================================
 RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
 retrieving revision 1.58
 diff -u -p -r1.58 fetch.mk
 --- mk/fetch/fetch.mk	20 Jan 2013 13:22:59 -0000	1.58
 +++ mk/fetch/fetch.mk	6 Feb 2013 13:54:03 -0000
 @@ -239,10 +239,10 @@ _FETCH_AFTER_ARGS.custom=	${FETCH_AFTER_
  _FETCH_RESUME_ARGS.custom=	${FETCH_RESUME_ARGS}
  _FETCH_OUTPUT_ARGS.custom=	${FETCH_OUTPUT_ARGS}

 -_FETCH_CMD=	${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
 +_FETCH_CMD=	unset FTPUSERAGENT && 				\
 +		${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
  		CP=${TOOLS_CP:Q}				\
  		ECHO=${TOOLS_ECHO:Q}				\
 -		FTPUSERAGENT=${FTPUSERAGENT:Q}			\
  		FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q}	\
  		FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
  		FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q}   \

 (btw FTP_VERSION has not been bumped after recent ssl addition)
 ---
 Izumi Tsutsui

From: Daniel Horecki <morr@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Wed, 06 Feb 2013 21:14:52 +0100

 Izumi Tsutsui <tsutsui@ceres.dti.ne.jp> writes:

 Hi,

 > The following reply was made to PR pkg/47525; it has been noted by GNATS.
 >
 > From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
 > To: gnats-bugs@NetBSD.org
 > Cc: obache@NetBSD.org, tsutsui@ceres.dti.ne.jp
 > Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
 > Date: Wed, 6 Feb 2013 23:01:50 +0900
 >
 >  >  I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
 >  >  but it is the origin of the fault.
 >  
 >  Why?  It's valid usage and documented environment in ftp(1) man page.
 >  ftp(1) is not only for pkgsrc.
 >  
 >  >  morr@'s change may rise new other faults, because it will work well with
 >  >  x11/rxvt-unicode/Makefile r1.41, adding FTPUSERAGENT="NetBSD/ftp".
 >  >  If other sites also has same problem (empty user-agent is not welcomed),
 >  >  package using such site as MASTER_SITES will also fail to fetch,
 >  >  because usually, FTPUSERAGENT environment variable is empty.
 >  
 >  ftp(1) uses FTPUSERAGENT only if getenv(3) returns !NULL.
 >  http://nxr.netbsd.org/xref/src/usr.bin/ftp/fetch.c?r=1.201#857
 >  ---
 >  		if ((useragent=getenv("FTPUSERAGENT")) != NULL) {
 >  			fetch_printf(fin, "User-Agent: %s\r\n", useragent);
 >  		} else {
 >  			fetch_printf(fin, "User-Agent: %s/%s\r\n",
 >  			    FTP_PRODUCT, FTP_VERSION);
 >  		}
 >  ---
 >  
 >  If FTPUSERAGENT is set but empty, ftp(1) uses empty USERAGENT.
 >  If FTPUSERAGENT is unset, ftp(1) uses the default FTP_PRODUCT
 >  and FTP_VERSION (which is "NetBSD-ftp/20100605") and it has worked
 >  as you see with the default settings.
 >  
 >  The following patch makes both rxvt-unicode and sourceforge sites happy,
 >  but I wonder if we should rather fix mk/fetch/fetch script directly.
 >  
 >  Index: mk/fetch/fetch.mk
 >  ===================================================================
 >  RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
 >  retrieving revision 1.58
 >  diff -u -p -r1.58 fetch.mk
 >  --- mk/fetch/fetch.mk	20 Jan 2013 13:22:59 -0000	1.58
 >  +++ mk/fetch/fetch.mk	6 Feb 2013 13:54:03 -0000
 >  @@ -239,10 +239,10 @@ _FETCH_AFTER_ARGS.custom=	${FETCH_AFTER_
 >   _FETCH_RESUME_ARGS.custom=	${FETCH_RESUME_ARGS}
 >   _FETCH_OUTPUT_ARGS.custom=	${FETCH_OUTPUT_ARGS}
 >   
 >  -_FETCH_CMD=	${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
 >  +_FETCH_CMD=	unset FTPUSERAGENT && 				\
 >  +		${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
 >   		CP=${TOOLS_CP:Q}				\
 >   		ECHO=${TOOLS_ECHO:Q}				\
 >  -		FTPUSERAGENT=${FTPUSERAGENT:Q}			\
 >   		FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q}	\
 >   		FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
 >   		FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q}   \
 >  
 >  (btw FTP_VERSION has not been bumped after recent ssl addition)
 >  ---
 >  Izumi Tsutsui
 >  

 I'm affraid that your patch won't work (make doesn't know anything about unset) but 
 reverting fetch.mk to 1.56 and changing script alone will work:

 Index: mk/fetch/fetch
 ===================================================================
 RCS file: /cvsroot/pkgsrc/mk/fetch/fetch,v
 retrieving revision 1.14
 diff -u -r1.14 fetch
 --- mk/fetch/fetch	2 Aug 2009 02:38:52 -0000	1.14
 +++ mk/fetch/fetch	6 Feb 2013 20:11:28 -0000
 @@ -115,6 +115,8 @@
  : ${TOUCH:=touch}
  : ${WC:=wc}

 +unset FTPUSERAGENT
 +
  self="${0##*/}"

  usage() {
 Index: mk/fetch/fetch.mk
 ===================================================================
 RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
 retrieving revision 1.58
 diff -u -r1.58 fetch.mk
 --- mk/fetch/fetch.mk	20 Jan 2013 13:22:59 -0000	1.58
 +++ mk/fetch/fetch.mk	6 Feb 2013 20:11:28 -0000
 @@ -242,7 +242,6 @@
  _FETCH_CMD=	${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
  		CP=${TOOLS_CP:Q}				\
  		ECHO=${TOOLS_ECHO:Q}				\
 -		FTPUSERAGENT=${FTPUSERAGENT:Q}			\
  		FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q}	\
  		FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
  		FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q}   \


 Could you try that? 
 I could add guards for this unset, so only it will be used when FETCH_CMD is set to ftp.

 Regards,

 -- 
 Daniel Horecki
 http://morr.pl http://linux.pl http://netbsd.pl http://netbsd.org
 HAIL ERIS!
 BOFH since 1999.

From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets
 ownFTPUSERAGENT
Date: Thu, 07 Feb 2013 09:10:14 +0900

 On Wed, 06 Feb 2013 23:05:03 +0900, Izumi Tsutsui <tsutsui@ceres.dti.ne.jp> wrote:

 >  >  I don't know why you want to use such a bogus ("Mozilla/5.0") user agent,
 >  >  but it is the origin of the fault.
 > Why?  It's valid usage and documented environment in ftp(1) man page.
 >  ftp(1) is not only for pkgsrc.

 ftp(1) is not a Moziilla (compatible) user agent, so it's not a valid value.

 -- 
 OBATA Akio / obache@NetBSD.org

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: pkg/47525: fetch from sourceforge fails if user sets ownFTPUSERAGENT
Date: Thu, 7 Feb 2013 18:48:01 +0900

 >  I'm affraid that your patch won't work (make doesn't know anything about unset) but 
 >  reverting fetch.mk to 1.56 and changing script alone will work:
 >  
 >  Index: mk/fetch/fetch
 >  ===================================================================
 >  RCS file: /cvsroot/pkgsrc/mk/fetch/fetch,v
 >  retrieving revision 1.14
 >  diff -u -r1.14 fetch
 >  --- mk/fetch/fetch	2 Aug 2009 02:38:52 -0000	1.14
 >  +++ mk/fetch/fetch	6 Feb 2013 20:11:28 -0000
 >  @@ -115,6 +115,8 @@
 >   : ${TOUCH:=touch}
 >   : ${WC:=wc}
 >   
 >  +unset FTPUSERAGENT
 >  +
 >   self="${0##*/}"
 >   
 >   usage() {
 >  Index: mk/fetch/fetch.mk
 >  ===================================================================
 >  RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
 >  retrieving revision 1.58
 >  diff -u -r1.58 fetch.mk
 >  --- mk/fetch/fetch.mk	20 Jan 2013 13:22:59 -0000	1.58
 >  +++ mk/fetch/fetch.mk	6 Feb 2013 20:11:28 -0000
 >  @@ -242,7 +242,6 @@
 >   _FETCH_CMD=	${PKGSRC_SETENV} CHECKSUM=${_CHECKSUM_CMD:Q}	\
 >   		CP=${TOOLS_CP:Q}				\
 >   		ECHO=${TOOLS_ECHO:Q}				\
 >  -		FTPUSERAGENT=${FTPUSERAGENT:Q}			\
 >   		FETCH_CMD=${_FETCH_CMD.${FETCH_USING}:Q}	\
 >   		FETCH_BEFORE_ARGS=${_FETCH_BEFORE_ARGS.${FETCH_USING}:Q} \
 >   		FETCH_AFTER_ARGS=${_FETCH_AFTER_ARGS.${FETCH_USING}:Q}   \
 >  
 >  
 >  Could you try that? 

 The following tests also work with your diff:

 % ( rm -rf distfiles/rxvt-unicode-* && cd x11/rxvt-unicode && env FTPUSERAGENT= make fetch )

 % ( rm -rf distfiles/rxvt-unicode-* && cd x11/rxvt-unicode && env FTPUSERAGENT='Mozilla/5.0' make fetch )

 % ( rm -rf distfiles/mlterm-* && cd x11/mlterm && env FTPUSERAGENT= make fetch )

 % ( rm -rf distfiles/mlterm-* && cd x11/mlterm && env FTPUSERAGENT='Mozilla/5.0' make fetch )


 >  I could add guards for this unset, so only it will be used when FETCH_CMD is set to ftp.

 That's also fine.

 ---
 Izumi Tsutsui

State-Changed-From-To: open->closed
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Thu, 07 Feb 2013 22:33:59 +0900
State-Changed-Why:
obache@ says setting FTPUSERAGENT is invalid.


From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/47525 CVS commit: pkgsrc/mk/fetch
Date: Thu, 7 Feb 2013 13:59:10 +0000

 Module Name:	pkgsrc
 Committed By:	obache
 Date:		Thu Feb  7 13:59:10 UTC 2013

 Modified Files:
 	pkgsrc/mk/fetch: fetch.mk

 Log Message:
 FTPUSERAGENT is a environment variable for ftp(1), not a make variable.
 No need to pass here.

 related to PR 47525.


 To generate a diff of this commit:
 cvs rdiff -u -r1.58 -r1.59 pkgsrc/mk/fetch/fetch.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.