NetBSD Problem Report #39208
From www@NetBSD.org Fri Jul 25 17:39:45 2008
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id 6ECFB63B99A
for <gnats-bugs@gnats.netbsd.org>; Fri, 25 Jul 2008 17:39:45 +0000 (UTC)
Message-Id: <20080725173945.1E58763B91E@narn.NetBSD.org>
Date: Fri, 25 Jul 2008 17:39:45 +0000 (UTC)
From: brian.de.alwis@usask.ca
Reply-To: brian.de.alwis@usask.ca
To: gnats-bugs@NetBSD.org
Subject: Using curl for FETCH_CMD is broken
X-Send-Pr-Version: www-1.0
>Number: 39208
>Category: pkg
>Synopsis: Using curl for FETCH_CMD is broken
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: pkg-manager
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Jul 25 17:40:00 +0000 2008
>Closed-Date: Wed Sep 24 05:38:29 +0000 2008
>Last-Modified: Wed Sep 24 05:38:29 +0000 2008
>Originator: Brian de Alwis
>Release: pkgsrc-current
>Organization:
University of Saskatchewan
>Environment:
Darwin beast 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386
>Description:
I posted about this to tech-pkg a while back, but forgot to follow-up wth a PR (Message-Id < DA8F765B-0FF5-43F1-9E8D-461C647880BC@usask.ca>).
I have been trying to use FETCH_CMD=curl as curl comes with a default set of certificates and thus a good candidate for fetching files using https. But I've encountered a few problems and in the process of trying to get it to work, I've found the description of FETCH_OUTPUT_ARGS to be unclear, and also found that the example using wget in the pkgsrc guide doesn't actually work.
curl, by default, sends its output to stdout. curl can be directed to save the content to an explicitly specified file using '-o filename'. mk/fetch/fetch describes FETCH_OUTPUT_ARGS as:
# FETCH_OUTPUT_ARGS
# These options specify the name of the local file
# that will hold the contents of the fetched file.
This wording is a bit unclear -- the variable doesn't specify the local file, but provides the arguments necessary to specify the output file. From looking at the example in the pkgsrc guide and the mk/fetch/fetch code, I think this is better worded as:
# FETCH_OUTPUT_ARGS
# These options are used to specify the name of the local file
# that will hold the contents of the fetched file.
FETCH_OUTPUT_ARGS doesn't actually seem to be used except for resumed fetches (which is not the default). mk/fetch/fetch does the fetch using:
if ${TEST} -n "$resume"; then
fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${FETCH_RESUME_ARGS} ${FETCH_OUTPUT_ARGS} $outputfile $site$file"
else
fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} $site$file ${FETCH_AFTER_ARGS}"
fi
I think it makes more sense to specify `${FETCH_OUTPUT_ARGS} $outputfile' for both cases (patch appended below). After all, the two commands should be mirrors except for the FETCH_RESUME_ARGS, right?
With this change, I can use curl to fetch files using the following:
FETCH_CMD= curl
FETCH_BEFORE_ARGS=
FETCH_OUTPUT_ARGS= -o
FETCH_RESUME_ARGS=
Without the patch, curl cannot be used as a fetch command.
>How-To-Repeat:
Install net/curl
$ cd x11/gtk2
$ rm ../../distfiles/gtk+-2*
$ make FETCH_CMD=curl FETCH_BEFORE_ARGS="" FETCH_OUTPUT_ARGS=-o FETCH_RESUME_ARGS="" fetch
see the file output go to the console rather than to a file
>Fix:
Only in /scratch/dev/cvs-pkgsrc.org/mk/fetch/: CVS
diff -ruwb /scratch/dev/cvs-pkgsrc.org/mk/fetch/fetch mk/fetch/fetch
--- /scratch/dev/cvs-pkgsrc.org/mk/fetch/fetch 2007-08-04 19:49:47.000000000 -0600
+++ mk/fetch/fetch 2008-02-29 18:36:32.000000000 -0600
@@ -99,8 +99,9 @@
# previous file transfer.
#
# FETCH_OUTPUT_ARGS
-# These options specify the name of the local file
-# that will hold the contents of the fetched file.
+# These options are used to specify the name of
+# the local file that will hold the contents of the
+# fetched file.
#
######################################################################
@@ -262,7 +263,7 @@
if ${TEST} -n "$resume"; then
fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${FETCH_RESUME_ARGS} ${FETCH_OUTPUT_ARGS} $outputfile $site$file"
else
- fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} $site$file ${FETCH_AFTER_ARGS}"
+ fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${FETCH_OUTPUT_ARGS} $outputfile $site$file ${FETCH_AFTER_ARGS}"
fi
${TEST} -z "$verbose" || ${ECHO} "$fetch_cmd"
$fetch_cmd )
>Release-Note:
>Audit-Trail:
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/39208: Using curl for FETCH_CMD is broken
Date: Fri, 25 Jul 2008 19:44:47 +0200
On Fri, Jul 25, 2008 at 05:40:00PM +0000, brian.de.alwis@usask.ca wrote:
> curl, by default, sends its output to stdout.
You should use -O for curl.
Joerg
From: Brian de Alwis <brian.de.alwis@usask.ca>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/39208: Using curl for FETCH_CMD is broken
Date: Fri, 25 Jul 2008 13:04:43 -0600
On 25-Jul-2008, at 11:45 AM, Joerg Sonnenberger wrote:
> You should use -O for curl.
Ok, that works. Thanks. The description for FETCH_OUTPUT_ARGS is
still difficult to understand though.
--
Brian de Alwis | HCI Lab | University of Saskatchewan
State-Changed-From-To: open->feedback
State-Changed-By: obache@NetBSD.org
State-Changed-When: Sun, 07 Sep 2008 08:19:55 +0000
State-Changed-Why:
Settings for FETCH_CMD=curl was added.
Would you try just FETCH_CMD=curl on your environment and feedback?
From: OBATA Akio <obache@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/39208 CVS commit: pkgsrc/mk/fetch
Date: Sun, 7 Sep 2008 08:17:40 +0000 (UTC)
Module Name: pkgsrc
Committed By: obache
Date: Sun Sep 7 08:17:40 UTC 2008
Modified Files:
pkgsrc/mk/fetch: fetch.mk
Log Message:
Add FETCH_*_ARGS templates for curl.
Should be solution for PR 39208.
To generate a diff of this commit:
cvs rdiff -r1.34 -r1.35 pkgsrc/mk/fetch/fetch.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Brian de Alwis <brian.de.alwis@usask.ca>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/39208 (Using curl for FETCH_CMD is broken)
Date: Tue, 23 Sep 2008 12:48:53 -0600
On 7-Sep-2008, at 2:19 AM, obache@NetBSD.org wrote:
> Settings for FETCH_CMD=curl was added.
> Would you try just FETCH_CMD=curl on your environment and feedback?
Sorry for the delayed response, but I've been away for 3 weeks.
Your changes seems to work for me (and looks like a great
improvement). Thanks!
BTW: there's a small typo in mk/fetch/fetch.mk, where "honor" is
mispelled as "honer"
Brian.
--
Brian de Alwis | HCI Lab | University of Saskatchewan
State-Changed-From-To: feedback->closed
State-Changed-By: obache@NetBSD.org
State-Changed-When: Wed, 24 Sep 2008 05:38:29 +0000
State-Changed-Why:
Submitter confirmed fixed.
Thanks for the feedback and pointed out a typo!
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.