NetBSD Problem Report #34695
From mishka@netbsd.org Mon Oct 2 14:37:16 2006
Return-Path: <mishka@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id DD15A63B911
for <gnats-bugs@gnats.NetBSD.org>; Mon, 2 Oct 2006 14:37:15 +0000 (UTC)
Message-Id: <20061002143715.D198C63B135@mail.netbsd.org>
Date: Mon, 2 Oct 2006 14:37:15 +0000 (UTC)
From: mishka@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: Master sites list randomization
X-Send-Pr-Version: 3.95
>Number: 34695
>Category: pkg
>Synopsis: Master sites list randomization
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: mishka
>State: closed
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Mon Oct 02 14:40:00 +0000 2006
>Closed-Date: Sat Nov 25 16:50:55 +0000 2006
>Last-Modified: Sat Nov 25 16:50:55 +0000 2006
>Originator: Mike M. Volokhov
>Release: pkgsrc-current as of 20060902
>Organization:
NONE
>Environment:
No matter
>Description:
Hello!
As you may note, static list of package master sites
accomplished with some known problems. Huge load on top of
the list sites from pkgsrc users, and died sites (also at
the top of list) are most annoying.
>How-To-Repeat:
make fetch
>Fix:
At this moment you can shuffle master sites with MASTER_SORT
and MASTER_SORT_REGEX. But you may add some extra randomization
with just implemented MASTER_SORT_RANDOM=yes|no feature.
It will randomize sites just before any other ordering, so
if you altready using some sorting, your sites will be
reshuffled in each group of your selection. To randomize
a whole list use MASTER_SORT_RANDOM=yes only.
To see randomization in action, just add ``| tee /tmp/sites''
to _RAND_SITES_CMD variable.
Also, I'd likely move _MASTER_SITE_BACKUP from unsorted
list to sorted one, so they all will be used more as backup
sites.
The patch:
Index: defaults/mk.conf
===================================================================
RCS file: /cvsroot/pkgsrc/mk/defaults/mk.conf,v
retrieving revision 1.131
diff -u -r1.131 mk.conf
--- defaults/mk.conf 27 Sep 2006 15:18:15 -0000 1.131
+++ defaults/mk.conf 2 Oct 2006 14:14:45 -0000
@@ -397,6 +397,16 @@
# Possible: Regexps as in awk(1)
# Default: none
+.if defined(PKG_DEVELOPER)
+MASTER_SORT_RANDOM?= NO
+.else
+MASTER_SORT_RANDOM?= YES
+.endif
+# If set to YES or yes, a list of master sites will be randomly intermixed.
+# Also, both MASTER_SORT and MASTER_SORT_REGEX may be applied later.
+# Possible: yes, no / not defined
+# Default: NO if PKG_DEVELOPER is defined, YES otherwise
+
#PATCH_DEBUG=
# Used to debug patches as they are applied
# Possible: defined, not defined
Index: fetch/fetch.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
retrieving revision 1.20
diff -u -r1.20 fetch.mk
--- fetch/fetch.mk 27 Jul 2006 18:48:03 -0000 1.20
+++ fetch/fetch.mk 2 Oct 2006 14:14:45 -0000
@@ -30,10 +30,20 @@
_BUILD_DEFS+= _DISTFILES _PATCHFILES
# Set up _ORDERED_SITES to work out the exact list of sites for every file,
-# using the dynamic sites script, or sorting according to the master site
-# list or the patterns in MASTER_SORT or MASTER_SORT_REGEX as appropriate.
+# using the dynamic sites script, or ordering according to the master site
+# list, MASTER_SORT_RANDOM randomization feature, or the patterns in
+# MASTER_SORT or MASTER_SORT_REGEX as appropriate.
# No actual sorting is done until _ORDERED_SITES is expanded.
#
+.if defined(MASTER_SORT_RANDOM) && !empty(MASTER_SORT_RANDOM:M[yY][eE][sS])
+_MASTER_RAND_AWK= { srand(systime()); n = split($$0, site); \
+ for (i = n; i > 0; i--) { \
+ ir = int(rand() * i + 1); \
+ t = site[i]; site[i] = site[ir]; site[ir] = t; \
+ print site[i]; } }
+_RAND_SITES_CMD= | ${AWK} '${_MASTER_RAND_AWK}'
+.endif
+
.if defined(MASTER_SORT) || defined(MASTER_SORT_REGEX)
MASTER_SORT?=
MASTER_SORT_REGEX?=
@@ -44,9 +54,12 @@
_MASTER_SORT_AWK+= /${srt:C/\//\\\//g}/ { good["${srt:S/\\/\\\\/g}"] = good["${srt:S/\\/\\\\/g}"] " " $$0 ; next; }
. endfor
_MASTER_SORT_AWK+= { rest = rest " " $$0; } END { n=split(gl, gla); for(i=1;i<=n;i++) { print good[gla[i]]; } print rest; }
+_SORT_SITES_CMD+= | ${AWK} '${_MASTER_SORT_AWK}'
+.endif
-_SORT_SITES_CMD= ${ECHO} $$unsorted_sites | ${AWK} '${_MASTER_SORT_AWK}'
-_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_CMD:S/\\/\\\\/g:C/"/\"/g}`
+.if defined(_RAND_SITES_CMD) || defined(_SORT_SITES_CMD)
+_SORT_SITES_FULL_CMD= ${ECHO} $$unsorted_sites ${_RAND_SITES_CMD} ${_SORT_SITES_CMD}
+_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_FULL_CMD:S/\\/\\\\/g:C/"/\"/g}`
.else
_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} $$unsorted_sites
.endif
@@ -245,8 +258,8 @@
fi; \
done
${_PKG_SILENT}${_PKG_DEBUG}set -e; \
- unsorted_sites="${SITES.${.TARGET:T:S/=/--/}} ${_MASTER_SITE_BACKUP}"; \
- sites="${_ORDERED_SITES}"; \
+ unsorted_sites="${SITES.${.TARGET:T:S/=/--/}}"; \
+ sites="${_ORDERED_SITES} ${_MASTER_SITE_BACKUP}"; \
cd ${.TARGET:H:S/\/${DIST_SUBDIR}$//} && \
${_FETCH_CMD} ${_FETCH_ARGS} ${.TARGET:T} $$sites
${_PKG_SILENT}${_PKG_DEBUG} \
>Release-Note:
>Audit-Trail:
From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:25:35 +0000
The awk code used in origianl patch is somewhat NetBSD dependent - other
awk implementations may missing some features, like systime(). So, taking
Solaris' nawk as a lowest denominator, please use the patch below:
Index: fetch.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
retrieving revision 1.20
diff -u -r1.20 fetch.mk
--- fetch.mk 27 Jul 2006 18:48:03 -0000 1.20
+++ fetch.mk 5 Oct 2006 15:05:54 -0000
@@ -30,10 +30,21 @@
_BUILD_DEFS+= _DISTFILES _PATCHFILES
# Set up _ORDERED_SITES to work out the exact list of sites for every file,
-# using the dynamic sites script, or sorting according to the master site
-# list or the patterns in MASTER_SORT or MASTER_SORT_REGEX as appropriate.
+# using the dynamic sites script, or ordering according to the master site
+# list, MASTER_SORT_RANDOM randomization feature, or the patterns in
+# MASTER_SORT or MASTER_SORT_REGEX as appropriate.
# No actual sorting is done until _ORDERED_SITES is expanded.
#
+.if defined(MASTER_SORT_RANDOM) && !empty(MASTER_SORT_RANDOM:M[yY][eE][sS])
+_MASTER_RAND_AWK= BEGIN { srand(seed) } { \
+ n = split($$0, site); \
+ for (i = n; i > 0; i--) { \
+ ir = int(rand() * i + 1); \
+ t = site[i]; site[i] = site[ir]; site[ir] = t; \
+ print site[i]; } }
+_RAND_SITES_CMD= | ${AWK} -v seed=$$$$ '${_MASTER_RAND_AWK}' | tee /tmp/sites
+.endif
+
.if defined(MASTER_SORT) || defined(MASTER_SORT_REGEX)
MASTER_SORT?=
MASTER_SORT_REGEX?=
@@ -44,9 +55,12 @@
_MASTER_SORT_AWK+= /${srt:C/\//\\\//g}/ { good["${srt:S/\\/\\\\/g}"] = good["${srt:S/\\/\\\\/g}"] " " $$0 ; next; }
. endfor
_MASTER_SORT_AWK+= { rest = rest " " $$0; } END { n=split(gl, gla); for(i=1;i<=n;i++) { print good[gla[i]]; } print rest; }
+_SORT_SITES_CMD+= | ${AWK} '${_MASTER_SORT_AWK}'
+.endif
-_SORT_SITES_CMD= ${ECHO} $$unsorted_sites | ${AWK} '${_MASTER_SORT_AWK}'
-_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_CMD:S/\\/\\\\/g:C/"/\"/g}`
+.if defined(_RAND_SITES_CMD) || defined(_SORT_SITES_CMD)
+_SORT_SITES_FULL_CMD= ${ECHO} $$unsorted_sites ${_RAND_SITES_CMD} ${_SORT_SITES_CMD}
+_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_FULL_CMD:S/\\/\\\\/g:C/"/\"/g}`
.else
_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} $$unsorted_sites
.endif
@@ -245,8 +259,8 @@
fi; \
done
${_PKG_SILENT}${_PKG_DEBUG}set -e; \
- unsorted_sites="${SITES.${.TARGET:T:S/=/--/}} ${_MASTER_SITE_BACKUP}"; \
- sites="${_ORDERED_SITES}"; \
+ unsorted_sites="${SITES.${.TARGET:T:S/=/--/}}"; \
+ sites="${_ORDERED_SITES} ${_MASTER_SITE_BACKUP}"; \
cd ${.TARGET:H:S/\/${DIST_SUBDIR}$//} && \
${_FETCH_CMD} ${_FETCH_ARGS} ${.TARGET:T} $$sites
${_PKG_SILENT}${_PKG_DEBUG} \
The defaults/mk.conf patch still with no changes.
--
Mishka.
P.S. Thanks to salo for warning me on this problem.
From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:38:49 +0000
Gah! In the latest patch _RAND_SITES_CMD variable must be without
'tee' ath the end, of course:
_RAND_SITES_CMD= | ${AWK} -v seed=$$$$ '${_MASTER_RAND_AWK}'
--
Mishka.
From: "Mike M. Volokhov" <mishka@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:16:44 +0300
The single objection I've seen is that awk-based randomization logic
might not work with non NetBSD awks (for example, on Solaris). So, can
anybody confirm that the following script is workable on non NetBSD
platform?
{
srand(systime());
n = split($0, words);
for (i = n; i > 0; i--) {
ir = int(rand() * i + 1);
t = words[i];
words[i] = words[ir];
words[ir] = t;
print words[i];
}
}
Usage:
echo "one two three four five six seven eight nine ten" | awk -f random.awk
--
Mishka.
From: Roland Illig <rillig@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/34695
Date: Fri, 06 Oct 2006 21:16:23 +0200
Mike M. Volokhov wrote:
> The following reply was made to PR pkg/34695; it has been noted by GNATS.
>
> From: "Mike M. Volokhov" <mishka@NetBSD.org>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: pkg/34695
> Date: Thu, 5 Oct 2006 15:16:44 +0300
>
> The single objection I've seen is that awk-based randomization logic
> might not work with non NetBSD awks (for example, on Solaris). So, can
> anybody confirm that the following script is workable on non NetBSD
> platform?
>
> {
> srand(systime());
> n = split($0, words);
> for (i = n; i > 0; i--) {
> ir = int(rand() * i + 1);
> t = words[i];
> words[i] = words[ir];
> words[ir] = t;
> print words[i];
> }
> }
>
> Usage:
> echo "one two three four five six seven eight nine ten" | awk -f random.awk
rzdspc8:~: uname -a
SunOS rzdspc8 5.10 Generic_118833-18 sun4u sparc SUNW,Sun-Fire-V440
rzdspc8:~: alias e="echo \"one two three four five six seven eight nine
ten\""
rzdspc8:~: e | /usr/bin/awk -f r.awk
awk: syntax error near line 2
awk: illegal statement near line 2
awk: syntax error near line 5
awk: illegal statement near line 5
rzdspc8:~: e | /usr/pkg/2006Q2/bin/nawk -f r.awk
/usr/pkg/2006Q2/bin/nawk: calling undefined function systime
input record number 1, file
source line number 2
rzdspc8:~: e | /usr/bin/nawk -f r.awk
/usr/bin/nawk: calling undefined function systime
input record number 1
source line number 2
rzdspc8:~: e | /usr/pkg/2006Q2/bin/gawk -f r.awk
seven
one
four
five
two
three
eight
nine
six
ten
rzdspc8:~: e | /opt/sfw/bin/gawk -f r.awk
nine
four
five
ten
three
six
two
seven
eight
one
^D
Roland
Responsible-Changed-From-To: pkg-manager->mishka
Responsible-Changed-By: mishka@netbsd.org
Responsible-Changed-When: Mon, 09 Oct 2006 08:35:53 +0000
Responsible-Changed-Why:
I'll take it.
From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org, rillig@NetBSD.org
Subject: Re: pkg/34695
Date: Mon, 9 Oct 2006 08:42:55 +0000
Roland Illig <rillig@NetBSD.org> wrote:
[snip]
Roland, many thanks for testing this. Actually, the original message was
postponed so it has been joined into Gnats after I've found some Solaris
people. Thanks anyway. (Now I'll know whom to ping about Solaris ;-)
> rzdspc8:~: uname -a
> SunOS rzdspc8 5.10 Generic_118833-18 sun4u sparc SUNW,Sun-Fire-V440
I'll ignore this one - pkgsrc relies on nawk.
The code was fixed to avoid systime() and new patch was provied.
--
Mishka.
From: "Mike M. Volokhov" <mishka@NetBSD.org>
To: gnats-bugs@netbsd.org, gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/34695
Date: Sat, 25 Nov 2006 18:34:08 +0200
Module Name: pkgsrc
Committed By: mishka
Date: Sun Oct 8 11:37:39 UTC 2006
Modified Files:
pkgsrc/mk/defaults: mk.conf
pkgsrc/mk/fetch: fetch.mk
Log Message:
pkg/34695: Static list of package master sites may (and often will)
lead to overloads of very first distribution site. Moreover, if first
site in the list is not available (often seen for sourceforge mirrors)
you have to wait for timeout each time. To distribute load on master
distribution sites and to make second problem not so annoying randomly
intermix list of MASTER_SITES with MASTER_SORT_RANDOM feature. Any of
MASTER_SORT and MASTER_SORT_REGEX can be applied later.
The feature is turned ON by default and is disabled for PKG_DEVELOPERs
or if MASTER_SORT_RANDOM=no.
To generate a diff of this commit:
cvs rdiff -r1.132 -r1.133 pkgsrc/mk/defaults/mk.conf
cvs rdiff -r1.21 -r1.22 pkgsrc/mk/fetch/fetch.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: mishka@netbsd.org
State-Changed-When: Sat, 25 Nov 2006 16:50:55 +0000
State-Changed-Why:
Patch has been commited; no objections arrived.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.