NetBSD Problem Report #34695

From mishka@netbsd.org  Mon Oct  2 14:37:16 2006
Return-Path: <mishka@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id DD15A63B911
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  2 Oct 2006 14:37:15 +0000 (UTC)
Message-Id: <20061002143715.D198C63B135@mail.netbsd.org>
Date: Mon,  2 Oct 2006 14:37:15 +0000 (UTC)
From: mishka@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: Master sites list randomization
X-Send-Pr-Version: 3.95

>Number:         34695
>Category:       pkg
>Synopsis:       Master sites list randomization
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    mishka
>State:          closed
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Mon Oct 02 14:40:00 +0000 2006
>Closed-Date:    Sat Nov 25 16:50:55 +0000 2006
>Last-Modified:  Sat Nov 25 16:50:55 +0000 2006
>Originator:     Mike M. Volokhov
>Release:        pkgsrc-current as of 20060902
>Organization:
	NONE
>Environment:
No matter
>Description:

	Hello!

	As you may note, static list of package master sites
	accomplished with some known problems. Huge load on top of
	the list sites from pkgsrc users, and died sites (also at
	the top of list) are most annoying.

>How-To-Repeat:

	make fetch

>Fix:

	At this moment you can shuffle master sites with MASTER_SORT
	and MASTER_SORT_REGEX. But you may add some extra randomization
	with just implemented MASTER_SORT_RANDOM=yes|no feature.
	It will randomize sites just before any other ordering, so
	if you altready using some sorting, your sites will be
	reshuffled in each group of your selection.  To randomize
	a whole list use MASTER_SORT_RANDOM=yes only.

	To see randomization in action, just add ``| tee /tmp/sites''
	to _RAND_SITES_CMD variable.

	Also, I'd likely move _MASTER_SITE_BACKUP from unsorted
	list to sorted one, so they all will be used more as backup
	sites.

	The patch:

Index: defaults/mk.conf
===================================================================
RCS file: /cvsroot/pkgsrc/mk/defaults/mk.conf,v
retrieving revision 1.131
diff -u -r1.131 mk.conf
--- defaults/mk.conf	27 Sep 2006 15:18:15 -0000	1.131
+++ defaults/mk.conf	2 Oct 2006 14:14:45 -0000
@@ -397,6 +397,16 @@
 # Possible: Regexps as in awk(1)
 # Default: none

+.if defined(PKG_DEVELOPER)
+MASTER_SORT_RANDOM?=	NO
+.else
+MASTER_SORT_RANDOM?=	YES
+.endif
+# If set to YES or yes, a list of master sites will be randomly intermixed.
+# Also, both MASTER_SORT and MASTER_SORT_REGEX may be applied later.
+# Possible: yes, no / not defined
+# Default: NO if PKG_DEVELOPER is defined, YES otherwise
+
 #PATCH_DEBUG=
 # Used to debug patches as they are applied
 # Possible: defined, not defined
Index: fetch/fetch.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
retrieving revision 1.20
diff -u -r1.20 fetch.mk
--- fetch/fetch.mk	27 Jul 2006 18:48:03 -0000	1.20
+++ fetch/fetch.mk	2 Oct 2006 14:14:45 -0000
@@ -30,10 +30,20 @@
 _BUILD_DEFS+=	_DISTFILES _PATCHFILES

 # Set up _ORDERED_SITES to work out the exact list of sites for every file,
-# using the dynamic sites script, or sorting according to the master site
-# list or the patterns in MASTER_SORT or MASTER_SORT_REGEX as appropriate.
+# using the dynamic sites script, or ordering according to the master site
+# list, MASTER_SORT_RANDOM randomization feature, or the patterns in
+# MASTER_SORT or MASTER_SORT_REGEX as appropriate.
 # No actual sorting is done until _ORDERED_SITES is expanded.
 #
+.if defined(MASTER_SORT_RANDOM) && !empty(MASTER_SORT_RANDOM:M[yY][eE][sS])
+_MASTER_RAND_AWK= { srand(systime()); n = split($$0, site); \
+		    for (i = n; i > 0; i--) { \
+			ir = int(rand() * i + 1); \
+			t = site[i]; site[i] = site[ir]; site[ir] = t; \
+			print site[i]; } }
+_RAND_SITES_CMD= | ${AWK} '${_MASTER_RAND_AWK}'
+.endif
+
 .if defined(MASTER_SORT) || defined(MASTER_SORT_REGEX)
 MASTER_SORT?=
 MASTER_SORT_REGEX?=
@@ -44,9 +54,12 @@
 _MASTER_SORT_AWK+= /${srt:C/\//\\\//g}/ { good["${srt:S/\\/\\\\/g}"] = good["${srt:S/\\/\\\\/g}"] " " $$0 ; next; }
 .  endfor
 _MASTER_SORT_AWK+= { rest = rest " " $$0; } END { n=split(gl, gla); for(i=1;i<=n;i++) { print good[gla[i]]; } print rest; }
+_SORT_SITES_CMD+= | ${AWK} '${_MASTER_SORT_AWK}'
+.endif

-_SORT_SITES_CMD= ${ECHO} $$unsorted_sites | ${AWK} '${_MASTER_SORT_AWK}'
-_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_CMD:S/\\/\\\\/g:C/"/\"/g}`
+.if defined(_RAND_SITES_CMD) || defined(_SORT_SITES_CMD)
+_SORT_SITES_FULL_CMD= ${ECHO} $$unsorted_sites ${_RAND_SITES_CMD} ${_SORT_SITES_CMD}
+_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_FULL_CMD:S/\\/\\\\/g:C/"/\"/g}`
 .else
 _ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} $$unsorted_sites
 .endif
@@ -245,8 +258,8 @@
 		fi;							\
 	done
 	${_PKG_SILENT}${_PKG_DEBUG}set -e;				\
-	unsorted_sites="${SITES.${.TARGET:T:S/=/--/}} ${_MASTER_SITE_BACKUP}"; \
-	sites="${_ORDERED_SITES}";					\
+	unsorted_sites="${SITES.${.TARGET:T:S/=/--/}}";			\
+	sites="${_ORDERED_SITES} ${_MASTER_SITE_BACKUP}";		\
 	cd ${.TARGET:H:S/\/${DIST_SUBDIR}$//} &&			\
 	${_FETCH_CMD} ${_FETCH_ARGS} ${.TARGET:T} $$sites
 	${_PKG_SILENT}${_PKG_DEBUG}					\

>Release-Note:

>Audit-Trail:
From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:25:35 +0000

 The awk code used in origianl patch is somewhat NetBSD dependent - other
 awk implementations may missing some features, like systime(). So, taking
 Solaris' nawk as a lowest denominator, please use the patch below:


 Index: fetch.mk
 ===================================================================
 RCS file: /cvsroot/pkgsrc/mk/fetch/fetch.mk,v
 retrieving revision 1.20
 diff -u -r1.20 fetch.mk
 --- fetch.mk	27 Jul 2006 18:48:03 -0000	1.20
 +++ fetch.mk	5 Oct 2006 15:05:54 -0000
 @@ -30,10 +30,21 @@
  _BUILD_DEFS+=	_DISTFILES _PATCHFILES

  # Set up _ORDERED_SITES to work out the exact list of sites for every file,
 -# using the dynamic sites script, or sorting according to the master site
 -# list or the patterns in MASTER_SORT or MASTER_SORT_REGEX as appropriate.
 +# using the dynamic sites script, or ordering according to the master site
 +# list, MASTER_SORT_RANDOM randomization feature, or the patterns in
 +# MASTER_SORT or MASTER_SORT_REGEX as appropriate.
  # No actual sorting is done until _ORDERED_SITES is expanded.
  #
 +.if defined(MASTER_SORT_RANDOM) && !empty(MASTER_SORT_RANDOM:M[yY][eE][sS])
 +_MASTER_RAND_AWK= BEGIN { srand(seed) } {				\
 +		n = split($$0, site);					\
 +		for (i = n; i > 0; i--) {				\
 +			ir = int(rand() * i + 1);			\
 +			t = site[i]; site[i] = site[ir]; site[ir] = t;	\
 +			print site[i]; } }
 +_RAND_SITES_CMD= | ${AWK} -v seed=$$$$ '${_MASTER_RAND_AWK}' | tee /tmp/sites
 +.endif
 +
  .if defined(MASTER_SORT) || defined(MASTER_SORT_REGEX)
  MASTER_SORT?=
  MASTER_SORT_REGEX?=
 @@ -44,9 +55,12 @@
  _MASTER_SORT_AWK+= /${srt:C/\//\\\//g}/ { good["${srt:S/\\/\\\\/g}"] = good["${srt:S/\\/\\\\/g}"] " " $$0 ; next; }
  .  endfor
  _MASTER_SORT_AWK+= { rest = rest " " $$0; } END { n=split(gl, gla); for(i=1;i<=n;i++) { print good[gla[i]]; } print rest; }
 +_SORT_SITES_CMD+= | ${AWK} '${_MASTER_SORT_AWK}'
 +.endif

 -_SORT_SITES_CMD= ${ECHO} $$unsorted_sites | ${AWK} '${_MASTER_SORT_AWK}'
 -_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_CMD:S/\\/\\\\/g:C/"/\"/g}`
 +.if defined(_RAND_SITES_CMD) || defined(_SORT_SITES_CMD)
 +_SORT_SITES_FULL_CMD= ${ECHO} $$unsorted_sites ${_RAND_SITES_CMD} ${_SORT_SITES_CMD}
 +_ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} `${_SORT_SITES_FULL_CMD:S/\\/\\\\/g:C/"/\"/g}`
  .else
  _ORDERED_SITES= ${_MASTER_SITE_OVERRIDE} $$unsorted_sites
  .endif
 @@ -245,8 +259,8 @@
  		fi;							\
  	done
  	${_PKG_SILENT}${_PKG_DEBUG}set -e;				\
 -	unsorted_sites="${SITES.${.TARGET:T:S/=/--/}} ${_MASTER_SITE_BACKUP}"; \
 -	sites="${_ORDERED_SITES}";					\
 +	unsorted_sites="${SITES.${.TARGET:T:S/=/--/}}";			\
 +	sites="${_ORDERED_SITES} ${_MASTER_SITE_BACKUP}";		\
  	cd ${.TARGET:H:S/\/${DIST_SUBDIR}$//} &&			\
  	${_FETCH_CMD} ${_FETCH_ARGS} ${.TARGET:T} $$sites
  	${_PKG_SILENT}${_PKG_DEBUG}					\


 The defaults/mk.conf patch still with no changes.

 --
 Mishka.

 P.S. Thanks to salo for warning me on this problem.

From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:38:49 +0000

 Gah! In the latest patch _RAND_SITES_CMD variable must be without
 'tee' ath the end, of course:

 	_RAND_SITES_CMD= | ${AWK} -v seed=$$$$ '${_MASTER_RAND_AWK}'

 --
 Mishka.

From: "Mike M. Volokhov" <mishka@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34695
Date: Thu, 5 Oct 2006 15:16:44 +0300

 The single objection I've seen is that awk-based randomization logic
 might not work with non NetBSD awks (for example, on Solaris). So, can
 anybody confirm that the following script is workable on non NetBSD
 platform?

 {
         srand(systime());
         n = split($0, words);
         for (i = n; i > 0; i--) {
                 ir = int(rand() * i + 1);
                 t = words[i];
                 words[i] = words[ir];
                 words[ir] = t;
                 print words[i];
         }
 }

 Usage:
 echo "one two three four five six seven eight nine ten" | awk -f random.awk

 --
 Mishka.

From: Roland Illig <rillig@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34695
Date: Fri, 06 Oct 2006 21:16:23 +0200

 Mike M. Volokhov wrote:
 > The following reply was made to PR pkg/34695; it has been noted by GNATS.
 > 
 > From: "Mike M. Volokhov" <mishka@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: pkg/34695
 > Date: Thu, 5 Oct 2006 15:16:44 +0300
 > 
 >  The single objection I've seen is that awk-based randomization logic
 >  might not work with non NetBSD awks (for example, on Solaris). So, can
 >  anybody confirm that the following script is workable on non NetBSD
 >  platform?
 >  
 >  {
 >          srand(systime());
 >          n = split($0, words);
 >          for (i = n; i > 0; i--) {
 >                  ir = int(rand() * i + 1);
 >                  t = words[i];
 >                  words[i] = words[ir];
 >                  words[ir] = t;
 >                  print words[i];
 >          }
 >  }
 >  
 >  Usage:
 >  echo "one two three four five six seven eight nine ten" | awk -f random.awk

 rzdspc8:~: uname -a
 SunOS rzdspc8 5.10 Generic_118833-18 sun4u sparc SUNW,Sun-Fire-V440

 rzdspc8:~: alias e="echo \"one two three four five six seven eight nine 
 ten\""

 rzdspc8:~: e | /usr/bin/awk -f r.awk
 awk: syntax error near line 2
 awk: illegal statement near line 2
 awk: syntax error near line 5
 awk: illegal statement near line 5

 rzdspc8:~: e | /usr/pkg/2006Q2/bin/nawk -f r.awk
 /usr/pkg/2006Q2/bin/nawk: calling undefined function systime
   input record number 1, file
   source line number 2

 rzdspc8:~: e | /usr/bin/nawk -f r.awk
 /usr/bin/nawk: calling undefined function systime
   input record number 1
   source line number 2

 rzdspc8:~: e | /usr/pkg/2006Q2/bin/gawk -f r.awk
 seven
 one
 four
 five
 two
 three
 eight
 nine
 six
 ten

 rzdspc8:~: e | /opt/sfw/bin/gawk -f r.awk
 nine
 four
 five
 ten
 three
 six
 two
 seven
 eight
 one

 ^D

 Roland

Responsible-Changed-From-To: pkg-manager->mishka
Responsible-Changed-By: mishka@netbsd.org
Responsible-Changed-When: Mon, 09 Oct 2006 08:35:53 +0000
Responsible-Changed-Why:
I'll take it.


From: "Mike M. Volokhov" <mishka@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org,
	pkgsrc-bugs@netbsd.org, rillig@NetBSD.org
Subject: Re: pkg/34695
Date: Mon, 9 Oct 2006 08:42:55 +0000

 Roland Illig <rillig@NetBSD.org> wrote:
 [snip]

 Roland, many thanks for testing this. Actually, the original message was
 postponed so it has been joined into Gnats after I've found some Solaris
 people. Thanks anyway. (Now I'll know whom to ping about Solaris ;-)

 >  rzdspc8:~: uname -a
 >  SunOS rzdspc8 5.10 Generic_118833-18 sun4u sparc SUNW,Sun-Fire-V440

 I'll ignore this one - pkgsrc relies on nawk.

 The code was fixed to avoid systime() and new patch was provied.

 --
 Mishka.

From: "Mike M. Volokhov" <mishka@NetBSD.org>
To: gnats-bugs@netbsd.org, gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34695
Date: Sat, 25 Nov 2006 18:34:08 +0200

 Module Name:    pkgsrc
 Committed By:   mishka
 Date:           Sun Oct  8 11:37:39 UTC 2006

 Modified Files:
         pkgsrc/mk/defaults: mk.conf
         pkgsrc/mk/fetch: fetch.mk

 Log Message:
 pkg/34695: Static list of package master sites may (and often will)
 lead to overloads of very first distribution site. Moreover, if first
 site in the list is not available (often seen for sourceforge mirrors)
 you have to wait for timeout each time. To distribute load on master
 distribution sites and to make second problem not so annoying randomly
 intermix list of MASTER_SITES with MASTER_SORT_RANDOM feature. Any of
 MASTER_SORT and MASTER_SORT_REGEX can be applied later.

 The feature is turned ON by default and is disabled for PKG_DEVELOPERs
 or if MASTER_SORT_RANDOM=no.


 To generate a diff of this commit:
 cvs rdiff -r1.132 -r1.133 pkgsrc/mk/defaults/mk.conf
 cvs rdiff -r1.21 -r1.22 pkgsrc/mk/fetch/fetch.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: mishka@netbsd.org
State-Changed-When: Sat, 25 Nov 2006 16:50:55 +0000
State-Changed-Why:
Patch has been commited; no objections arrived.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.