NetBSD Problem Report #36298

From cheusov@tut.by  Tue May  8 22:54:33 2007
Return-Path: <cheusov@tut.by>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id A1BA963B853
	for <gnats-bugs@gnats.netbsd.org>; Tue,  8 May 2007 22:54:33 +0000 (UTC)
Message-Id: <s93hcqnncxn.fsf@chen.chizhovka.net>
Date: Wed, 09 May 2007 01:55:00 +0300
From: cheusov@tut.by
Reply-To:
To: gnats-bugs@NetBSD.org
Subject: improvements of pkgtools/pkg_chk
X-Send-Pr-Version: 3.95

>Number:         36298
>Category:       pkg
>Synopsis:       improvements for pkgtools/pkg_chk
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    tnn
>State:          closed
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Tue May 08 22:55:00 +0000 2007
>Closed-Date:    Sun Feb 17 14:03:54 +0000 2008
>Last-Modified:  Sun Feb 17 14:03:54 +0000 2008
>Originator:     cheusov@tut.by
>Release:        NetBSD 4.0_BETA2
>Organization:
Best regards, Aleksey Cheusov.
>Environment:
System: NetBSD chen.chizhovka.net 4.0_BETA2 NetBSD 4.0_BETA2 (GENERIC) #15: Sat Apr 28 23:42:26 EEST 2007 cheusov@chen.chizhovka.net:/srv/src/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:
Fixes:
1) pkg_chk -L /dev/null
   or
   pkg_chk -L /dev/stderr
   or something similar
   run with root priviledges may completely break the system
2) check for mktemp result
3) scanning $PACKAGES/All needs too many fork(2) and this may significantly
   slow down the pkg_chk especially on platforms with slow fork(2)
   such as Interix. Even on NetBSD/i386 that has very fast fork
   xargs saves about 15% of the total time.

  test: pkg_chk-1.85 -abn on my old system (884 .tbz packages in PACKAGES/All/)
  101.00s real    81.27s user    32.00s system

  pkg_chk -abn with xargs
   86.88s real    69.87s user    18.64s system

4) pkg_info -B for hundreds or thousands packages is extreamly slow operation
   (see above). I suggest using cache by setting PKG_BIN_CACHE in mk.conf.

   pkg_chk -abn with xargs and cache 
    4.07s real     3.75s user     2.36s system

>How-To-Repeat:

>Fix:

Index: files/pkg_chk.sh
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/files/pkg_chk.sh,v
retrieving revision 1.48
diff -u -r1.48 pkg_chk.sh
--- files/pkg_chk.sh	20 Apr 2007 09:50:21 -0000	1.48
+++ files/pkg_chk.sh	8 May 2007 22:43:34 -0000
@@ -176,13 +176,13 @@
 	extract_make_vars Makefile \
 		AWK GREP GZIP_CMD ID PACKAGES PKGCHK_CONF PKGCHK_NOTAGS \
 		PKGCHK_TAGS PKGCHK_UPDATE_CONF PKG_ADD PKG_DBDIR PKG_DELETE \
-		PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT
+		PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT PKG_BIN_CACHE
 	if [ -z "$PACKAGES" ];then
 	    PACKAGES=$PKGSRCDIR/packages
 	fi
     elif [ $MAKECONF != /dev/null ] ; then
 	extract_make_vars $MAKECONF PACKAGES PKGCHK_CONF PKGCHK_UPDATE_CONF \
-			PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX
+			PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX PKG_BIN_CACHE
 	if [ -z "$PACKAGES" ] ; then
 	    PACKAGES=`pwd`
 	fi
@@ -680,6 +680,9 @@
 fi

 MY_TMPDIR=`mktemp -d ${TMPDIR-/tmp}/${0##*/}.XXXXXX`
+if [ -z "$MY_TMPDIR" ]; then
+    exit 1
+fi
 MY_TMPFILE=$MY_TMPDIR/tmp

 # Hide PKG_PATH to avoid breakage in 'make' calls
@@ -726,7 +729,7 @@
 fi

 if [ -n "$opt_L" ] ; then
-    rm -f $opt_L
+    printf '' > $opt_L
 fi

 basedir=$(pwd)
@@ -741,6 +744,43 @@
     PACKAGES="$PACKAGES/All"
 fi

+list_bin_pkgs (){
+    ls -t $PACKAGES | grep "$PKG_SUFX_RE"'$' |
+    awk "{print \"$PACKAGES/\" \$0}"
+}
+
+bin_pkg_info_is_up_to_date (){
+    if test -f "$PKG_BIN_CACHE" && \
+	test "$PKG_BIN_CACHE" -nt "$PACKAGES"
+    then
+	return 0
+    else
+	return 1
+    fi
+}
+
+get_bin_pkg_info (){
+    if test -z "$PKG_BIN_CACHE"; then
+	list_bin_pkgs | xargs ${PKG_INFO} -. -B
+    elif bin_pkg_info_is_up_to_date; then
+	cat "$PKG_BIN_CACHE"
+    else
+	trap "rm -f $PKG_BIN_CACHE; exit 1" 0 1 2
+
+	list_bin_pkgs | xargs ${PKG_INFO} -. -B |
+	tee "$PKG_BIN_CACHE"
+
+	trap - 0 1 2
+    fi
+}
+
+bin_pkg_info2pkgdb (){
+    ${SED} -n -e "s|$PKG_SUFX_RE:*\$||" \
+	      -e 's|^PKGPATH=||p' \
+	      -e 's|^Information.* .*/||p' |
+    ${AWK} '{getline name; printf "%s:%s ", name, $0}'
+}
+
 if [ "`${ID} -u`" = 0 ] ; then
     SU_CMD=
 fi
@@ -775,12 +815,7 @@
 	*)
 	    if [ -d "$PACKAGES" ] ; then
 		msg_progress Scan $PACKAGES
-		cd $PACKAGES
-		for f in `ls -t | grep "$PKG_SUFX_RE"'$'` ; do # Sort by time to pick up newest first
-		    PKGDIR=`${PKG_INFO} -. -B $PACKAGES/$f|${AWK} -F= '$1=="PKGPATH"{print $2}'`
-		    PKGNAME=`echo $f | ${SED} "s/$PKG_SUFX"'$//'`
-		    PKGDB="${PKGDB} $PKGDIR:$PKGNAME"
-		done
+		PKGDB=$(get_bin_pkg_info | bin_pkg_info2pkgdb)
 		PKGSRCDIR=NONE
 	    fi;;
     esac

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: pkg-manager->abs
Responsible-Changed-By: wiz@netbsd.org
Responsible-Changed-When: Wed, 09 May 2007 18:47:21 +0000
Responsible-Changed-Why:
Over to maintainer.


From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org, pkg-manager@netbsd.org, pkgsrc-bugs@netbsd.org,
	gnats-admin@netbsd.org, wiz@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Sun, 20 May 2007 18:32:07 +0300

 > Synopsis: improvements for pkgtools/pkg_chk

 Some notes about patch.
 The code that updates a cache is probably better to implement
 in a separate script, i.e. it will be better if pkg_chk provide
 _A WAY_ to use external caching program,
 not to implement it yourself. The same for -s mode.

 -- 
 Best regards, Aleksey Cheusov.

From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org, pkg-manager@netbsd.org, pkgsrc-bugs@netbsd.org,
	gnats-admin@netbsd.org, wiz@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Wed, 20 Jun 2007 21:10:41 +0300

 --=-=-=

 > Synopsis: improvements for pkgtools/pkg_chk

 New patch is in attachment.

 Cache format is changed to that of pkg_summary.gz.
 External tool (wip/pkg_update_summary)
 discussed in pkgsrc-users@ (Subject: pkg_summary)
 is used for keeping pkg_summary.gz up-to-date


 --=-=-=
 Content-Type: text/x-patch
 Content-Disposition: attachment; filename=pkg_chk_summary.diff
 Content-Description: patch for using/updating pkg_summary.gz

 ? pkg_chk-1.85.diff
 ? pkg_chk_summary.diff
 Index: Makefile
 ===================================================================
 RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/Makefile,v
 retrieving revision 1.54
 diff -u -r1.54 Makefile
 --- Makefile	19 Jun 2007 16:08:47 -0000	1.54
 +++ Makefile	20 Jun 2007 18:00:37 -0000
 @@ -12,6 +12,8 @@

  PKG_INSTALLATION_TYPES=	overwrite pkgviews

 +DEPENDS+=	pkg_update_summary-[0-9]*:../../wip/pkg_update_summary
 +
  WRKSRC=		${WRKDIR}
  EXTRACT_ONLY=	# empty
  USE_LANGUAGES=	# empty
 Index: files/pkg_chk.sh
 ===================================================================
 RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/files/pkg_chk.sh,v
 retrieving revision 1.49
 diff -u -r1.49 pkg_chk.sh
 --- files/pkg_chk.sh	19 Jun 2007 13:45:50 -0000	1.49
 +++ files/pkg_chk.sh	20 Jun 2007 18:00:37 -0000
 @@ -176,18 +176,27 @@
  	extract_make_vars Makefile \
  		AWK GREP GZIP_CMD ID PACKAGES PKGCHK_CONF PKGCHK_NOTAGS \
  		PKGCHK_TAGS PKGCHK_UPDATE_CONF PKG_ADD PKG_DBDIR PKG_DELETE \
 -		PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT
 +		PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT PKG_BIN_CACHE
  	if [ -z "$PACKAGES" ];then
  	    PACKAGES=$PKGSRCDIR/packages
  	fi
      elif [ $MAKECONF != /dev/null ] ; then
  	extract_make_vars $MAKECONF PACKAGES PKGCHK_CONF PKGCHK_UPDATE_CONF \
 -			PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX
 +			PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX PKG_BIN_CACHE
  	if [ -z "$PACKAGES" ] ; then
  	    PACKAGES=`pwd`
  	fi
      fi

 +    # compress and uncompress commands
 +    if echo "$PKG_BIN_CACHE" | $GREP '[.]gz$' > /dev/null; then
 +	COMPRESS="$GZIP_CMD -c"
 +	DECOMPRESS="$GZIP_CMD -dc"
 +    else
 +	COMPRESS='cat'
 +	DECOMPRESS='cat'
 +    fi
 +
      # .tgz/.tbz to regexp
      PKG_SUFX_RE=`echo $PKG_SUFX | sed 's/[.]/[.]/'`

 @@ -680,6 +689,9 @@
  fi

  MY_TMPDIR=`mktemp -d ${TMPDIR-/tmp}/${0##*/}.XXXXXX`
 +if [ -z "$MY_TMPDIR" ]; then
 +    exit 1
 +fi
  MY_TMPFILE=$MY_TMPDIR/tmp

  # Hide PKG_PATH to avoid breakage in 'make' calls
 @@ -726,7 +738,7 @@
  fi

  if [ -n "$opt_L" ] ; then
 -    rm -f $opt_L
 +    printf '' > $opt_L
  fi

  basedir=$(pwd)
 @@ -741,6 +753,40 @@
      PACKAGES="$PACKAGES/All"
  fi

 +list_bin_pkgs (){
 +    ls -t $PACKAGES | grep "$PKG_SUFX_RE"'$' |
 +    awk "{print \"$PACKAGES/\" \$0}"
 +}
 +
 +bin_pkg_info_is_up_to_date (){
 +    if test -f "$PKG_BIN_CACHE" && \
 +	test "$PKG_BIN_CACHE" -nt "$PACKAGES"
 +    then
 +	return 0
 +    else
 +	return 1
 +    fi
 +}
 +
 +get_bin_pkg_info (){
 +    if test -z "$PKG_BIN_CACHE"; then
 +	list_bin_pkgs | xargs ${PKG_INFO} -X
 +    elif bin_pkg_info_is_up_to_date; then
 +	$DECOMPRESS "$PKG_BIN_CACHE"
 +    else
 +	pkg_update_summary "$PKG_BIN_CACHE" "$PACKAGES" \
 +	    "$DECOMPRESS" "$COMPRESS"
 +
 +	$DECOMPRESS "$PKG_BIN_CACHE"
 +    fi
 +}
 +
 +bin_pkg_info2pkgdb (){
 +    ${SED} -n -e 's|^PKGPATH=||p' \
 +	      -e 's|^PKGNAME=||p' |
 +    ${AWK} '{getline name; printf "%s:%s ", name, $0}'
 +}
 +
  if [ "`${ID} -u`" = 0 ] ; then
      SU_CMD=
  fi
 @@ -775,13 +821,7 @@
  	*)
  	    if [ -d "$PACKAGES" ] ; then
  		msg_progress Scan $PACKAGES
 -		cd $PACKAGES
 -		for f in `ls -t | grep "$PKG_SUFX_RE"'$'` ; do # Sort by time to pick up newest first
 -		    PKGDIR=`${PKG_INFO} -. -B $PACKAGES/$f|${AWK} -F= '$1=="PKGPATH"{print $2}'`
 -		    PKGNAME=`echo $f | ${SED} "s/$PKG_SUFX"'$//'`
 -		    PKGDB="${PKGDB} $PKGDIR:$PKGNAME"
 -		done
 -		PKGSRCDIR=NONE
 +		PKGDB=$(get_bin_pkg_info | bin_pkg_info2pkgdb)
  	    fi;;
      esac
  fi

 --=-=-=


 -- 
 Best regards, Aleksey Cheusov.

 --=-=-=--

From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org,  gnats-admin@netbsd.org,  pkgsrc-bugs@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Sat, 16 Feb 2008 22:24:41 +0200

 Is there anybody? I hope yes.

 I think items 1) are 3) are obvious and should be applied.
 Item 4)
 a) xargs - I think this is obvious too
 b) using cache - disputable. file:// style of url may be better
    but it is not supported
 c) updating cache from inside pkg_chk - disputable too.

 -- 
 Best regards, Aleksey Cheusov.

Responsible-Changed-From-To: abs->tnn
Responsible-Changed-By: tnn@narn.netbsd.org
Responsible-Changed-When: Sat, 16 Feb 2008 20:39:12 +0000
Responsible-Changed-Why:
I'll deal with items 1-3. Number 4 will need to be discussed further.


State-Changed-From-To: open->feedback
State-Changed-By: tnn@narn.netbsd.org
State-Changed-When: Sun, 17 Feb 2008 00:58:30 +0000
State-Changed-Why:
I've implemented some of the suggested changes, but some abit different
from your patches. My limited testing indicates that it works, but can
you please verify?


State-Changed-From-To: feedback->closed
State-Changed-By: tnn@narn.netbsd.org
State-Changed-When: Sun, 17 Feb 2008 14:03:54 +0000
State-Changed-Why:
Most of the patch went in the tree. The remainder depends on
stuff in pkgsrc-wip and the diff doesn't apply cleanly after
parts of it went in. If the remaining stuff is still of
interest, Please make a new diff and open a new PR or
mailing list thread with some explanation of the cache thing.
Thanks, -T


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.