NetBSD Problem Report #36298
From cheusov@tut.by Tue May 8 22:54:33 2007
Return-Path: <cheusov@tut.by>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id A1BA963B853
for <gnats-bugs@gnats.netbsd.org>; Tue, 8 May 2007 22:54:33 +0000 (UTC)
Message-Id: <s93hcqnncxn.fsf@chen.chizhovka.net>
Date: Wed, 09 May 2007 01:55:00 +0300
From: cheusov@tut.by
Reply-To:
To: gnats-bugs@NetBSD.org
Subject: improvements of pkgtools/pkg_chk
X-Send-Pr-Version: 3.95
>Number: 36298
>Category: pkg
>Synopsis: improvements for pkgtools/pkg_chk
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: tnn
>State: closed
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Tue May 08 22:55:00 +0000 2007
>Closed-Date: Sun Feb 17 14:03:54 +0000 2008
>Last-Modified: Sun Feb 17 14:03:54 +0000 2008
>Originator: cheusov@tut.by
>Release: NetBSD 4.0_BETA2
>Organization:
Best regards, Aleksey Cheusov.
>Environment:
System: NetBSD chen.chizhovka.net 4.0_BETA2 NetBSD 4.0_BETA2 (GENERIC) #15: Sat Apr 28 23:42:26 EEST 2007 cheusov@chen.chizhovka.net:/srv/src/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:
Fixes:
1) pkg_chk -L /dev/null
or
pkg_chk -L /dev/stderr
or something similar
run with root priviledges may completely break the system
2) check for mktemp result
3) scanning $PACKAGES/All needs too many fork(2) and this may significantly
slow down the pkg_chk especially on platforms with slow fork(2)
such as Interix. Even on NetBSD/i386 that has very fast fork
xargs saves about 15% of the total time.
test: pkg_chk-1.85 -abn on my old system (884 .tbz packages in PACKAGES/All/)
101.00s real 81.27s user 32.00s system
pkg_chk -abn with xargs
86.88s real 69.87s user 18.64s system
4) pkg_info -B for hundreds or thousands packages is extreamly slow operation
(see above). I suggest using cache by setting PKG_BIN_CACHE in mk.conf.
pkg_chk -abn with xargs and cache
4.07s real 3.75s user 2.36s system
>How-To-Repeat:
>Fix:
Index: files/pkg_chk.sh
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/files/pkg_chk.sh,v
retrieving revision 1.48
diff -u -r1.48 pkg_chk.sh
--- files/pkg_chk.sh 20 Apr 2007 09:50:21 -0000 1.48
+++ files/pkg_chk.sh 8 May 2007 22:43:34 -0000
@@ -176,13 +176,13 @@
extract_make_vars Makefile \
AWK GREP GZIP_CMD ID PACKAGES PKGCHK_CONF PKGCHK_NOTAGS \
PKGCHK_TAGS PKGCHK_UPDATE_CONF PKG_ADD PKG_DBDIR PKG_DELETE \
- PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT
+ PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT PKG_BIN_CACHE
if [ -z "$PACKAGES" ];then
PACKAGES=$PKGSRCDIR/packages
fi
elif [ $MAKECONF != /dev/null ] ; then
extract_make_vars $MAKECONF PACKAGES PKGCHK_CONF PKGCHK_UPDATE_CONF \
- PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX
+ PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX PKG_BIN_CACHE
if [ -z "$PACKAGES" ] ; then
PACKAGES=`pwd`
fi
@@ -680,6 +680,9 @@
fi
MY_TMPDIR=`mktemp -d ${TMPDIR-/tmp}/${0##*/}.XXXXXX`
+if [ -z "$MY_TMPDIR" ]; then
+ exit 1
+fi
MY_TMPFILE=$MY_TMPDIR/tmp
# Hide PKG_PATH to avoid breakage in 'make' calls
@@ -726,7 +729,7 @@
fi
if [ -n "$opt_L" ] ; then
- rm -f $opt_L
+ printf '' > $opt_L
fi
basedir=$(pwd)
@@ -741,6 +744,43 @@
PACKAGES="$PACKAGES/All"
fi
+list_bin_pkgs (){
+ ls -t $PACKAGES | grep "$PKG_SUFX_RE"'$' |
+ awk "{print \"$PACKAGES/\" \$0}"
+}
+
+bin_pkg_info_is_up_to_date (){
+ if test -f "$PKG_BIN_CACHE" && \
+ test "$PKG_BIN_CACHE" -nt "$PACKAGES"
+ then
+ return 0
+ else
+ return 1
+ fi
+}
+
+get_bin_pkg_info (){
+ if test -z "$PKG_BIN_CACHE"; then
+ list_bin_pkgs | xargs ${PKG_INFO} -. -B
+ elif bin_pkg_info_is_up_to_date; then
+ cat "$PKG_BIN_CACHE"
+ else
+ trap "rm -f $PKG_BIN_CACHE; exit 1" 0 1 2
+
+ list_bin_pkgs | xargs ${PKG_INFO} -. -B |
+ tee "$PKG_BIN_CACHE"
+
+ trap - 0 1 2
+ fi
+}
+
+bin_pkg_info2pkgdb (){
+ ${SED} -n -e "s|$PKG_SUFX_RE:*\$||" \
+ -e 's|^PKGPATH=||p' \
+ -e 's|^Information.* .*/||p' |
+ ${AWK} '{getline name; printf "%s:%s ", name, $0}'
+}
+
if [ "`${ID} -u`" = 0 ] ; then
SU_CMD=
fi
@@ -775,12 +815,7 @@
*)
if [ -d "$PACKAGES" ] ; then
msg_progress Scan $PACKAGES
- cd $PACKAGES
- for f in `ls -t | grep "$PKG_SUFX_RE"'$'` ; do # Sort by time to pick up newest first
- PKGDIR=`${PKG_INFO} -. -B $PACKAGES/$f|${AWK} -F= '$1=="PKGPATH"{print $2}'`
- PKGNAME=`echo $f | ${SED} "s/$PKG_SUFX"'$//'`
- PKGDB="${PKGDB} $PKGDIR:$PKGNAME"
- done
+ PKGDB=$(get_bin_pkg_info | bin_pkg_info2pkgdb)
PKGSRCDIR=NONE
fi;;
esac
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->abs
Responsible-Changed-By: wiz@netbsd.org
Responsible-Changed-When: Wed, 09 May 2007 18:47:21 +0000
Responsible-Changed-Why:
Over to maintainer.
From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org, pkg-manager@netbsd.org, pkgsrc-bugs@netbsd.org,
gnats-admin@netbsd.org, wiz@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Sun, 20 May 2007 18:32:07 +0300
> Synopsis: improvements for pkgtools/pkg_chk
Some notes about patch.
The code that updates a cache is probably better to implement
in a separate script, i.e. it will be better if pkg_chk provide
_A WAY_ to use external caching program,
not to implement it yourself. The same for -s mode.
--
Best regards, Aleksey Cheusov.
From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org, pkg-manager@netbsd.org, pkgsrc-bugs@netbsd.org,
gnats-admin@netbsd.org, wiz@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Wed, 20 Jun 2007 21:10:41 +0300
--=-=-=
> Synopsis: improvements for pkgtools/pkg_chk
New patch is in attachment.
Cache format is changed to that of pkg_summary.gz.
External tool (wip/pkg_update_summary)
discussed in pkgsrc-users@ (Subject: pkg_summary)
is used for keeping pkg_summary.gz up-to-date
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment; filename=pkg_chk_summary.diff
Content-Description: patch for using/updating pkg_summary.gz
? pkg_chk-1.85.diff
? pkg_chk_summary.diff
Index: Makefile
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/Makefile,v
retrieving revision 1.54
diff -u -r1.54 Makefile
--- Makefile 19 Jun 2007 16:08:47 -0000 1.54
+++ Makefile 20 Jun 2007 18:00:37 -0000
@@ -12,6 +12,8 @@
PKG_INSTALLATION_TYPES= overwrite pkgviews
+DEPENDS+= pkg_update_summary-[0-9]*:../../wip/pkg_update_summary
+
WRKSRC= ${WRKDIR}
EXTRACT_ONLY= # empty
USE_LANGUAGES= # empty
Index: files/pkg_chk.sh
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_chk/files/pkg_chk.sh,v
retrieving revision 1.49
diff -u -r1.49 pkg_chk.sh
--- files/pkg_chk.sh 19 Jun 2007 13:45:50 -0000 1.49
+++ files/pkg_chk.sh 20 Jun 2007 18:00:37 -0000
@@ -176,18 +176,27 @@
extract_make_vars Makefile \
AWK GREP GZIP_CMD ID PACKAGES PKGCHK_CONF PKGCHK_NOTAGS \
PKGCHK_TAGS PKGCHK_UPDATE_CONF PKG_ADD PKG_DBDIR PKG_DELETE \
- PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT
+ PKG_INFO PKG_SUFX SED SORT SU_CMD TSORT PKG_BIN_CACHE
if [ -z "$PACKAGES" ];then
PACKAGES=$PKGSRCDIR/packages
fi
elif [ $MAKECONF != /dev/null ] ; then
extract_make_vars $MAKECONF PACKAGES PKGCHK_CONF PKGCHK_UPDATE_CONF \
- PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX
+ PKGCHK_TAGS PKGCHK_NOTAGS PKG_SUFX PKG_BIN_CACHE
if [ -z "$PACKAGES" ] ; then
PACKAGES=`pwd`
fi
fi
+ # compress and uncompress commands
+ if echo "$PKG_BIN_CACHE" | $GREP '[.]gz$' > /dev/null; then
+ COMPRESS="$GZIP_CMD -c"
+ DECOMPRESS="$GZIP_CMD -dc"
+ else
+ COMPRESS='cat'
+ DECOMPRESS='cat'
+ fi
+
# .tgz/.tbz to regexp
PKG_SUFX_RE=`echo $PKG_SUFX | sed 's/[.]/[.]/'`
@@ -680,6 +689,9 @@
fi
MY_TMPDIR=`mktemp -d ${TMPDIR-/tmp}/${0##*/}.XXXXXX`
+if [ -z "$MY_TMPDIR" ]; then
+ exit 1
+fi
MY_TMPFILE=$MY_TMPDIR/tmp
# Hide PKG_PATH to avoid breakage in 'make' calls
@@ -726,7 +738,7 @@
fi
if [ -n "$opt_L" ] ; then
- rm -f $opt_L
+ printf '' > $opt_L
fi
basedir=$(pwd)
@@ -741,6 +753,40 @@
PACKAGES="$PACKAGES/All"
fi
+list_bin_pkgs (){
+ ls -t $PACKAGES | grep "$PKG_SUFX_RE"'$' |
+ awk "{print \"$PACKAGES/\" \$0}"
+}
+
+bin_pkg_info_is_up_to_date (){
+ if test -f "$PKG_BIN_CACHE" && \
+ test "$PKG_BIN_CACHE" -nt "$PACKAGES"
+ then
+ return 0
+ else
+ return 1
+ fi
+}
+
+get_bin_pkg_info (){
+ if test -z "$PKG_BIN_CACHE"; then
+ list_bin_pkgs | xargs ${PKG_INFO} -X
+ elif bin_pkg_info_is_up_to_date; then
+ $DECOMPRESS "$PKG_BIN_CACHE"
+ else
+ pkg_update_summary "$PKG_BIN_CACHE" "$PACKAGES" \
+ "$DECOMPRESS" "$COMPRESS"
+
+ $DECOMPRESS "$PKG_BIN_CACHE"
+ fi
+}
+
+bin_pkg_info2pkgdb (){
+ ${SED} -n -e 's|^PKGPATH=||p' \
+ -e 's|^PKGNAME=||p' |
+ ${AWK} '{getline name; printf "%s:%s ", name, $0}'
+}
+
if [ "`${ID} -u`" = 0 ] ; then
SU_CMD=
fi
@@ -775,13 +821,7 @@
*)
if [ -d "$PACKAGES" ] ; then
msg_progress Scan $PACKAGES
- cd $PACKAGES
- for f in `ls -t | grep "$PKG_SUFX_RE"'$'` ; do # Sort by time to pick up newest first
- PKGDIR=`${PKG_INFO} -. -B $PACKAGES/$f|${AWK} -F= '$1=="PKGPATH"{print $2}'`
- PKGNAME=`echo $f | ${SED} "s/$PKG_SUFX"'$//'`
- PKGDB="${PKGDB} $PKGDIR:$PKGNAME"
- done
- PKGSRCDIR=NONE
+ PKGDB=$(get_bin_pkg_info | bin_pkg_info2pkgdb)
fi;;
esac
fi
--=-=-=
--
Best regards, Aleksey Cheusov.
--=-=-=--
From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: abs@NetBSD.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/36298 (improvements for pkgtools/pkg_chk)
Date: Sat, 16 Feb 2008 22:24:41 +0200
Is there anybody? I hope yes.
I think items 1) are 3) are obvious and should be applied.
Item 4)
a) xargs - I think this is obvious too
b) using cache - disputable. file:// style of url may be better
but it is not supported
c) updating cache from inside pkg_chk - disputable too.
--
Best regards, Aleksey Cheusov.
Responsible-Changed-From-To: abs->tnn
Responsible-Changed-By: tnn@narn.netbsd.org
Responsible-Changed-When: Sat, 16 Feb 2008 20:39:12 +0000
Responsible-Changed-Why:
I'll deal with items 1-3. Number 4 will need to be discussed further.
State-Changed-From-To: open->feedback
State-Changed-By: tnn@narn.netbsd.org
State-Changed-When: Sun, 17 Feb 2008 00:58:30 +0000
State-Changed-Why:
I've implemented some of the suggested changes, but some abit different
from your patches. My limited testing indicates that it works, but can
you please verify?
State-Changed-From-To: feedback->closed
State-Changed-By: tnn@narn.netbsd.org
State-Changed-When: Sun, 17 Feb 2008 14:03:54 +0000
State-Changed-Why:
Most of the patch went in the tree. The remainder depends on
stuff in pkgsrc-wip and the diff doesn't apply cleanly after
parts of it went in. If the remaining stuff is still of
interest, Please make a new diff and open a new PR or
mailing list thread with some explanation of the cache thing.
Thanks, -T
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.