NetBSD Problem Report #49117
From www@NetBSD.org Fri Aug 15 09:20:38 2014
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 06AE8AD52A
for <gnats-bugs@gnats.NetBSD.org>; Fri, 15 Aug 2014 09:20:38 +0000 (UTC)
Message-Id: <20140815092035.E6E78AD5DC@mollari.NetBSD.org>
Date: Fri, 15 Aug 2014 09:20:33 +0000 (UTC)
From: rchibois@gmail.com
Reply-To: rchibois@gmail.com
To: gnats-bugs@NetBSD.org
Subject: new pkg_info file list option
X-Send-Pr-Version: www-1.0
>Number: 49117
>Category: pkg
>Synopsis: new pkg_info file list option
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: pkg-manager
>State: closed
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Fri Aug 15 09:25:00 +0000 2014
>Closed-Date: Wed Mar 02 08:43:13 +0000 2016
>Last-Modified: Wed Mar 02 08:43:13 +0000 2016
>Originator: Rémy Chibois
>Release: 2013Q4
>Organization:
>Environment:
Darwin macbook 14.0.0 Darwin Kernel Version 14.0.0: Wed Jul 16 00:46:31 PDT 2014; root:xnu-2782.1.43.0.2~2/RELEASE_X86_64 x86_64
>Description:
There is currently no way to search for non-installed packages containing a file. A simple (compressed) file in the spirit of pkg_summary.gz and stored in binary package repositories would allow this. This file is a collection of lines like:
PACKAGE_NAME: /path/to/file
The suggested new pkg_info option, -Y, would be used like:
pkg_info -qY *.tgz | gzip -9 > pkg_files.gz
I already contacted pkgin's author and proposed a new 'search-file' command implementation with cached download of a per-repository pkg_files.gz
(please see https://github.com/chybz/pkgin for details).
Please find below a patch against pkg_install that implements this option.
A working implementation is available in this pkgsrc tree:
https://github.com/yrmt/pkgsrc
Thank you.
>How-To-Repeat:
non relevant -- change request
>Fix:
diff --git a/pkgtools/pkg_install/files/info/info.h b/pkgtools/pkg_install/files/info/info.h
index f8b6b4d..6912ba9 100644
--- a/pkgtools/pkg_install/files/info/info.h
+++ b/pkgtools/pkg_install/files/info/info.h
@@ -101,6 +101,7 @@ struct pkg_meta {
#define SHOW_BI_VAR 0x40000
#define SHOW_SUMMARY 0x80000
#define SHOW_FULL_REQBY 0x100000
+#define SHOW_PKG_FILES 0x200000
enum which {
WHICH_ALL,
@@ -123,6 +124,7 @@ void show_file(const char *, const char *, Boolean);
void show_var(const char *, const char *);
void show_plist(const char *, package_t *, pl_ent_t);
void show_files(const char *, package_t *);
+void show_pkg_files(const char *, package_t *);
void show_depends(const char *, package_t *);
void show_bld_depends(const char *, package_t *);
void show_index(const char *, const char *);
diff --git a/pkgtools/pkg_install/files/info/main.c b/pkgtools/pkg_install/files/info/main.c
index 82b0e1d..16a7c38 100644
--- a/pkgtools/pkg_install/files/info/main.c
+++ b/pkgtools/pkg_install/files/info/main.c
@@ -41,7 +41,7 @@ __RCSID("$NetBSD: main.c,v 1.31 2012/12/17 04:34:02 agc Exp $");
#include "lib.h"
#include "info.h"
-static const char Options[] = ".aBbcDde:E:fFhIiK:kLl:mNnpQ:qrRsSuvVX";
+static const char Options[] = ".aBbcDde:E:fFhIiK:kLl:mNnpQ:qrRsSuvVXY";
int Flags = 0;
enum which Which = WHICH_LIST;
@@ -55,7 +55,7 @@ static void
usage(void)
{
fprintf(stderr, "%s\n%s\n%s\n%s\n",
- "usage: pkg_info [-BbcDdFfhIikLmNnpqRrSsVvX] [-E pkg-name] [-e pkg-name]",
+ "usage: pkg_info [-BbcDdFfhIikLmNnpqRrSsVvXY] [-E pkg-name] [-e pkg-name]",
" [-K pkg_dbdir] [-l prefix] pkg-name ...",
" pkg_info [-a | -u] [flags]",
" pkg_info [-Q variable] pkg-name ...");
@@ -202,6 +202,10 @@ main(int argc, char **argv)
Flags |= SHOW_SUMMARY;
break;
+ case 'Y':
+ Flags |= SHOW_PKG_FILES;
+ break;
+
case 'h':
case '?':
default:
diff --git a/pkgtools/pkg_install/files/info/perform.c b/pkgtools/pkg_install/files/info/perform.c
index 8278158..f68afee 100644
--- a/pkgtools/pkg_install/files/info/perform.c
+++ b/pkgtools/pkg_install/files/info/perform.c
@@ -473,6 +473,9 @@ pkg_do(const char *pkg)
if (Flags & SHOW_FILES) {
show_files("Files:\n", &plist);
}
+ if (Flags & SHOW_PKG_FILES) {
+ show_pkg_files("Files:\n", &plist);
+ }
if ((Flags & SHOW_BUILD_VERSION) && meta->meta_build_version) {
show_file(meta->meta_build_version, "Build version:\n",
TRUE);
diff --git a/pkgtools/pkg_install/files/info/pkg_info.1 b/pkgtools/pkg_install/files/info/pkg_info.1
index 16f5fd8..f094432 100644
--- a/pkgtools/pkg_install/files/info/pkg_info.1
+++ b/pkgtools/pkg_install/files/info/pkg_info.1
@@ -25,7 +25,7 @@
.Nd a utility for displaying information on software packages
.Sh SYNOPSIS
.Nm
-.Op Fl BbcDdFfhIikLmNnpqRrSsVvX
+.Op Fl BbcDdFfhIikLmNnpqRrSsVvXY
.Op Fl E Ar pkg-name
.Op Fl e Ar pkg-name
.Op Fl K Ar pkg_dbdir
@@ -210,6 +210,12 @@ described in
.Xr pkg_summary 5 .
Its primary use is to contain all information about the contents of a
(remote) binary package repository needed by package managing software.
+.It Fl Y
+Similar as
+.Fl L .
+, but prefixes each line with package name.
+Its primary use is to provide a global searchable file list of a
+(remote) binary package repository needed by package managing software.
.El
.Sh TECHNICAL DETAILS
Package info is either extracted from package files named on the
diff --git a/pkgtools/pkg_install/files/info/show.c b/pkgtools/pkg_install/files/info/show.c
index daf3bdb..accb6d3 100644
--- a/pkgtools/pkg_install/files/info/show.c
+++ b/pkgtools/pkg_install/files/info/show.c
@@ -247,6 +247,48 @@ show_files(const char *title, package_t *plist)
}
/*
+ * Show all files in the packing list (except ignored ones) for pkg_files.txt
+ *
+ */
+void
+show_pkg_files(const char *title, package_t *plist)
+{
+ plist_t *p;
+ Boolean ign;
+ const char *dir = ".";
+ const char *pkgname = "PKG";
+
+ if (!Quiet) {
+ printf("%s%s", InfoPrefix, title);
+ }
+ for (ign = FALSE, p = plist->head; p; p = p->next) {
+ if (p->type == PLIST_NAME) {
+ pkgname = p->name;
+ break;
+ }
+ }
+ for (ign = FALSE, p = plist->head; p; p = p->next) {
+ switch (p->type) {
+ case PLIST_FILE:
+ if (!ign) {
+ printf("%s: %s%s%s\n", pkgname, dir,
+ (strcmp(dir, "/") == 0) ? "" : "/", p->name);
+ }
+ ign = FALSE;
+ break;
+ case PLIST_CWD:
+ dir = p->name;
+ break;
+ case PLIST_IGNORE:
+ ign = TRUE;
+ break;
+ default:
+ break;
+ }
+ }
+}
+
+/*
* Show dependencies (packages this pkg requires)
*/
void
>Release-Note:
>Audit-Trail:
From: Aleksey Cheusov <vle@gmx.net>
To: gnats-bugs@netbsd.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/49117: new pkg_info file list option
Date: Fri, 15 Aug 2014 20:08:55 +0300
--001a11c12b5a85ba850500ae13ab
Content-Type: text/plain; charset=UTF-8
>
> There is currently no way to search for non-installed packages containing
> a file. A simple (compressed) file in the spirit of pkg_summary.gz and
> stored in binary package repositories would allow this.
JFYI: pkgtools/nih, a package manager for pkgsrc, provides a functionality
you need.
First, it is able to scan installed package by filename (not only exact
match!):
0 cheusov>nih search -i f:s:bin/nih
pkgtools/nih - Package manager for pkgsrc
0 cheusov>
Second, one can also search for packages in pkgsrc tree via pkg_online
service (dict://dict.mova.org:26280)
updated on regular basis. It is very fast.
0 cheusov>nih search -o f:s:bin/nih
pkgtools/nih - Package manager for pkgsrc
wip/nih-current(nih) - Package manager for pkgsrc
0 cheusov>
If you don't trust remote services, you can scan pkgsrc tree locally on
your own machine.
nih refresh -p # scanning
nih search -p f:s:bin/nih
The very first scan takes hours of work.
Usually, subsequent ones work for dozens of second or just a few minutes.
If pkg_summary.gz file contained PLIST entries, you can also search for
package by filename in this file.
nih search -b f:s:bin/nih
f:s: above means "Look for packages by Filename with Substring search
strategy".
For more information, see
http://mova.org/~cheusov/pub/mlug/2012-11/package-search.pdf
--001a11c12b5a85ba850500ae13ab
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex">
There is currently no way to search for non-installed packages containing a=
file. A simple (compressed) file in the spirit of pkg_summary.gz and store=
d in binary package repositories would allow this. </blockquote><br></div>
JFYI: pkgtools/nih, a package manager for pkgsrc, provides a functionality =
you need.<br></div><div class=3D"gmail_extra"><br>First, it is able to scan=
installed package by filename (not only exact match!):<br><br>=C2=A0=C2=A0=
=C2=A0 0 cheusov>nih search -i f:s:bin/nih<br>
=C2=A0=C2=A0=C2=A0 pkgtools/nih=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Package manager for pkgsrc<br>=C2=A0=
=C2=A0=C2=A0 0 cheusov><br><br></div><div class=3D"gmail_extra">Second, =
one can also search for packages in pkgsrc tree via pkg_online service (dic=
t://<a href=3D"http://dict.mova.org:26280">dict.mova.org:26280</a>)<br>
</div><div class=3D"gmail_extra">updated on regular basis. It is very fast.=
<br><br>=C2=A0=C2=A0=C2=A0 0 cheusov>nih search -o f:s:bin/nih<br>=C2=A0=
=C2=A0=C2=A0 pkgtools/nih=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Package manager for pkgsrc<br>=C2=A0=C2=A0=
=C2=A0 wip/nih-current(nih)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Package manager=
for pkgsrc<br>
=C2=A0=C2=A0=C2=A0 0 cheusov><br><br></div><div class=3D"gmail_extra">If=
you don't trust remote services, you can scan pkgsrc tree locally on y=
our own machine.<br><br></div><div class=3D"gmail_extra">=C2=A0=C2=A0=C2=A0=
nih refresh -p # scanning<br>
</div><div class=3D"gmail_extra">=C2=A0=C2=A0=C2=A0 nih search -p f:s:bin/n=
ih<br><br></div><div class=3D"gmail_extra">The very first scan takes hours =
of work.<br>Usually, subsequent ones work for dozens of second or just a fe=
w minutes.<br><br>
</div><div class=3D"gmail_extra">If pkg_summary.gz file contained PLIST ent=
ries, you can also search for package by filename in this file.<br><br>=C2=
=A0=C2=A0=C2=A0 nih search -b f:s:bin/nih<br><br></div><div class=3D"gmail_=
extra">f:s: above means "Look for packages by Filename with Substring =
search strategy".<br>
</div><div class=3D"gmail_extra">For more information, see <a href=3D"http:=
//mova.org/~cheusov/pub/mlug/2012-11/package-search.pdf">http://mova.org/~c=
heusov/pub/mlug/2012-11/package-search.pdf</a><br><br></div></div>
--001a11c12b5a85ba850500ae13ab--
From: =?utf-8?Q?R=C3=A9my_Chibois?= <rchibois@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org,
gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org,
Aleksey Cheusov <vle@gmx.net>
Subject: Re: pkg/49117: new pkg_info file list option
Date: Fri, 15 Aug 2014 20:17:50 +0200
>> There is currently no way to search for non-installed packages =
containing
>> a file. A simple (compressed) file in the spirit of pkg_summary.gz =
and
>> stored in binary package repositories would allow this.
>=20
>=20
> JFYI: pkgtools/nih, a package manager for pkgsrc, provides a =
functionality
> you need.
Thanks for this information, Aleksey.
The proposed option is aimed at binary package management and a =
companion
tool to pkgin, that is without the need to fetch the entire pkgsrc tree.
The resulting file, pkg_files.gz, generates in minutes and is also =
usable
from the shell using standard tools (zgrep, zfgrep, ...).
While all the work involved in creating such list could be easily done
using shell tools walking the pkgsrc tree, that would be inefficient
(thousands of PLIST processing and reformating).
This is why I think it should be integrated in pkg_info where all
information and tools are available for processing a binary package.
This could also be done in a separate tool but will require duplicating
code from pkg_info.
That=E2=80=99s my idea, though, but the changes seem simple enough and =
would
provide useful information for other binary package tools.
--=20
R=C3=A9my=
From: Aleksey Cheusov <vle@gmx.net>
To: =?UTF-8?Q?R=C3=A9my_Chibois?= <rchibois@gmail.com>
Cc: gnats-bugs@netbsd.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org
Subject: Re: pkg/49117: new pkg_info file list option
Date: Fri, 15 Aug 2014 23:13:55 +0300
--001a1139872022a8480500b0a94a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On Fri, Aug 15, 2014 at 9:17 PM, R=C3=A9my Chibois <rchibois@gmail.com> wro=
te:
> >> There is currently no way to search for non-installed packages
> containing
> >> a file. A simple (compressed) file in the spirit of pkg_summary.gz and
> >> stored in binary package repositories would allow this.
> >
> >
> > JFYI: pkgtools/nih, a package manager for pkgsrc, provides a
> functionality
> > you need.
>
> Thanks for this information, Aleksey.
>
> The proposed option is aimed at binary package management and a companion
> tool to pkgin, that is without the need to fetch the entire pkgsrc tree.
>
Well, "nih search" is actually not about scanning pkgsrc tree.
As I said, it provides search in binary repository as well.
Scanning pkgsrc tree and search in it is just one option.
I mentioned "nih" because it has powerful search capabilities. It supports
search in binary repository.
The only problem is that bulk build tools (distbb, pbulk) don't provide
something like your pkg_files.gz,
so, searching in PLIST in this mode is not available.
As about your original proposal, I'd like to see pkg_files.gz file in
pkg_summary(5) format, that is KEY=3DVALUE.
pkgtools/pkg_summary-utils contains pkg_bin_summary script built on top of
"pkg_info -XL" and implements
things similar to your pkg_info -Y.
0 cheusov>pkg_bin_summary -f PKGNAME,PKGPATH,PLIST pkglint
PKGNAME=3Dpkglint-4.144
PKGPATH=3Dpkgtools/pkglint
PLIST=3D/opt/pkgsrc4unix/bin/pkglint
PLIST=3D/opt/pkgsrc4unix/bin/plist-clash
PLIST=3D/opt/pkgsrc4unix/man/man1/pkglint.1
PLIST=3D/opt/pkgsrc4unix/share/pkglint/deprecated.map
PLIST=3D/opt/pkgsrc4unix/share/pkglint/makevars.map
0 cheusov>pkg_bin_summary -f PKGNAME,PKGPATH,PLIST
/srv/pkgsrc_bin/Linux/RHEL/x86_64/6/current/All/pkg-config-0.28.tgz
PKGNAME=3Dpkg-config-0.28
PKGPATH=3Ddevel/pkg-config
PLIST=3D/opt/pkgsrc4unix/bin/pkg-config
PLIST=3D/opt/pkgsrc4unix/man/man1/pkg-config.1
PLIST=3D/opt/pkgsrc4unix/share/aclocal/pkg.m4
PLIST=3D/opt/pkgsrc4unix/share/doc/pkg-config/pkg-config-guide.html
0 cheusov>
Obviously, you can search in pkg_summary(5) files just like in pkg_files.gz=
.
In my view the goal #1 should be pbulk generating pkg_files.gz file in
pkg_summary(5) format.
Patching pkg_info is not necessary. We already have "pkg_info -BL" and even
"pkg_bin_summary"
built on top of it.
P.S. Search in pkg_summary(5) files is also already implemented in
pkg_grep_summary (pkgtools/pkg_summary-utils).
--001a1139872022a8480500b0a94a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On F=
ri, Aug 15, 2014 at 9:17 PM, R=C3=A9my Chibois <span dir=3D"ltr"><<a hre=
f=3D"mailto:rchibois@gmail.com" target=3D"_blank">rchibois@gmail.com</a>>=
;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex"><div class=3D"">>> =
There is currently no way to search for non-installed packages containing<b=
r>
>> a file. A simple (compressed) file in the spirit of pkg_summary.gz=
and<br>
>> stored in binary package repositories would allow this.<br>
><br>
><br>
> JFYI: pkgtools/nih, a package manager for pkgsrc, provides a functiona=
lity<br>
> you need.<br>
<br>
</div>Thanks for this information, Aleksey.<br>
<br>
The proposed option is aimed at binary package management and a companion<b=
r>
tool to pkgin, that is without the need to fetch the entire pkgsrc tree.<br=
></blockquote><div><br></div><div>Well, "nih search" is actually =
not about scanning pkgsrc tree. <br></div><div>As I said, it provides searc=
h in binary repository as well.<br>
Scanning pkgsrc tree and search in it is just one option.</div><div>I menti=
oned "nih" because it has powerful search capabilities. It suppor=
ts search in binary repository.<br>The only problem is that bulk build tool=
s (distbb, pbulk) don't provide something like your pkg_files.gz,<br>
</div><div>so, searching in PLIST in this mode is not available.<br></div><=
div><br></div><div>As about your original proposal, I'd like to see pkg=
_files.gz file in pkg_summary(5) format, that is KEY=3DVALUE.<br></div><div=
>
pkgtools/pkg_summary-utils contains pkg_bin_summary script built on top of =
"pkg_info -XL" and implements<br></div><div>things similar to you=
r pkg_info -Y.<br><br>0 cheusov>pkg_bin_summary -f PKGNAME,PKGPATH,PLIST=
pkglint<br>
PKGNAME=3Dpkglint-4.144<br>PKGPATH=3Dpkgtools/pkglint<br>PLIST=3D/opt/pkgsr=
c4unix/bin/pkglint<br>PLIST=3D/opt/pkgsrc4unix/bin/plist-clash<br>PLIST=3D/=
opt/pkgsrc4unix/man/man1/pkglint.1<br>PLIST=3D/opt/pkgsrc4unix/share/pkglin=
t/deprecated.map<br>
PLIST=3D/opt/pkgsrc4unix/share/pkglint/makevars.map<br><br>0 cheusov>pkg=
_bin_summary -f PKGNAME,PKGPATH,PLIST /srv/pkgsrc_bin/Linux/RHEL/x86_64/6/c=
urrent/All/pkg-config-0.28.tgz <br>PKGNAME=3Dpkg-config-0.28<br>PKGPATH=3Dd=
evel/pkg-config<br>
PLIST=3D/opt/pkgsrc4unix/bin/pkg-config<br>PLIST=3D/opt/pkgsrc4unix/man/man=
1/pkg-config.1<br>PLIST=3D/opt/pkgsrc4unix/share/aclocal/pkg.m4<br>PLIST=3D=
/opt/pkgsrc4unix/share/doc/pkg-config/pkg-config-guide.html<br><br>0 cheuso=
v><br>
</div><br></div><div class=3D"gmail_quote"></div><div class=3D"gmail_quote"=
>Obviously, you can search in pkg_summary(5) files just like in pkg_files.g=
z.<br><br></div><div class=3D"gmail_quote">In my view the goal #1 should be=
pbulk generating pkg_files.gz file in pkg_summary(5) format.<br>
<br></div><div class=3D"gmail_quote">Patching pkg_info is not necessary. We=
already have "pkg_info -BL" and even "pkg_bin_summary"=
<br>built on top of it.<br></div><div class=3D"gmail_quote"><br></div><div =
class=3D"gmail_quote">
P.S. Search in pkg_summary(5) files is also already implemented in pkg_grep=
_summary (pkgtools/pkg_summary-utils).<br><br></div></div></div>
--001a1139872022a8480500b0a94a--
From: =?utf-8?Q?R=C3=A9my_Chibois?= <rchibois@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org,
gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org,
Aleksey Cheusov <vle@gmx.net>
Subject: Re: pkg/49117: new pkg_info file list option
Date: Sat, 16 Aug 2014 11:15:52 +0200
>>>> There is currently no way to search for non-installed packages
>> containing
>>>> a file. A simple (compressed) file in the spirit of pkg_summary.gz =
and
>>>> stored in binary package repositories would allow this.
>>>=20
>>>=20
>>> JFYI: pkgtools/nih, a package manager for pkgsrc, provides a
>> functionality
>>> you need.
>>=20
>> Thanks for this information, Aleksey.
>>=20
>> The proposed option is aimed at binary package management and a =
companion
>> tool to pkgin, that is without the need to fetch the entire pkgsrc =
tree.
>>=20
>=20
> Well, "nih search" is actually not about scanning pkgsrc tree.
> As I said, it provides search in binary repository as well.
> Scanning pkgsrc tree and search in it is just one option.
> I mentioned "nih" because it has powerful search capabilities. It =
supports
> search in binary repository.
> The only problem is that bulk build tools (distbb, pbulk) don't =
provide
> something like your pkg_files.gz,
> so, searching in PLIST in this mode is not available.
>=20
> As about your original proposal, I'd like to see pkg_files.gz file in
> pkg_summary(5) format, that is KEY=3D3DVALUE.
> pkgtools/pkg_summary-utils contains pkg_bin_summary script built on =
top of
> "pkg_info -XL" and implements
> things similar to your pkg_info -Y.
Perhaps I=E2=80=99ve been misleading about my intent.
The main goal is to have a list in the proposed format (PKGNAME: PATH)
because with a simple regex it gives both packages and matching files
on the same line.
This file should be built server-side when a binary repository is
built/updated and made available to public use for binary package
managers which would update/download it once in a while.
For example:
<pbulk build>
cd /path/to/binary/packages
pkg_summary -X *.tgz | gzip -9 > pkg_summary.gz
pkg_info -qY *.tgz | gzip -9 > pkg_files.gz
Once downloaded, this file is usable for fast searches using common
tools without requiring network access and answers the question
=E2=80=9Cgive me a list of packages and files matching a file regex=E2=80=9D=
.
While I=E2=80=99ve already looked at the various tools available in =
pkgsrc,
nothing provides the required format and extra processing steps
are required to process PLIST files, that is:
- grab package name
- iterate over or sed PLIST and print =E2=80=9CPKGNAME: PATH=E2=80=9D
and to use it:
- scan output of available tools and search for package name
- output PLIST in required format until new package name
So, to summarise a bit, pkg_files.gz:
- targets binary package users who dot not necessarily have
the full pkgsrc tree
- is built server-side using a patched pkg_info
- is downloaded/updated by a binary package manager
(or curl, wget, ...)
- is usable by binary package users via their binary package
manager or with common tools locally on their machine
- needs network access only once in a while
I found the most efficient way to achieve this was to add a small
piece of code to pkg_info using its existing code base, but I
perfectly understand that such modification is perhaps too
specific or implicates other things I don=E2=80=99t foresee.
Do you think it=E2=80=99s preferable I make a separate tool ?
Thank you.
--=20
R=C3=A9my
From: Aleksey Cheusov <vle@gmx.net>
To: =?UTF-8?Q?R=C3=A9my_Chibois?= <rchibois@gmail.com>
Cc: gnats-bugs@netbsd.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org
Subject: Re: pkg/49117: new pkg_info file list option
Date: Sat, 16 Aug 2014 15:50:37 +0300
--001a113a39369d7d350500be95f1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
>
> The main goal is to have a list in the proposed format (PKGNAME: PATH)
> because with a simple regex it gives both packages and matching files
> on the same line.
>
I understand this goal and I still think this format is very bad.
We already have pkg_summary(5) format which is more generic and universal.
As I said you can easily grep pkg_summary(5) files with a help of
pkg_grep_summary,
a part of pkgtools/pkg_summary-utils package. See below.
So, we just don't need yet another format.
0 cheusov>pkg_bin_summary -f PKGNAME,PLIST > summary.txt
0 cheusov>pkg_grep_summary -f PKGNAME -t substring PLIST pkglint <
summary.txt
PKGNAME=3Dpkglint-4.144
0 cheusov>
This file should be built server-side when a binary repository is
> built/updated and made available to public use for binary package
> managers which would update/download it once in a while.
>
Yes, I absolutely agree with this goal but with different format for
pkg_files.gz
and without patching pkg_info(8). Output of PLIST can easily be done
in pbulk using pkg_info -XL and trivial AWK script.
> While I=E2=80=99ve already looked at the various tools available in pkgsr=
c,
> nothing provides the required format and extra processing steps
> are required to process PLIST files, that is:
>
Have a look at pkgtools/pkg_summary-utils
> So, to summarise a bit, pkg_files.gz:
> - targets binary package users who dot not necessarily have
> the full pkgsrc tree
>
See above.
- is built server-side using a patched pkg_info
>
Patching pkg_info is not necessary. It is just good enough (_XL).
> - is downloaded/updated by a binary package manager
> (or curl, wget, ...)
>
Yes, we really need something like pkg_files.gz containing PLIST entries.
> - is usable by binary package users via their binary package
> manager or with common tools locally on their machine
>
Yes.
> I found the most efficient way to achieve this was to add a small
> piece of code to pkg_info using its existing code base, but I
> perfectly understand that such modification is perhaps too
> specific or implicates other things I don=E2=80=99t foresee.
>
> Do you think it=E2=80=99s preferable I make a separate tool ?
>
There is no need for yet another tool. It already exists.
Have a look at pkg_bin_summary. I doubt pbulk developers will use it,
but they can easily generate PKGNAME+PKGPATH+PLIST file
with existing pkg_info and very simple awk script.
--001a113a39369d7d350500be95f1
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex">
The main goal is to have a list in the proposed format (PKGNAME: PATH)<br>
because with a simple regex it gives both packages and matching files<br>
on the same line.<br></blockquote><div><br></div><div>I understand this goa=
l and I still think this format is very bad.<br>We already have pkg_summary=
(5) format which is more generic and universal.<br></div><div>As I said you=
can easily grep pkg_summary(5) files with a help of pkg_grep_summary,<br>
</div><div>a part of pkgtools/pkg_summary-utils package. See below.<br>So, =
we just don't need yet another format.<br></div><div><br>0 cheusov>p=
kg_bin_summary -f PKGNAME,PLIST > summary.txt<br>0 cheusov>pkg_grep_s=
ummary -f PKGNAME -t substring PLIST pkglint < summary.txt <br>
PKGNAME=3Dpkglint-4.144<br><br>0 cheusov><br><br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">
This file should be built server-side when a binary repository is<br>
built/updated and made available to public use for binary package<br>
managers which would update/download it once in a while.<br></blockquote><d=
iv><br></div><div>Yes, I absolutely agree with this goal but with different=
format for pkg_files.gz<br></div><div>and without patching pkg_info(8). Ou=
tput of PLIST can easily be done<br>
</div><div>in pbulk using pkg_info -XL and trivial AWK script.<br></div><di=
v><br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1=
ex">
While I=E2=80=99ve already looked at the various tools available in pkgsrc,=
<br>
nothing provides the required format and extra processing steps<br>
are required to process PLIST files, that is:<br></blockquote><div><br></di=
v><div>Have a look at pkgtools/pkg_summary-utils<br>=C2=A0<br></div><blockq=
uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p=
x solid rgb(204,204,204);padding-left:1ex">
So, to summarise a bit, pkg_files.gz:<br>
- targets binary package users who dot not necessarily have<br>
=C2=A0 the full pkgsrc tree<br></blockquote><div><br>See above.<br><br></di=
v><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;borde=
r-left:1px solid rgb(204,204,204);padding-left:1ex">
- is built server-side using a patched pkg_info<br></blockquote><div><br>Pa=
tching pkg_info is not necessary. It is just good enough (_XL).<br>=C2=A0<b=
r></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex=
;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- is downloaded/updated by a binary package manager<br>
=C2=A0 (or curl, wget, ...)<br></blockquote><div><br></div><div>Yes, we rea=
lly need something like pkg_files.gz containing PLIST entries.<br>=C2=A0<br=
></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;=
border-left:1px solid rgb(204,204,204);padding-left:1ex">
- is usable by binary package users via their binary package<br>
=C2=A0 manager or with common tools locally on their machine<br></blockquot=
e><div>Yes.<br>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D"m=
argin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left=
:1ex">I found the most efficient way to achieve this was to add a small<br>
piece of code to pkg_info using its existing code base, but I<br>
perfectly understand that such modification is perhaps too<br>
specific or implicates other things I don=E2=80=99t foresee.<br>
<br>
Do you think it=E2=80=99s preferable I make a separate tool ?<br></blockquo=
te></div><br></div><div class=3D"gmail_extra">There is no need for yet anot=
her tool. It already exists.<br></div><div class=3D"gmail_extra">Have a loo=
k at pkg_bin_summary. I doubt pbulk developers will use it,<br>
but they can easily generate PKGNAME+PKGPATH+PLIST file<br>with existing pk=
g_info and very simple awk script.<br><br></div></div>
--001a113a39369d7d350500be95f1--
From: =?utf-8?Q?R=C3=A9my_Chibois?= <rchibois@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org,
gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org,
Aleksey Cheusov <vle@gmx.net>
Subject: Re: pkg/49117: new pkg_info file list option
Date: Sun, 17 Aug 2014 12:16:24 +0200
>> The main goal is to have a list in the proposed format (PKGNAME: =
PATH)
>> because with a simple regex it gives both packages and matching files
>> on the same line.
>>=20
>=20
> I understand this goal and I still think this format is very bad.
> We already have pkg_summary(5) format which is more generic and =
universal.
I totally agree with generic and universal here but still don=E2=80=99t =
see
why the format is =E2=80=9Cvery=E2=80=9D bad...
> As I said you can easily grep pkg_summary(5) files with a help of
> pkg_grep_summary,
> a part of pkgtools/pkg_summary-utils package. See below.
> So, we just don't need yet another format.
>=20
> 0 cheusov>pkg_bin_summary -f PKGNAME,PLIST > summary.txt
> 0 cheusov>pkg_grep_summary -f PKGNAME -t substring PLIST pkglint <
> summary.txt
> PKGNAME=3D3Dpkglint-4.144
That=E2=80=99s not that I don=E2=80=99t want to use existing tools, but =
the example
commands above still don=E2=80=99t provide both the package name and
matching file. I already tried pkg_summary-utils:
$ pkg_bin_summary -f PKGNAME,PLIST > summary.txt
$ pkg_grep_summary -f PKGNAME -t re PLIST =
"/usr/pkg/include/.*string\.hpp" < summary.txt=20
PKGNAME=3Dboost-headers-1.55.0nb1
PKGNAME=3Dmsgpack-0.5.7
PKGNAME=3Dobby-0.4.7nb2
PKGNAME=3Dxqilla-2.2.4
I get the package name (despite blank lines) but not the matching file.
(please note that I=E2=80=99m not an AWK expert)
$ zgrep "/usr/pkg/include/.*string\.hpp=E2=80=9D pkg_files.gz
boost-headers-1.55.0nb1: /usr/pkg/include/boost/algorithm/string.hpp
boost-headers-1.55.0nb1: /usr/pkg/include/boost/chrono/clock_string.hpp
boost-headers-1.55.0nb1: =
/usr/pkg/include/boost/chrono/io/utility/to_string.hpp
boost-headers-1.55.0nb1: /usr/pkg/include/boost/container/string.hpp
[...]
boost-headers-1.55.0nb1: =
/usr/pkg/include/boost/wave/util/flex_string.hpp
msgpack-0.5.7: /usr/pkg/include/msgpack/type/string.hpp
obby-0.4.7nb2: /usr/pkg/include/obby/format_string.hpp
xqilla-2.2.4: /usr/pkg/include/xqilla/functions/FunctionSubstring.hpp
For something like a source package include dependency scan (the real
intended purpose that led me proposing pkg_files.gz), we can now
tell the end user that =E2=80=9C#include <string.hpp>=E2=80=9D (or =
whatever =E2=80=9Cmalformed=E2=80=9D
include directive using incorrect -I paths) should be fixed.
> Yes, I absolutely agree with this goal but with different format for
> pkg_files.gz
> and without patching pkg_info(8). Output of PLIST can easily be done
> in pbulk using pkg_info -XL and trivial AWK script.
Okay but what about binary package managers ? Should they reimplement
the logic found in AWK scripts or system()/pipe() a command line
(with path issues, added dependencies, ...). A simple file read loop
using regexec still seems a lot simpler to me, but I agree that=E2=80=99s
not the main goal.
> Yes, we really need something like pkg_files.gz containing PLIST =
entries.
Okay, we agree on this. Can you please give me some guidelines on
how proceed and propose this feature (in whatever format you find
approriate, I=E2=80=99ll reprocess it myself)?
Thank you for your time and patience.
--=20
R=C3=A9my=
State-Changed-From-To: open->closed
State-Changed-By: youri@NetBSD.org
State-Changed-When: Wed, 02 Mar 2016 08:43:13 +0000
State-Changed-Why:
After some research, it has been decided that it is not possible to implement this in pkgin in a non intrusive way.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home