NetBSD Problem Report #35340

From www@NetBSD.org  Sat Dec 30 04:34:02 2006
Return-Path: <www@NetBSD.org>
Received: by narn.NetBSD.org (Postfix, from userid 31301)
	id BCDA963B88A; Sat, 30 Dec 2006 04:34:01 +0000 (UTC)
Message-Id: <20061230043401.BCDA963B88A@narn.NetBSD.org>
Date: Sat, 30 Dec 2006 04:34:01 +0000 (UTC)
From: truist-netbsd-sendpr@truist.com
Reply-To: truist-netbsd-sendpr@truist.com
To: gnats-bugs@NetBSD.org
Subject: pkgsrc OWN_DIRS bug where empty directories are pruned too earnestly
X-Send-Pr-Version: www-1.0

>Number:         35340
>Category:       pkg
>Synopsis:       pkgsrc OWN_DIRS bug where empty directories are pruned too earnestly
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    dholland
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Dec 30 04:35:00 +0000 2006
>Closed-Date:    
>Last-Modified:  Sat May 14 21:25:00 +0000 2022
>Originator:     Nathan Arthur
>Release:        3.0_STABLE on i386
>Organization:
truist.com
>Environment:
NetBSD home.truist.com 3.0_STABLE NetBSD 3.0_STABLE (TRUIST) #1: Fri Aug 18 08:01:55 EDT 2006  root@home.truist.com:/usr/obj/sys/arch/i386/compile/TRUIST i386

>Description:
I use pkgsrc's bulk build scripts to do bulk rebuilds of all the packages I use on my NetBSD 3.0 machine.  I don't build all packages - just the subset that I need.  I do the builds in a chroot sandbox (via mk/bulk/mksandbox) and I've discovered a problem where the build of the qmail-qfilter package is deleting the sandbox's /var/tmp directory (which then causes later package builds the fail).  Per the email thread below, it seems to be deleted during pkg_delete because of something to do with OWN_DIRS_PERMS.  I've dug as far as I can into the scripts to try to understand what's going wrong, and I don't know enough to go further.  Amitai (below) asked that I send-pr.

This only seems to happen if /var/tmp is empty at the time that qmail-qfilter is pkg_delete'd, which explains why this doesn't show up for regular package builds.   It also doesn't seem to be affecting the regular netbsd bulk rebuilds, presumably because there is something in /var/tmp at the time that this happens.

I can happily give more detail about the process here, but I think all the salient details are above and below.  For now, I've worked around this by just putting a file in /var/tmp, but it seems like an issue that should probably be fixed.

-------- Original Message --------
Subject:     Re: You broke my bulk build!
Date:     Sun, 22 Oct 2006 19:29:40 -0400 (EDT)
From:     Amitai Schlair <schmonz@schmonz.com>
To:     Nathan Arthur <truist@truist.com>
References:     <453BED63.4090205@truist.com>


Nathan Arthur wrote:

> 1) Create the sandbox (which creates an appropriate /var/tmp)
> 2) chroot into it
> 3) Manually build qmail-qfilter, into a package
> 4) Install the qmail-qfilter package
> 5) At this point, /var/tmp is still there, and has an empty qmail-qfilter directory in it.
> 6) pkg_delete qmail-qfilter
> 7) /var/tmp is gone!

Wacky. At a glance, that shouldn't happen. The relevant line of
pkgsrc/mail/qmail-qfilter/Makefile is:

   OWN_DIRS_PERMS+=    ${QMAIL_QFILTER_TMPDIR} qmaild ${ROOT_GROUP} 700

And the relevant lines in pkgsrc/mk/defaults/mk.conf are:

   QMAIL_QFILTER_TMPDIR?= ${VARBASE}/tmp/qmail-qfilter
   # The temporary directory for qmail-qfilter.
   # Possible: a directory writable only by qmail-qfilter's UID
   # Default: ${VARBASE}/tmp/qmail-qfilter

OWN_DIRS is handled somewhere in pkgsrc/mk/install or pkgsrc/mk/pkginstall (I forget which), which are standard shell scripts that get executed by pkg_add and pkg_delete. What's supposed to happen is, package defines OWN_DIRS, it gets created at install time, if it's empty at deinstall time it's removed (there's a related variable that instead instructs the user to remove it). So it sounds like this logic is pruning empty directories a little too earnestly up the filesystem.

>How-To-Repeat:
1) Create the sandbox (which creates an appropriate /var/tmp)
2) chroot into it
3) Manually build qmail-qfilter, into a package
4) Install the qmail-qfilter package
5) At this point, /var/tmp is still there, and has an empty qmail-qfilter directory in it.
6) pkg_delete qmail-qfilter
7) /var/tmp is gone!

>Fix:
Unknown

>Release-Note:

>Audit-Trail:
From: Nathan Arthur <truist@rainskit.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/35340
Date: Wed, 16 Jul 2008 02:15:53 -0400

 I can confirm that this problem still exists, as of today, using these 
 same steps.

 Thanks,
 Nathan

From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/35340
Date: Wed, 16 Jul 2008 15:56:36 +0900

 Can you check tail of /var/db/pkg/qmail-qfilter-2.1nb1/+DIRS,
 lines start with "# DIR:"?

State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 18 Jan 2009 09:25:34 +0000
State-Changed-Why:
A question was asked.


State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 18 Jan 2009 09:27:12 +0000
State-Changed-Why:
Feedback mail bounced.


Responsible-Changed-From-To: pkg-manager->dholland
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Sun, 09 Aug 2015 22:55:56 +0000
Responsible-Changed-Why:
I can replicate this with a simple test package.


From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/35340 (pkgsrc OWN_DIRS bug where empty directories are
 pruned too earnestly)
Date: Sun, 9 Aug 2015 23:14:57 +0000

 On Sun, Aug 09, 2015 at 10:55:56PM +0000, dholland@NetBSD.org wrote:
  > I can replicate this with a simple test package.

 The problem is not that the wrong things are listed at the end of
 +DIRS; the problem is that +DIRS uses rmdir -p to remove dirs, so that
 any empty ancestor directories are pruned regardless of whether pkgsrc
 owns them or not.

 This could be changed; however, because +DIRS also uses mkdir -p, this
 may cause intermediate dirs created by pkgsrc to get left around.

 I note that +DIRS sets MKDIR to mkdir -p, and then uses ${MKDIR} -p in
 places, suggesting that this may not have been originally intended.

 It seems to me that the right fix is to not use either rmdir -p or
 mkdir -p, and to fix packages that don't explicitly create ancestor
 directories. (That is, if you have
    OWN_DIRS_PERMS+=	/var/thingy/boojum root wheel 755
    OWN_DIRS_PERMS+=	/var/thingy/snark  root wheel 755
 without an explicit entry for /var/thingy, nothing will create
 /var/thingy and installation of the binary package will fail.)
 This is probably a good thing, at least for OWN_DIRS, as the
 permissions on directories created by mkdir -p depend on the umask and
 maybe other things and aren't necessarily as repeatable as we'd like.

 However, this introduces two other issues: (a) where do we stop (do we
 need OWN_DIRS for /var? surely not; but what about e.g. /var/db, which
 probably doesn't exist on say Linux?) and (b) does mucking with the
 +DIRS script affect other directory creation that we aren't prepared
 to deal with or where using mkdir -p is necessary and appropriate? I
 don't know...

 btw, the replication example is to just add e.g.
    OWN_DIRS+=	/var/thingy/boojum root wheel 755
 to something simple and install/deinstall. Currently creating
 /var/thingy beforehand is not necessary, because it will be created by
 mkdir -p.

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/35340 (pkgsrc OWN_DIRS bug where empty directories are
 pruned too earnestly)
Date: Sat, 14 May 2022 21:22:31 +0000

 On Sun, Aug 09, 2015 at 11:15:00PM +0000, David Holland wrote:
  >  On Sun, Aug 09, 2015 at 10:55:56PM +0000, dholland@NetBSD.org wrote:
  >   > I can replicate this with a simple test package.
  >  
  >  The problem is not that the wrong things are listed at the end of
  >  +DIRS; the problem is that +DIRS uses rmdir -p to remove dirs, so that
  >  any empty ancestor directories are pruned regardless of whether pkgsrc
  >  owns them or not.

 This still happens, btw.

 (Though the package in the original report seems to no longer be
 affected, as AFAICT it no longer uses OWN_DIRS.)

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

Home	PR Database Search