NetBSD Problem Report #24255

Received: (qmail 17466 invoked by uid 605); 26 Jan 2004 22:32:13 -0000
Message-Id: <20040126223213.17465.qmail@mail.netbsd.org>
Date: 26 Jan 2004 22:32:13 -0000
From: Christian Biere <christianbiere@gmx.de>
Sender: gnats-bugs-owner@NetBSD.org
To: gnats-bugs@gnats.netbsd.org
Subject: pkg_delete removes meta data even if deinstall fails
X-Send-Pr-Version: 3.95

>Number:         24255
>Category:       pkg
>Synopsis:       pkg_delete removes meta data even if deinstall fails
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    joerg
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jan 26 22:33:00 +0000 2004
>Closed-Date:    
>Last-Modified:  Sun May 01 04:50:02 +0000 2011
>Originator:     Christian Biere
>Release:        NetBSD 1.6ZH
>Organization:
>Environment:
System: NetBSD cyclonus 1.6ZH NetBSD 1.6ZH (STARSCREAM) #2: Sun Jan 25 22:37:39 CET 2004 bin@cyclonus:/usr/build/src/sys/arch/i386/compile/STARSCREAM i386
>Description:

pkg_delete deletes all meta data /var/db/pkg/PACKAGE for a package even if
the directories/files couldn't be deleted due to missing permissions, a
read-only mounted partition etc, wrong moon-phase etc. This can result in a
lot of orphaned files under /usr/pkg.

Instead pkg_delete should bail out or at least ask how to procede if some of
the deinstallation procedures fail.

>How-To-Repeat:

$ find /usr/pkg -type f -print0|xargs -0 chflags -h uchg # paranoia
$ pkg_delete PACKAGE

>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->joerg
Responsible-Changed-By: joerg@netbsd.org
Responsible-Changed-When: Wed, 04 Oct 2006 14:53:33 +0000
Responsible-Changed-Why:
pkg_install issue.


State-Changed-From-To: open->feedback
State-Changed-By: asau@NetBSD.org
State-Changed-When: Tue, 14 Dec 2010 19:01:03 +0000
State-Changed-Why:
There were massive improvements in pkg_install since PR was filed.
Does the problem persist?


From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall fails)
Date: Fri, 25 Mar 2011 20:36:52 +0200

 > There were massive improvements in pkg_install since PR was filed.
 > Does the problem persist?

 Yes, the problem still persists.

    asrock# chflags -h uchg /usr/pkg/bin/distbb
    asrock# pkg_delete distbb
    pkg_delete: Couldn't remove /usr/pkg/bin/distbb
    asrock# echo $?
    0
    asrock# pkg_delete distbb
    pkg_delete: No matching package for basename `distbb' of `distbb'
    asrock# ls -la /usr/pkg/bin/distbb
    -rwxr-xr-x  1 root  wheel  7056 Jan 21 01:23 /usr/pkg/bin/distbb
    asrock#

 How about swapping files removal and pkgdb operations?

 -- 
 Best regards, Aleksey Cheusov.

State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 21 Apr 2011 03:49:38 +0000
State-Changed-Why:
Problem still exists. Also, submitter wrote to the gnats junkmail address
to report that he can't test.


From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
 fails)
Date: Thu, 21 Apr 2011 05:49:01 -0400

 On Thu, 21 Apr 2011 03:49:39 +0000 (UTC)
 dholland@NetBSD.org wrote:

 > Synopsis: pkg_delete removes meta data even if deinstall fails
 > 
 > State-Changed-From-To: feedback->open
 > State-Changed-By: dholland@NetBSD.org
 > State-Changed-When: Thu, 21 Apr 2011 03:49:38 +0000
 > State-Changed-Why:
 > Problem still exists. Also, submitter wrote to the gnats junkmail address
 > to report that he can't test.

 Just so I have a clear view of what is ideally wanted:

 Currently, some files of a package might still subsist while the db
 metadata is lost/deleted.

 It would be possible to instead leave the meta information pending on
 error, at the risk of having some files lost/missing with inconsistent
 files/PLIST.

 I doubt it, but could access(2) reliably be used as a first pass before
 the unlink(2)/rmdir(2) phase to fail before performing any
 non-reversible operation?

 Alternatively, should packages also support a "partial/broken" state,
 where metadata would be updated to reflect actual remaining files after
 an incomplete removal?  Possibly keeping the original PLIST for
 reference but adding a PARTIAL-PLIST for such packages?


 Or should a transactional/atomic system be expected instead?

 In which case I could see various imperfect solutions (FFS not
 helping much here):

 - Obtain a global exclusive pkg advisory lock

 - Backup the package files and attempt to delete the original files; if
 everything is successful, delete the backup and metadata, otherwise
 restore the deleted files from the backup and preserve the metadata.
 This should take into account file permissions, flags, PaX flags, etc.
 As drawbacks, some space would be required to store the temporary
 backup, and some effort made to sync all file permissions and flags,
 code which might be duplicated among the various backup utilities
 supporting some or all of these flags (not to mention the OS-specifics).
 Perhaps acceptable with a dependency on pax(1)?

 - Instead of deleting files, move them to a temporary "trashcan"
 directory tree on the same filesystem (on write failure, fail
 immediately); if all files were successfully moved, ensure that their
 permissions allow deletion and recursively delete the temporary tree,
 then the metadata.
 On error move successfully-moved files back where they belong,
 restoring their permissions, and leave the metadata as-is, since we
 restored the tree state.  Seems flaky; I'm not sure this would
 succeed in all cases where package removal now fails.  We also could
 end up with a growing trash directory if files could be moved yet could
 not be deleted.

 - fsync(2) and release the pkg lock


 Unfortunately, even the above solutions might yield inconsistent state
 in the advent of a crash or user interrupt.  NetBSD doesn't have
 transactional FS and support syscalls, and even if it did this might
 not exist on most other pkgsrc-supported OSs...


 Other more far-fetched "atomic" support built on non-transactional FS:

 I guess that a variant could be using a NetBSD-supported transactional
 "trashcan" per file system along with its atomic-simulated
 auto-restore/async-delete system.  Such a trashcan might include two
 sections, one for user-deleted files and another for system-deleted
 files.  Files would be stored with their original path/flags.
 User-deleted files would require explicit "empty/restore" commands, yet
 files destined for system-removal would be enclosed in transactions and
 could be restored at crash/reboot in case of an incomplete transaction,
 with a thread/daemon responsible for asynchroneously deleting files
 deleted in complete transactions.
 Transactions metadata could be implemented on top of bdb, sqlite or a
 custom system supporting a transactional log (using fdatasync(2) at
 reasonable intervals and automatic log file rotation/removal, etc).
 pkgsrc when on NetBSD could then make use of that "atomic multi-delete
 operation", despite it being non-portable.  Perhaps that some files
 might still be dangling over time and that a nightly script should warn
 about it.

 A variant of the above might be directly provided by pkgsrc, with
 reliance on the required portable tools such as sqlite used for package
 metadata and transactional-log, with $PKGDIR/tmp/trash used as a
 temporary trash directory, an asynchroneous files cleanup cron event or
 daemon, with a boot rollback/recovery script.
 Unfortunately complicates pkgsrc, adds new tool dependencies and makes
 it more system-invasive...  Perhaps that instead of having a
 boot-time recovery script and a cleanup cron event/daemon, every pkg_*
 command could as a prologue perform those verification/pending
 operations as needed, to minimise system invasion, with failed
 procedures logged to stderr as a warning before attempting the
 requested operation, with some loss of performance...

 Other suggestions?

 Thanks,
 -- 
 Matt

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
	Christian Biere <christianbiere@gmx.de>
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
 fails)
Date: Thu, 21 Apr 2011 13:27:03 +0200

 On Thu, Apr 21, 2011 at 09:50:04AM +0000, Matthew Mondor wrote:
 >  I doubt it, but could access(2) reliably be used as a first pass before
 >  the unlink(2)/rmdir(2) phase to fail before performing any
 >  non-reversible operation?

 AFAIK, no. Many file systems don't check for read-only mounts etc. in
 this case.

 Frankly, I consider this a clear "won't fix".

 Joerg

From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
 fails)
Date: Thu, 21 Apr 2011 17:03:18 -0400

 On Thu, 21 Apr 2011 13:27:03 +0200
 Joerg Sonnenberger <joerg@britannica.bec.de> wrote:

 > Frankly, I consider this a clear "won't fix".

 That's the shortest path :) personally, this hasn't been a problem for
 me, and I prefer the possibility of dangling pkg files without package
 metadata vs a metadata entry with missing files...

 Hmm since the submitter can't test anymore, does this bother anyone else
 enough to prevent this PR from being switched to closed or analyzed
 state?

 Thanks,
 -- 
 Matt

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
	Christian Biere <christianbiere@gmx.de>
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall fails)
Date: Sun, 1 May 2011 04:47:00 +0000

 On Thu, Apr 21, 2011 at 09:05:03PM +0000, Matthew Mondor wrote:
  >  Hmm since the submitter can't test anymore, does this bother anyone else
  >  enough to prevent this PR from being switched to closed or analyzed
  >  state?

 I dunno, I'd rather that if something belched while it was being
 removed we didn't delete the only available record of which particular
 wreckage in /usr/pkg belonged to it.

 ISTM offhand that it shouldn't be too hard to mark the package as
 partially deleted in the metadata first, delete the contents, then
 delete the metadata.

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

Home	PR Database Search