NetBSD Problem Report #24255
Received: (qmail 17466 invoked by uid 605); 26 Jan 2004 22:32:13 -0000
Message-Id: <20040126223213.17465.qmail@mail.netbsd.org>
Date: 26 Jan 2004 22:32:13 -0000
From: Christian Biere <christianbiere@gmx.de>
Sender: gnats-bugs-owner@NetBSD.org
To: gnats-bugs@gnats.netbsd.org
Subject: pkg_delete removes meta data even if deinstall fails
X-Send-Pr-Version: 3.95
>Number: 24255
>Category: pkg
>Synopsis: pkg_delete removes meta data even if deinstall fails
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: joerg
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Jan 26 22:33:00 +0000 2004
>Closed-Date:
>Last-Modified: Sun May 01 04:50:02 +0000 2011
>Originator: Christian Biere
>Release: NetBSD 1.6ZH
>Organization:
>Environment:
System: NetBSD cyclonus 1.6ZH NetBSD 1.6ZH (STARSCREAM) #2: Sun Jan 25 22:37:39 CET 2004 bin@cyclonus:/usr/build/src/sys/arch/i386/compile/STARSCREAM i386
>Description:
pkg_delete deletes all meta data /var/db/pkg/PACKAGE for a package even if
the directories/files couldn't be deleted due to missing permissions, a
read-only mounted partition etc, wrong moon-phase etc. This can result in a
lot of orphaned files under /usr/pkg.
Instead pkg_delete should bail out or at least ask how to procede if some of
the deinstallation procedures fail.
>How-To-Repeat:
$ find /usr/pkg -type f -print0|xargs -0 chflags -h uchg # paranoia
$ pkg_delete PACKAGE
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->joerg
Responsible-Changed-By: joerg@netbsd.org
Responsible-Changed-When: Wed, 04 Oct 2006 14:53:33 +0000
Responsible-Changed-Why:
pkg_install issue.
State-Changed-From-To: open->feedback
State-Changed-By: asau@NetBSD.org
State-Changed-When: Tue, 14 Dec 2010 19:01:03 +0000
State-Changed-Why:
There were massive improvements in pkg_install since PR was filed.
Does the problem persist?
From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall fails)
Date: Fri, 25 Mar 2011 20:36:52 +0200
> There were massive improvements in pkg_install since PR was filed.
> Does the problem persist?
Yes, the problem still persists.
asrock# chflags -h uchg /usr/pkg/bin/distbb
asrock# pkg_delete distbb
pkg_delete: Couldn't remove /usr/pkg/bin/distbb
asrock# echo $?
0
asrock# pkg_delete distbb
pkg_delete: No matching package for basename `distbb' of `distbb'
asrock# ls -la /usr/pkg/bin/distbb
-rwxr-xr-x 1 root wheel 7056 Jan 21 01:23 /usr/pkg/bin/distbb
asrock#
How about swapping files removal and pkgdb operations?
--
Best regards, Aleksey Cheusov.
State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 21 Apr 2011 03:49:38 +0000
State-Changed-Why:
Problem still exists. Also, submitter wrote to the gnats junkmail address
to report that he can't test.
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
fails)
Date: Thu, 21 Apr 2011 05:49:01 -0400
On Thu, 21 Apr 2011 03:49:39 +0000 (UTC)
dholland@NetBSD.org wrote:
> Synopsis: pkg_delete removes meta data even if deinstall fails
>
> State-Changed-From-To: feedback->open
> State-Changed-By: dholland@NetBSD.org
> State-Changed-When: Thu, 21 Apr 2011 03:49:38 +0000
> State-Changed-Why:
> Problem still exists. Also, submitter wrote to the gnats junkmail address
> to report that he can't test.
Just so I have a clear view of what is ideally wanted:
Currently, some files of a package might still subsist while the db
metadata is lost/deleted.
It would be possible to instead leave the meta information pending on
error, at the risk of having some files lost/missing with inconsistent
files/PLIST.
I doubt it, but could access(2) reliably be used as a first pass before
the unlink(2)/rmdir(2) phase to fail before performing any
non-reversible operation?
Alternatively, should packages also support a "partial/broken" state,
where metadata would be updated to reflect actual remaining files after
an incomplete removal? Possibly keeping the original PLIST for
reference but adding a PARTIAL-PLIST for such packages?
Or should a transactional/atomic system be expected instead?
In which case I could see various imperfect solutions (FFS not
helping much here):
- Obtain a global exclusive pkg advisory lock
- Backup the package files and attempt to delete the original files; if
everything is successful, delete the backup and metadata, otherwise
restore the deleted files from the backup and preserve the metadata.
This should take into account file permissions, flags, PaX flags, etc.
As drawbacks, some space would be required to store the temporary
backup, and some effort made to sync all file permissions and flags,
code which might be duplicated among the various backup utilities
supporting some or all of these flags (not to mention the OS-specifics).
Perhaps acceptable with a dependency on pax(1)?
- Instead of deleting files, move them to a temporary "trashcan"
directory tree on the same filesystem (on write failure, fail
immediately); if all files were successfully moved, ensure that their
permissions allow deletion and recursively delete the temporary tree,
then the metadata.
On error move successfully-moved files back where they belong,
restoring their permissions, and leave the metadata as-is, since we
restored the tree state. Seems flaky; I'm not sure this would
succeed in all cases where package removal now fails. We also could
end up with a growing trash directory if files could be moved yet could
not be deleted.
- fsync(2) and release the pkg lock
Unfortunately, even the above solutions might yield inconsistent state
in the advent of a crash or user interrupt. NetBSD doesn't have
transactional FS and support syscalls, and even if it did this might
not exist on most other pkgsrc-supported OSs...
Other more far-fetched "atomic" support built on non-transactional FS:
I guess that a variant could be using a NetBSD-supported transactional
"trashcan" per file system along with its atomic-simulated
auto-restore/async-delete system. Such a trashcan might include two
sections, one for user-deleted files and another for system-deleted
files. Files would be stored with their original path/flags.
User-deleted files would require explicit "empty/restore" commands, yet
files destined for system-removal would be enclosed in transactions and
could be restored at crash/reboot in case of an incomplete transaction,
with a thread/daemon responsible for asynchroneously deleting files
deleted in complete transactions.
Transactions metadata could be implemented on top of bdb, sqlite or a
custom system supporting a transactional log (using fdatasync(2) at
reasonable intervals and automatic log file rotation/removal, etc).
pkgsrc when on NetBSD could then make use of that "atomic multi-delete
operation", despite it being non-portable. Perhaps that some files
might still be dangling over time and that a nightly script should warn
about it.
A variant of the above might be directly provided by pkgsrc, with
reliance on the required portable tools such as sqlite used for package
metadata and transactional-log, with $PKGDIR/tmp/trash used as a
temporary trash directory, an asynchroneous files cleanup cron event or
daemon, with a boot rollback/recovery script.
Unfortunately complicates pkgsrc, adds new tool dependencies and makes
it more system-invasive... Perhaps that instead of having a
boot-time recovery script and a cleanup cron event/daemon, every pkg_*
command could as a prologue perform those verification/pending
operations as needed, to minimise system invasion, with failed
procedures logged to stderr as a warning before attempting the
requested operation, with some loss of performance...
Other suggestions?
Thanks,
--
Matt
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
Christian Biere <christianbiere@gmx.de>
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
fails)
Date: Thu, 21 Apr 2011 13:27:03 +0200
On Thu, Apr 21, 2011 at 09:50:04AM +0000, Matthew Mondor wrote:
> I doubt it, but could access(2) reliably be used as a first pass before
> the unlink(2)/rmdir(2) phase to fail before performing any
> non-reversible operation?
AFAIK, no. Many file systems don't check for read-only mounts etc. in
this case.
Frankly, I consider this a clear "won't fix".
Joerg
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
fails)
Date: Thu, 21 Apr 2011 17:03:18 -0400
On Thu, 21 Apr 2011 13:27:03 +0200
Joerg Sonnenberger <joerg@britannica.bec.de> wrote:
> Frankly, I consider this a clear "won't fix".
That's the shortest path :) personally, this hasn't been a problem for
me, and I prefer the possibility of dangling pkg files without package
metadata vs a metadata entry with missing files...
Hmm since the submitter can't test anymore, does this bother anyone else
enough to prevent this PR from being switched to closed or analyzed
state?
Thanks,
--
Matt
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
Christian Biere <christianbiere@gmx.de>
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall fails)
Date: Sun, 1 May 2011 04:47:00 +0000
On Thu, Apr 21, 2011 at 09:05:03PM +0000, Matthew Mondor wrote:
> Hmm since the submitter can't test anymore, does this bother anyone else
> enough to prevent this PR from being switched to closed or analyzed
> state?
I dunno, I'd rather that if something belched while it was being
removed we didn't delete the only available record of which particular
wreckage in /usr/pkg belonged to it.
ISTM offhand that it shouldn't be too hard to mark the package as
partially deleted in the metadata first, delete the contents, then
delete the metadata.
--
David A. Holland
dholland@netbsd.org
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.