NetBSD Problem Report #51657
From www@NetBSD.org Sat Nov 26 14:06:17 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 4B7597A32B
for <gnats-bugs@gnats.NetBSD.org>; Sat, 26 Nov 2016 14:06:17 +0000 (UTC)
Message-Id: <20161126140616.0FDDB7A33E@mollari.NetBSD.org>
Date: Sat, 26 Nov 2016 14:06:16 +0000 (UTC)
From: oskar.skog.finland@gmail.com
Reply-To: oskar.skog.finland@gmail.com
To: gnats-bugs@NetBSD.org
Subject: Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
X-Send-Pr-Version: www-1.0
>Number: 51657
>Category: pkg
>Synopsis: Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: pkg-manager
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat Nov 26 14:10:00 +0000 2016
>Last-Modified: Mon Aug 14 14:10:01 +0000 2017
>Originator: Oskar Skog
>Release: 6.1
>Organization:
>Environment:
$ uname -a
NetBSD .multi.fi 6.1 NetBSD 6.1 (GENERIC) i386
$ # VirtualBox x86-32, Python is 2.7.10 (2015-09-25)
$ # Not actually copy-pasted (the VM is on :1 while my browser is at :0)
>Description:
If subprocess.Popen() has ever been called, the interpreter will hang after the program has terminated (ie. at EOF, sys.exit() or quit()).
>How-To-Repeat:
#!/usr/pkg/bin/python
import subprocess
subprocess.Popen(['true'])
print('Program finished.')
# sys.exit(0) # also hangs
# quit() # also hangs
# Hangs at EOF
>Fix:
Using os._exit() which will bypass all the Python code and immediately exit the process. But this has the disadvantage that really weird things can happen if you try to pipe the script into something else. (Buffering perhaps?)
>Audit-Trail:
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
to subproccess.Popen()
Date: Sun, 27 Nov 2016 20:17:31 +0000
On Sat, Nov 26, 2016 at 02:10:00PM +0000, oskar.skog.finland@gmail.com wrote:
> If subprocess.Popen() has ever been called, the interpreter will
> hang after the program has terminated (ie. at EOF, sys.exit() or
> quit()).
(a) is this a Python bug? (e.g. does it happen on other OSes?)
(b) what does ps -l show for the hanging process?
(c) if you run the script with ktrace, what are the last few things
that appear when you run kdump to show the trace?
--
David A. Holland
dholland@netbsd.org
From: Oskar Skog <oskar.skog.finland@gmail.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
to subproccess.Popen()
Date: Sun, 27 Nov 2016 22:43:55 +0200
> (a) is this a Python bug? (e.g. does it happen on other OSes?)
It does not happen on any other OSes I have tested on. Those OSes
include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
GNU/Linux distributions.
> (b) what does ps -l show for the hanging process?
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
1000 605 612 27555 27 0 9092 4616 - R ttyE0 4:27.21 /usr/pkg/bin/pyth
command is /usr/pkg/python ./bug
> (c) if you run the script with ktrace, what are the last few things
> that appear when you run kdump to show the trace?
670 1 python2.7 GIO fd 1 wrote 18 bytes
"Program finished.\n"
670 1 python2.7 RET write 18/0x12
670 1 python2.7 CALL
__sigaction_sigtramp(SIGINT,0xbfbfec7c,0xbfbfec64,0xbb9ba1a0,2)
670 1 python2.7 RET __sigaction_sigtramp 0
670 1 python2.7 CALL __wait450(0x297,0xbfbfe588,1,0)
670 1 python2.7 RET __wait450 0
670 1 python2.7 PSIG SIGINT SIG_DFL: code=SI_NOINFO
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org,
pkgsrc-bugs@netbsd.org, oskar.skog.finland@gmail.com
Cc:
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
Date: Mon, 28 Nov 2016 07:57:17 -0500
On Nov 27, 8:45pm, oskar.skog.finland@gmail.com (Oskar Skog) wrote:
-- Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a ca
| The following reply was made to PR pkg/51657; it has been noted by GNATS.
|
| From: Oskar Skog <oskar.skog.finland@gmail.com>
| To: gnats-bugs@netbsd.org
| Cc:
| Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
| to subproccess.Popen()
| Date: Sun, 27 Nov 2016 22:43:55 +0200
|
| > (a) is this a Python bug? (e.g. does it happen on other OSes?)
|
| It does not happen on any other OSes I have tested on. Those OSes
| include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
| distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
| GNU/Linux distributions.
I can't reproduce it in current in bare metal of virtualbox.
christos
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
to subproccess.Popen()
Date: Mon, 28 Nov 2016 19:28:22 +0000
On Mon, Nov 28, 2016 at 01:00:01PM +0000, Christos Zoulas wrote:
> | It does not happen on any other OSes I have tested on. Those OSes
> | include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
> | distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
> | GNU/Linux distributions.
>
> I can't reproduce it in current in bare metal of virtualbox.
I wonder if this is a manifestation of that issue with pty closing.
Did the changes for that get into -6?
--
David A. Holland
dholland@netbsd.org
From: Havard Eidnes <he@NetBSD.org>
To: Oskar.Skog.Finland@gmail.com
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a
call to subproccess.Popen()
Date: Mon, 14 Aug 2017 14:40:58 +0200 (CEST)
Hi,
I'm wondering... We're seeing something which looks a little
similar with use of ansible on NetBSD/amd64 7.0_STABLE from Nov
25 2016; its connection module also uses subproccess.Popen().
What we're seeing is processes stuck in "DE" state, i.e. both
exiting and in short-term IO-wait, partial "ps axlw" output:
1071 22773 1 0 0 0 0 0 - DE pts/70- =
0:00.00 (python2.7)
1071 267 1 0 0 0 0 0 - DE pts/71- =
0:00.00 (python2.7)
1071 5273 1 0 0 0 0 0 - DE pts/71- =
0:00.00 (python2.7)
These processes open pty's for doing IO, and we're also seeing
output stuck on ptys, partial "pstat -t" output:
pts/32 0 22 1024 1216 256 13 OC 0 0 termi=
os
pts/33 0 22 1024 1216 256 14 OC 0 0 termi=
os
pts/34 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/35 0 22 1024 1216 256 14 OC 0 0 termi=
os
pts/36 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/37 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/38 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/39 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/40 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/41 0 22 1024 1216 256 4 OC 0 0 termi=
os
pts/42 0 22 1024 1216 256 4 OC 0 0 termi=
os
...
pts/74 3 0 81 1216 256 3 OC 0 0 termi=
os
pts/75 0 0 95 1216 256 0 OC 0 0 termi=
os
pts/76 0 0 95 1216 256 0 OC 0 0 termi=
os
pts/77 0 0 95 1216 256 0 OC 0 0 termi=
os
pts/78 0 0 95 1216 256 0 OC 0 0 termi=
os
pts/79 0 0 95 1216 256 0 OC 0 0 termi=
os
fstat for the three processes above show:
: {11} fstat -p 22773
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
nn python2.7 22773 wd /home 4101726 drwxr-xr-x 1024 r =
nn python2.7 22773 1* pipe 0xfffffe8aee118968 -> 0x0 w
nn python2.7 22773 2* pipe 0xfffffe8efba98a58 -> 0x0 w
nn python2.7 22773 4 /dev/pts 152 crw-rw-rw- ptc,74 rw
nn python2.7 22773 8* kqueue pending 0
nn python2.7 22773 10 / 62406 -rw-r--r-- 63 r =
nn python2.7 22773 12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
40f0
: {12} fstat -p 267
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
nn python2.7 267 wd /home 4101726 drwxr-xr-x 1024 r =
nn python2.7 267 1* pipe 0xfffffe81011a3a00 -> 0x0 w
nn python2.7 267 2* pipe 0xfffffe8848716698 -> 0x0 w
nn python2.7 267 4 /dev/pts 154 crw-rw-rw- ptc,75 rw
nn python2.7 267 8* kqueue pending 0
nn python2.7 267 10 / 62406 -rw-r--r-- 63 r =
nn python2.7 267 12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
4ec0
: {13} fstat -p 5273
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
nn python2.7 5273 wd /home 4101726 drwxr-xr-x 1024 r =
nn python2.7 5273 1* pipe 0xfffffe84b1001808 -> 0x0 w
nn python2.7 5273 2* pipe 0xfffffe886827b198 -> 0x0 w
nn python2.7 5273 4 /dev/pts 158 crw-rw-rw- ptc,77 rw
nn python2.7 5273 8* kqueue pending 0
nn python2.7 5273 10 / 62406 -rw-r--r-- 63 r =
nn python2.7 5273 12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
44d0
: {14}
Would you care to share your "bug" python script, so I can take a
closer look?
Best regards,
- H=E5vard
From: Havard Eidnes <he@NetBSD.org>
To: Oskar.Skog.Finland@gmail.com
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a
call to subproccess.Popen()
Date: Mon, 14 Aug 2017 16:06:03 +0200 (CEST)
> Would you care to share your "bug" python script, so I can take a
> closer look?
Ignore this part, I now see the "How-To-Repeat" section.
And ... in my case, I also can't reproduce it with that script.
Regards,
- H=E5vard
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.