NetBSD Problem Report #51657

From www@NetBSD.org  Sat Nov 26 14:06:17 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 4B7597A32B
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 26 Nov 2016 14:06:17 +0000 (UTC)
Message-Id: <20161126140616.0FDDB7A33E@mollari.NetBSD.org>
Date: Sat, 26 Nov 2016 14:06:16 +0000 (UTC)
From: oskar.skog.finland@gmail.com
Reply-To: oskar.skog.finland@gmail.com
To: gnats-bugs@NetBSD.org
Subject: Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
X-Send-Pr-Version: www-1.0

>Number:         51657
>Category:       pkg
>Synopsis:       Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Nov 26 14:10:00 +0000 2016
>Last-Modified:  Mon Aug 14 14:10:01 +0000 2017
>Originator:     Oskar Skog
>Release:        6.1
>Organization:
>Environment:
$ uname -a
NetBSD .multi.fi 6.1 NetBSD 6.1 (GENERIC) i386
$ # VirtualBox x86-32, Python is 2.7.10 (2015-09-25)
$ # Not actually copy-pasted (the VM is on :1 while my browser is at :0)
>Description:
If subprocess.Popen() has ever been called, the interpreter will hang after the program has terminated (ie. at EOF, sys.exit() or quit()).

>How-To-Repeat:
#!/usr/pkg/bin/python
import subprocess
subprocess.Popen(['true'])
print('Program finished.')
# sys.exit(0) # also hangs
# quit() # also hangs
# Hangs at EOF
>Fix:
Using os._exit() which will bypass all the Python code and immediately exit the process. But this has the disadvantage that really weird things can happen if you try to pipe the script into something else. (Buffering perhaps?)

>Audit-Trail:
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
 to subproccess.Popen()
Date: Sun, 27 Nov 2016 20:17:31 +0000

 On Sat, Nov 26, 2016 at 02:10:00PM +0000, oskar.skog.finland@gmail.com wrote:
  > If subprocess.Popen() has ever been called, the interpreter will
  > hang after the program has terminated (ie. at EOF, sys.exit() or
  > quit()).

 (a) is this a Python bug? (e.g. does it happen on other OSes?)

 (b) what does ps -l show for the hanging process?

 (c) if you run the script with ktrace, what are the last few things
 that appear when you run kdump to show the trace?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Oskar Skog <oskar.skog.finland@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
 to subproccess.Popen()
Date: Sun, 27 Nov 2016 22:43:55 +0200

 > (a) is this a Python bug? (e.g. does it happen on other OSes?)

 It does not happen on any other OSes I have tested on.  Those OSes
 include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
 distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
 GNU/Linux distributions.


 > (b) what does ps -l show for the hanging process?
  UID PID PPID   CPU PRI NI  VSZ  RSS WCHAN STAT TTY      TIME COMMAND
 1000 605  612 27555  27  0 9092 4616 -     R    ttyE0 4:27.21 /usr/pkg/bin/pyth

 command is /usr/pkg/python ./bug


 >  (c) if you run the script with ktrace, what are the last few things
 > that appear when you run kdump to show the trace?

    670      1 python2.7 GIO   fd 1 wrote 18 bytes
        "Program finished.\n"
    670      1 python2.7 RET   write 18/0x12
    670      1 python2.7 CALL
 __sigaction_sigtramp(SIGINT,0xbfbfec7c,0xbfbfec64,0xbb9ba1a0,2)
    670      1 python2.7 RET   __sigaction_sigtramp 0
    670      1 python2.7 CALL  __wait450(0x297,0xbfbfe588,1,0)
    670      1 python2.7 RET   __wait450 0
    670      1 python2.7 PSIG  SIGINT SIG_DFL: code=SI_NOINFO

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org, 
	pkgsrc-bugs@netbsd.org, oskar.skog.finland@gmail.com
Cc: 
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call to subproccess.Popen()
Date: Mon, 28 Nov 2016 07:57:17 -0500

 On Nov 27,  8:45pm, oskar.skog.finland@gmail.com (Oskar Skog) wrote:
 -- Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a ca

 | The following reply was made to PR pkg/51657; it has been noted by GNATS.
 | 
 | From: Oskar Skog <oskar.skog.finland@gmail.com>
 | To: gnats-bugs@netbsd.org
 | Cc: 
 | Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
 |  to subproccess.Popen()
 | Date: Sun, 27 Nov 2016 22:43:55 +0200
 | 
 |  > (a) is this a Python bug? (e.g. does it happen on other OSes?)
 |  
 |  It does not happen on any other OSes I have tested on.  Those OSes
 |  include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
 |  distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
 |  GNU/Linux distributions.

 I can't reproduce it in current in bare metal of virtualbox.

 christos

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a call
 to subproccess.Popen()
Date: Mon, 28 Nov 2016 19:28:22 +0000

 On Mon, Nov 28, 2016 at 01:00:01PM +0000, Christos Zoulas wrote:
  >  |  It does not happen on any other OSes I have tested on.  Those OSes
  >  |  include FreeBSD 9 and 10, OpenBSD 5.8, Minix 3.3, openindiana (illumos
  >  |  distribution (OpenSolaris derivate)), Haiku (BeOS clone) and various
  >  |  GNU/Linux distributions.
  >  
  >  I can't reproduce it in current in bare metal of virtualbox.

 I wonder if this is a manifestation of that issue with pty closing.
 Did the changes for that get into -6?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Havard Eidnes <he@NetBSD.org>
To: Oskar.Skog.Finland@gmail.com
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a
 call to subproccess.Popen()
Date: Mon, 14 Aug 2017 14:40:58 +0200 (CEST)

 Hi,

 I'm wondering...  We're seeing something which looks a little
 similar with use of ansible on NetBSD/amd64 7.0_STABLE from Nov
 25 2016; its connection module also uses subproccess.Popen().

 What we're seeing is processes stuck in "DE" state, i.e. both
 exiting and in short-term IO-wait, partial "ps axlw" output:

 1071 22773     1     0   0  0       0       0 -       DE   pts/70-     =
 0:00.00 (python2.7)
 1071   267     1     0   0  0       0       0 -       DE   pts/71-     =
 0:00.00 (python2.7)
 1071  5273     1     0   0  0       0       0 -       DE   pts/71-     =
 0:00.00 (python2.7)

 These processes open pty's for doing IO, and we're also seeing
 output stuck on ptys, partial "pstat -t" output:

 pts/32   0  22 1024 1216 256     13 OC                    0     0 termi=
 os
 pts/33   0  22 1024 1216 256     14 OC                    0     0 termi=
 os
 pts/34   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/35   0  22 1024 1216 256     14 OC                    0     0 termi=
 os
 pts/36   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/37   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/38   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/39   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/40   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/41   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 pts/42   0  22 1024 1216 256      4 OC                    0     0 termi=
 os
 ...
 pts/74   3   0  81 1216 256       3 OC                    0     0 termi=
 os
 pts/75   0   0  95 1216 256       0 OC                    0     0 termi=
 os
 pts/76   0   0  95 1216 256       0 OC                    0     0 termi=
 os
 pts/77   0   0  95 1216 256       0 OC                    0     0 termi=
 os
 pts/78   0   0  95 1216 256       0 OC                    0     0 termi=
 os
 pts/79   0   0  95 1216 256       0 OC                    0     0 termi=
 os

 fstat for the three processes above show:

 : {11} fstat -p 22773
 USER     CMD          PID   FD MOUNT       INUM MODE         SZ|DV R/W
 nn  python2.7  22773   wd /home    4101726 drwxr-xr-x    1024 r =

 nn  python2.7  22773    1* pipe 0xfffffe8aee118968 -> 0x0 w
 nn  python2.7  22773    2* pipe 0xfffffe8efba98a58 -> 0x0 w
 nn  python2.7  22773    4 /dev/pts     152 crw-rw-rw-   ptc,74 rw
 nn  python2.7  22773    8* kqueue pending 0
 nn  python2.7  22773   10 /          62406 -rw-r--r--      63 r =

 nn  python2.7  22773   12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
 40f0
 : {12} fstat -p 267
 USER     CMD          PID   FD MOUNT       INUM MODE         SZ|DV R/W
 nn  python2.7    267   wd /home    4101726 drwxr-xr-x    1024 r =

 nn  python2.7    267    1* pipe 0xfffffe81011a3a00 -> 0x0 w
 nn  python2.7    267    2* pipe 0xfffffe8848716698 -> 0x0 w
 nn  python2.7    267    4 /dev/pts     154 crw-rw-rw-   ptc,75 rw
 nn  python2.7    267    8* kqueue pending 0
 nn  python2.7    267   10 /          62406 -rw-r--r--      63 r =

 nn  python2.7    267   12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
 4ec0
 : {13} fstat -p 5273
 USER     CMD          PID   FD MOUNT       INUM MODE         SZ|DV R/W
 nn  python2.7   5273   wd /home    4101726 drwxr-xr-x    1024 r =

 nn  python2.7   5273    1* pipe 0xfffffe84b1001808 -> 0x0 w
 nn  python2.7   5273    2* pipe 0xfffffe886827b198 -> 0x0 w
 nn  python2.7   5273    4 /dev/pts     158 crw-rw-rw-   ptc,77 rw
 nn  python2.7   5273    8* kqueue pending 0
 nn  python2.7   5273   10 /          62406 -rw-r--r--      63 r =

 nn  python2.7   5273   12* misc ops=3D0xffffffff80c3c820 0xfffffe8fec80=
 44d0
 : {14}

 Would you care to share your "bug" python script, so I can take a
 closer look?

 Best regards,

 - H=E5vard

From: Havard Eidnes <he@NetBSD.org>
To: Oskar.Skog.Finland@gmail.com
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/51657: Python (2.7): interpreter hangs on exit after a
 call to subproccess.Popen()
Date: Mon, 14 Aug 2017 16:06:03 +0200 (CEST)

 > Would you care to share your "bug" python script, so I can take a
 > closer look?

 Ignore this part, I now see the "How-To-Repeat" section.

 And ... in my case, I also can't reproduce it with that script.

 Regards,

 - H=E5vard

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.