NetBSD Problem Report #52793

From martin@duskware.de  Fri Dec  8 16:27:30 2017
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id AF7867A177
	for <gnats-bugs@gnats.NetBSD.org>; Fri,  8 Dec 2017 16:27:30 +0000 (UTC)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: Firefox cores at exit in mesa callback
X-Send-Pr-Version: 3.95

>Number:         52793
>Category:       lib
>Synopsis:       Firefox cores at exit in mesa callback
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Dec 08 16:30:00 +0000 2017
>Closed-Date:    Sun Aug 18 09:05:30 +0000 2019
>Last-Modified:  Sun Aug 18 09:05:30 +0000 2019
>Originator:     Martin Husemann
>Release:        NetBSD 8.99.9
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD night-owl.duskware.de 8.99.9 NetBSD 8.99.9 (NIGHT-OWL) #562: Thu Dec 7 12:00:12 CET 2017 martin@night-owl.duskware.de:/usr/src/sys/arch/amd64/compile/NIGHT-OWL amd64
Architecture: x86_64
Machine: amd64
>Description:

With -current and firefox-57.0.1:
 - start firefox
 - quit firefox

notice there is a core file in your home directory.

Reading symbols from firefox...(no debugging symbols found)...done.
[New process 1]
[New process 2]
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f7fda01010a in ?? ()
[Current thread is 1 (process 1)]
(gdb) bt
#0  0x00007f7fda01010a in ?? ()
#1  0x00007f7fd9266e6d in _mesa_map_function_spec (spec=0x7f7fd95a910b "", 
    spec@entry=0x7f7fd95a90d2 "iiiiiip")
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/remap.c:109
#2  0x00007f7fd9266fb9 in _mesa_do_init_remap_table (size=747, 
    remap=0x7f7fd95a3360, pool=<optimized out>)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/remap.c:203
#3  _mesa_init_remap_table ()
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/remap.c:217
#4  0x00007f7fd92c7a65 in one_time_init (ctx=0x7f7ff6f59000)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/context.c:423
#5  _mesa_initialize_context (ctx=ctx@entry=0x7f7ff6f59000, 
    api=api@entry=API_OPENGL_COMPAT, visual=visual@entry=0x7f7fffffd7d0, 
    share_list=share_list@entry=0x0, 
    driverFunctions=driverFunctions@entry=0x7f7fffffd210)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/context.c:1104
#6  0x00007f7fd92c835e in _mesa_create_context (
    api=api@entry=API_OPENGL_COMPAT, visual=visual@entry=0x7f7fffffd7d0, 
    share_list=share_list@entry=0x0, 
    driverFunctions=driverFunctions@entry=0x7f7fffffd210)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/context.c:1225
#7  0x00007f7fd91772d3 in st_create_context (api=api@entry=API_OPENGL_COMPAT, 
    pipe=pipe@entry=0x7f7ff7341000, visual=visual@entry=0x7f7fffffd7d0, 
    share=share@entry=0x0, options=options@entry=0x7f7fffffd908)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/state_tracker/st_context.c:266
#8  0x00007f7fd9174582 in st_api_create_context (stapi=<optimized out>, 
    smapi=0x7f7ff731a1f0, attribs=0x7f7fffffd8e0, error=0x7f7fffffd8dc, 
    shared_stctxi=0x0)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/state_tracker/st_manager.c:665
#9  0x00007f7fd90be022 in dri_create_context (api=<optimized out>, 
    visual=0x7f7ff731cbe0, cPriv=0x7f7ff7312bf0, 
    major_version=<optimized out>, minor_version=<optimized out>, 
    flags=<optimized out>, notify_reset=false, error=0x7f7fffffd9fc, 
    sharedContextPrivate=0x0)
    at /usr/xsrc/external/mit/MesaLib/dist/src/gallium/state_trackers/dri/dri_context.c:112
#10 0x00007f7fd90c4c16 in driCreateContextAttribs (screen=0x7f7ff731a140, 
    api=api@entry=0, config=0x7f7ff731cbe0, shared=<optimized out>, 
    num_attribs=num_attribs@entry=0, attribs=attribs@entry=0x0, 
    error=error@entry=0x7f7fffffd9fc, data=0x7f7ff73370b0)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/drivers/dri/common/dri_util.c:435
#11 0x00007f7fd90c4d2f in driCreateNewContextForAPI (data=<optimized out>, 
    shared=<optimized out>, config=<optimized out>, api=0, 
    screen=<optimized out>)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/drivers/dri/common/dri_util.c:464
#12 driCreateNewContext (screen=<optimized out>, config=<optimized out>, 
    shared=<optimized out>, data=<optimized out>)
    at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/drivers/dri/common/dri_util.c:472
#13 0x00007f7fdbc58b02 in dri2_create_context (base=0x7f7ff7333040, 
    config_base=0x7f7ff6f09c60, shareList=<optimized out>, renderType=32788)
    at /usr/xsrc/external/mit/MesaLib/dist/src/glx/dri2_glx.c:223
#14 0x00007f7fdbc52871 in CreateContext (dpy=dpy@entry=0x7f7ff7301000, 
    generic_id=520, config=0x7f7ff6f09c60, 
    shareList_user=shareList_user@entry=0x0, allowDirect=allowDirect@entry=1, 
    code=code@entry=3, renderType=32788, screen=0)
    at /usr/xsrc/external/mit/MesaLib/dist/src/glx/glxcmds.c:299
#15 0x00007f7fdbc52b2e in glXCreateContext (dpy=0x7f7ff7301000, 
    vis=0x7f7ff7393c00, shareList=0x0, allowDirect=1)
    at /usr/xsrc/external/mit/MesaLib/dist/src/glx/glxcmds.c:430
#16 0x00007f7fe6adb906 in ?? () from /usr/pkg/lib/firefox/libxul.so
#17 0x00007f7fe6adbb2f in ?? () from /usr/pkg/lib/firefox/libxul.so
#18 0x00007f7fe6ad3345 in ?? () from /usr/pkg/lib/firefox/libxul.so
#19 0x00007f7fe6ad7651 in ?? () from /usr/pkg/lib/firefox/libxul.so
#20 0x00007f7fe6ad7d8c in ?? () from /usr/pkg/lib/firefox/libxul.so
...

>How-To-Repeat:
s/a

>Fix:
n/a

>Release-Note:

>Audit-Trail:
From: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/52793: Firefox cores at exit in mesa callback
Date: Fri, 8 Dec 2017 20:17:14 +0000

 I could reproduce it, but not reliably. the one backtrace I got might be
 helpful.

 Thread 1 received signal SIGSEGV, Segmentation fault.
 0x00007aa80a2b6216 in ?? () from /usr/lib/libc.so.12
 (gdb) bt
 #0  0x00007aa80a2b6216 in ?? () from /usr/lib/libc.so.12
 #1  0x00007aa80a2b779e in free () from /usr/lib/libc.so.12
 #2  0x00007aa7e7726e3a in _mesa_glsl_release_builtin_functions() () from /usr/X11R7/lib/modules/dri/nouveau_dri.so
 #3  0x00007aa7e7b02953 in _mesa_destroy_shader_compiler () from /usr/X11R7/lib/modules/dri/nouveau_dri.so
 #4  0x00007aa7e7663349 in ?? () from /usr/X11R7/lib/modules/dri/nouveau_dri.so
 #5  0x00007aa80b5286c0 in ?? ()
 #6  0x00007aa7e7b10129 in _fini () from /usr/X11R7/lib/modules/dri/nouveau_dri.so
 #7  0x0000000000000000 in ?? ()

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/52793: Firefox cores at exit in mesa callback
Date: Sat, 9 Dec 2017 15:19:59 +0100

 A bit more gdb output:

    0x7f7fd9266e47 <_mesa_map_function_spec+77>: mov    %rbx,(%r12)
    0x7f7fd9266e4b <_mesa_map_function_spec+81>: add    $0x1,%ebp
    0x7f7fd9266e4e <_mesa_map_function_spec+84>: add    $0x8,%r12
    0x7f7fd9266e52 <_mesa_map_function_spec+88>: cmp    $0x10,%ebp
    0x7f7fd9266e55 <_mesa_map_function_spec+91>: 
     jne    0x7f7fd9266e35 <_mesa_map_function_spec+59>
    0x7f7fd9266e57 <_mesa_map_function_spec+93>: movslq %ebp,%rbp
    0x7f7fd9266e5a <_mesa_map_function_spec+96>: movq   $0x0,(%rsp,%rbp,8)
    0x7f7fd9266e62 <_mesa_map_function_spec+104>:        mov    %r13,%rsi
    0x7f7fd9266e65 <_mesa_map_function_spec+107>:        mov    %r14,%rdi
    0x7f7fd9266e68 <_mesa_map_function_spec+110>:        
     callq  0x7f7fd9062e30 <_glapi_add_dispatch@plt>
 (gdb) 
 => 0x7f7fd9266e6d <_mesa_map_function_spec+115>:        add    $0x90,%rsp
    0x7f7fd9266e74 <_mesa_map_function_spec+122>:        pop    %rbx
    0x7f7fd9266e75 <_mesa_map_function_spec+123>:        pop    %rbp
    0x7f7fd9266e76 <_mesa_map_function_spec+124>:        pop    %r12
    0x7f7fd9266e78 <_mesa_map_function_spec+126>:        pop    %r13
    0x7f7fd9266e7a <_mesa_map_function_spec+128>:        pop    %r14
    0x7f7fd9266e7c <_mesa_map_function_spec+130>:        retq   

 rax            0x19     25
 rbx            0x7f7fd95a910b   140187084165387
 rcx            0x96969696ffffbdad       -7595718146229551699
 rdx            0x2      2
 rsi            0x7f7fd95a90d2   140187084165330
 rdi            0x7f7fffffcc50   140187732528208
 rbp            0x2      0x2
 rsp            0x7f7fffffcc50   0x7f7fffffcc50
 r8             0x101010101010101        72340172838076673
 r9             0x8080808080808080       -9187201950435737472
 r10            0xffffffffffffffff       -1
 r11            0xdfffffffffffffff       -2305843009213693953
 r12            0x7f7fffffcc60   140187732528224
 r13            0x7f7fd95a90d2   140187084165330
 r14            0x7f7fffffcc50   140187732528208
 r15            0x0      0
 rip            0x7f7fd9266e6d   0x7f7fd9266e6d <_mesa_map_function_spec+115>
 eflags         0x10246  [ PF ZF IF RF ]
 cs             0x47     71
 ss             0x3f     63
 ds             0x23     35
 es             0x23     35
 fs             0x0      0
 gs             0x0      0
 (gdb) up
 #2  0x00007f7fd9266fb9 in _mesa_do_init_remap_table (size=747, 
     remap=0x7f7fd95a3360, pool=<optimized out>)
     at /usr/xsrc/external/mit/MesaLib/dist/src/mesa/main/remap.c:203
 203           offset = _mesa_map_function_spec(spec);
 (gdb) p spec
 $3 = 0x7f7fd95a90d2 "iiiiiip"
 (gdb) list
 198     
 199           /* sanity check */
 200           ASSERT(i == remap[i].remap_index);
 201           spec = _mesa_function_pool + remap[i].pool_index;
 202     
 203           offset = _mesa_map_function_spec(spec);
 204           /* store the dispatch offset in the remap table */
 205           driDispatchRemapTable[i] = offset;
 206           if (offset < 0) {
 207              const char *name = spec + strlen(spec) + 1;

 The firefox code this comes from is:

 298     /** \returns true in the child glxtest process, false in the parent process */
 299     bool fire_glxtest_process()
 300     {
 301       int pfd[2];
 302       if (pipe(pfd) == -1) {
 303           perror("pipe");
 304           return false;
 305       }
 306     #if MOZ_WIDGET_GTK == 2
 307       int gtkpfd[2];
 308       if (pipe(gtkpfd) == -1) {
 309           perror("pipe");
 310           return false;
 311       }
 312     #endif
 313       pid_t pid = fork();
 314       if (pid < 0) {
 315           perror("fork");
 316           close(pfd[0]);
 317           close(pfd[1]);
 318     #if MOZ_WIDGET_GTK == 2
 319           close(gtkpfd[0]);
 320           close(gtkpfd[1]);
 321     #endif
 322           return false;
 323       }
 324       // The child exits early to avoid running the full shutdown sequence and avoid conflicting with threads
 325       // we have already spawned (like the profiler).
 326       if (pid == 0) {
 327           close(pfd[0]);
 328           write_end_of_the_pipe = pfd[1];
 329     #if MOZ_WIDGET_GTK == 2
 330           close(gtkpfd[0]);
 331           gtk_write_end_of_the_pipe = gtkpfd[1];
 332     #endif
 333           glxtest();
 334           close(pfd[1]);
 335     #if MOZ_WIDGET_GTK == 2
 336           close(gtkpfd[1]);
 337     #endif
 338           _exit(0);
 339       }
 340     
 341       close(pfd[1]);
 342       mozilla::widget::glxtest_pipe = pfd[0];
 343       mozilla::widget::glxtest_pid = pid;
 344     #if MOZ_WIDGET_GTK == 2
 345       close(gtkpfd[1]);
 346       gtk_read_end_of_the_pipe = gtkpfd[0];
 347     #endif
 348       return false;
 349     }

 and we are in glxtest() at line 333, which is:

 122     void glxtest()
 123     {
 124       // we want to redirect to /dev/null stdout, stderr, and while we're at it,
 125       // any PR logging file descriptors. To that effect, we redirect all positive
 126       // file descriptors up to what open() returns here. In particular, 1 is stdout and 2 is stderr.
 127       int fd = open("/dev/null", O_WRONLY);
 128       for (int i = 1; i < fd; i++)
 129         dup2(fd, i);
 130       close(fd);
 131     
 132     #if MOZ_WIDGET_GTK == 2
 133       // On Gtk+2 builds, try to get the Gtk+3 version if it's installed, and
 134       // use that in nsSystemInfo for secondaryLibrary. Better safe than sorry,
 135       // we want to load the Gtk+3 library in a subprocess, and since we already
 136       // have such a subprocess for the GLX test, we piggy back on it.
 137       void *gtk3 = dlopen("libgtk-3.so.0", RTLD_LOCAL | RTLD_LAZY);
 138       if (gtk3) {
 139         auto gtk_get_major_version = reinterpret_cast<guint (*)(void)>(
 140           dlsym(gtk3, "gtk_get_major_version"));
 141         auto gtk_get_minor_version = reinterpret_cast<guint (*)(void)>(
 142           dlsym(gtk3, "gtk_get_minor_version"));
 143         auto gtk_get_micro_version = reinterpret_cast<guint (*)(void)>(
 144           dlsym(gtk3, "gtk_get_micro_version"));
 145     
 146         if (gtk_get_major_version && gtk_get_minor_version &&
 147             gtk_get_micro_version) {
 148           // 64 bytes is going to be well enough for "GTK " followed by 3 integers
 149           // separated with dots.
 150           char gtkver[64];
 151           int len = snprintf(gtkver, sizeof(gtkver), "GTK %u.%u.%u",
 152                              gtk_get_major_version(), gtk_get_minor_version(),
 153                              gtk_get_micro_version());
 154           if (len > 0 && size_t(len) < sizeof(gtkver)) {
 155             mozilla::Unused << write(gtk_write_end_of_the_pipe, gtkver, len);
 156           }
 157         }
 158       }
 159     #endif
 160     
 161     
 162       if (getenv("MOZ_AVOID_OPENGL_ALTOGETHER"))
 163         fatal_error("The MOZ_AVOID_OPENGL_ALTOGETHER environment variable is defined");
 164     
 165       ///// Open libGL and load needed symbols /////
 166     #if defined(__OpenBSD__) || defined(__NetBSD__)
 167       #define LIBGL_FILENAME "libGL.so"
 168     #else
 169       #define LIBGL_FILENAME "libGL.so.1"
 170     #endif
 171       void *libgl = dlopen(LIBGL_FILENAME, RTLD_LAZY);
 172       if (!libgl)
 173         fatal_error("Unable to load " LIBGL_FILENAME);
 174     
 175       typedef void* (* PFNGLXGETPROCADDRESS) (const char *);
 176       PFNGLXGETPROCADDRESS glXGetProcAddress = cast<PFNGLXGETPROCADDRESS>(dlsym(libgl, "glXGetProcAddress"));
 177     
 178       if (!glXGetProcAddress)
 179         fatal_error("Unable to find glXGetProcAddress in " LIBGL_FILENAME);
 180     
 181       typedef GLXFBConfig* (* PFNGLXQUERYEXTENSION) (Display *, int *, int *);
 182       PFNGLXQUERYEXTENSION glXQueryExtension = cast<PFNGLXQUERYEXTENSION>(glXGetProcAddress("glXQueryExtension"));
 183     
 184       typedef GLXFBConfig* (* PFNGLXQUERYVERSION) (Display *, int *, int *);
 185       PFNGLXQUERYVERSION glXQueryVersion = cast<PFNGLXQUERYVERSION>(dlsym(libgl, "glXQueryVersion"));
 186     
 187       typedef XVisualInfo* (* PFNGLXCHOOSEVISUAL) (Display *, int, int *);
 188       PFNGLXCHOOSEVISUAL glXChooseVisual = cast<PFNGLXCHOOSEVISUAL>(glXGetProcAddress("glXChooseVisual"));
 189     
 190       typedef GLXContext (* PFNGLXCREATECONTEXT) (Display *, XVisualInfo *, GLXContext, Bool);
 191       PFNGLXCREATECONTEXT glXCreateContext = cast<PFNGLXCREATECONTEXT>(glXGetProcAddress("glXCreateContext"));
 192     
 193       typedef Bool (* PFNGLXMAKECURRENT) (Display*, GLXDrawable, GLXContext);
 194       PFNGLXMAKECURRENT glXMakeCurrent = cast<PFNGLXMAKECURRENT>(glXGetProcAddress("glXMakeCurrent"));
 195     
 196       typedef void (* PFNGLXDESTROYCONTEXT) (Display*, GLXContext);
 197       PFNGLXDESTROYCONTEXT glXDestroyContext = cast<PFNGLXDESTROYCONTEXT>(glXGetProcAddress("glXDestroyContext"));
 198     
 199       typedef GLubyte* (* PFNGLGETSTRING) (GLenum);
 200       PFNGLGETSTRING glGetString = cast<PFNGLGETSTRING>(glXGetProcAddress("glGetString"));
 201     
 202       if (!glXQueryExtension ||
 203           !glXQueryVersion ||
 204           !glXChooseVisual ||
 205           !glXCreateContext ||
 206           !glXMakeCurrent ||
 207           !glXDestroyContext ||
 208           !glGetString)
 209       {
 210         fatal_error("glXGetProcAddress couldn't find required functions");
 211       }
 212       ///// Open a connection to the X server /////
 213       Display *dpy = XOpenDisplay(nullptr);
 214       if (!dpy)
 215         fatal_error("Unable to open a connection to the X server");
 216     
 217       ///// Check that the GLX extension is present /////
 218       if (!glXQueryExtension(dpy, nullptr, nullptr))
 219         fatal_error("GLX extension missing");
 220     
 221       XSetErrorHandler(x_error_handler);
 222     
 223       ///// Get a visual /////
 224        int attribs[] = {
 225           GLX_RGBA,
 226           GLX_RED_SIZE, 1,
 227           GLX_GREEN_SIZE, 1,
 228           GLX_BLUE_SIZE, 1,
 229           None };
 230       XVisualInfo *vInfo = glXChooseVisual(dpy, DefaultScreen(dpy), attribs);
 231       if (!vInfo)
 232         fatal_error("No visuals found");
 233     
 234       // using a X11 Window instead of a GLXPixmap does not crash
 235       // fglrx in indirect rendering. bug 680644
 236       Window window;
 237       XSetWindowAttributes swa;
 238       swa.colormap = XCreateColormap(dpy, RootWindow(dpy, vInfo->screen),
 239                                      vInfo->visual, AllocNone);
 240     
 241       swa.border_pixel = 0;
 242       window = XCreateWindow(dpy, RootWindow(dpy, vInfo->screen),
 243                            0, 0, 16, 16,
 244                            0, vInfo->depth, InputOutput, vInfo->visual,
 245                            CWBorderPixel | CWColormap, &swa);
 246     
 247       ///// Get a GL context and make it current //////
 248       GLXContext context = glXCreateContext(dpy, vInfo, nullptr, True);
 249       glXMakeCurrent(dpy, window, context);
 250     
 251       ///// Look for this symbol to determine texture_from_pixmap support /////
 252       void* glXBindTexImageEXT = glXGetProcAddress("glXBindTexImageEXT");
 253     
 254       ///// Get GL vendor/renderer/versions strings /////
 255       enum { bufsize = 1024 };
 256       char buf[bufsize];
 257       const GLubyte *vendorString = glGetString(GL_VENDOR);
 258       const GLubyte *rendererString = glGetString(GL_RENDERER);
 259       const GLubyte *versionString = glGetString(GL_VERSION);
 260     
 261       if (!vendorString || !rendererString || !versionString)
 262         fatal_error("glGetString returned null");
 263     
 264       int length = snprintf(buf, bufsize,
 265                             "VENDOR\n%s\nRENDERER\n%s\nVERSION\n%s\nTFP\n%s\n",
 266                             vendorString,
 267                             rendererString,
 268                             versionString,
 269                             glXBindTexImageEXT ? "TRUE" : "FALSE");
 270       if (length >= bufsize)
 271         fatal_error("GL strings length too large for buffer size");
 272     
 273       ///// Clean up. Indeed, the parent process might fail to kill us (e.g. if it doesn't need to check GL info)
 274       ///// so we might be staying alive for longer than expected, so it's important to consume as little memory as
 275       ///// possible. Also we want to check that we're able to do that too without generating X errors.
 276       glXMakeCurrent(dpy, None, nullptr); // must release the GL context before destroying it
 277       glXDestroyContext(dpy, context);
 278       XDestroyWindow(dpy, window);
 279       XFreeColormap(dpy, swa.colormap);
 280     
 281     #ifdef NS_FREE_PERMANENT_DATA // conditionally defined in nscore.h, don't forget to #include it above
 282       XCloseDisplay(dpy);
 283     #else
 284       // This XSync call wanted to be instead:
 285       //   XCloseDisplay(dpy);
 286       // but this can cause 1-minute stalls on certain setups using Nouveau, see bug 973192
 287       XSync(dpy, False);
 288     #endif
 289     
 290       dlclose(libgl);
 291     
 292       ///// Finally write data to the pipe
 293       mozilla::Unused << write(write_end_of_the_pipe, buf, length);
 294     }
 295     
 296     }
 297     

 and we are in the glXCreateContext() call in line 248.

 Martin

State-Changed-From-To: open->feedback
State-Changed-By: maya@NetBSD.org
State-Changed-When: Sun, 18 Aug 2019 06:54:01 +0000
State-Changed-Why:
Is this still an issue after mesa changes? which mesa? I seem to recall it was an issue with atexit in dlopen'd libraries.


State-Changed-From-To: feedback->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sun, 18 Aug 2019 09:05:30 +0000
State-Changed-Why:
confirmed fixed


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.