NetBSD Problem Report #60782

From www@netbsd.org  Thu Sep 24 14:59:47 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id AE9BA1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 24 Sep 2026 14:59:47 +0000 (UTC)
Message-Id: <20260924145946.5A7821A923C@mollari.NetBSD.org>
Date: Thu, 24 Sep 2026 14:59:46 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: SIGIO says socket ready for send before send does
X-Send-Pr-Version: www-1.0
X-From4GNATS: "campbell+netbsd@mumble.net via gnats" <gnats-admin@NetBSD.org>

>Number:         60782
>Category:       kern
>Synopsis:       SIGIO says socket ready for send before send does
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Sep 24 15:00:00 +0000 2026
>Last-Modified:  Thu Sep 24 22:35:01 +0000 2026
>Originator:     Taylor R Campbell
>Release:        current, 11, 10, ...
>Organization:
The NetBSD Foun*** I/O available
>Environment:
>Description:

	Consider a local socket pair,connecting processes A and B,
	whose buffer is currently full in the direction from A (sender)
	to B (receiver).

	If process A has subscribed to SIGIO on one socket, and process
	B receives a single byte on the peer socket, the system will
	send SIGIO with si_code=POLL_OUT si_band=POLLOUT to process A.

	But if process A then tries to send anything, even a single
	byte, it will block, because the default low water mark of the
	socket is >>1, and send will block until the space available in
	the buffer is at least the low water mark.

	Perhaps we should avoid sending SIGIO notifications claiming
	that the socket is writable when it is not, in fact, writable.

>How-To-Repeat:
#ifdef __linux__
#define _GNU_SOURCE		/* F_SETSIG */
#endif

#include <sys/ioctl.h>
#include <sys/socket.h>

#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <poll.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

struct context {
	sigset_t mask;
	pthread_barrier_t bar;
	int s[2];
};

static void
on_sigio(int signo)
{
	fprintf(stderr, "[signal] %d (%s)\n", signo, strsignal(signo));
}

static void *
start_receiver(void *cookie)
{
	struct context *C = cookie;
	int lowat, sndbuf, rcvbuf, nreadable;
	socklen_t optlen;
	char *buf, *p;
	size_t resid, n;
	ssize_t nrcvd;

	/*
	 * Block SIGIO in this thread -- only the main thread will
	 * process SIGIO.
	 */
	if (pthread_sigmask(SIG_BLOCK, &C->mask, NULL) == -1)
		err(EXIT_FAILURE, "pthread_sigmask(SIG_BLOCK)");

	/*
	 * Get the sender's low water mark.  This is the minimum space
	 * that must be free in order to unblock write operations by
	 * the sender.
	 */
	optlen = sizeof(lowat);
	if (getsockopt(C->s[0], SOL_SOCKET, SO_SNDLOWAT, &lowat, &optlen) ==
	    -1)
		err(EXIT_FAILURE, "getsockopt(SOL_SOCKET, SO_SNDLOWAT)");

	/*
	 * Get the sender's send buffer size and the receiver's receive
	 * buffer size just out of curiosity.
	 */
	optlen = sizeof(sndbuf);
	if (getsockopt(C->s[0], SOL_SOCKET, SO_SNDBUF, &sndbuf, &optlen) ==
	    -1)
		err(EXIT_FAILURE, "getsockopt(SOL_SOCKET, SO_SNDBUF)");
	optlen = sizeof(rcvbuf);
	if (getsockopt(C->s[1], SOL_SOCKET, SO_RCVBUF, &rcvbuf, &optlen) ==
	    -1)
		err(EXIT_FAILURE, "getsockopt(SOL_SOCKET, SO_RCVBUF)");

	fprintf(stderr, "[thread] lowat=%d sndbuf=%d rcvbuf=%d\n",
	    lowat, sndbuf, rcvbuf);

	/*
	 * Wait for the main thread to be ready.
	 */
	(void)pthread_barrier_wait(&C->bar);

	/*
	 * Find exactly how many bytes are readable,
	 */
	if (ioctl(C->s[1], FIONREAD, &nreadable) == -1)
		err(EXIT_FAILURE, "ioctl(FIONREAD)");
	fprintf(stderr, "[thread] FIONREAD %d\n", nreadable);
	if (nreadable <= 0)
		errx(EXIT_FAILURE, "invalid FIONREAD");
	resid = nreadable;

	/*
	 * Allocate a buffer to free enough space up to the low water
	 * mark.
	 */
	buf = malloc(resid);
	if (buf == NULL)
		err(EXIT_FAILURE, "malloc");
	p = buf;

	/*
	 * Receive a single byte.  This is not enough to unblock the
	 * sender, but it may be enough to deliver SIGIO -- oops.
	 */
	fprintf(stderr, "[thread] reading 1\n");
	nrcvd = recv(C->s[1], p, 1, /*flags*/0);
	if (nrcvd == -1)
		err(EXIT_FAILURE, "recv 1");
	fprintf(stderr, "[thread] read %zd\n", nrcvd);
	p += (size_t)nrcvd;
	resid -= (size_t)nrcvd;

	/*
	 * After a delay, receive up to the low water mark if it's
	 * above 1.  This is enough to unblock the sender.
	 */
	if (lowat > 1) {
		sleep(1);
		fprintf(stderr, "[thread] slept\n");
		while ((size_t)nreadable - resid < (size_t)lowat) {
			n = lowat - (nreadable - resid);
			fprintf(stderr, "[thread] reading %zu\n", n);
			nrcvd = recv(C->s[1], p, n, /*flags*/0);
			if (nrcvd == -1)
				err(EXIT_FAILURE, "recv %zu",
				    lowat - (nreadable - resid));
			if (nrcvd == 0)
				errx(EXIT_FAILURE, "eof");
			fprintf(stderr, "[thread] read %zd\n", nrcvd);
			resid -= (size_t)nrcvd;
		}
	}
	fprintf(stderr, "[thread] read to low water mark\n");

	/*
	 * After a delay, receive the rest.
	 */
	sleep(1);
	fprintf(stderr, "[thread] slept again\n");
	while (resid) {
		n = resid;
		if (n > 32768)
			n = 32768;
		fprintf(stderr, "[thread] reading %zu\n", n);
		nrcvd = recv(C->s[1], p, n, /*flags*/0);
		if (nrcvd == -1)
			err(EXIT_FAILURE, "recv %zu", nrcvd);
		if (nrcvd == 0)
			errx(EXIT_FAILURE, "eof");
		fprintf(stderr, "[thread] read %zd\n", nrcvd);
		resid -= (size_t)nrcvd;
		usleep(100000);
	}

	return NULL;
}

int
main(void)
{
	char buf[PIPE_BUF] = {0};
	struct context ctx, *C = &ctx;
	int flags;
	sigset_t omask;
	pthread_t t;
	int signo;
	siginfo_t si;
	ssize_t nsent;
	int error, status = 0;

	/*
	 * Create a socket pair to test.
	 */
	if (socketpair(AF_LOCAL, SOCK_STREAM, 0, C->s) == -1)
		err(EXIT_FAILURE, "socketpair");

	/*
	 * Block SIGIO; we will process it with sigtimedwait.
	 */
	if (sigemptyset(&C->mask) == -1)
		err(EXIT_FAILURE, "sigemptyset");
	if (sigaddset(&C->mask, SIGIO) == -1)
		err(EXIT_FAILURE, "sigaddset");
	if (pthread_sigmask(SIG_BLOCK, &C->mask, &omask) == -1)
		err(EXIT_FAILURE, "pthread_sigmask(SIG_BLOCK)");

	/*
	 * Subscribe to SIGIO from the socket and set up nonblocking
	 * I/O for convenience.
	 *
	 * Set up a signal handler that does nothing just so the
	 * signals aren't discarded before sigtimedwait() can get to
	 * them, because of SIGIO's default disposition to ignore.
	 */
	if (signal(SIGIO, &on_sigio) == SIG_ERR)
		err(EXIT_FAILURE, "signal(SIGIO)");
	if (fcntl(C->s[0], F_SETOWN, getpid()) == -1)
		err(EXIT_FAILURE, "fcntl(F_SETOWN)");
#ifdef __linux__
	/*
	 * Request SIGIO with siginfo_t populated (si_code = POLL_*,
	 * si_fd, si_band).
	 */
	if (fcntl(C->s[0], F_SETSIG, SIGIO) == -1)
		err(EXIT_FAILURE, "fcntl(F_GETSIG, SIGIO)");
#endif
	flags = fcntl(C->s[0], F_GETFL);
	if (flags == -1)
		err(EXIT_FAILURE, "fcntl(F_GETFL)");
	if (fcntl(C->s[0], F_SETFL, flags|O_ASYNC|O_NONBLOCK) == -1)
		err(EXIT_FAILURE, "fcntl(F_SETFL)");

	/*
	 * Fill the socket's buffer and spawn a thread to consume it
	 * from the peer socket C->s[1].
	 */
	while (send(C->s[0], buf, sizeof(buf), 0) != -1)
		continue;
	error = pthread_barrier_init(&C->bar, NULL, 2);
	if (error) {
		errno = error;
		err(EXIT_FAILURE, "pthread_barrier_init");
	}
	error = pthread_create(&t, NULL, &start_receiver, C);
	if (error) {
		errno = error;
		err(EXIT_FAILURE, "pthread_create");
	}

	/*
	 * Wait for the thread to be ready.
	 */
	(void)pthread_barrier_wait(&C->bar);

	/*
	 * Wait until we receive SIGIO.
	 */
	signo = sigtimedwait(&C->mask, &si, &(const struct timespec){3,0});
	if (signo == -1)
		err(EXIT_FAILURE, "sigtimedwait");
	printf("signo=%d (%s) code=%d band=0x%lx\n", signo, strsignal(signo),
	    si.si_code, si.si_band);
	fflush(stdout);
	if (si.si_code != POLL_OUT || (si.si_band & (POLLOUT|POLLWRNORM)) == 0)
		errx(EXIT_FAILURE, "unexpected SIGIO code and band");

	/*
	 * Now that we received SIGIO, we should be ready to write at
	 * least a single byte, right?
	 */
	fprintf(stderr, "trying to send\n");
	nsent = send(C->s[0], buf, 1, 0);
	if (nsent == -1) {
		warn("SIGIO said ready but send didn't");
		status = 1;
	} else {
		fprintf(stderr, "sent %zd\n", nsent);
	}

	/*
	 * Wait for the thread to definitely receive up to the low
	 * water mark, and then try sending a byte again.
	 */
	sleep(2);
	fprintf(stderr, "trying to send again\n");
	nsent = send(C->s[0], buf, 1, 0);
	if (nsent == -1)
		err(EXIT_FAILURE, "SIGIO said ready but send didn't");
	else
		fprintf(stderr, "sent %zd\n", nsent);

	error = pthread_join(t, NULL);
	if (error) {
		errno = error;
		err(EXIT_FAILURE, "pthread_join");
	}
	error = pthread_barrier_destroy(&C->bar);
	if (error) {
		errno = error;
		err(EXIT_FAILURE, "pthread_barrier_destroy");
	}

	return status;
}

>Fix:

	Yes, please!

>Audit-Trail:
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/60782: SIGIO says socket ready for send before send does
Date: Thu, 24 Sep 2026 21:17:18 -0000 (UTC)

 gnats-admin@NetBSD.org ("campbell+netbsd@mumble.net via gnats") writes:

 >>Number:         60782
 >>Category:       kern
 >>Synopsis:       SIGIO says socket ready for send before send does
 >>Confidential:   no
 >>Severity:       serious
 >>Priority:       medium
 >>Responsible:    kern-bug-people
 >>State:          open
 >>Class:          sw-bug
 >>Submitter-Id:   net
 >>Arrival-Date:   Thu Sep 24 15:00:00 +0000 2026
 >>Originator:     Taylor R Campbell
 >>Release:        current, 11, 10, ...
 >>Organization:
 >The NetBSD Foun*** I/O available
 >>Environment:
 >>Description:

 >	Consider a local socket pair,connecting processes A and B,
 >	whose buffer is currently full in the direction from A (sender)
 >	to B (receiver).

 >	If process A has subscribed to SIGIO on one socket, and process
 >	B receives a single byte on the peer socket, the system will
 >	send SIGIO with si_code=POLL_OUT si_band=POLLOUT to process A.

 >	But if process A then tries to send anything, even a single
 >	byte, it will block, because the default low water mark of the
 >	socket is >>1, and send will block until the space available in
 >	the buffer is at least the low water mark.

 >	Perhaps we should avoid sending SIGIO notifications claiming
 >	that the socket is writable when it is not, in fact, writable.



 In unp_rcvd the receiver updates two hiwater marks for the sender:

 mbmax is increased for the memory used by mbufs consumed by the reader:

      snd->sb_mbmax += unp->unp_mbcnt - rcv->sb_mbcnt;
      unp->unp_mbcnt = rcv->sb_mbcnt;

 hiwat is increased for the number of bytes consumed by the reader:

      newhiwat = snd->sb_hiwat + unp->unp_cc - rcv->sb_cc;
      (void)chgsbsize(so2->so_uidinfo,
          &snd->sb_hiwat, newhiwat, RLIM_INFINITY);
      unp->unp_cc = rcv->sb_cc;


 The sender blocks when it tries to write more than there is free space
 in the buffer. But the free space calculation takes both hiwater marks
 into account in sbspace():

         if (sb->sb_hiwat <= sb->sb_cc || sb->sb_mbmax <= sb->sb_mbcnt) 
                 return 0;

 For the test program, the hiwat value doesn't matter. cc on the sender
 side is 0 as previous writes are already in the receive buffer and
 hiwat has been bumped by the read to 1. But the mbuf highwater value
 stays at 0 until the first mbuf has been consumed.

 The first mbuf in the chain contains a header. That's why on amd64 with
 MSIZE=512 you need to read at least 400 bytes (MHLEN) to free the first mbuf.


 Sending SIGIO and waking up a blocked writer is done at the same time,
 but a blocked writer just blocks again without noticable issue.

 poll(2) checks sbspace(sb) >= sb->sb_lowat. So it signals a writable
 descriptor at a different time. If sbspace(sb) > 0 a send smaller
 than sb_lowat may even succeed when, according to poll(2) it's not
 yet possible.


 Maybe changing the test in sbspace to:

 	if (sb->sb_hiwat <= sb->sb_cc)
 		return 0;
 	if (sb->sb_mbmax <= sb->sb_mbcnt) 
 		return 1;

 already helps.

From: Taylor R Campbell <riastradh@NetBSD.org>
To: mlelstv@serpens.de (Michael van Elst)
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/60782: SIGIO says socket ready for send before send does
Date: Thu, 24 Sep 2026 22:31:40 +0000

 > Date: Thu, 24 Sep 2026 21:17:18 -0000 (UTC)
 > From: mlelstv@serpens.de (Michael van Elst)
 > 
 > Sending SIGIO and waking up a blocked writer is done at the same time,
 > but a blocked writer just blocks again without noticable issue.

 I think the basic issue is that the wakeup path in recv(2) that leads
 to SIGIO is simply not conditional on anything about the buffer state.
 I bet nobody noticed this because it doesn't cause an actual return
 from poll -- just another pass through polscan or whatever, which
 might hurt performance but doesn't hurt correctness.

 Perhaps sorwakeup/sowwakeup should just check soreadable/sowritable
 before calling sowakeup.

 > poll(2) checks sbspace(sb) >= sb->sb_lowat. So it signals a writable
 > descriptor at a different time. If sbspace(sb) > 0 a send smaller
 > than sb_lowat may even succeed when, according to poll(2) it's not
 > yet possible.
 > 
 > 
 > Maybe changing the test in sbspace to:
 > 
 > 	if (sb->sb_hiwat <= sb->sb_cc)
 > 		return 0;
 > 	if (sb->sb_mbmax <= sb->sb_mbcnt) 
 > 		return 1;
 > 
 > already helps.

 Why would changing sbspace affect this?  I think it is not used
 anywhere in the decision of whether to send SIGIO or not.  (But maybe
 I missed something -- I only took a cursory skim at this part while
 lifting the SIGIO rock and finding a whole colony of bugs under it!)

Home	PR Database Search