I think the principal reason for requiring a pipe is so that splice only has to understand N combinations of source/sink pairs, rather than N^2 combinations. Some of the mechanics, like page borrowing and gifting, follow naturally and deliberately from using the pipe as a pivot.

I think the concept remains relatively elegant in the Unix tradition (90+% value by exposing an extremely simple kernel primitive to user space), especially as compared to sendfile, but also io_uring. splice falls apart for non-blocking I/O, though. And at least sendfile has an offset parameter for block-like objects (e.g. files).

io_uring solves all of this by implementing a Go-like work scheduler in the kernel and then exposing a command queue to user space. (Both Go and io_uring use a pool of worker threads that can perform synchronous blocking task, but opportunistically make use of readiness polling or asynchronous completion for specific source/sink objects to minimize the number of active worker threads.) That's elegant in its own way, but it's totally at odds with the traditional Unix kernel/user space architecture.

How does it fall apart for non-blocking I/O?

Also splice supports offsets (to me your comment implied it doesn't).

You're right. It's been awhile since I played around with it, and confused some of the discussion here and on LWN with the splice prototype.

Regarding non-blocking I/O, the biggest issue is splice'ing to or from a regular file I/O can still block, as file I/O in Linux is not asynchronous. For buffered I/O, io_uring gets around this by first checking if an operation would block, and if so scheduling the operation for a worker thread to perform. User space can't implement this performance hack without running into TOCTTOU issues, so you have to choose between possibly blocking, or pessimistically always pushing the operation to a thread pool.

Then there are other issues, IIRC, like the fact that readiness polling on sockets doesn't obey low water marks. Specifically, you can't set SO_SNDLOWAT on Linux; it's fixed at 1 byte. For something like splice this makes it more complex to efficiently track through polling when borrowed/gifted pages have been retired from the send queue. I think the proper method now is polling on timestamps events in the socket error queue, but this is also an indirect method. In short, it's just very difficult to track the state of pages that have been spliced, which can be problematic for splicing from a regular file (e.g. the Samba issues), but especially also for managing vmsplice buffers. In synchronous contexts, and especially simple I/O chains like a command utility implements, these aren't likely to be huge stumbling blocks; but when concurrently juggling many I/O contexts and trying to use your resources as efficiently as possible, they stick out.

I think these and other issues could be fixed or addressed, either with kernel changes or a user space library (or both), but in well over a decade nobody has; and now we have io_uring, all but guaranteeing they will never get fixed.

Thanks for the detailed response!