Why Fewer Syscalls Don't Make io_uring Faster Than epoll

Moving a network service from epoll to io_uring can reduce its system-call count while leaving QPS unchanged. P99 latency can even get worse.

There is no contradiction. A system call is only one line item in an I/O path. Data copies, protocol processing, scheduling, cache locality, application work, and queueing all remain. io_uring changes how applications exchange requests and completions with the kernel; it does not make the rest of that work disappear.

The useful question is therefore not which API is newer. It is whether the cost that io_uring removes is material on your critical path.

Readiness and completion are different contracts

epoll is a readiness interface. For a typical receive path, an event loop:

1. waits in epoll_wait() until one or more sockets become readable;
2. calls read() or recv() on each ready descriptor.

One epoll_wait() can return many events, so the wait cost is already batched. The actual reads and writes still use their own system calls. With edge-triggered operation, the application normally drains a descriptor until it gets EAGAIN.

io_uring uses shared submission and completion rings:

• the application writes submission queue entries, or SQEs, describing operations;
• the kernel writes completion queue entries, or CQEs, with their results.

Preparing an SQE and consuming an existing CQE do not themselves require crossing into the kernel. In the normal non-SQPOLL mode, however, the application generally still calls io_uring_enter() to publish work, wait for completions, or both. Ordinary network receive and send operations still traverse the protocol stack and normally still copy payload data unless a specific zero-copy facility applies.

The core gain is lower coordination overhead and more opportunities to batch—not a blanket promise of zero system calls or zero copies.

Where the syscall reduction comes from

Batch submission and waiting

A single io_uring_enter() can submit multiple SQEs. With IORING_ENTER_GETEVENTS, the same call can also wait for a minimum number of completions. CQEs are then consumed through the shared completion ring.

If an application can accumulate useful batches, the kernel-entry cost is spread across several operations. If the workload is permanently “submit one, wait for one,” there is little to amortize.

Batching also creates a latency trade-off. Waiting to build a larger batch can improve throughput while making the oldest request in that batch wait longer. A throughput-oriented storage engine may accept that. A latency-sensitive RPC loop may not.

Multishot operations and provided buffers

Normal SQEs are one-shot: one submission eventually produces one completion. Multishot operations keep one request armed and can generate multiple CQEs. Linux added multishot poll in 5.13, multishot accept in 5.19, and multishot receive in 6.0.

For a network server, multishot accept avoids resubmitting an accept after every connection. Multishot receive can keep receiving on a socket. As long as a CQE carries IORING_CQE_F_MORE, the application knows that the original operation remains active.

Receive operations can use a provided buffer ring. Instead of binding a dedicated buffer to every outstanding receive, the application supplies a pool and the kernel selects one when data arrives. The CQE identifies the selected buffer, which the application processes and returns to the pool. This is particularly useful when many connections are mostly idle or message sizes vary.

These mechanisms reduce repeated submissions and buffer reservation. They also add state that the application must manage correctly: CQ capacity, buffer replenishment, cancellation, and the final CQE that no longer carries IORING_CQE_F_MORE.

SQPOLL

With IORING_SETUP_SQPOLL, the kernel creates a thread that polls the submission queue. While that thread is active, publishing SQEs does not require an io_uring_enter() for every batch.

The missing system calls are paid for with CPU time. The polling thread consumes scheduling capacity, belongs to the creating process’s cgroup, and may compete with application threads in a CPU-limited container. After the configured idle period it sleeps and sets IORING_SQ_NEED_WAKEUP; the application must enter the kernel to wake it again.

SQPOLL is most convincing under sustained submission load with an explicit CPU budget. Sparse traffic often makes ordinary batched submission the better bargain.

Three larger costs remain

A network service must adopt the completion model

A mature epoll server often has a carefully tuned Reactor: one loop owns a stable connection set, performs parsing and application work, and preserves cache locality. SO_REUSEPORT, CPU affinity, and interrupt placement may already keep most work local.

Replacing epoll_wait() with an io_uring poll request while leaving the rest of that architecture unchanged uses only a small part of the new API. The system-call count may fall, but the application also acquires additional queue and completion state.

A more complete design can combine batching, multishot accept and receive, provided buffers, and an appropriate send path such as IORING_OP_SEND_ZC. Zero-copy send can fall back to copying, and the application must observe notification CQEs before reusing buffers.

Flags such as IORING_SETUP_COOP_TASKRUN and IORING_SETUP_SINGLE_ISSUER can reduce particular synchronization and task-work costs when they match the event-loop design. They are not generic “make it faster” switches. Likewise, registered NAPI busy polling can trade more CPU for lower receive latency; its value depends on the NIC, driver, queue affinity, and load.

File I/O does not follow one execution path

Regular files are not useful epoll targets. They normally appear ready even when the next access may wait for storage, and epoll_ctl() generally rejects them.

io_uring gives file I/O one submission/completion interface, but the kernel can execute different operations differently. A buffered read that hits the page cache may complete immediately. A miss may initiate asynchronous work through the filesystem and block layers. An operation that cannot safely complete inline may be delegated to io-wq.

io-wq prevents the submitting thread from blocking, but worker scheduling and concurrency still cost CPU and can amplify contention. Whether an operation uses a native asynchronous path or a worker depends on the operation, filesystem, cache state, flags, and kernel version.

Deep queues of direct I/O to NVMe are a strong io_uring case. Small reads that mostly hit the page cache may be dominated by copying and application processing, leaving less for the interface to improve. This is also why mmap() versus read() has no universal winner.

Registered resources move work; they do not erase it

Every asynchronous operation needs stable references to files and user memory. io_uring can pre-register both:

• fixed files replace repeated descriptor lookup with indices into a registered table;
• fixed buffers pre-pin and map memory so repeated direct I/O avoids rebuilding that state.

Fixed buffers are not automatically zero-copy. They reduce validation, pinning, and mapping work. The data path still depends on the operation.

Registration also consumes long-lived resources. Pinned pages occupy memory, buffer pools need capacity and lifecycle controls, and fixed file tables must be updated. The smaller and more frequent the I/O, and the more stable the resource reuse, the easier it is to recover that upfront cost.

When epoll remains the right answer

Keep epoll when most of these are true:

• the service is network-heavy and already has a stable Reactor design;
• CPU time is dominated by application logic, serialization, encryption, or copying;
• requests do not form useful batches, or queueing delay is unacceptable;
• the deployment cannot pin a sufficiently new kernel and liburing version;
• profiling does not show epoll_wait(), read(), write(), repeated registration, or mapping work as major costs.

In that situation, improving affinity, buffering, event-loop ownership, and application code is usually more direct than replacing the I/O interface.

When io_uring deserves a prototype

Evaluate io_uring when the workload has one or more of these properties:

• a storage engine needs deep NVMe queues and frequent direct I/O;
• a network server has enough active connections to use batching, multishot operations, and provided buffers;
• one runtime needs to coordinate network, file, timer, and other operations through a completion model;
• profiling shows syscall, repeated submission, resource lookup, or buffer mapping overhead is material;
• the team can control kernel versions and continuously tune queue depth, affinity, and buffer pools.

PostgreSQL 18 illustrates the boundary well. Its asynchronous I/O subsystem can use worker processes, io_uring, or synchronous execution for eligible work. The interface is one execution strategy; the benefit comes from upper-layer scanning, prefetching, and concurrency making use of it.

Benchmark the complete application

A useful comparison records at least:

• kernel, liburing, filesystem, and NIC driver versions;
• request sizes, read/write ratio, active-connection ratio, and bursts;
• ring size, queue depth, batch size, and wait policy;
• SQPOLL, multishot, provided-buffer, fixed-file, fixed-buffer, and zero-copy settings;
• CPU and NUMA affinity for application threads, SQPOLL, interrupts, and storage queues;
• throughput, mean latency, P99/P999, CPU time, syscalls, context switches, and cache misses.

Benchmark the real protocol and application path, not an empty loop. A syscall may dominate a microbenchmark and become noise once parsing, copying, encryption, and business work are restored.

io_uring is valuable because it lowers coordination costs and exposes composable asynchronous operations. It is not necessarily faster than epoll because those costs may not be the bottleneck—and because every optimization introduces its own queueing, CPU, memory, and lifecycle trade-offs.

Fewer system calls prove that one cost fell. A migration succeeds only when throughput, tail latency, and resource consumption improve together.

References

1. Linux man-pages, io_uring(7)
2. Linux man-pages, io_uring_enter(2)
3. Linux man-pages, io_uring_sqpoll(7)
4. Linux man-pages, io_uring_multishot(7)
5. Linux man-pages, io_uring_provided_buffers(7)
6. Linux man-pages, io_uring_registered_buffers(7)
7. Jens Axboe, io_uring and networking in 2023
8. PostgreSQL 18, Asynchronous I/O configuration