What Actually Happens During a Linux System Call on x86-64?

Calling read(), write(), or connect() looks like calling an ordinary C function: pass arguments, execute code, receive a return value. Under load, the boundary becomes visible. A service can show high sys%, and a profile can contain thousands of stacks entering through entry_SYSCALL_64.

Be precise about what those observations mean. sys% includes all CPU time spent in the kernel—not just the fixed cost of crossing the privilege boundary, but also VFS work, protocol processing, copying, faults, and drivers. Seeing entry_SYSCALL_64 identifies the entrance, not the expensive room further inside.

This article follows the traditional SYSCALL/SYSRET path on x86-64 using Linux 6.12 as the source baseline. Systems with Intel FRED enabled use a different event-delivery architecture, and older kernels differ in dispatch details.

The privilege boundary

x86 defines four privilege rings. Linux normally uses two:

• Ring 3 for applications, which cannot change control registers, disable interrupts, or access supervisor-only pages;
• Ring 0 for the kernel, which can execute privileged instructions and access kernel mappings, while still being constrained by page permissions and mechanisms such as SMEP and SMAP.

User code cannot raise its own privilege. It must enter through a controlled path. A system call is the common path for file, network, memory-management, and process operations.

Some read-only operations avoid the transition. Time queries such as clock_gettime() can often execute through the vDSO, which the kernel maps into user space. We will return to that optimization after walking the normal path.

From int 0x80 to syscall

Early 32-bit x86 Linux commonly entered the kernel with int 0x80. The processor looked up an IDT gate, performed privilege checks, switched stacks through the TSS, pushed user state, loaded the target code segment and instruction pointer, and started the kernel handler.

The 64-bit syscall instruction is a faster contract. During boot, Linux programs model-specific registers:

• MSR_LSTAR contains the address of entry_SYSCALL_64;
• MSR_STAR supplies code-segment values for entry and return;
• MSR_SF_MASK specifies flags to clear on entry.

When user space executes syscall, the CPU saves the next user RIP in RCX, saves RFLAGS in R11, masks selected flags, loads the kernel entry address, and changes privilege. It does not switch to the kernel stack or build a complete register frame. Linux entry code must do that work.

Stage 1: the calling convention

For read(fd, buf, count), the x86-64 syscall ABI puts:

• syscall number 0 in RAX;
• fd, buf, and count in RDI, RSI, and RDX;
• later arguments in R10, R8, and R9.

The fourth argument uses R10, not the C ABI’s RCX, because the instruction uses RCX to preserve the return address.

After syscall, execution begins at entry_SYSCALL_64 in arch/x86/entry/entry_64.S.

Stage 2: make kernel execution safe

At the first instruction, the CPU is privileged but most register values and the stack pointer still describe user space. The entry code must establish a trusted execution environment. A simplified outline is:

swapgs
movq  %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
movq  PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp

/* Build pt_regs on the kernel stack. */
pushq $__USER_DS
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
pushq %r11
pushq $__USER_CS
pushq %rcx
pushq %rax
PUSH_AND_CLEAR_REGS

movq  %rsp, %rdi
movslq %eax, %rsi
call  do_syscall_64

Several distinct costs hide in this outline.

swapgs switches the GS base so the kernel can address per-CPU data. The user stack pointer is saved, then RSP moves to the current task’s kernel stack. The entry path constructs struct pt_regs, preserving the user return state and general-purpose registers.

When Kernel Page-Table Isolation is enabled, SWITCH_TO_KERNEL_CR3 changes from the restricted user page-table view to one containing the full kernel mapping. Without KPTI, that macro does not perform the same page-table transition.

The entry path also expands mitigations according to processor capabilities and boot policy. IBRS_ENTER, return untraining, branch-history clearing, and register clearing do not execute identically on every machine. This is one reason a universal “a syscall costs N nanoseconds” number is misleading.

Stage 3: entry work and dispatch

do_syscall_64() first runs generic entry handling. Depending on the task, that can involve seccomp, ptrace, auditing, and tracepoints. It then validates the syscall number and dispatches to the generated x86-64 syscall code.

Since Linux 6.9, x86-64 dispatch uses generated switch-based functions such as x64_sys_call() rather than forcing the main path through the sys_call_table function-pointer array. The array still exists for facilities that need metadata, but it is no longer the normal dispatch mechanism described by many older diagrams.

For read(), the generated ABI wrapper extracts arguments from pt_regs and enters the VFS path. This is where the system call’s actual work begins: descriptor lookup, access checks, filesystem or socket operations, waiting, copying data, and updating offsets.

On systems with SMAP enabled, the kernel normally cannot access user pages even at Ring 0. Copy helpers temporarily permit that access around operations that move data to or from the user buffer. Calls that never touch a user buffer do not pay that exact cost.

Stage 4: work that must happen before return

Completing the service function is not the same as being ready to execute user code. syscall_exit_to_user_mode() performs one-time syscall exit work and enters the common return-to-user preparation loop.

That loop handles flags that may have been set while the call ran. Important examples include:

• rescheduling when TIF_NEED_RESCHED is set;
• pending signals and notification work;
• task work scheduled for return to user mode;
• uprobes, live patching, and architecture-specific tasks.

The loop disables interrupts and checks again because handling one item can create another. Only when the required work is drained can the architecture-specific exit continue.

A system call does not inherently imply a scheduler context switch. If the same task enters the kernel, completes its work, and returns, no other task ran. A context switch occurs only if the path blocks or the return work invokes the scheduler and another task is selected.

Stage 5: SYSRET when it is safe, IRET when it is not

The kernel checks whether the saved state satisfies the restrictions of the fast return path. Among other conditions:

• RCX and R11 must match the saved RIP and RFLAGS;
• code and stack segments must have the expected user values;
• RIP must be a valid user address;
• flags that SYSRET cannot restore safely must not require the general path.

If the checks fail, Linux uses iretq. If they pass, Linux restores registers, switches through the safe exit stack, changes back to the user page-table view when KPTI is enabled, restores the user GS base, and executes sysretq to resume at the address in RCX.

Why 2018 changed the cost

Meltdown and Spectre disclosures changed the fixed cost of this boundary on many machines.

KPTI maintains a restricted mapping while user code runs. The user view maps the process address space and only the small amount of kernel entry data needed to cross the boundary. The kernel view maps the full kernel address space. A typical syscall therefore switches CR3 on entry and again on exit.

Process-Context Identifiers reduce the damage. With PCID, user and kernel translations have different tags, so changing CR3 does not necessarily flush the complete TLB. Targeted invalidations can still be needed, and processors without suitable support pay a different price.

Spectre-family mitigations add other conditional work around entry, dispatch, and return. Their exact combination depends on microarchitecture, kernel configuration, microcode, and boot parameters such as mitigations= and pti=.

For a tiny call that does almost no service work, these fixed costs can dominate. For a call that performs storage I/O, copies megabytes, or executes a large protocol path, they become a smaller fraction of the total.

vDSO: avoid the boundary when the answer is already available

Some operations need kernel-maintained state but not kernel execution on every call. Linux maps a Virtual Dynamic Shared Object and related data pages into each process:

• [vdso] contains callable code;
• [vvar] exposes read-only timekeeping data maintained by the kernel.

For a supported clock and clock source, libc’s clock_gettime() can read the time base, combine it with a hardware counter, and return entirely in Ring 3. Unsupported clocks or conditions fall back to a real system call.

You can see the mappings with:

grep -E 'vvar|vdso' /proc/self/maps

The older vsyscall mechanism used a fixed virtual address, which was hostile to address randomization. Modern systems commonly retain it only in emulation mode for old binaries.

General I/O cannot be moved into a vDSO because the kernel still must validate resources and perform work. The engineering answer there is usually amortization: buffering, vectorized I/O, batching, or a submission/completion design such as io_uring.

A useful cost model

Split a syscall into four categories:

1. Boundary cost: privilege transition, stack switch, register save/restore, dispatch, and return checks.
2. Conditional protection cost: KPTI page-table switches and processor-specific speculative-execution mitigations.
3. Service cost: VFS, networking, drivers, waiting, validation, and copying.
4. Microarchitectural disruption: instruction/data cache pressure, TLB effects, and lost locality when user execution resumes.

When sys% is high, ask which category is responsible. Tiny unbuffered reads may repeat boundary cost. Futex contention may spend time sleeping and waking. Page faults are exceptions rather than syscalls, even though entry and mitigation machinery can overlap. High-frequency time reads may have missed the vDSO fast path.

Understanding the boundary prevents a common optimization mistake: reducing the number of entries while leaving the expensive work unchanged—or merely moving it somewhere harder to observe.

References

1. Linux 6.12, arch/x86/entry/entry_64.S
2. Linux 6.12, arch/x86/entry/common.c
3. Linux 6.12, kernel/entry/common.c
4. Linux kernel documentation, Page Table Isolation
5. Linux man-pages, vdso(7)
6. Linux man-pages, syscall(2)