In modern Kubernetes environments, achieving deep observability into microservices often requires sidecars or heavy instrumentation, which adds latency and overhead. extended Berkeley Packet Filter (eBPF) revolutionizes this by allowing safe, sandboxed execution of C programs directly in the Linux kernel. This enables zero-instrumentation monitoring of network traffic, syscalls, and file system operations.

The Challenge: Tracing TCP Connect Latency

Let's build a highly targeted eBPF probe to monitor TCP connection attempts (tcp_v4_connect). This is crucial for diagnosing network bottlenecks between pods where standard metrics fail to provide granular kernel-level timings.

The eBPF C Program

We write the probe in restricted C, which will be compiled to eBPF bytecode using LLVM/Clang.

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

// Define a BPF map to store the start time of the connect call, keyed by PID
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, u64);
} start SEC(".maps");

// Kprobe triggered on entry to tcp_v4_connect
SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_enter, struct sock *sk)
{
    u64 ts = bpf_ktime_get_ns();
    u32 pid = bpf_get_current_pid_tgid() >> 32;

    bpf_map_update_elem(&start, &pid, &ts, BPF_ANY);
    return 0;
}

// Kretprobe triggered on exit of tcp_v4_connect
SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(tcp_v4_connect_exit, int ret)
{
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *tsp, delta_us;

    // Lookup the start time
    tsp = bpf_map_lookup_elem(&start, &pid);
    if (!tsp) {
        return 0; // Missed entry
    }

    // Calculate latency in microseconds
    delta_us = (bpf_ktime_get_ns() - *tsp) / 1000;
    
    // bpf_printk logs to /sys/kernel/debug/tracing/trace_pipe
    bpf_printk("PID %d TCP connect took %llu us, ret = %d\n", pid, delta_us, ret);

    bpf_map_delete_elem(&start, &pid);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Code Annotations & Kernel Interactions

• #include <vmlinux.h>: Auto-generated from the kernel via BTF (BPF Type Format). It provides all internal kernel structs, avoiding manual header inclusion nightmares.
• BPF_MAP_TYPE_HASH: eBPF programs cannot use global variables for dynamic state. We define a BPF map to safely share state (start timestamps) between the entry and exit probes.
• bpf_ktime_get_ns(): A core eBPF helper that fetches a highly accurate monotonic timestamp from the kernel.
• bpf_get_current_pid_tgid(): Returns the Thread Group ID (PID in user space) in the upper 32 bits, and the Thread ID in the lower 32. We bitshift right 32 times to get the PID.
• SEC("kprobe/..."): These macros define the ELF sections. The loader (like libbpf) reads these sections to attach the program to the correct kernel tracepoints.

By loading this probe onto your Kubernetes nodes using a DaemonSet and user-space agent (e.g., written in Go using cilium/ebpf), you achieve sub-millisecond observability into network latency, independent of the pod's language runtime.