Writing eBPF Probes in C for Advanced Kubernetes Observability

Traditional Kubernetes observability usually relies on sidecar proxies or application-level tracing, both of which introduce CPU and memory overhead. eBPF avoids this by running sandboxed C programs directly inside the Linux kernel, enabling zero-instrumentation tracing for syscalls, network events, and file platform operations.

Tracing TCP Connect Latency

Standard pod metrics usually miss what happens deep in the networking stack. By tracing tcp_v4_connect, we can measure exact TCP connection setup times across nodes without modifying application code or container images.

The eBPF C Program

The probe below is written in restricted C and compiled into eBPF bytecode using Clang/LLVM.

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
// Define a BPF map to store the start time of the connect call, keyed by PID
struct {
 __uint(type, BPF_MAP_TYPE_HASH);
 __uint(max_entries, 10240);
 __type(key, u32);
 __type(value, u64);
} start SEC(".maps");
// Kprobe triggered on entry to tcp_v4_connect
SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_enter, struct sock *sk)
{
 u64 ts = bpf_ktime_get_ns();
 u32 pid = bpf_get_current_pid_tgid() >> 32;
bpf_map_update_elem(&start, &pid, &ts, BPF_ANY);
 return 0;
}
// Kretprobe triggered on exit of tcp_v4_connect
SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(tcp_v4_connect_exit, int ret)
{
 u32 pid = bpf_get_current_pid_tgid() >> 32;
 u64 *tsp, delta_us;
// Lookup the start time
 tsp = bpf_map_lookup_elem(&start, &pid);
 if (!tsp) {
 return 0; // Missed entry
 }
// Calculate latency in microseconds
 delta_us = (bpf_ktime_get_ns() - *tsp) / 1000;
// bpf_printk logs to /sys/kernel/debug/tracing/trace_pipe
 bpf_printk("PID %d TCP connect took %llu us, ret = %d\n", pid, delta_us, ret);
bpf_map_delete_elem(&start, &pid);
 return 0;
}
char LICENSE[] SEC("license") = "GPL";

Key Mechanics & Kernel Integration

• #include : Auto-generated from kernel BTF (BPF Type Format). It provides definitions for internal kernel structs without needing multiple kernel header files.
• BPF_MAP_TYPE_HASH: eBPF programs cannot leverage global variables for dynamic state. Hash maps allow entry and exit probes to share timestamps across process IDs.
• bpf_ktime_get_ns(): Fetches high-resolution monotonic timestamps from the kernel clock.
• bpf_get_current_pid_tgid(): Returns the thread group ID in the upper 32 bits and thread ID in the lower 32 bits. Right-shifting by 32 extracts the user-space PID.
• SEC("kprobe/...") / SEC("kretprobe/..."): Macros defining ELF section names. Loaders like libbpf read these to attach probes to kernel functions.

Deploying this probe via a Kubernetes DaemonSet alongside a user-space agent (for example, written in Go using cilium/ebpf) provides low-overhead network latency tracing across all pods on a node.