What's the Delay? Understanding Latency Across the Network

Network latency directly affects the performance of many applications that run over the Internet. While significant effort is spent on reducing network latency, the fundamental capability to observe latency remains limited in many network environments. Network operators who rely on active measurements do not get a complete view of the latency encountered by ordinary application traffic, while the coarse-grained measurements frequently used by network researchers fail to capture the rapid latency fluctuations in emerging wireless networks. In this thesis, we explore the potential of technologies like eBPF and hardware timestamps to address the limited observability of latency in modern networks. To this end, we use eBPF to design two lightweight in-kernel passive monitoring solutions that network operators can use to monitor the end-to-end network latency and track the latency within the local endhost network stack. Through deployments in an Internet service provider network and across servers in a global content distribution network, we demonstrate the feasibility of continuously monitoring the latency in production. Our analysis of the monitoring data highlights that last-mile access remains a significant source of latency in end-users' Internet connections, and shows how large latency spikes can occur already in the early parts of web servers' packet processing pipeline. Additionally, we develop methodologies that combine high-frequency active measurements with accurate hardware timestamps to gain a more detailed understanding of the latency characteristics in wireless networks. Using our high-fidelity measurements to study Starlink, we reveal how its link-layer scheduling and apparent use of front-drop queueing impact latency. By studying latency in various network environments, we thus advance our understanding of network latency, and by designing new tools to measure latency, we provide a foundation for future research on network latency.