We analyse two complementary datasets to quantify the latency variation experienced by internet end-users: (i) a large-scale active measurement dataset (from the Measurement Lab Network Diagnostic Tool) which shed light on long-term trends and regional differences; and (ii) passive measurement data from an access aggregation link which is used to analyse the edge links closest to the user.

The analysis shows that variation in latency is both common and of significant magnitude, with two thirds of samples exceeding 100\,ms of variation. The variation is seen within single connections as well as between connections to the same client. The distribution of experienced latency variation is heavy-tailed, with the most affected clients seeing an order of magnitude larger variation than the least affected. In addition, there are large differences between regions, both within and between continents. Despite consistent improvements in throughput, most regions show no reduction in latency variation over time, and in one region it even increases.

We examine load-induced queueing latency as a possible cause for the variation in latency and find that both datasets readily exhibit symptoms of queueing latency correlated with network load. Additionally, when this queueing latency does occur, it is of significant magnitude, more than 200\,ms in the median. This indicates that load-induced queueing contributes significantly to the overall latency variation.

Several new active queue management (AQM) and hybrid AQM/fairness queueing algorithms have been proposed recently. They seek to ensure low queueing delay and high network goodput without requiring parameter tuning of the algorithms themselves. However, extensive experimental evaluations of these algorithms are still lacking. This paper evaluates a selection of bottleneck queue management schemes in a test-bed representative of residential Internet connections of both symmetrical and asymmetrical bandwidths as well as WiFi. Latency under load and the performance of VoIP and web traffic patterns are evaluated under steady state conditions. Furthermore, the impact of the algorithms on fairness between TCP flows with different RTTs, and also the transient behaviour of the algorithms at flow startup is examined. The results show that while the AQM algorithms can significantly improve steady state performance, they exacerbate TCP flow unfairness. In addition, the evaluated AQMs severely struggle to quickly control queueing latency at flow startup, which can lead to large latency spikes that hurt the perceived performance. The fairness queueing algorithms almost completely alleviate the algorithm performance problems, providing the best balance of low latency and high throughput in the tested scenarios. However, on WiFi the performance of all the tested algorithms is hampered by large amounts of queueing in lower layers of the network stack inducing significant latency outside of the algorithms’ control.

The FQ-CoDel queue management algorithm was recently published as an IETF RFC.It achieves low latency especially for low-volume (or sparse) traffic flowscompeting with bulk flows. However, the exact conditions for when a particularflow is considered to be sparse has not been well-explored.

In this work, we analyse the performance characteristics of the sparse flowoptimisation of FQ-CoDel, formulating the constraints that flows must satisfy tobe considered sparse in a given scenario. We also formulate expressions for theexpected queueing latency for sparse flows.

Then, using a numerical example, we show that for a given link and a given typeof sparse flows (VoIP traffic), the number of sparse flows that a givenbottleneck can service with low sparse flow latency is only dependent on thenumber of backlogged bulk flows at the bottleneck. Furthermore, as long as themaximum number of sparse flows is not exceeded, all sparse flows can expect avery low queueing latency through the bottleneck.

The last several years has seen a renewed interest in smart queue management tocurb excessive network queueing delay, as people have realised the prevalence of bufferbloat in real networks.

However, for an effective deployment at today's last mile connections, animproved queueing algorithm is not enough in itself, as often the bottleneckqueue is situated in legacy systems that cannot be upgraded. In addition,features such as per-user fairness and the ability to de-prioritise backgroundtraffic are often desirable in a home gateway.

In this paper we present Common Applications Kept Enhanced (CAKE), a comprehensive network queue management system designed specifically for homeInternet gateways. CAKE packs several compelling features into an integratedsolution, thus easing deployment. These features include: bandwidth shaping withoverhead compensation for various link layers; reasonable DiffServ handling;improved flow hashing with both per-flow and per-host queueing fairness; andfiltering of TCP ACKs.

Our evaluation shows that these features offer compelling advantages, andthat CAKE has the potential to significantly improve performance of last-mileinternet connections.

With more devices connected, delays and jitter at the WiFi hop become more prevalent, and correct functioning during network congestion becomes more important. However, two important performance issues prevent modern WiFi from reaching its potential: increased latency under load caused by excessive queueing (i.e. bufferbloat) and the 802.11 performance anomaly.

To remedy these issues, we present a novel two-part solution. We design a new queueing scheme that eliminates bufferbloat in the wireless setting. Leveraging this queueing scheme, we then design an airtime fairness scheduler that operates at the access point and doesn't require any changes to clients.

We evaluate our solution using both a theoretical model and experiments in a testbed environment, formulating a suitable analytical model in the process. We show that our solution achieves an order of magnitude reduction in latency under load, large improvements in multi-station throughput, and nearly perfect airtime fairness for both TCP and downstream UDP traffic. Further experiments with application traffic confirm that the solution provides significant performance gains for real-world traffic.We develop a production quality implementation of our solution in the Linux kernel, the platform powering most access points outside of the managed enterprise setting. The implementation has been accepted into the mainline kernel distribution, making it available for deployment on billions of devices running Linux today.

As WiFi grows ever more popular, airtime contention becomes an increasing problem. One way to alleviate this is through network policy enforcement. Unfortunately, WiFi lacks protocol support for configuring policies for its usage, and since network-wide coordination cannot generally be ensured, enforcing policy is challenging. However, as we have shown in previous work, an access point can influence the behaviour of connected devices by changing its scheduling of transmission opportunities, which can be used to achieve airtime fairness. In this work, we show that this mechanism can be extended to successfully enforce airtime usage policies in WiFi networks. We implement this as an extension our previous airtime fairness work, and present PoliFi, the resulting policy enforcement system. Our evaluation shows that PoliFi makes it possible to express a range of useful policies. These include prioritisation of specific devices; balancing groups of devices for sharing between different logical networks or network slices; and limiting groups of devices to implement guest networks or other low-priority services. We also show how these can be used to improve the performance of a real-world DASH video streaming application.

In recent years, the Linux kernel has adopted an algorithm calledTCP Small Queues (TSQ) for reducing queueing latency by controlling buffering in the networking stack.This solution consists of a back-pressure mechanism that limitsthe number of TCP segments within the sender TCP/IP stack, waitingfor packets to actually be transmitted onto the wire beforeenqueueing further segments.Unfortunately, TSQ prevents the frameaggregation mechanism in the IEEE 802.11n/ac standards from achieving itsmaximum aggregation, because not enough packets are available in the queue to buildaggregates from, which severely limits achievable throughput over wirelesslinks.This paper demonstrates this limitation of TSQ in wireless networks and proposesControlled TSQ (CoTSQ), a solution that improves TSQ so that it controls the amountof data buffered while allowing the IEEE 802.11n/ac aggregation logic to fullyexploit the available channel and achieve high throughput. Results on a real testbed show that CoTSQ leadsto a doubling of throughput on 802.11n and up to an order of magnitudeimprovement in 802.11ac networks, with a negligible latency increase.

Programmable packet processing is increasingly implemented using kernel bypass  techniques, where a userspace application takes complete control of the  networking hardware to avoid expensive context switches between kernel and  userspace. However, as the operating system is bypassed, so are its  application isolation and security mechanisms; and well-tested configuration,  deployment and management tools cease to function.  To overcome this limitation, we present the design of a novel approach to  programmable packet processing, called the eXpress Data Path (XDP). In XDP,  the operating system kernel itself provides a safe execution environment for  custom packet processing applications, executed in device driver context. XDP  is part of the mainline Linux kernel and provides a fully integrated solution  working in concert with the kernel's networking stack. Applications are  written in higher level languages such as C and compiled into custom byte code  which the kernel statically analyses for safety, and translates into native  instructions.  We show that XDP achieves single-core packet processing performance as high as  24 million packets per second, and illustrate the flexibility of the  programming model through three example use cases: layer-3 routing, inline  DDoS protection and layer-4 load balancing.

Running network performance experiments on real systems is essential for a complete understanding of protocols and systems connected to the internet. However, the process of running experiments can be tedious and error-prone. In particular, ensuring reproducibility across different systems is difficult, and comparing different test runs from an experiment can be non-trivial.In this paper, we present a tool, called Flent, designed to make experimental evaluations of networks more reliable and easier to perform. Flent works by composing well-known benchmarking tools to, e.g., run tests consisting of several bulk data flows combined with simultaneous latency measurements. Tests are specified in source code, and several common tests are included with the tool. In addition, Flent contains features to automate test runs, collect relevant metadata and interactively plot and explore datasets.We showcase Flent's capabilities by performing a set of experiments evaluating the new BBR congestion control algorithm, using Flent's capabilities to reproduce experiments both in a controlled testbed and across the public internet. Our evaluation reveals several interesting features of BBR's performance.