Introduction
Research has shown that Multipath TCP (MPTCP) improves the quality of a TCP connection by exploiting multiple paths, but its adoption in the wide area network is still fledgling. While MPTCP-TCP proxying is often employed as a practical solution, the performance of a split-connection proxy is suboptimal; it wastes CPU cycles on content relaying between two connections while it does not efficiently leverage multiple CPU cores in packet processing.
We present FlexCP, a high-performance MPTCP-TCP proxy based on the following properties. First, FlexCP operates by translating the two protocols on a packet level. This approach not only avoids the overhead of flow reassembly and memory copying, but it greatly simplifies the implementation as the proxy stays away from reliable data transfer, socket buffer management, and per-hop congestion/flow control. Second, FlexCP maintains connection-to-core affinity for multiple subflows of the same MPTCP connection and its corresponding TCP connection by leveraging SmartNIC. This enables a lock-free implementation for packet processing, which significantly improves the performance. Our evaluation demonstrates that FlexCP achieves 281 Gbps of connection proxying on a single machine, outperforming existing proxies by up to 6.3x in terms of throughput while it incurs little extra latency over direct TCP/MPTCP connections.
Overhead in MPTCP-TCP Proxy
While the MPTCP-TCP proxy can be a promising solution for providing multi-connectivity across heterogeneous access networks, it suffers from significantly lower throughput. We find that the performance gap between an MPTCP-TCP proxy and end-to-end (E2E) TCP and MPTCP scenarios is huge, while between MPTCP-TCP and TCP-TCP proxies is not significantly large. This implies that the performance bottleneck of a proxy primarily originates from the nature of proxy model itself, rather than inherent protocol overheads.
Connection-terminating vs Splitless Proxy (FlexCP!)
The root cause for suboptimal performance of the protocol-translating proxy lies in its connection-terminating (or split-connection) architecture. In this design, a significant portion of CPU cycles are wasted on connection-level proxying which includes reliable data transfer, buffer management, flow reassemble, congestion and flow control.
Instead we present FlexCP, a scalable MPTCP-TCP proxy on a multicore system for multi-homed clients. it relays the two logical connections of heterogeneous protocols with a single physical connection that translates the headers on a packet level without connection termination. This effectively avoids the TCP protocol conformance overhead of each segment and substantially improves the performance.
Toward Scalability in Multicore
While this “splitless-connection architecture” avoids the performance overheads, it introduces a number of new challenges mainly due to the intricacy of the MPTCP protocol. First, MPTCP-TCP proxying must maintain a per-connection state for consistency when mapping the TCP SN space to the MPTCP SSN space, which is drastically different from packet-level SN translation in TCP-TCP proxies that can operate completely in a stateless manner. Figure below displays two of corner case: translating retransmission and SACK packet correctly. We propose our own D-tree structure to manage per-connection state and enable packet-level header translation in splitless proxying.
(left) forwarding retransmitted packet into multiple different subflows
(right) SACK packet translation from an MPTCP subflow to the TCP connection
Second, as illustrated in the figure below, the proxy cannot blindly apply symmetric RSS to multiple MPTCP subflows since different subflows would have distinct 4-tuples, resulting in heterogeneous RSS hash values. Consequently, packets belonging to different subflows of the same MPTCP connection would be steered to different CPU cores that causes the resource contention.
To address the problem, we exploit SmartNIC which carefully steers the subflows of the same MPTCP connection to a single CPU core and ensures high processing scalability on a multicore system without locking. Please refer the paper for the detail of FlexCP design.
Evaluation for FlexCP
A figure below compares the throughputs over an increasing number of CPU cores of the proxy, with 256 concurrent connections per proxy core. The performance of FlexCP scales well with multiple cores and it achieves 281 Gbps with 24 CPU cores, outperforming the nginx proxy and haproxy by 3.2x to 6.3x and 2.4x to 4.4x, respectively. Furthermore, it is notable that FlexCP achieves throughputs comparable to those of E2E MPTCP shown in figure above, performing slightly better until 16 cores.
We also compares the performance of FlexCP with and without flow-to-core affinity, by configuring FlexCP to do packet-level traffic steering so that the ACKs from the clients are delivered through different subflow paths. The result (left below) shows that ensuring flow-to-core affinity improves the throughput by up to 45% as the number of CPU cores increases.
The figure right above shows whether FlexCP can adapt to the dynamic change of the subflow path quality, with two distinct physical 100Gbps paths. Initially, all MPTCP connections use only one subflow path for 5 seconds. Then, the second path becomes available, and each MPTCP connection adds a new subflow on it. Then, after 5 seconds, the second path exhibits a very high latency. We observe that the aggregate throughput is near 100 Gbps for the first five seconds, and then both paths are fully utilized for the next five seconds with load-balancing mode. Then FlexCP detects a very high delay on the second path so that it steers all flows to use the subflow on the first path again.
Publications
FlexCP: A Scalable Multipath TCP Proxy for Cellular Networks
Duckwoo Kim, YoungGyun Moon, Jaehyun Hwang, Kyoungsoo Park
In The 19th International Conference on emerging Networking EXperiments and Technologies
(CoNEXT ’23)
December 2023