The Internet of today has intermediary devices known as middleboxes that perform more functions than the normal packet forwarding function of a router. Security middleboxes are a subset of these middleboxes and face an increasingly difficult task to perform their functions correctly. These middleboxes make many assumptions about the traffic that may not hold true any longer with the advent of new protocols such as MPTCP and technologies like end-to-end encryption.

The work in this thesis focuses on security middleboxes and the challenges they face. We develop methods and solutions to help these security middleboxes continue to function correctly. In particular, we investigate the case of using MPTCP over traditional security infrastructure as well as the case of end-to-end encryption. We study how practical it is to evade a security middlebox by fragmenting and sending traffic across multiple paths using MPTCP. We then go on to propose possible solutions to detect such attacks and implement them. The potential MPTCP scenario where security middleboxes only have access to part of the traffic is also investigated and addressed. Moreover, the thesis contributes a machine learning based approach to help security middleboxes detect malware in encrypted traffic without decryption.

The Internet of today has intermediary devices known as middleboxes thatperform more functions than the normal packet forwarding function of arouter. Security middleboxes are a subset of these middleboxes and face anincreasingly difficult task to perform their functions correctly in the wake ofemerging protocols and technologies on the Internet. Security middleboxesmake many assumptions about the traffic, e.g., they assume that traffic froma single connection always arrives over the same path and they often expectto observe plaintext data. These along with many other assumptions may nothold true any longer with the advent of new protocols such as MPTCP andtechnologies like end-to-end encryption.

The work in this thesis focuses on security middleboxes and the challengesthey face in performing their functions in an evolving Internet where newnetworking protocols and technologies are regularly introduced. We developmethods and solutions to help these security middleboxes continue to functioncorrectly. In particular, we investigate the case of using MPTCP overtraditional security infrastructure as well as the case of end-to-end encryption.

We study how practical it is to evade a security middlebox by fragmentingand sending traffic across multiple paths using MPTCP. Attack traffic that isgenerated from a self-developed tool is used to evaluate such attacks to showthat these attacks are feasible. We then go on to propose possible solutionsto detect such attacks and implement them. The potential MPTCP scenariowhere security middleboxes only have access to part of the traffic is also investigated.Furthermore, we propose and implement an algorithm to performintrusion detection in such situations. Moreover, the thesis contributes a machinelearning based approach to help security middleboxes detect malware inencrypted traffic without decryption.