The detailed performance characteristics of networking equipment is to a large extent a function of the software that controls the underlying hardware components. Most networking equipment is regularly updated with new software versions. By studying performance changes related to such changes in software, it is possible to identify particular software versions that affect the performance of the system. Consequently, having automated methods for detecting changes in network equipment performance is crucial. In this work we study the change point detection problem arising when the placement in time of software updates is known a priori, but the presence of any performance implications on any of the thousands of performance indicators that can be collected is unknown. The ability to improve the automated detection of such change points by clustering the monitored systems according to the set of collected indicators has not been fully evaluated. We here report our experience with employing clustering, together with a bootstrap-based change point detection, across a range of performance indicators. We evaluate four variations of clustering approaches, and demonstrate the resulting improvement in change point detection sensitivity.