A compression driver is an electro-acoustic transducer with considerably higher efficiency than direct radiating loudspeakers, thanks to the increased radiation resistance caused by a large vibrating diaphragm placed in a compression chamber with small openings. The transition section between compression chamber and output waveguide, the phase plug, must be carefully designed to avoid irregularities in the output sound pressure level (SPL) as a function of frequency. Here we present a shape optimization method based on an implicit level-set description and adjoint sensitivity analysis, which enables a large number of design parameters and vast design freedom. The CutFEM approach, a fictitious domain finite element method, removes the need for mesh updates and makes the method robust and computationally inexpensive. Numerical experiments for a generic annular diaphragm compression driver are presented, with optimized designs showing only minor frequency irregularities. Two different objective functions are considered: one for maximum SPL and one where the SPL is fitted to that of a hypothetical ideal design; the latter approach is found to be more effective in reducing irregularities. Visco-thermal boundary-layer losses are included in a post-processing step, and, though the influence of losses is clearly noticeable, the overall performance is similar and the optimized designs still outperform the original design.