A scale-invariant two-component Fermi gas in a time-dependent isotropic harmonic potential is investigated. The exact time evolution of the density distribution in position space in any spatial dimension is obtained. Two experimentally relevant examples, an abrupt change and a periodic modulation of the trapping frequency, are solved. Small deviations from scale invariance are addressed within first-order perturbation theory. The consequences for experiments with ultracold quantum gases, such as the excitation of a tower of undamped breathing modes and an alternative for measuring the Tan contact, are discussed.