We investigate the rich quantum phase diagram of Wegner's theory of discrete Ising gauge fields interacting with U (1) symmetric single-component fermion matter hopping on a two-dimensional square lattice. In particular limits, the model reduces to (i) pure Z(2) even and odd gauge theories, (ii) free fermions in a static background of deconfined Z(2) gauge fields, and (iii) the kinetic Rokhsar-Kivelson quantum dimer model at a generic dimer filling. We develop a local transformation that maps the lattice gauge theory onto a model of Z(2) gauge-invariant spin 1/2 degrees of freedom. Using the mapping, we perform numerical density matrix renormalization group calculations that corroborate our understanding of the limits identified above. Moreover, in the absence of the magnetic plaquette term, we reveal signatures of topologically ordered Dirac semimetal and staggered Mott insulator phases at half filling. At strong coupling, the lattice gauge theory displays fracton phenomenology with isolated fermions being completely frozen and dimers exhibiting restricted mobility. In that limit, we predict that in the ground state, dimers form compact clusters, whose hopping is suppressed exponentially in their size. We determine the band structure of the smallest clusters numerically using exact diagonalization. The rich phenomenology discussed in this paper can be probed in analog and digital quantum simulators of discrete gauge theories and in Kitaev spin-orbital liquids.