The eigenvalue problems for generalized natural modes of an inhomogeneous dielectric waveguide without a sharp boundary and a step-index dielectric waveguide with a smooth boundary of cross-section are formulated as problems for the set of time-harmonic Maxwell equations with partial radiation conditions at infinity in the cross-sectional plane. The original problems are reduced by the integral equation method to nonlinear spectral problems with Fredholm integral operators. Properties of the spectrum are investigated. The Galerkin and collocation methods for the calculations of generalized natural modes are proposed and convergence of the methods is proved. Some results of numerical experiments are discussed.