The linear and nonlinear optical (NLO) properties of a series of quaternary molybdenum tellurite crystals are studied using a first-principles computational approach and an empirical dipole model. The calculated second harmonic generation (SHG) coefficients are consistent with the experimental values and the calculations show that their magnitude is independent of the dipole moment. Using a flexible dipole model based on the concept of bond-valence it is shown that the size of the NLO effects is determined by the compliance with the dipole moment in response to external perturbation rather than the intrinsic dipole moment of the structure. The study revises the common belief that SHG effects depend on the dipole moment inherent to the unperturbed structure and will help guide the search for new NLO crystals including those that are not necessarily polar.