Most theories of soft matter elasticity assume that the local strain in a sample after deformation is identical everywhere and equal to the macroscopic strain, or equivalently that the deformation is affine. We discuss the elasticity of hydrogels of crosslinked polymers with special attention to affine and non-affine theories of elasticity. Experimental procedures to measure non-affine deformations are also described. Entropic theories, which account for gel elasticity based on stretching out individual polymer chains, predict affine deformations. In contrast, simulations of network deformation that result in bending of the stiff constituent filaments generally predict non-affine behavior. Results from experiments show significant non-affine deformation in hydrogels even when they are formed by flexible polymers for which bending would appear to be negligible compared to stretching. However, this finding is not necessarily an experimental proof of the non-affine model for elasticity. We emphasize the insights gained from experiments using confocal rheoscope and show that, in addition to filament bending, sample micro-inhomogeneity can be a significant alternative source of non-affine deformation.