Using quantum dot fluorescence imaging, we investigated the interference of surface plasmon beams in branched silver nanowire structures. Depending on the phases and polarizations of the incident light, interferences of plasmon beams modulate the plasmon propagation in the branched structures and the output light intensity in the distal ends. The interference visibility is strongly dependent on the incident polarization at the main wire terminal, and the mechanism is revealed by quantum dot fluorescence imaging of the near field distribution of propagating plasmons. The near field distribution pattern resulting from the beating of different plasmon modes plays a critical role in the plasmon interference. The overlap of the antinode in the near field pattern with the connection junction in the nanowire structure is required for a large interference visibility, since the overlap makes the electric field intensity difference of the two plasmon beams smaller. It is found that the plasmon interference is strongly dependent on the polarization of the excitation light at the main wire terminal, but weakly dependent on the polarization at the branch wire terminal.