A facile low-temperature hydrothermal process to synthesize α-Fe/Fe3O4 nanocomposite is reported. TEM and HRTEM revealed that the α-Fe/Fe3O4 nanocomposite was composed of catenulate α-Fe and lamellar structured Fe3O4. The weight ratio of α-Fe in the α-Fe/Fe3O4 nanocomposite is 35.6%. The α-Fe/Fe3O4 nanocomposite demonstrates an extremely high Congo red (CR) removal efficiency from waste water showing almost complete removal within 3 min. For 100 mg L−1 of CR aqueous solution, the maximum CR removal can reach 1297.06 mg g−1. The large saturation magnetization (80.5 emu g−1) of the nanocomposite allows fast separation of α-Fe/Fe3O4 nanoparticles loaded with CR from the liquid suspension. The synergistic effect of the nanocomposite may contribute to the enhanced CR removal ability, because the CR can be removed by reduction reaction and adsorption at the same time. Based on the degradation products identified by UV-Vis spectra, XRD and FTIR spectra, a possible degradation mechanism of CR on the α-Fe/Fe3O4 composite was proposed. The significantly reduced treatment time required to remove the CR and the simple, low-cost and pollution-free preparation method make α-Fe/Fe3O4 nanocomposite promising for highly efficient removal of dyes from waste water.