Photovoltaic (PV) technologies for solar energy conversion represent promising routes to green and renewable energy generation. Despite relevant PV technologies being available for more than half a century, the production of solar energy remains costly, largely owing to low power conversion efficiencies of solar cells. The main difficulty in improving the efficiency of PV energy conversion lies in the spectral mismatch between the energy distribution of photons in the incident solar spectrum and the bandgap of a semiconductor material. In recent years, luminescent materials, which are capable of converting a broad spectrum of light into photons of a particular wavelength, have been synthesized and used to minimize the losses in the solar-cell-based energy conversion process. In this review, we will survey recent progress in the development of spectral converters, with a particular emphasis on lanthanide-based upconversion, quantum-cutting and down-shifting materials, for PV applications. In addition, we will also present technical challenges that arise in developing cost-effective high-performance solar cells based on these luminescent materials.