Side chains and fluorine (F) substituents on conjugated polymers have shown significant impact on the photovoltaic properties of polymer-based bulk heterojunction (BHJ) solar cells, but their respective impact is largely studied independently. In order to disentangle the effect of side chains and F substituents, we comprehensively investigate a series of conjugated polymers with an identical backbone (PNDT–DTBT) but different combinations of side chains and F substituents. Surprisingly, these seemingly marginal changes to the polymer backbone strongly influence the morphology and structure in BHJ thin films (e.g., domain size/purity and the relative orientation of polymer crystallites), as manifested by resonant soft X-ray scattering (R-SoXS) and grazing-incidence wide-angle X-ray scattering (GI-WAXS), thereby exerting significant impact on the photovoltaic properties of these conjugated polymer-based BHJ cells. Devices based on the polymer with long bulky side chains and F substituents (C8,4-C6,2F) simultaneously exhibit large open circuit voltage (V_oc), high short circuit current (J_sc) and good fill factor (FF), with an efficiency as high as 5.6% for this series of PNDT–DTBT polymers.