Understanding the critical roles of ligands (e.g. thiolates, SR) in the formation of metal nanoclusters of specific sizes has long been an intriguing task since the report of ligand exchange-induced transformation of Au38(SR)24 into Au36(SR′)24. Herein, we conduct a systematic study of ligand exchange on Au38(SC2H4Ph)24 with 21 incoming thiols and reveal that the size/structure preference is dependent on the substituent site. Specifically, ortho-substituted benzenethiols preserve the structure of Au38(SR)24, while para- or non-substituted benzenethiols cause its transformation into Au36(SR)24. Strong electron-donating or -withdrawing groups do not make a difference, but they will inhibit full ligand exchange. Moreover, the crystal structure of Au38(SR)24 (SR = 2,4-dimethylbenzenethiolate) exhibits distinctive π⋯π stacking and “anagostic” interactions (indicated by substantially short Au⋯H distances). Theoretical calculations reveal the increased energies of frontier orbitals for aromatic ligand-protected Au38, indicating decreased electronic stability. However, this adverse effect could be compensated for by the Au⋯H–C interactions, which improve the geometric stability when ortho-substituted benzenethiols are used. Overall, this work reveals the substituent site effects based on the Au38 model, and highlights the long-neglected “anagostic” interactions on the surface of Au-SR NCs which improve the structural stability.