Although it is frequently hypothesized that surface (like surface charge) and physical characteristics (like particle size) play important roles in cellular interactions of nanoparticles (NPs), a systematic study probing this issue is missing. Hence, a comparative cytotoxicity study, quantifying nine different cellular endpoints, was performed with a broad series of monodisperse, well characterized silicon (Si) and germanium (Ge) NPs with various surface functionalizations. Human colonic adenocarcinoma Caco-2 and rat alveolar macrophage NR8383 cells were used to clarify the toxicity of this series of NPs. The surface coatings on the NPs appeared to dominate the cytotoxicity: the cationic NPs exhibited cytotoxicity, whereas the carboxylic acid-terminated and hydrophilic PEG- or dextran-terminated NPs did not. Within the cationic Si NPs, smaller Si NPs were more toxic than bigger ones. Manganese-doped (1% Mn) Si NPs did not show any added toxicity, which favors their further development for bioimaging. Iron-doped (1% Fe) Si NPs showed some added toxicity, which may be due to the leaching of Fe3+ ions from the core. A silica coating seemed to impart toxicity, in line with the reported toxicity of silica. Intracellular mitochondria seem to be the target for the toxic NPs since a dose-, surface charge- and size-dependent imbalance of the mitochondrial membrane potential was observed. Such an imbalance led to a series of other cellular events for cationic NPs, like decreased mitochondrial membrane potential (ΔΨm) and ATP production, induction of ROS generation, increased cytoplasmic Ca2+ content, production of TNF-α and enhanced caspase-3 activity. Taken together, the results explain the toxicity of Si NPs/Ge NPs largely by their surface characteristics, provide insight into the mode of action underlying the observed cytotoxicity, and give directions on synthesizing biocompatible Si and Ge NPs, as this is crucial for bioimaging and other applications in for example the field of medicine.