Cellular uptake, endosomal/lysosomal escape, and the effective dissociation from the carrier are a series of hurdles for specific genes to be delivered both in vitro and in vivo. To construct siRNA delivery systems, poly(allylamine hydrochloride) (PAH) and siRNA were alternately assembled on the surface of 11.8 ± 0.9 nm Au nanoparticles (GNP), stabilized by denatured bovine serum albumin, by the ionic layer-by-layer (LbL) self-assembly method. By manipulating the outmost PAH layer, GNP–PAH vectors with different surface electric potentials were prepared. Then, the surface potential-dependent cytotoxicity of the resultant GNP–PAH particles was evaluated via sulforhodamine B (SRB) assay, while the surface potential-dependent cellular uptake efficiency was quantitatively analyzed by using the flow cytometry method based on carboxyfluorescein (FAM)-labeled siRNA. It was revealed that the GNP–PAH particles with surface potential of +25 mV exhibited the optimal cellular uptake efficiency and cytotoxicity for human breast cancer MCF-7 cells. Following these results, two more positively charged polyelectrolytes with different protonating abilities in comparison with PAH, i.e., polyethylenimine (PEI), and poly(diallyl dimethyl ammonium chloride) (PDDA), were chosen to fabricate similarly structured vectors. Confocal fluorescence microscopy studies indicated that siRNA delivered by GNP–PAH and GNP–PEI systems was better released than that delivered by the GNP–PDDA system. Further flow cytometric assays based on immunofluorescence staining of the epidermal growth factor receptor (EGFR) revealed that EGFR siRNA delivered by GNP–PAH and GNP–PEI exhibited similar down-regulation effects on EGFR expression in MCF-7 cells. The following dual fluorescence flow cytometry assays by co-staining phosphatidylserine and DNA suggested the EGFR siRNA delivered by GNP–PAH exhibited an improved silencing effect in comparison with that delivered by the commercial transfection reagent Lipofectamine 2000.