Understanding of the interplay of generally known colloidal transformations under conditions of test media (TM) used during cultivation of organisms and biological effect (=ecotoxicological) studies is still limited, although this knowledge is required for an adequate interpretation of test outcomes and for a comparison among different studies. In this context, we investigated the aggregation and dissolution dynamics of citrate-stabilized silver nanoparticles (Ag NPs) by varying the composition of three TM (ASTM, SAM-5S, and R2A, used during bioassays with Daphnia magna, Gammarus fossarum, and bacterial biofilms, respectively) in the presence and absence of two types of natural organic matter (NOM), namely, Suwanee River humic acid (SRHA) and seaweed extract (SW). Each original test medium induced reaction-limited aggregation of Ag NPs, and aggregation increased from R2A to SAM-5S and ASTM. In addition to the differences in aggregation dynamics, the concentration and speciation of Ag(I) differed between the three TM, whereby SAM-5S and ASTM are comparable with respect to the nature of the aggregation process but clearly differ from the R2A medium. Furthermore, Cl−, mainly present in SAM-5S, induced NP stabilization. The release of silver ions from Ag NPs was controlled by the presence of NOM and organic constituents of TM and by interactions with Cl− and Br−. The degree of aggregation, formation of interparticle cation–NOM bridges or stabilization was larger for Ca2+ than for Mg2+ due to the stronger ability of Ca2+ to interact with citrate or NOM compared to Mg2+. These observations and the dependence of aggregation rates on the particle concentration renders the interpretation of dose–response relationships challenging, but they may open perspectives for targeted ecotoxicological testing by modifications of TM composition.