Colloidal nanodiamonds (NDs) have emerged as highly versatile platforms for the controlled delivery of therapeutics, proteins, DNA, and other assorted biological agents. The most common mechanism of drug loading onto the ND surface depends mainly on electrostatic interactions. Although a few reports have been published on using NaCl salt to increase the drug loading onto NDs, no comprehensive mechanistic study with a wide range of anions and cations has been reported. In this work, the Hofmeister effect of inorganic salts and amino acids with different isoelectric points was employed to understand the mechanism of doxorubicin (DOXH+) loading onto NDs with different sizes. Inorganic salts including NaCl, NaNO3, Na2SO4, KCl, CaCl2, (NH4)2SO4 and amino acids with an isoelectric point above 7 (positively charged at neutral pH) increase the DOXH+ loading onto small size NDs (SNDs, 5–10 nm). On the other hand, amino acids with an isoelectric point below 7 (negatively charged at neutral pH) increase the DOXH+ loading onto large size NDs (LNDs, 80–100 nm).