We proposed an effective strategy for evaluating the targeting specificity of an antibody-conjugated quantum dot (QD) nanoprobe in a coculture system mimicking an in vivo-like tumor microenvironment in which cancer cells grow with normal cells. Analysis of the images was performed with automated confocal microscopy. We have employed a melanoma–melanocyte coculture model to assess the specific binding of QDs conjugated with melanoma antibodies. Conjugation of antibodies to the QD significantly improved the melanoma specificity, while unconjugated antibody alone suffered from non-specific binding to melanocytes. Concentration-dependent binding and competitive inhibition studies with QD–antibody conjugates reproducibly proved the specificity to melanoma cells against melanocytes. The specificity and targeting efficiency of nanoprobes evaluated in a simple coculture model may provide a reasonable assessment for the in vitro diagnosis of early stage melanoma development before in vivo studies. Further, a rapid and sensitive cancer cell detection system demonstrated herein may allow for the development of high-throughput screening platforms for early cancer diagnosis and anti-cancer therapeutics.