Abstract: In order to study performance characteristics of untrained, amateur singers, we developed an online tool through which coders could evaluate real-world vocal performances of “Amazing Grace” from a Smule dataset. Coders from Stanford University used the online evaluation tool to deliver judgments of age and gender of the performers, as well as skill, likeability, and expressiveness of the vocal performances. Initial results show subjective evaluations of skill, likeability, and expressiveness are highly correlated, and coders rarely gave the highest possible score in any of the three metrics. This online evaluation tool can be used in future computational studies of vocal performance.