Five Stars, Still Foreign: Stereotypes in Favorable Evaluations

Abstract: Category membership affects how audiences evaluate producers, but most research focuses on ratings rather than language. This study examines how evaluation language differs across demographic groups. Using a method that combines word embeddings with automatic text coding, we analyze stakeholder reviews to identify which linguistic features distinguish descriptions of instructors from different ethnic groups. The central finding is counterintuitive: among highly-rated instructors, stereotypic language is most common, not least. Nationality-based language appears most often in reviews of high-performing Asian instructors. Five-star reviews mention accent, national origin, or foreignness for minority instructors but not for White instructors receiving equally positive ratings. This suggests that favorable ratings can coexist with stereotypic language. A positive rating does not mean the instructor escaped stereotyping; the stereotyping appears in the words rather than the score. For research on categories and evaluation, this implies that equal ratings do not mean equal treatment. The same numerical outcome can come with different verbal framing, and that framing may carry stereotypic content. The method introduced here provides a tool for detecting stereotypic language in large text corpora wherever stakeholders write open-ended evaluations.

Keywords: performance evaluation, ethnic bias, status characteristics, word embeddings, automatic coding, workplace discrimination