Benchmarks

Druglib Drug Reviews is a public corpus of patient-written reviews of prescription medications, each rating a drug's benefits on 1–10 ordinal scales (Gräßer et al., 2018). The texts predominantly discuss a range of personal experiences related to the effects of various medications, highlighting both positive and negative outcomes. Common themes include improvement in mood, relief from physical pain, and management of chronic conditions such as anxiety, insomnia, and acne. Many reviews express satisfaction with treatments that resulted in reduced symptoms like cramps, hypertension, and menstrual issues, while others detail adverse effects, including emotional instability and relationship problems. Overall, the narratives reflect a complex interplay of benefits and drawbacks, illustrating the varied impact of medications on individuals' quality of life.

R-squared, Pearson r, and RMSE agree that PsyProxy Psychology fits best; all three are computed on the same held-out target and move together.

Druglib Drug Reviews is a public corpus of patient-written reviews of prescription medications, each rating a drug's benefits and overall effectiveness on 1–10 ordinal scales. The texts predominantly discuss a range of personal experiences related to the effects of various medications, highlighting both positive and negative outcomes. Common themes include improvement in mood, relief from physical pain, and management of chronic conditions such as anxiety, insomnia, and acne. Many reviews express satisfaction with treatments that resulted in reduced symptoms like cramps, hypertension, and menstrual issues, while others detail adverse effects, including emotional instability and relationship problems. Overall, the narratives reflect a complex interplay of benefits and drawbacks, illustrating the varied impact of medications on individuals' quality of life.

The leader switches because the metrics reward different things: QW-kappa (PsyProxy Health) rewards getting close on the ordinal scale, within-one accuracy (PsyProxy Sociology) credits near-misses, and exact accuracy (Top2Vec) credits only exact hits, which a coarser model can win by predicting the modal category.

Disneyland Reviews is a public TripAdvisor corpus (Kaggle, Chillar Anand) of 42,656 guest reviews across three Disneyland branches (Paris, California, Hong Kong). The reviews predominantly discuss a range of experiences related to visiting the park, highlighting both positive and negative aspects. Common topics include the variety of rides available, with many noting the appeal for children and families, as well as the enjoyment of parades and fireworks that contribute to the overall atmosphere. Complaints frequently arise regarding long wait times for attractions and the high cost of food and souvenirs, which some reviewers feel detracts from the experience. Additionally, the cleanliness of the park and the friendliness of the staff are often praised, though some visitors express concerns about overcrowding, especially during peak seasons. Overall, the reviews reflect a mix of nostalgia and practical advice for navigating the park effectively.

R-squared, Pearson r, and RMSE agree that PsyProxy Sociology fits best; all three are computed on the same held-out target and move together.

Disneyland Reviews is a public TripAdvisor corpus (Kaggle: Chillar Anand) of 29,026 guest reviews across two Disneyland branches (California and Hong Kong). The texts predominantly discuss a range of experiences and sentiments related to visits to the amusement park, highlighting both positive and negative aspects. Common themes include ride experiences, with many reviews praising the excitement of various attractions while also noting the long wait times associated with popular rides. Visitors frequently express concerns about costs, mentioning the high prices of food and souvenirs, as well as the overall expense of admission. The cleanliness and staff friendliness are often commended, contributing to a generally positive atmosphere despite complaints about crowds and logistical challenges, particularly for families with young children. Additionally, some reviews suggest strategies for maximizing enjoyment, such as arriving early or utilizing FastPass options to minimize waiting.

The leader switches because the metrics measure different things: AUC (PsyProxy Psychology) is threshold-free ranking, FVE (PsyProxy Psychology) is variance explained, and F1 (HDP) is a thresholded score that depends on the decision cutoff and class balance. A model can rank slightly worse yet win F1 by placing more of its predictions on the majority side of the cutoff.

Amazon Video Games Reviews is a public Kaggle corpus of consumer reviews scraped from Amazon's video-games category, each with a star rating, verified-purchase flag, and free-text body. The texts in the sampled corpus predominantly discuss a variety of experiences related to gameplay mechanics, installation issues, and product quality. Many reviews express disappointment with specific games due to perceived shortcomings, such as poor graphics, limited multiplayer options, and authentication problems. Positive sentiments often highlight enjoyable gameplay and immersive storylines, while some reviews focus on the functionality and aesthetics of gaming accessories like controllers and headsets. Additionally, there are mentions of customer service experiences and the impact of game content on different age groups, reflecting a broad spectrum of user engagement with video games and related products.

R-squared, Pearson r, and RMSE agree that PsyProxy Sociology fits best; all three are computed on the same held-out target and move together.

Amazon Video Games Reviews is a public Kaggle corpus of 199,992 consumer reviews scraped from Amazon's video-games category, each with a verified or not-verified purchase flag, and free-text body. The texts in the sampled corpus predominantly discuss a variety of experiences related to gameplay mechanics, installation issues, and product quality. Many reviews express disappointment with specific games due to perceived shortcomings, such as poor graphics, limited multiplayer options, and authentication problems. Positive sentiments often highlight enjoyable gameplay and immersive storylines, while some reviews focus on the functionality and aesthetics of gaming accessories like controllers and headsets. Additionally, there are mentions of customer service experiences and the impact of game content on different age groups, reflecting a broad spectrum of user engagement with video games and related products.

AUC, FVE, and F1 agree that PsyProxy Psychology is best. AUC is threshold-free ranking, FVE is variance explained, and F1 is thresholded classification; one model dominating all three is an unambiguous result.

IMDB Movie Reviews (ACL) is a public corpus of 50,000 movie reviews scraped from IMDB, labeled positive or negative sentiment. The sampled texts from the corpus predominantly explore a variety of themes related to film critique, character development, and storytelling quality. Many reviews express strong opinions about the acting performances and directorial choices, often highlighting the disparity between expectations and the final product. Common sentiments include disappointment with sequels or remakes that fail to capture the essence of their predecessors, as well as frustration with clichéd plots and predictable narratives. Additionally, there are discussions surrounding the emotional impact of films, particularly those that tackle serious themes such as family dynamics and social issues, revealing a spectrum of viewer reactions from admiration to outright disdain. Overall, the texts reflect a rich tapestry of personal experiences and critical assessments that engage with the cinematic landscape.

AUC, FVE, and F1 agree that ChatGPT wrapper is best. AUC is threshold-free ranking, FVE is variance explained, and F1 is thresholded classification; a model that dominates all three is an unambiguous result.

CFPB Consumer Complaints is a public corpus from the U.S. Consumer Financial Protection Bureau database of consumer complaints filed against financial-services companies, paired with the company's response classification (explanation vs monetary relief). The texts in the corpus predominantly address issues related to inaccurate credit reporting, identity theft, and disputed accounts. Many individuals express concerns about unauthorized charges and fraudulent accounts appearing on their credit reports, which they assert are damaging their creditworthiness. Common themes include requests for validation of debts, disputes over erroneous personal information, and frustrations with the lack of response from credit reporting agencies. Additionally, there are mentions of legal rights under the Fair Credit Reporting Act and the potential for pursuing legal action if issues remain unresolved. Overall, the texts reflect a significant level of distress and urgency regarding the management of personal credit information.

Consumer complaints is the one task in this table where a criterion system other than PsyProxy leads the headline metric: HDP, a topic model, tops AUC and FVE under both screens, while PsyProxy leads on the thresholded F1. That fits the target: complaint categories are largely about subject matter, which topic mixtures capture directly, rather than about the psychology of the writer. The margin is small (HDP's restrictive AUC of 0.664 is within about 0.01 of the best PsyProxy lens) and variance explained is low for every method on this hard target.

Reddit r/SuicideWatch Posts (UMD) is a corpus of Reddit posts labeled binary as expressing suicidal ideation or not, sampled from r/SuicideWatch and matched control subreddits. The texts within the corpus frequently explore themes of suicidal thoughts, mental health struggles, and feelings of isolation or emptiness. Many individuals share their experiences with depression, detailing moments of despair and the impact of trauma on their lives, often expressing a desire for relief from their emotional pain. There are also discussions about support systems, with some posts highlighting the importance of friendships and connections that provide temporary solace, while others reflect on feelings of guilt and failure in the context of lost relationships. Additionally, some texts convey moments of personal achievement or coping strategies, such as sobriety or engaging in hobbies, contrasting with the pervasive sense of hopelessness that characterizes many narratives.

AUC, FVE, and F1 agree that ChatGPT wrapper is best. AUC is threshold-free ranking, FVE is variance explained, and F1 is thresholded classification.

LIAR is a public corpus of 12,836 fact-checked political statements labeled on a six-level truth scale (pants-on-fire to true). The texts predominantly address a range of political and social issues, often focusing on tax relief, healthcare reform, and education funding. Recurring themes include the financial implications of government policies, such as the impact of budget cuts on school districts and the consequences of tax structures on low-income populations. Additionally, the texts highlight concerns regarding gun control and border security, as well as the socioeconomic challenges faced by specific demographics, including unemployed veterans and minority communities. The statements reflect a critical examination of political figures and their decisions, often questioning the accuracy of claims made in public discourse.

PsyProxy is best on all three metrics; different lenses top different ones (PsyProxy Sociology on QW-kappa, PsyProxy Sociology on within-one accuracy, PsyProxy Health on exact accuracy), trading places by small margins, so the method, not any single lens, is the winner here.