Web2 days ago · %0 Conference Proceedings %T CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models %A Nangia, Nikita %A Vania, … WebCrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one …

crows-pairs/crows_pairs_anonymized.csv at master - Github

WebCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models ... CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. … WebWe build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages while also characterizing biases that are specific to each country and language. We introduce 1,679 sentence pairs in French that cover stereotypes in ten types of bias like gender and age. 1,467 sentence ... in and out winery

lm-evaluation-harness/crowspairs.py at master - Github

WebCrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re … WebCrowS-Pairs. Introduced by Nangia et al. in CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. CrowS-Pairs has 1508 … WebThe four benchmark datasets we consider 1) are de-signed to test NLP systems on two tasks—language modeling and coreference resolution, 2) consist of pairs of contrastive sentences (§2.1), and 3) are accompanied by aggregating metrics (§2.2). The datasets also vary in how the sentence pairs were constructed (by subject matter experts, dvbe search