diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 8c2d8abb..3e2cab89 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,3 +1,3 @@ -We feel that a welcoming community is important and we ask that you follow Twitter's +We feel that a welcoming community is important and we ask that you follow our [Open Source Code of Conduct](https://github.com/twitter/.github/blob/main/code-of-conduct.md) in all interactions with the community. diff --git a/README.md b/README.md index 4f600c04..1f172bf7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Twitter Community Notes +# Community Notes ![](/documentation/images/help-rate-this-note-expanded.png) @@ -8,13 +8,13 @@ This repository is a place for us to transparently host our content, algorithms, The folder `/sourcecode` holds the [open-source code](https://github.com/twitter/communitynotes/tree/main/sourcecode) powering Community Notes under the hood. -The folder `/documentation` holds the [Markdown content](https://github.com/twitter/communitynotes/tree/main/documentation) that is used to generate our [documentation website](https://communitynotes.twitter.com/guide). +The folder `/documentation` holds the [Markdown content](https://github.com/twitter/communitynotes/tree/main/documentation) that is used to generate our [documentation website](https://communitynotes.x.com/guide). Here you can also find our [research paper](https://github.com/twitter/communitynotes/blob/main/birdwatch_paper_2022_10_27.pdf). ## About Community Notes -Community Notes aims to create a better informed world, by empowering people on Twitter to add helpful notes to Tweets that might be misleading. +Community Notes aims to create a better informed world, by empowering people on X to add helpful notes to posts that might be misleading. We're building it in the open, with the public’s input, and we’re taking significant steps to make Community Notes transparent. @@ -24,7 +24,7 @@ Our goal is to expand Community Notes globally. We want anyone to be able to par As there are important nuances in each market, we’ll expand the contributor base country-by-country. We’ll add contributors from a first new country soon. -[Sign up here](https://twitter.com/i/flow/join-birdwatch) +[Sign up here](https://x.com/i/flow/join-birdwatch) ## How to contribute to this repo @@ -34,15 +34,15 @@ Thank you for your interest in contributing to Community Notes! Currently, we wi * Documentation * Open issues -Note that we aren’t currently accepting changes that alter existing APIs, as there is other utility and production infrastructure code at Twitter that depends on these APIs remaining stable. +Note that we aren’t currently accepting changes that alter existing APIs, as there is other utility and production infrastructure code at X that depends on these APIs remaining stable. -We are also exploring ways to make it easier for people to contribute directly to the core algorithm. For example, by making available testing and evaluation frameworks that would allow open source contributors to evaluate the impact of their PRs on note quality. +We are also exploring ways to make it easier for people to contribute directly to the core algorithm. For example, by making available testing and evaluation frameworks that would allow open source contributors to evaluate the impact of their PRs on note quality. --- ### Documentation website -The markdown files in this repo are the source of truth for the content in our documentation website (aka "Community Notes Guide"). They are always updated here first, then ingested by Twitter's internal tools, translated, and published in [communitynotes.twitter.com/guide](https://communitynotes.twitter.com/guide). +The markdown files in this repo are the source of truth for the content in our documentation website (aka "Community Notes Guide"). They are always updated here first, then ingested by X's internal tools, translated, and published in [communitynotes.x.com/guide](https://communitynotes.x.com/guide). ### Community Notes open-source code @@ -50,7 +50,7 @@ The algorithm that powers Community Notes can be found on the [sourcecode folder ### Community Notes data -All notes, ratings, and contributor data are [publicly available and published daily here](https://twitter.com/i/communitynotes/download-data). Instructions on how to use them can be found in the [Community Notes Guide](https://communitynotes.twitter.com/guide/under-the-hood/download-data/). +All notes, ratings, and contributor data are [publicly available and published daily here](https://x.com/i/communitynotes/download-data). Instructions on how to use them can be found in the [Community Notes Guide](https://communitynotes.x.com/guide/under-the-hood/download-data/). ### Community Notes paper diff --git a/documentation/_redirects/404.html b/documentation/_redirects/404.html index 5f30c024..6542bf85 100644 --- a/documentation/_redirects/404.html +++ b/documentation/_redirects/404.html @@ -2,7 +2,7 @@ - Redirecting to communitynotes.twitter.com/guide + Redirecting to communitynotes.x.com/guide diff --git a/documentation/about/challenges.md b/documentation/about/challenges.md index dd6ee509..602df1e1 100644 --- a/documentation/about/challenges.md +++ b/documentation/about/challenges.md @@ -7,7 +7,7 @@ navWeight: 2 We know there are many challenges involved in building an open, participatory system like Community Notes — from making it resistant to manipulation attempts, to ensuring it isn’t dominated by a simple majority or biased because of the distribution of contributors. -We've been building Community Notes (formerly called Birdwatch) in [public since January 2021](https://blog.twitter.com/en_us/topics/product/2021/introducing-birdwatch-a-community-based-approach-to-misinformation), and have intentionally designed it to mitigate potential risks. We've seen [encouraging results](https://blog.twitter.com/en_us/topics/product/2022/birdwatch-getting-new-onboarding-process-more-visible-notes), but we're constantly designing for challenges that could arise. +We've been building Community Notes (formerly called Birdwatch) in [public since January 2021](https://blog.x.com/en_us/topics/product/2021/introducing-birdwatch-a-community-based-approach-to-misinformation), and have intentionally designed it to mitigate potential risks. We've seen [encouraging results](https://blog.x.com/en_us/topics/product/2022/birdwatch-getting-new-onboarding-process-more-visible-notes), but we're constantly designing for challenges that could arise. Here are a handful of particular challenges we are aware of as well as steps we are taking to address them: @@ -17,8 +17,8 @@ Attempts at coordinated manipulation represent a crucial risk for open rating sy The program currently takes multiple steps to reduce the potential for this type of manipulation: -- First, all Twitter accounts must meet the [eligibility criteria](../contributing/signing-up.md) to become a Community Notes contributor. For example, having a unique, verified phone number. These criteria are designed to help prevent the creation of large numbers of fake or sock puppet contributor accounts that could be used for inauthentic rating. -- Second, Community Notes doesn't work like many engagement-based ranking systems, where popular content gains the most visibility and people can coordinate to mass upvote or downvote content they don't like or agree with. Instead, Community Notes uses a bridging algorithm — for a note to be shown on a Tweet, it needs to be found helpful by people who have tended to [disagree in their past ratings](../contributing/diversity-of-perspectives.md). +- First, all X accounts must meet the [eligibility criteria](../contributing/signing-up.md) to become a Community Notes contributor. For example, having a unique, verified phone number. These criteria are designed to help prevent the creation of large numbers of fake or sock puppet contributor accounts that could be used for inauthentic rating. +- Second, Community Notes doesn't work like many engagement-based ranking systems, where popular content gains the most visibility and people can coordinate to mass upvote or downvote content they don't like or agree with. Instead, Community Notes uses a bridging algorithm — for a note to be shown on a post, it needs to be found helpful by people who have tended to [disagree in their past ratings](../contributing/diversity-of-perspectives.md). [Academic](https://www.belfercenter.org/publication/bridging-based-ranking) [research](https://www.google.com/books/edition/Breaking_the_Social_Media_Prism/ORMCEAAAQBAJ?hl=en&gbpv=0) indicates that bridging-based ranking can help to identify content that is healthier and higher quality, and reduce the risk of elevating polarizing content. @@ -36,21 +36,21 @@ Community Notes will be most effective if the context it produces can be found t - Second, Community Notes can proactively seek ratings from contributors who are likely to provide a different perspective based on their rating history. This is currently done in the [Needs Your Help tab](../contributing/rating-notes.md), and we are exploring new ways to quickly collect ratings on notes from a wide range of contributors. -- Third, to help ensure that people of diverse backgrounds and viewpoints feel safe and empowered to participate, Community Notes has implemented program [aliases](../contributing/aliases.md) that aren’t publicly associated with contributors’ Twitter accounts. This can help prevent one-sided-ness by providing more diverse contributors with a voice in the system. +- Third, to help ensure that people of diverse backgrounds and viewpoints feel safe and empowered to participate, Community Notes has implemented program [aliases](../contributing/aliases.md) that aren’t publicly associated with contributors’ X accounts. This can help prevent one-sided-ness by providing more diverse contributors with a voice in the system. -- Finally, we regularly survey representative samples of Twitter customers who are not Community Notes contributors to assess whether a broad range of people on Twitter are likely to find the context in Community Notes to be helpful, and whether the notes can be informative to people of different points of view. +- Finally, we regularly survey representative samples of customers who are not Community Notes contributors to assess whether a broad range of people on X are likely to find the context in Community Notes to be helpful, and whether the notes can be informative to people of different points of view. - This is one indicator of Community Notes' ability to be of value to people from a [wide range of perspectives](../contributing/diversity-of-perspectives.md) vs. to be biased towards one group or viewpoint. Twitter customers who aren’t enrolled Community Notes contributors can also provide rating feedback on notes they see on Twitter. This provides an additional indicator of note helpfulness observed over time. + This is one indicator of Community Notes' ability to be of value to people from a [wide range of perspectives](../contributing/diversity-of-perspectives.md) vs. to be biased towards one group or viewpoint. Customers who aren’t enrolled Community Notes contributors can also provide rating feedback on notes they see on X. This provides an additional indicator of note helpfulness observed over time. ### Avoiding Harassment It’s crucial that people feel safe contributing to Community Notes and aren’t harassed for their contributions. It’s also important that Community Notes itself does not become a vector for harassment. Here are measures Community Notes takes to keep everyone safe: -- First, as described above, all contributors get a new, auto-generated [display name (or alias)](../contributing/aliases.md) when they join Community Notes. These aliases are not publicly associated with contributors’ Twitter accounts, so everyone can write and rate notes privately. This inhibits public identification and harassment of contributors. +- First, as described above, all contributors get a new, auto-generated [display name (or alias)](../contributing/aliases.md) when they join Community Notes. These aliases are not publicly associated with contributors’ X accounts, so everyone can write and rate notes privately. This inhibits public identification and harassment of contributors. -- Second, contributors have an open communication line with the Community Notes team to report any issues they experience (they can reach us by DM [@CommunityNotes](https://twitter.com/communitynotes)). Community Notes has a dedicated community manager who gathers and responds to contributors’ feedback or concerns via this handle. This provides a way for contributors to flag potential issues to the Community Notes team. +- Second, contributors have an open communication line with the Community Notes team to report any issues they experience (they can reach us by DM [@CommunityNotes](https://x.com/communitynotes)). Community Notes has a dedicated community manager who gathers and responds to contributors’ feedback or concerns via this handle. This provides a way for contributors to flag potential issues to the Community Notes team. -- Finally, all Community Notes contributions are subject to Twitter [Rules](https://help.twitter.com/rules-and-policies/twitter-rules), [Terms of Service](https://twitter.com/tos) and [Privacy Policy](https://twitter.com/privacy). If you think note might not abide by the rules, you can report it by clicking or tapping the ••• menu on a note, and then selecting "Report”, or by using the [Report a Community Note](https://help.twitter.com/en/forms/community-note) form. This provides a mechanism to address violating content in notes. +- Finally, all Community Notes contributions are subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules), [Terms of Service](https://x.com/tos) and [Privacy Policy](https://x.com/privacy). If you think note might not abide by the rules, you can report it by clicking or tapping the ••• menu on a note, and then selecting "Report”, or by using the [Report a Community Note](https://help.x.com/en/forms/community-note) form. This provides a mechanism to address violating content in notes. ### Reducing the impacts of low quality contributions on the system @@ -69,4 +69,4 @@ As Community Notes grows and evolves, we will continue to iterate to ensure it r ## Feedback? Ideas? -We welcome feedback on these or additional risks and challenges, as well as ideas for addressing them. Please DM us at [@CommunityNotes](http://twitter.com/communitynotes). +We welcome feedback on these or additional risks and challenges, as well as ideas for addressing them. Please DM us at [@CommunityNotes](http://x.com/communitynotes). diff --git a/documentation/about/faq.md b/documentation/about/faq.md index 29f486e7..aac0d497 100644 --- a/documentation/about/faq.md +++ b/documentation/about/faq.md @@ -12,7 +12,7 @@ Anyone can sign up to become contributors. Accounts must meet a few criteria to {% /accordionItem %} {% accordionItem title="Are contributions anonymous? Or will my name be attached to my notes?" %} -We want everyone to feel comfortable contributing to Community Notes. Because of that, all contributors get a new, auto-generated display name (or alias) when they join Community Notes. These aliases are not publicly associated with contributors’ Twitter accounts, so everyone can write and rate notes privately. +We want everyone to feel comfortable contributing to Community Notes. Because of that, all contributors get a new, auto-generated display name (or alias) when they join Community Notes. These aliases are not publicly associated with contributors’ X accounts, so everyone can write and rate notes privately. You can learn more about aliases [here](../contributing/aliases.md). {% /accordionItem %} @@ -25,21 +25,21 @@ We believe regular people can valuably contribute to identifying and adding help - [Bhuiyan, Zhang, Sehat, and Mitra 2020](https://arxiv.org/pdf/2008.09533.pdf) - [Kim and Walker 2020](https://misinforeview.hks.harvard.edu/article/leveraging-volunteer-fact-checking-to-identify-misinformation-about-covid-19-in-social-media/) -In our pilot test of Community Notes, we evaluated notes that would be shown on Tweets, and have found: +In our pilot test of Community Notes, we evaluated notes that would be shown on posts, and have found: -- The majority of people surveyed on Twitter found notes helpful -- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading Tweet after reading the note about it, compared to those who saw a Tweet without a note +- The majority of people surveyed on X found notes helpful +- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading post after reading the note about it, compared to those who saw a post without a note - Most notes have been rated highly on accuracy by professional reviewers; it has been rare to find a note that reviewers agree is inaccurate -The people on Twitter span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. +The people on X span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. {% /accordionItem %} -{% accordionItem title="What Tweets are eligible to be annotated in Community Notes?" %} -Any Tweet can receive a Community Note. +{% accordionItem title="What posts are eligible to be annotated in Community Notes?" %} +Any post can receive a Community Note. {% /accordionItem %} -{% accordionItem title="There’s a Community Note on one of my Tweets. What does that mean?" %} -Tweet authors with Community Notes on their Tweets can request additional review or report notes. [Learn more](../contributing/additional-review.md). +{% accordionItem title="There’s a Community Note on one of my posts. What does that mean?" %} +Post authors with Community Notes on their posts can request additional review or report notes. [Learn more](../contributing/additional-review.md). {% /accordionItem %} {% accordionItem title="How do I opt-out of the program?" %} @@ -47,40 +47,40 @@ If you join the program but later want to leave, DM @CommunityNotes to let the t {% /accordionItem %} {% /accordionSection %} -## Twitter policies and rules +## Policies and rules {% accordionSection %} -{% accordionItem title="Will Community Notes contributions affect content on Twitter? Will it be used to label or remove Tweets? " %} -Notes that have been rated helpful by enough contributors from [different points of view](../contributing/diversity-of-perspectives.md) will appear directly on Tweets. Beyond that, notes do not affect display of Tweets or enforcement of Twitter Rules. Our current focus is gaining confidence that Community Notes produces context people from [different points of view](../contributing/diversity-of-perspectives.md) find helpful and informative. +{% accordionItem title="Will Community Notes contributions affect content on X? Will it be used to label or remove posts? " %} +Notes that have been rated helpful by enough contributors from [different points of view](../contributing/diversity-of-perspectives.md) will appear directly on posts. Beyond that, notes do not affect display of posts or enforcement of X's [Rules](https://help.x.com/rules-and-policies/twitter-rules). Our current focus is gaining confidence that Community Notes produces context people from [different points of view](../contributing/diversity-of-perspectives.md) find helpful and informative. {% /accordionItem %} -{% accordionItem title="How does Community Notes relate to Twitter’s other policies?" %} -Community Notes is designed to add context that is informative and helpful to people from different points of view. It is intended to complement Twitter's other policies and rules. +{% accordionItem title="How does Community Notes relate to X's other policies?" %} +Community Notes is designed to add context that is informative and helpful to people from different points of view. It is intended to complement our other policies and rules. {% /accordionItem %} {% accordionItem title="Can people be removed from the Community Notes program? What if someone is writing harmful content in their notes?" %} -All contributors and all Community Notes contributions are subject to the Twitter Rules. Failure to abide by the rules can result in removal from the Community Notes program, and/or other remediations. +All contributors and all Community Notes contributions are subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules). Failure to abide by the rules can result in removal from the Community Notes program, and/or other remediations. -If you believe a note may have violated the Twitter Rules, you can report it by clicking or tapping the ••• menu on a note, and then selecting "Report”, or by using [this form](https://help.twitter.com/en/forms/community-note). +If you believe a note may have violated X's [Rules](https://help.x.com/rules-and-policies/twitter-rules), you can report it by clicking or tapping the ••• menu on a note, and then selecting "Report”, or by using [this form](https://help.x.com/en/forms/community-note). Contributors who consistently write unhelpful notes may also have their [ability to write new notes temporarily locked](../contributing/writing-ability.md). {% /accordionItem %} {% accordionItem title="How will attempts to abuse or manipulate Community Notes be prevented?" %} -Community Notes will only be successful if the context it produces is found to be helpful and appropriate by a wide range of people with diverse views. This won’t be true if it can be taken over by a single group or ideology, or used in an abusive manner. We believe this is possible, and will be experimenting with different mechanics and incentives than Twitter to help make it happen. Learn more about how we are approaching these issues [here](./challenges.md). +Community Notes will only be successful if the context it produces is found to be helpful and appropriate by a wide range of people with diverse views. This won’t be true if it can be taken over by a single group or ideology, or used in an abusive manner. We believe this is possible, and will be experimenting with different mechanics and incentives to help make it happen. Learn more about how we are approaching these issues [here](./challenges.md). {% /accordionItem %} {% /accordionSection %} ## Research {% accordionSection %} -{% accordionItem title="Will researchers and others outside Twitter be able to download and analyze the data? " %} -All Community Notes' data are made publicly available on the [Download Data](https://twitter.com/i/communitynotes/download-data) page of the Community Notes site so that anyone has free access to analyze the data, identify problems, and spot opportunities to make Community Notes better. [You can learn more about our data here.](../under-the-hood/download-data.md) +{% accordionItem title="Will researchers and others outside X be able to download and analyze the data? " %} +All Community Notes' data are made publicly available on the [Download Data](https://x.com/i/communitynotes/download-data) page of the Community Notes site so that anyone has free access to analyze the data, identify problems, and spot opportunities to make Community Notes better. [You can learn more about our data here.](../under-the-hood/download-data.md) {% /accordionItem %} -{% accordionItem title="Is Twitter working with any outside organizations to build this product?" %} -A wealth of relevant knowledge exists beyond Twitter’s virtual walls, so we are drawing on external research and collaborating with a variety of groups and organizations for their expertise and input. In building Community Notes, we’ve drawn on input from academic advisors whose expertise includes crowdsourcing, online juries, political science, polarization and more. Their input has helped us shape what Community Notes is today. We’re also [collaborating with professional reviewers](https://twitter.com/communitynotes/status/1422293696041603081) as part of our analysis of the helpfulness & quality of information elevated by Community Notes. +{% accordionItem title="Is X working with any outside organizations to build this product?" %} +A wealth of relevant knowledge exists beyond our virtual walls, so we are drawing on external research and collaborating with a variety of groups and organizations for their expertise and input. In building Community Notes, we’ve drawn on input from academic advisors whose expertise includes crowdsourcing, online juries, political science, polarization and more. Their input has helped us shape what Community Notes is today. We’re also [collaborating with professional reviewers](https://x.com/communitynotes/status/1422293696041603081) as part of our analysis of the helpfulness & quality of information elevated by Community Notes. {% /accordionItem %} {% /accordionSection %} -Have more questions? Feel free to Tweet or message our team [@CommunityNotes](https://twitter.com/communitynotes) +Have more questions? Feel free to post or message our team [@CommunityNotes](https://x.com/communitynotes) diff --git a/documentation/about/introduction.md b/documentation/about/introduction.md index 295e6ede..46cbc95d 100644 --- a/documentation/about/introduction.md +++ b/documentation/about/introduction.md @@ -1,27 +1,27 @@ --- title: Introduction -description: Learn everything about Community Notes, Twitter's open-source program to create a better-informed world. +description: Learn everything about Community Notes, X's open-source program to create a better-informed world. --- -![Screenshot of a mobile device showing a Tweet with a Community Note.](../images/help-rate-this-note-expanded.png) +![Screenshot of a mobile device showing a post with a Community Note.](../images/help-rate-this-note-expanded.png) -## Community Notes: a collaborative way to add helpful context to Tweets and keep people better informed +## Community Notes: a collaborative way to add helpful context to posts and keep people better informed -Community Notes aims to create a better-informed world, by empowering people on Twitter to collaboratively add helpful notes to Tweets that might be misleading. +Community Notes aims to create a better-informed world, by empowering people on X to collaboratively add helpful notes to posts that might be misleading. {% inline wrap="false" %} {% atmImage widthControl="width" width="30" src="./images/people.svg" alt="Vector icon of 3 people together" /%} {% stack %} **Contributors write and rate notes** -Contributors are people on Twitter, just like you, who [sign up](../contributing/signing-up.md) to write and rate notes. The more people that participate, the better the program becomes. +Contributors are people on X, just like you, who [sign up](../contributing/signing-up.md) to write and rate notes. The more people that participate, the better the program becomes. {% /stack %} {% /inline %} {% inline wrap="false" %} {% atmImage widthControl="width" width="30" src="./images/rate.svg" alt="Vector icon of a star, half-filled with color" /%} {% stack %} -**Only notes rated helpful by people from diverse perspectives appear on Tweets** +**Only notes rated helpful by people from diverse perspectives appear on posts** Community Notes doesn't work by majority rules. To identify notes that are helpful to a wide range of people, notes require agreement between contributors who have sometimes disagreed in their past ratings. This helps prevent one-sided ratings. @@ -32,9 +32,9 @@ Learn more about how Community Notes handles [diverse perspectives](../contribut {% inline wrap="false" %} {% atmImage widthControl="width" width="30" src="./images/world.svg" alt="Vector icon of the earth" /%} {% stack %} -**Twitter doesn’t choose what shows up, the people do** +**X doesn’t choose what shows up, the people do** -Twitter doesn’t write, rate or moderate notes (unless they break the Twitter rules.) We believe giving people a voice to make these choices together is a fair and effective way to add information that helps people stay better informed. +X doesn’t write, rate or moderate notes (unless they break X's [Rules](https://help.x.com/rules-and-policies/twitter-rules).) We believe giving people a voice to make these choices together is a fair and effective way to add information that helps people stay better informed. {% /stack %} {% /inline %} @@ -50,24 +50,24 @@ It’s important for people to understand how Community Notes works, and to be a {% box backgroundColor="gray-100" cornerRadius="radius-16" topPadding="space-12" bottomPadding="space-12" leftPadding="space-24" rightPadding="space-24" %} {% inline align="space-between" alignY="center" %} **Become a Community Notes contributor** -{% button buttonTitle="Sign up here" buttonHref="https://twitter.com/i/flow/join-birdwatch" /%} +{% button buttonTitle="Sign up here" buttonHref="https://x.com/i/flow/join-birdwatch" /%} {% /inline %} {% /box %} ## Frequently asked questions {% accordionSection %} -{% accordionItem title="How does a Tweet get a note?" %} +{% accordionItem title="How does a post get a note?" %} -Contributors can suggest a note on any Tweet. Notes are then rated for helpfulness by other contributors. Notes are only shown on Tweets if they are rated helpful by enough people from different perspectives. See how Community Notes defines and uses [differences of perspectives here](../contributing/diversity-of-perspectives.md). +Contributors can suggest a note on any post. Notes are then rated for helpfulness by other contributors. Notes are only shown on posts if they are rated helpful by enough people from different perspectives. See how Community Notes defines and uses [differences of perspectives here](../contributing/diversity-of-perspectives.md). {% /accordionItem %} {% accordionItem title="How does Community Notes prevent abuse?" %} -Community Notes works differently than the rest of Twitter. It is not a popularity contest. It aims to find notes that many people from different points of view will find helpful. It takes into account not only how many ratings a note has received, but also whether people who rated it helpful seem to come from different perspectives. Because notes need to genuinely be found helpful by people who tend to disagree, the program is more likely to identify notes that many people find helpful. Read the full details of [how this works](../contributing/diversity-of-perspectives.md). +Community Notes works differently than the rest of the platform. It is not a popularity contest. It aims to find notes that many people from different points of view will find helpful. It takes into account not only how many ratings a note has received, but also whether people who rated it helpful seem to come from different perspectives. Because notes need to genuinely be found helpful by people who tend to disagree, the program is more likely to identify notes that many people find helpful. Read the full details of [how this works](../contributing/diversity-of-perspectives.md). -Notes are also subject to Twitter rules and can be reported. +Notes are also subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules) and can be reported. {% /accordionItem %} @@ -80,19 +80,19 @@ We believe regular people can valuably contribute to identifying and adding help - [Bhuiyan, Zhang, Sehat, and Mitra 2020](https://arxiv.org/pdf/2008.09533.pdf) - [Kim and Walker 2020](https://misinforeview.hks.harvard.edu/article/leveraging-volunteer-fact-checking-to-identify-misinformation-about-covid-19-in-social-media/) -In our pilot test of Community Notes, we evaluated notes would be shown on Tweets, and have found: +In our pilot test of Community Notes, we evaluated notes would be shown on posts, and have found: -- The majority of people surveyed on Twitter found notes helpful. -- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading Tweet after reading the note about it, compared to those who saw a Tweet without a note. +- The majority of people surveyed on X found notes helpful. +- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading post after reading the note about it, compared to those who saw a post without a note. - Most notes have been rated highly on accuracy by professional reviewers; it has been rare to find a note that reviewers agree is inaccurate. -The people on Twitter span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. +The people on X span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. {% /accordionItem %} -{% accordionItem title="I have a note on my Tweet. What can I do?" %} +{% accordionItem title="I have a note on my post. What can I do?" %} -As a Tweet author, if you disagree that a note provides important context about your Tweet, you can request [additional review](../contributing/additional-review.md). +As a post author, if you disagree that a note provides important context about your post, you can request [additional review](../contributing/additional-review.md). {% /accordionItem %} diff --git a/documentation/contributing/additional-review.md b/documentation/contributing/additional-review.md index 4ea3c889..e64ed31f 100644 --- a/documentation/contributing/additional-review.md +++ b/documentation/contributing/additional-review.md @@ -1,30 +1,30 @@ --- title: Additional review -description: Request additional review of Community Notes on your Tweets. +description: Request additional review of Community Notes on your posts. navWeight: 12 --- # Additional Community review -[Community Notes](../index.md) allows volunteer contributors on Twitter to collaboratively add context (called “notes”) to Tweets they believe could be misleading. +[Community Notes](../index.md) allows volunteer contributors on X to collaboratively add context (called “notes”) to posts they believe could be misleading. -If you believe a note rated “helpful” on your Tweet doesn’t add helpful context or shouldn’t be there, you can request additional review. Reviews aren’t done by Twitter — they are done by regular people who use Twitter and have signed up to become Community Notes Contributors. +If you believe a note rated “helpful” on your post doesn’t add helpful context or shouldn’t be there, you can request additional review. Reviews aren’t done by X — they are done by regular people who use X and have signed up to become Community Notes Contributors. ## How additional reviews work: -- Tweet authors can request additional contributor review of notes that are rated helpful and are being shown on their Tweet. -- When an author requests additional review, the note is shown to contributors on the Community Notes site, which is separate from the Twitter apps and where they may then rate the note. +- Post authors can request additional contributor review of notes that are rated helpful and are being shown on their post. +- When an author requests additional review, the note is shown to contributors on the Community Notes site, which is separate from the X apps and where they may then rate the note. - All ratings are done by Community Notes contributors, who voluntarily review notes, so there’s no guarantee that more contributors will review it or that the note's rating will change. -- Twitter does not determine the outcome of reviews. The aim of Community Notes is to empower people on Twitter to determine what additional context is helpful. -- If the additional ratings change a note's status such that it is no longer rated "helpful", the note will stop being shown on the Tweet. +- X does not determine the outcome of reviews. The aim of Community Notes is to empower people on X to determine what additional context is helpful. +- If the additional ratings change a note's status such that it is no longer rated "helpful", the note will stop being shown on the post. - Additional ratings will likely come in within 24 hours following an author's request, but status can change at any time as ratings are received. -- A review of one note does not impact the status of other existing notes or newly written notes on the Tweet. If new notes are added and earn the status of Helpful, they may be shown on the Tweet. -- Notes are subject to the Twitter Rules. If you believe a note violates them, you can report it to Twitter. +- A review of one note does not impact the status of other existing notes or newly written notes on the post. If new notes are added and earn the status of Helpful, they may be shown on the post. +- Notes are subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules). If you believe a note violates them, you can report it. ## Request additional review -1. Copy your Tweet's link +1. Copy your post's link - It should look like this: https://twitter.com/{{yourdisplayname}}/status/123456789 + It should look like this: https://x.com/{{yourdisplayname}}/status/123456789 2. Paste the link into the form below. diff --git a/documentation/contributing/aliases.md b/documentation/contributing/aliases.md index be11cc88..589d78ab 100644 --- a/documentation/contributing/aliases.md +++ b/documentation/contributing/aliases.md @@ -1,28 +1,28 @@ --- title: Community Notes Aliases -description: Aliases are auto-generated display names that contributors get when they join Community Notes on Twitter. +description: Aliases are auto-generated display names that contributors get when they join Community Notes. navWeight: 4 --- # Aliases ![Screenshot of a summary screen about Community Notes Aliases](../images/alias-01.png) -We want everyone to feel comfortable contributing to Community Notes. Aliases let contributors write and rate notes without sharing their Twitter usernames. +We want everyone to feel comfortable contributing to Community Notes. Aliases let contributors write and rate notes without sharing their X usernames. -A Community Notes alias is a new, auto-generated display name that contributors get when they join Community Notes. Aliases are not publicly associated with contributors’ Twitter accounts, so everyone can write and rate notes privately. +A Community Notes alias is a new, auto-generated display name that contributors get when they join Community Notes. Aliases are not publicly associated with contributors’ X accounts, so everyone can write and rate notes privately. Further, we believe aliases have the potential to: - **Reduce bias**. By keeping focus on the content of notes rather than who’s writing them, aliases have the potential to reduce bias that people might have around specific authors. -- **Reduce polarization**. [Recent research](https://twitter.com/chris_bail/status/1379453587558952960?s=20) indicates aliases might reduce polarization by helping people feel comfortable crossing partisan lines, or criticizing their own side without the prospect of peer pressure or retribution. +- **Reduce polarization**. [Recent research](https://x.com/chris_bail/status/1379453587558952960?s=20) indicates aliases might reduce polarization by helping people feel comfortable crossing partisan lines, or criticizing their own side without the prospect of peer pressure or retribution. ### Accountability -It’s important that the benefits of aliases don’t come at the expense of accountability. To that end: all Community Notes accounts have profile pages that make it easy to see one’s past contributions and their [Writing and Rating Impact](./writing-and-rating-impact.md). Additionally, Community Notes contributors are [accountable to ratings their contributions receive](https://twitter.com/communitynotes/status/1404519791394758657), giving weight to contributors whose notes and ratings are consistently found helpful by others. +It’s important that the benefits of aliases don’t come at the expense of accountability. To that end: all Community Notes accounts have profile pages that make it easy to see one’s past contributions and their [Writing and Rating Impact](./writing-and-rating-impact.md). Additionally, Community Notes contributors are [accountable to ratings their contributions receive](https://x.com/communitynotes/status/1404519791394758657), giving weight to contributors whose notes and ratings are consistently found helpful by others. ## How to choose your alias -Contributors can visit [this link](https://twitter.com/i/communitynotes/u/me) to choose an alias. You'll have 5 random options to pick from, and at this time choices cannot be changed. +Contributors can visit [this link](https://x.com/i/communitynotes/u/me) to choose an alias. You'll have 5 random options to pick from, and at this time choices cannot be changed. ![Screen presenting different alias options the user can pick from](../images/alias-02.png) @@ -32,4 +32,4 @@ Contributors can visit [this link](https://twitter.com/i/communitynotes/u/me) to With aliases, every contributor gets a public profile. On this page, everyone can see the notes people have written in the past, as well as their [Writing and Rating Impact](./writing-and-rating-impact.md). We'll continue to add information and evolve the profile pages as we learn more about what's important to contributors when using Community Notes. -Don't hesitate to [send us feedback](http://twitter.com/communitynotes) at any time. +Don't hesitate to [send us feedback](http://x.com/communitynotes) at any time. diff --git a/documentation/contributing/diversity-of-perspectives.md b/documentation/contributing/diversity-of-perspectives.md index 4441981b..006725dd 100644 --- a/documentation/contributing/diversity-of-perspectives.md +++ b/documentation/contributing/diversity-of-perspectives.md @@ -1,20 +1,20 @@ --- title: Diversity of perspectives -description: Learn more about how Community Notes identifies notes that many people on Twitter will find helpful, including people with different points of view. +description: Learn more about how Community Notes identifies notes that many people on X will find helpful, including people with different points of view. navWeight: 9 --- # Diversity of perspectives -Community Notes aims to identify notes that many people on Twitter will find helpful, including people with different points of view. +Community Notes aims to identify notes that many people on X will find helpful, including people with different points of view. To find notes that are helpful to the broadest possible set of people, Community Notes takes into account **not only how many** contributors rated a note as helpful or unhelpful, **but also whether people who rated it seem to come from different perspectives**. -Community Notes assesses "different perspectives" entirely based on how people have rated notes in the past; Community Notes does not ask about or use any other information to do this (e.g. demographics like location, gender, or political affiliation, or data from Twitter such as follows or Tweets). This is based on the intuition that Contributors who tend to rate the same notes similarly are likely to have more similar perspectives while contributors who rate notes differently are likely to have different perspectives. If people who typically disagree in their ratings agree that a given note is helpful, it's probably a good indicator the note is helpful to people from different points of view. +Community Notes assesses "different perspectives" entirely based on how people have rated notes in the past; Community Notes does not ask about or use any other information to do this (e.g. demographics like location, gender, or political affiliation, or data from X such as follows or posts). This is based on the intuition that Contributors who tend to rate the same notes similarly are likely to have more similar perspectives while contributors who rate notes differently are likely to have different perspectives. If people who typically disagree in their ratings agree that a given note is helpful, it's probably a good indicator the note is helpful to people from different points of view. This approach has a number of benefits. First, it reflects the reality that people’s views can be nuanced, rather than defined by demographics. Second, in support of our focus on transparency, it allows people working with [Community Notes public data](../under-the-hood/download-data.md) to replicate, analyze and audit how Community Notes works, as it allows Community Notes to run entirely on publicly available data. -We are constantly evaluating ways to improve this approach (and [welcome suggestions](./feedback.md)) This current method has shown promising results in helping find quality notes: in surveys of people on Twitter in the US, the majority of respondents found notes that earned a status of "Helpful" by Community Notes contributors to be “somewhat” or “extremely” helpful — this includes people from across the political spectrum. +We are constantly evaluating ways to improve this approach (and [welcome suggestions](./feedback.md)) This current method has shown promising results in helping find quality notes: in surveys of people who use X in the US, the majority of respondents found notes that earned a status of "Helpful" by Community Notes contributors to be “somewhat” or “extremely” helpful — this includes people from across the political spectrum. -In addition to the approach described here, we also work to understand specifically how helpful notes are to people from different political perspectives. For example, by analyzing Twitter’s follow, like, and retweet graphs. It's important to note that these analyses don't directly impact the ratings of specific notes. Instead, they serve as a quality measure that links to specific notes and aids in refining our open-source algorithms. [Learn more here](../under-the-hood/guardrails.md). +In addition to the approach described here, we also work to understand specifically how helpful notes are to people from different political perspectives. For example, by analyzing X's follow, like, and repost graphs. It's important to note that these analyses don't directly impact the ratings of specific notes. Instead, they serve as a quality measure that links to specific notes and aids in refining our open-source algorithms. [Learn more here](../under-the-hood/guardrails.md). If you're interested in the math and code powering Community Notes, take a look at our [Note Ranking: Under the Hood](../under-the-hood/note-ranking-code.md) section. diff --git a/documentation/contributing/examples.md b/documentation/contributing/examples.md index 4d037969..d193d222 100644 --- a/documentation/contributing/examples.md +++ b/documentation/contributing/examples.md @@ -1,17 +1,17 @@ --- title: Note-writing tips -description: Learn what makes a helpful Community Note on Twitter. +description: Learn what makes a helpful Community Note. navWeight: 11 --- # Examples and note-writing tips -Great notes provide helpful context to Tweets and leave people better informed. But what makes a note helpful? +Great notes provide helpful context to posts and leave people better informed. But what makes a note helpful? ## Helpful attributes - Cites high-quality sources - Easy to understand -- Directly addresses the Tweet’s claim +- Directly addresses the post’s claim - Provides important context - Neutral or unbiased language @@ -26,7 +26,7 @@ Helpful notes steer clear of all of these unhelpful attributes: - Typos or unclear language - Misses key points or irrelevant - Argumentative or biased language -- Note not needed on this Tweet +- Note not needed on this post - Harassment or abuse The following are examples of notes that illustrate what makes a note helpful (versus unhelpful): @@ -37,13 +37,13 @@ The following are examples of notes that illustrate what makes a note helpful (v {% tweet source="custom" displayName="Kian" username="naurelvr49" avatar="/content/dam/communitynotes-twitter/images/en/user-avatar.jpeg" text="Another big day at the Supreme Court ending with a police car being lit on fire. 😨" image="/content/dam/communitynotes-twitter/images/en/tweet-example-1.png" /%} {% /box %} -{% inlineCallout title="Helpful note" variant="success" children="

“This photo was taken in Toronto at a protest of the 2010 G20 summit where G20 officials met to discuss the world economy, not at the Supreme Court in Washington D.C. You can see the car in this photo being lit on fire via a local Toronto news broadcast: https://www.youtube.com/watch...”

" /%} +{% inlineCallout title="Helpful note" variant="success" children="

“This photo was taken in Toronto at a protest of the 2010 G20 summit where G20 officials met to discuss the world economy, not at the Supreme Court in Washington D.C. You can see the car in this photo being lit on fire via a local Toronto news broadcast: https://www.youtube.com/watch...”

" /%} -{% inlineCallout title="Helpful note" variant="success" children="

“The last time it was reported that a police car was lit on fire outside the Supreme Court was on 7/15/2020. This photo is not from that incident. This photo was taken in Toronto at a protest of the 2010 G20 summit. https://www.cbsnews.com/news/..., https://www.cp24.com/breaking/...”

" /%} +{% inlineCallout title="Helpful note" variant="success" children="

“The last time it was reported that a police car was lit on fire outside the Supreme Court was on 7/15/2020. This photo is not from that incident. This photo was taken in Toronto at a protest of the 2010 G20 summit. https://www.cbsnews.com/news/..., https://www.cp24.com/breaking/...”

" /%} {% inlineCallout title="Unhelpful note" variant="error" children="

“Wow, people really hate cops. They just park there car to go into the office and it’s suddenly fair game to be torched. Some of you need to relax.”

" /%} -{% inlineCallout title="Unhelpful note" variant="error" children="

“This is totally made up. There are no reports of a fire outside the Supreme Court today. Twitter should ban this account for spreading lies. https://www.supremecourt.gov/”

" /%} +{% inlineCallout title="Unhelpful note" variant="error" children="

“This is totally made up. There are no reports of a fire outside the Supreme Court today. X should ban this account for spreading lies. https://www.supremecourt.gov/”

" /%} ## Example 2 @@ -51,9 +51,9 @@ The following are examples of notes that illustrate what makes a note helpful (v {% tweet source="custom" displayName="Kian" username="naurelvr49" avatar="/content/dam/communitynotes-twitter/images/en/user-avatar.jpeg" text="Protesters in Medyka, Poland, a small town that borders Ukraine, respond to the number of Ukrainian refugees who have relocated to Poland since Russia invaded. They say that the extension of Ukrainian work visas is starting to limit job opportunities for native Poles." image="/content/dam/communitynotes-twitter/images/en/tweet-example-2.png" /%} {% /box %} -{% inlineCallout title="Helpful note" variant="success" children="

“Medkya is the largest crossing point for Ukrainian refugees into Poland but there have been no protests reported. In fact, the Poles have welcomed refugees and called for more resources to support the Ukrainian people. https://www.unhcr.org/en-us/news/stories...”

" /%} +{% inlineCallout title="Helpful note" variant="success" children="

“Medkya is the largest crossing point for Ukrainian refugees into Poland but there have been no protests reported. In fact, the Poles have welcomed refugees and called for more resources to support the Ukrainian people. https://www.unhcr.org/en-us/news/stories...”

" /%} -{% inlineCallout title="Helpful note" variant="success" children="

“A reverse image search places this photo at the 2/21/17 Women’s March in Portland, Oregon. And there are no recorded protests in response to Ukrainian refugees crossing into Medyka, Poland. https://www.npr.org/2022/05/19/...”

" /%} +{% inlineCallout title="Helpful note" variant="success" children="

“A reverse image search places this photo at the 2/21/17 Women’s March in Portland, Oregon. And there are no recorded protests in response to Ukrainian refugees crossing into Medyka, Poland. https://www.npr.org/2022/05/19/...”

" /%} {% inlineCallout title="Unhelpful note" variant="error" children="

“One of the protes signs is in English?! Unlikely this protest is in small Polish town where few people can speak English. https://en.wikipedia.org/wiki/...”

" /%} diff --git a/documentation/contributing/feedback.md b/documentation/contributing/feedback.md index d7c6acca..cd3922df 100644 --- a/documentation/contributing/feedback.md +++ b/documentation/contributing/feedback.md @@ -13,6 +13,6 @@ Community Notes is early in its development, and our goal is to build it in the - Ideas for product updates or features for our roadmap - Analyses done with our [public data](../under-the-hood/download-data.md) or feedback about the data. -Talk directly with the team building Community Notes — follow, Tweet or DM us [@CommunityNotes](https://twitter.com/communitynotes). We can't wait to hear from you. +Talk directly with the team building Community Notes — follow, post or DM us [@CommunityNotes](https://x.com/communitynotes). We can't wait to hear from you. -{% button buttonTitle="Tweet @CommunityNotes" buttonHref="https://twitter.com/communitynotes" /%} +{% button buttonTitle="Post @CommunityNotes" buttonHref="https://x.com/communitynotes" /%} diff --git a/documentation/contributing/getting-started.md b/documentation/contributing/getting-started.md index 3c425dac..1274fa65 100644 --- a/documentation/contributing/getting-started.md +++ b/documentation/contributing/getting-started.md @@ -1,6 +1,6 @@ --- title: Getting started -description: Step-by-step guide for getting started with Community Notes on Twitter. +description: Step-by-step guide for getting started with Community Notes on X. navWeight: 1 --- # Getting started @@ -9,30 +9,30 @@ navWeight: 1 Anyone who meets the [eligibility criteria](../contributing/signing-up.md) can sign up to become a Community Notes contributor. As there are important nuances in each market, we’ll expand the contributor base country-by-country. -{% button buttonTitle="Sign up" buttonHref="https://communitynotes.twitter.com/guide/en/contributing/sign-up" /%} +{% button buttonTitle="Sign up" buttonHref="https://communitynotes.x.com/guide/en/contributing/sign-up" /%} ## 2. Select an alias so you can keep your identity private As a Community Notes contributor, you can now rate the helpfulness of all notes. But first, choose an alias to contribute under (it’s like a pen name!) -{% button buttonTitle="Sign up" buttonHref="https://twitter.com/i/communitynotes/u/me" /%} +{% button buttonTitle="Sign up" buttonHref="https://x.com/i/communitynotes/u/me" /%} ## 3. Begin rating notes -Your ratings help determine which notes are made visible on Twitter. To start rating notes, navigate to Community Notes from your Twitter app navigation bar. There, you'll find the notes where your ratings can have the most impact. +Your ratings help determine which notes are made visible on X. To start rating notes, navigate to Community Notes from your X app navigation bar. There, you'll find the notes where your ratings can have the most impact. -![Twitter app with main navigation menu open and the Community Notes option highlighted.](../images/navigate-to-home.png) +![X app with main navigation menu open and the Community Notes option highlighted.](../images/navigate-to-home.png) -{% button buttonTitle="Rate notes" buttonHref="https://twitter.com/i/communitynotes/needs_your_help" /%} +{% button buttonTitle="Rate notes" buttonHref="https://x.com/i/communitynotes/needs_your_help" /%} ## 4. Soon, you'll unlock the ability to write your own notes! To unlock the ability to write notes, you need to earn a Rating Impact of 5. You earn Impact when your rating helps a note earn a helpful (or unhelpful) status! You can track your progress on the Rating Impact section of your Community Notes profile. ![Screen showing an anonymous contributor, their avatar, and alias name.](../images/rating-impact-welcome.png) -{% button buttonTitle="Track your impact" buttonHref="https://twitter.com/i/communitynotes/u/me" /%} +{% button buttonTitle="Track your impact" buttonHref="https://x.com/i/communitynotes/u/me" /%} ### Confused? Need a hand? Have feedback? -Don’t hesitate to get in touch with the Community Notes team! You can reach us by Tweeting to or DMing -[@CommunityNotes](https://twitter.com/communitynotes). We’re so glad you’re here! +Don’t hesitate to get in touch with the Community Notes team! You can reach us by posting to or DMing +[@CommunityNotes](https://x.com/communitynotes). We’re so glad you’re here! diff --git a/documentation/contributing/notes-on-media.md b/documentation/contributing/notes-on-media.md index 64f780f8..5ee8f1f9 100644 --- a/documentation/contributing/notes-on-media.md +++ b/documentation/contributing/notes-on-media.md @@ -1,24 +1,24 @@ --- title: Notes on Media -description: Community Notes on Tweets that contain images or videos +description: Community Notes on posts that contain images or videos navWeight: 11 --- # Notes on Media -Community Notes are frequently added to Tweets that feature images or videos. In many cases, these notes can provide valuable context, not just for a single Tweet, but for any Tweet containing the same media. +Community Notes are frequently added to posts that feature images or videos. In many cases, these notes can provide valuable context, not just for a single post, but for any post containing the same media. -Contributors with a Writing Impact score of 10 or above have the option to write notes about the media found within Tweets, as opposed to focusing on the specific Tweet. Contributors should select this option when they believe the context added would be helpful independently of the Tweet the note is attached to. +Contributors with a Writing Impact score of 10 or above have the option to write notes about the media found within posts, as opposed to focusing on the specific post. Contributors should select this option when they believe the context added would be helpful independently of the post the note is attached to. -![Note-writing form with option: Is your note about the Tweet or the image?](../images/notes-on-media-01.png) +![Note-writing form with option: Is your note about the post or the image?](../images/notes-on-media-01.png) -Tagging notes as “about the image” makes them visible on all Tweets that our system identifies as containing the same image. These notes, when deemed Helpful, accumulate view counts from all the Tweets they appear in, but only count as one Writing and Rating Impact for the author and raters. +Tagging notes as “about the image” makes them visible on all posts that our system identifies as containing the same image. These notes, when deemed Helpful, accumulate view counts from all the posts they appear in, but only count as one Writing and Rating Impact for the author and raters. -![Multiple Tweets with same image and the same note applied](../images/notes-on-media-02.png) +![Multiple posts with same image and the same note applied](../images/notes-on-media-02.png) -To ensure transparency, raters will see a disclaimer indicating that notes about the image may appear across multiple Tweets. This way, they'll be aware that the context provided by these notes extends beyond the specific Tweet they're currently viewing. +To ensure transparency, raters will see a disclaimer indicating that notes about the image may appear across multiple posts. This way, they'll be aware that the context provided by these notes extends beyond the specific post they're currently viewing. -When someone rates a media note, the rating is associated with the Tweet on which the note appeared. This allows Community Notes to identify cases where a note may not apply to a specific Tweet. +When someone rates a media note, the rating is associated with the post on which the note appeared. This allows Community Notes to identify cases where a note may not apply to a specific post. -![Rating screen with callout: This note is about the image and could be shown on all Tweets that contain the same image](../images/notes-on-media-03.png) +![Rating screen with callout: This note is about the image and could be shown on all posts that contain the same image](../images/notes-on-media-03.png) -Currently, this feature is experimental and only supports Tweets with a single image. We're actively working on expanding it to support Tweets with multiple images, GIFs, and videos. Stay tuned for updates. +Currently, this feature is experimental and only supports posts with a single image. We're actively working on expanding it to support posts with multiple images, GIFs, and videos. Stay tuned for updates. diff --git a/documentation/contributing/notes-on-twitter.md b/documentation/contributing/notes-on-twitter.md index 535959ac..3c6fb52e 100644 --- a/documentation/contributing/notes-on-twitter.md +++ b/documentation/contributing/notes-on-twitter.md @@ -1,26 +1,26 @@ --- -title: Notes shown on Twitter -description: How do Community Notes start showing on Twitter? +title: Notes shown on X +description: How do Community Notes start showing on X? navWeight: 10 --- -# Notes shown on Twitter +# Notes shown on X -Community Notes' goal is to show helpful context to people when they come across potentially misleading information on Twitter. When a note is rated Helpful by contributors, it starts being shown directly on the Tweet. Here's how they show up: +Community Notes' goal is to show helpful context to people when they come across potentially misleading information on X. When a note is rated Helpful by contributors, it starts being shown directly on the post. Here's how they show up: -![A Tweet with a Community Note.](../images/help-rate-this-note-expanded.png) +![a post with a Community Note.](../images/help-rate-this-note-expanded.png) -## How do notes start showing on Twitter? +## How do notes start showing on X? All proposed notes start with the status of **Needs More Ratings**, and are shown to Contributors in order to gather ratings. -If you are a Community Notes contributor, you'll see a prompt on Tweets that have proposed notes. Tap the prompt to rate those notes. +If you are a Community Notes contributor, you'll see a prompt on posts that have proposed notes. Tap the prompt to rate those notes. -![Tweet with a Community Note prompt](../images/help-rate-this-note.png) +![Post with a Community Note prompt](../images/help-rate-this-note.png) -If enough contributors from [different points of view](./diversity-of-perspectives.md) rate a note as **helpful**, it earns a status of "Helpful", and starts showing on Twitter. +If enough contributors from [different points of view](./diversity-of-perspectives.md) rate a note as **helpful**, it earns a status of "Helpful", and starts showing on X. -If enough contributors rate a note as **not helpful**, it reaches a status of "Not Helpful", and is not shown on Twitter. Contributors whose notes consistently reach this status may have their [writing ability temporarily locked](./writing-ability.md). +If enough contributors rate a note as **not helpful**, it reaches a status of "Not Helpful", and is not shown on X. Contributors whose notes consistently reach this status may have their [writing ability temporarily locked](./writing-ability.md). ![Three Community notes with different statuses](../images/note-statuses.png) -Note statuses are updated as new ratings come in, so notes may show on Twitter and then disappear. Statuses are locked after 2 weeks. +Note statuses are updated as new ratings come in, so notes may show on X and then disappear. Statuses are locked after 2 weeks. diff --git a/documentation/contributing/notifications.md b/documentation/contributing/notifications.md index 0938e1e6..e161748a 100644 --- a/documentation/contributing/notifications.md +++ b/documentation/contributing/notifications.md @@ -5,23 +5,23 @@ navWeight: 13 --- # Notifications -## Readers added a note to a Tweet to which you replied, Liked or Retweeted +## Readers added a note to a post to which you replied, Liked or reposted -Sometimes, people read a Tweet before a Community Note appears on it. To give people more context in those cases, Community Notes sends notifications to everyone who has replied to, Liked or Retweeted a Tweet after a note starts showing on it. +Sometimes, people read a post before a Community Note appears on it. To give people more context in those cases, Community Notes sends notifications to everyone who has replied to, Liked or reposted a post after a note starts showing on it. These appear in the notifications tab. They look like this: -![Twitter app with a notification showing someone a note that's been added to a Tweet to which they replied, Liked or Rewteeted](../images/tweet-liked-alert.png) +![X app with a notification showing someone a note that's been added to a post to which they replied, Liked or Rewteeted](../images/tweet-liked-alert.png) -These notifications are sent when a note has been showing on a Tweet for 24 hours, has been found particularly helpful (indicated by a [note intercept score](../under-the-hood/note-ranking-code.md) of 0.45 or higher), and was written within the past week. +These notifications are sent when a note has been showing on a post for 24 hours, has been found particularly helpful (indicated by a [note intercept score](../under-the-hood/note-ranking-code.md) of 0.45 or higher), and was written within the past week. ## Alerts when a note needs your help -Notes have the most impact when they get rated quickly and start showing on Tweets early. To ensure contributors don't miss the opportunity to rate notes on Tweets getting a lot of attention, Community Notes sends alerts requesting help from time to time. +Notes have the most impact when they get rated quickly and start showing on posts early. To ensure contributors don't miss the opportunity to rate notes on posts getting a lot of attention, Community Notes sends alerts requesting help from time to time. If you are a contributor, you'll see these alerts in your notifications tab. They look like this: -![Twitter app with a new notification at the top requesting help from the contributor](../images/alerts-tab.png) +![X app with a new notification at the top requesting help from the contributor](../images/alerts-tab.png) To learn more about how Community Notes decides which notes trigger these alerts, see [Under the Hood – Needs Your Help](../under-the-hood/timeline-tabs.md) @@ -31,7 +31,7 @@ The default setting for all contributors is "Often", which means you'll start by ![One screenshot showing a Community Notes Contributor profile page, highlighting the settings button on the top right. Another screenshot showing the settings screen where contributors can edit their alert frequency](../images/alerts-settings.png) -{% button buttonTitle="Open notification settings" buttonHref="https://www.twitter.com/i/communitynotes/notification_settings" /%} +{% button buttonTitle="Open notification settings" buttonHref="https://www.x.com/i/communitynotes/notification_settings" /%} ## Other notifications diff --git a/documentation/contributing/phone-faq.md b/documentation/contributing/phone-faq.md index 69fc4ec6..3cdc28ac 100644 --- a/documentation/contributing/phone-faq.md +++ b/documentation/contributing/phone-faq.md @@ -10,13 +10,13 @@ If you're having trouble with your phone number in Community Notes, here's what ## If your phone number is not verified -You can verify your phone number by following [Twitter's Help Center instructions](https://help.twitter.com/en/managing-your-account/how-to-add-a-phone-number-to-your-account). +You can verify your phone number by following [X's Help Center instructions](https://help.x.com/en/managing-your-account/how-to-add-a-phone-number-to-your-account). -## If your phone number is associated with more than one Twitter account +## If your phone number is associated with more than one X account -If your phone number is associated with multiple Twitter accounts, only the account that most recently verified that number will be able to access Community Notes. +If your phone number is associated with multiple X accounts, only the account that most recently verified that number will be able to access Community Notes. -You can verify your phone number on the account you want to use with Community Notes by following [Twitter's Help Center instructions](https://help.twitter.com/en/managing-your-account/how-to-add-a-phone-number-to-your-account). +You can verify your phone number on the account you want to use with Community Notes by following [X's Help Center instructions](https://help.x.com/en/managing-your-account/how-to-add-a-phone-number-to-your-account). You can optionally remove your phone number from other accounts by going into **Privacy & Settings > Your Account > Account Information > Phone** diff --git a/documentation/contributing/rating-notes.md b/documentation/contributing/rating-notes.md index 7b065797..9e60ea6e 100644 --- a/documentation/contributing/rating-notes.md +++ b/documentation/contributing/rating-notes.md @@ -5,25 +5,25 @@ navWeight: 5 --- # Rating notes -Ratings help identify which notes are helpful enough to show on Twitter and notes that are not helpful and should not be shown. +Ratings help identify which notes are helpful enough to show on X and notes that are not helpful and should not be shown. ## Rating notes on the Community Notes site -Go to the [**Community Notes Home page**](https://communitynotes.twitter.com) and choose any Tweet to see notes that contributors have added to it. There are three main timelines on this page: +Go to the [**Community Notes Home page**](https://communitynotes.x.com) and choose any post to see notes that contributors have added to it. There are three main timelines on this page: -- **Needs your help:** Contains Tweets with notes where you can have the most impact when rating. This tab is only visible to Community Notes contributors. +- **Needs your help:** Contains posts with notes where you can have the most impact when rating. This tab is only visible to Community Notes contributors. -- **New:** Contains Tweets with the most recent notes. +- **New:** Contains posts with the most recent notes. -- **Rated Helpful:** Contains Tweets with at least one note rated helpful by the community. +- **Rated Helpful:** Contains posts with at least one note rated helpful by the community. -![Community Notes home page, showing Tweets with notes to be rated](../images/home.png) +![Community Notes home page, showing posts with notes to be rated](../images/home.png) -## Rating notes on Twitter +## Rating notes on X -Community Notes contributors and people on Twitter may also see notes and prompts in their timeline when browsing Twitter. +Community Notes contributors and people on X may also see notes and prompts in their timeline when browsing X. -![Tweet with a community notes prompt](../images/notes-on-twitter.png) +![post with a community notes prompt](../images/notes-on-twitter.png) ## Rating Impact diff --git a/documentation/contributing/signing-up.md b/documentation/contributing/signing-up.md index ac65e2c8..b7fed28f 100644 --- a/documentation/contributing/signing-up.md +++ b/documentation/contributing/signing-up.md @@ -1,6 +1,6 @@ --- title: Signing up -description: Sign up now and become a Community Notes contributor on Twitter +description: Sign up now and become a Community Notes contributor on X navWeight: 2 --- # Signing up @@ -9,7 +9,7 @@ Sign ups are open to everyone. We want anyone to be able to participate and know As there are important nuances in each market, we’ll expand the contributor base country-by-country. -{% button buttonTitle="Sign up here" buttonHref="https://twitter.com/i/flow/join-birdwatch" /%} +{% button buttonTitle="Sign up here" buttonHref="https://x.com/i/flow/join-birdwatch" /%} --- @@ -17,11 +17,11 @@ As there are important nuances in each market, we’ll expand the contributor ba To become a Community Notes contributor, accounts must have: -1. **No recent notice of Twitter Rules violations** +1. **No recent notice of violations of X's [Rules](https://help.x.com/rules-and-policies/twitter-rules)** Intended to reduce the likelihood of abusive contributions. -2. **Joined Twitter at least 6 months ago** +2. **Joined X at least 6 months ago** Intended to reduce spam. @@ -40,14 +40,14 @@ We want anyone to be able to contribute to Community Notes, and may remove or mo ## Admissions -Our goal is to admit applicants on a periodic basis. We will admit all contributors who meet the required criteria, but if we have more applicants than slots, we will randomly admit accounts, prioritizing those more likely to participate due to having been recently active on Twitter, and those that tend to follow and engage with different tweets than existing contributors do — so as to reduce the likelihood that contributors would be predominantly from one ideology, background, or interest space. +Our goal is to admit applicants on a periodic basis. We will admit all contributors who meet the required criteria, but if we have more applicants than slots, we will randomly admit accounts, prioritizing those more likely to participate due to having been recently active on X, and those that tend to follow and engage with different posts than existing contributors do — so as to reduce the likelihood that contributors would be predominantly from one ideology, background, or interest space. ## Contributions are public -To promote transparency, all contributions to Community Notes are [anonymized](./aliases.md) and publicly visible on the Community Notes site, **even if an account’s Tweets are protected**. +To promote transparency, all contributions to Community Notes are [anonymized](./aliases.md) and publicly visible on the Community Notes site, **even if an account’s posts are protected**. ## Rules -Contributions are subject to Twitter [Rules](https://help.twitter.com/en/rules-and-policies/twitter-rules), [Terms of Service](https://twitter.com/tos) and [Privacy Policy](https://twitter.com/privacy). Failure to abide by the rules can result in removal from the Community Notes program, and/or other remediations. +Contributions are subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules), [Terms of Service](https://x.com/tos) and [Privacy Policy](https://x.com/privacy). Failure to abide by the rules can result in removal from the Community Notes program, and/or other remediations. Anyone can report notes they believe aren’t in accordance with those rules by clicking or tapping the ••• menu on a note, and then selecting "Report", or by using this form. diff --git a/documentation/contributing/top-contributors.md b/documentation/contributing/top-contributors.md new file mode 100644 index 00000000..c51b739f --- /dev/null +++ b/documentation/contributing/top-contributors.md @@ -0,0 +1,24 @@ +--- +title: Top Contributors +description: Learn more about top Community Note contributors +navWeight: 13 +--- +# Top Writers + +Top Writers are contributors recognized for writing a significant number of notes that are found helpful by others. To earn this status, contributors must meet the following criteria: + +- Writing Impact of 10 or more +- At least 4% of all their notes are currently rated “Helpful” + +Top Note Writers get access to: + +**Writing notes about media** +Top Writers can write notes about media featured on multiple posts, keeping many more people better informed. [Learn more](./notes-on-media.md). + +**Priority for note alerts** +Top Writers’ note proposals are more likely to trigger notifications to get rater’s attention. + +**Badge in alias profile** +Top Writers get a badge in their Community Notes profile. + +A contributor’s Top Writer status can always change as their notes are rated by others. diff --git a/documentation/contributing/values.md b/documentation/contributing/values.md index bb61b476..37aecf5f 100644 --- a/documentation/contributing/values.md +++ b/documentation/contributing/values.md @@ -5,11 +5,11 @@ navWeight: 3 --- # Values -Community Notes aims to provide context on Tweets in a way that a broad set of people with diverse perspectives find helpful. We ask that all Community Notes contributors agree to uphold three core values: +Community Notes aims to provide context on posts in a way that a broad set of people with diverse perspectives find helpful. We ask that all Community Notes contributors agree to uphold three core values: ## 1. Contribute to build understanding -Community Notes seeks to provide helpful context so people can stay better informed about Tweets they see. It’s not a place for quick dunks, personal opinions, or insults. Write clear, evidence-based notes to help everyone, even skeptics, better understand a Tweet and why it might be misleading. +Community Notes seeks to provide helpful context so people can stay better informed about posts they see. It’s not a place for quick dunks, personal opinions, or insults. Write clear, evidence-based notes to help everyone, even skeptics, better understand a post and why it might be misleading. ## 2. Be helpful, even to those who disagree @@ -17,4 +17,4 @@ Genuinely and constructively contribute to help others stay informed. Do not att ## 3. Act in good faith -Community Notes aims to help all people better understand issues discussed in Tweets, including people who may hold different viewpoints. Contribute in a way that even those who may disagree with you might find helpful and respectful. Avoid hateful, derogatory, or inflammatory language. +Community Notes aims to help all people better understand issues discussed in posts, including people who may hold different viewpoints. Contribute in a way that even those who may disagree with you might find helpful and respectful. Avoid hateful, derogatory, or inflammatory language. diff --git a/documentation/contributing/welcome.md b/documentation/contributing/welcome.md index ef52f0a2..b97394e1 100644 --- a/documentation/contributing/welcome.md +++ b/documentation/contributing/welcome.md @@ -10,15 +10,15 @@ You're now a Community Notes contributor 🎉 As a Community Notes contributor, you can now rate the helpfulness of all notes. But first, choose an alias to contribute under (it’s like a pen name!) -{% button buttonTitle="Select your alias" buttonHref="https://twitter.com/i/communitynotes/u/me" /%} +{% button buttonTitle="Select your alias" buttonHref="https://x.com/i/communitynotes/u/me" /%} ## 2. Begin rating notes -Your ratings help determine which notes are made visible on Twitter. To start rating notes, navigate to Community Notes from your Twitter app navigation bar. There, you'll find the notes where your ratings can have the most impact. +Your ratings help determine which notes are made visible on X. To start rating notes, navigate to Community Notes from your app's navigation bar. There, you'll find the notes where your ratings can have the most impact. -![Twitter app with the main navigation menu open. Community Notes option is highlighted.](../images/navigate-to-home.png) +![X app with the main navigation menu open. Community Notes option is highlighted.](../images/navigate-to-home.png) -{% button buttonTitle="Rate notes" buttonHref="https://twitter.com/i/communitynotes/needs_your_help" /%} +{% button buttonTitle="Rate notes" buttonHref="https://x.com/i/communitynotes/needs_your_help" /%} ## 3. Soon, you'll unlock the ability to write your own notes! @@ -26,9 +26,9 @@ To unlock the ability to write notes, you need to earn a Rating Impact of 5. You ![Screen showing an anonymous contributor profile. Highlighted section reads: 0 Rating Impact. You need an Impact of 5 to unlock note writing](../images/rating-impact-welcome.png) -{% button buttonTitle="Track your impact" buttonHref="https://twitter.com/i/communitynotes/u/me" /%} +{% button buttonTitle="Track your impact" buttonHref="https://x.com/i/communitynotes/u/me" /%} ### Confused? Need a hand? Have feedback? -Don’t hesitate to get in touch with the Community Notes team! You can reach us by Tweeting to or DMing -[@CommunityNotes](https://twitter.com/communitynotes). We’re glad you’re here! +Don’t hesitate to get in touch with the Community Notes team! You can reach us by posting to or DMing +[@CommunityNotes](https://x.com/communitynotes). We’re glad you’re here! diff --git a/documentation/contributing/writing-and-rating-impact.md b/documentation/contributing/writing-and-rating-impact.md index c7970ea1..8f36fbbd 100644 --- a/documentation/contributing/writing-and-rating-impact.md +++ b/documentation/contributing/writing-and-rating-impact.md @@ -35,11 +35,11 @@ Note statuses can change any time as more ratings come in, and can be overturned ### How do I increase my Rating Impact? -To increase this impact, you should look for notes that need more ratings, and rate them. The best ways to do this are by browsing the [Needs Your Help](https://twitter.com/i/communitynotes/needs_your_help) tab in Community Notes, and by looking out for alerts when a note needs your rating. +To increase this impact, you should look for notes that need more ratings, and rate them. The best ways to do this are by browsing the [Needs Your Help](https://x.com/i/communitynotes/needs_your_help) tab in Community Notes, and by looking out for alerts when a note needs your rating. 1. **Help identify notes as Helpful** - When you help a note earn the status of Helpful, it gets shown on Twitter as context, helping keep people informed. + When you help a note earn the status of Helpful, it gets shown on X as context, helping keep people informed. 2. **Help identify notes as Not Helpful** diff --git a/documentation/contributing/writing-notes.md b/documentation/contributing/writing-notes.md index 94d864e2..dbfbd1f3 100644 --- a/documentation/contributing/writing-notes.md +++ b/documentation/contributing/writing-notes.md @@ -1,25 +1,25 @@ --- title: Writing notes -description: A step-by-step guide on how to write Community Notes on Twitter. +description: A step-by-step guide on how to write Community Notes on X. navWeight: 6 --- # Writing notes -Anyone can read and rate Community Notes, but only contributors who've [unlocked the ability to write](./writing-ability.md) can add new notes to Tweets. Here's how to add a note: +Anyone can read and rate Community Notes, but only contributors who've [unlocked the ability to write](./writing-ability.md) can add new notes to posts. Here's how to add a note: -1. Click or tap the ••• menu on the Tweet's top right and then **Write a Community Note**. +1. Click or tap the ••• menu on the post's top right and then **Write a Community Note**. -![A Tweet with the options menu open. “Write a Community Note” listed as the last item](../images/writing-notes.png) +![a post with the options menu open. “Write a Community Note” listed as the last item](../images/writing-notes.png) -2. Answer the required multiple choice questions and write the context you feel would help others understand why that Tweet is or is not misleading. +2. Answer the required multiple choice questions and write the context you feel would help others understand why that post is or is not misleading. - If your note claims Tweet is potentially misleading and would benefit from added context, it will be eligible to show on the Tweet if it is rated as helpful by enough people. + If your note claims post is potentially misleading and would benefit from added context, it will be eligible to show on the post if it is rated as helpful by enough people. - If you claim the Tweet is not misleading, you note will be shown to other raters as additional information, but won't show on the Tweet if it earns a status of Helpful. + If you claim the post is not misleading, you note will be shown to other raters as additional information, but won't show on the post if it earns a status of Helpful. 3. After it’s submitted, the note will be available on the Community Notes site for other contributors to read and rate. -4. If your note earns the status of Helpful, it will be [shown on Twitter as context](./notes-on-twitter) and you'll earn [Writing Impact](./writing-and-rating-impact.md). +4. If your note earns the status of Helpful, it will be [shown on X as context](./notes-on-twitter) and you'll earn [Writing Impact](./writing-and-rating-impact.md). 5. You can delete your notes at any time by clicking or tapping the ••• icon on a note. @@ -43,4 +43,4 @@ For example: ### What makes a good Community Note? -{% button buttonTitle="See examples" buttonHref="https://communitynotes.twitter.com/guide/en/contributing/examples" /%} +{% button buttonTitle="See examples" buttonHref="https://communitynotes.x.com/guide/en/contributing/examples" /%} diff --git a/documentation/index.md b/documentation/index.md index 034654b2..9b76fd5b 100644 --- a/documentation/index.md +++ b/documentation/index.md @@ -1,27 +1,27 @@ --- title: Introduction -description: Learn everything about Community Notes, Twitter's open-source program to create a better-informed world. +description: Learn everything about Community Notes, X's open-source program to create a better-informed world. --- -![Screenshot of a mobile device showing a Tweet with a Community Note.](./images/help-rate-this-note-expanded.png) +![Screenshot of a mobile device showing a post with a Community Note.](./images/help-rate-this-note-expanded.png) -## Community Notes: a collaborative way to add helpful context to Tweets and keep people better informed +## Community Notes: a collaborative way to add helpful context to posts and keep people better informed -Community Notes aims to create a better-informed world, by empowering people on Twitter to collaboratively add helpful notes to Tweets that might be misleading. +Community Notes aims to create a better-informed world, by empowering people on X to collaboratively add helpful notes to posts that might be misleading. {% inline wrap="false" %} {% atmImage maxWidth="40" width="40" src="./images/people.svg" alt="Vector icon of 3 people together" /%} {% stack %} **Contributors write and rate notes** -Contributors are people on Twitter, just like you, who [sign up](./contributing/signing-up.md) to write and rate notes. The more people that participate, the better the program becomes. +Contributors are people on X, just like you, who [sign up](./contributing/signing-up.md) to write and rate notes. The more people that participate, the better the program becomes. {% /stack %} {% /inline %} {% inline wrap="false" %} {% atmImage src="./images/rate.svg" alt="Vector icon of a star, half-filled with color" maxWidth="40" width="40" /%} {% stack %} -**Only notes rated helpful by people from diverse perspectives appear on Tweets** +**Only notes rated helpful by people from diverse perspectives appear on posts** Community Notes doesn't work by majority rules. To identify notes that are helpful to a wide range of people, notes require agreement between contributors who have sometimes disagreed in their past ratings. This helps prevent one-sided ratings. @@ -32,9 +32,9 @@ Learn more about how Community Notes handles [diverse perspectives](./contributi {% inline wrap="false" %} {% atmImage maxWidth="40" width="40" src="./images/world.svg" alt="Vector icon of the earth" /%} {% stack %} -**Twitter doesn’t choose what shows up, the people do** +**X doesn’t choose what shows up, the people do** -Twitter doesn’t write, rate or moderate notes (unless they break the Twitter rules.) We believe giving people a voice to make these choices together is a fair and effective way to add information that helps people stay better informed. +X doesn’t write, rate or moderate notes (unless they break X's [Rules](https://help.x.com/rules-and-policies/twitter-rules).) We believe giving people a voice to make these choices together is a fair and effective way to add information that helps people stay better informed. {% /stack %} {% /inline %} @@ -50,24 +50,24 @@ It’s important for people to understand how Community Notes works, and to be a {% box backgroundColor="gray-100" cornerRadius="radius-16" topPadding="space-12" bottomPadding="space-12" leftPadding="space-24" rightPadding="space-24" %} {% inline align="space-between" alignY="center" %} **Become a Community Notes contributor** -{% button buttonTitle="Sign up here" buttonHref="https://twitter.com/i/flow/join-birdwatch" /%} +{% button buttonTitle="Sign up here" buttonHref="https://x.com/i/flow/join-birdwatch" /%} {% /inline %} {% /box %} ## Frequently asked questions {% accordionSection %} -{% accordionItem title="How does a Tweet get a note?" %} +{% accordionItem title="How does a post get a note?" %} -Contributors can suggest a note on any Tweet. Notes are then rated for helpfulness by other contributors. Notes are only shown on Tweets if they are rated helpful by enough people from different perspectives. See how Community Notes defines and uses [differences of perspectives here](./contributing/diversity-of-perspectives.md). +Contributors can suggest a note on any post. Notes are then rated for helpfulness by other contributors. Notes are only shown on posts if they are rated helpful by enough people from different perspectives. See how Community Notes defines and uses [differences of perspectives here](./contributing/diversity-of-perspectives.md). {% /accordionItem %} {% accordionItem title="How does Community Notes prevent abuse?" %} -Community Notes works differently than the rest of Twitter. It is not a popularity contest. It aims to find notes that many people from different points of view will find helpful. It takes into account not only how many ratings a note has received, but also whether people who rated it helpful seem to come from different perspectives. Because notes need to genuinely be found helpful by people who tend to disagree, the program is more likely to identify notes that many people find helpful. Read the full details of [how this works](./contributing/diversity-of-perspectives.md). +Community Notes works differently than the rest of the platform. It is not a popularity contest. It aims to find notes that many people from different points of view will find helpful. It takes into account not only how many ratings a note has received, but also whether people who rated it helpful seem to come from different perspectives. Because notes need to genuinely be found helpful by people who tend to disagree, the program is more likely to identify notes that many people find helpful. Read the full details of [how this works](./contributing/diversity-of-perspectives.md). -Notes are also subject to Twitter rules and can be reported. +Notes are also subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules) and can be reported. {% /accordionItem %} @@ -80,19 +80,19 @@ We believe regular people can valuably contribute to identifying and adding help - [Bhuiyan, Zhang, Sehat, and Mitra 2020](https://arxiv.org/pdf/2008.09533.pdf) - [Kim and Walker 2020](https://misinforeview.hks.harvard.edu/article/leveraging-volunteer-fact-checking-to-identify-misinformation-about-covid-19-in-social-media/) -In our pilot test of Community Notes, we evaluated notes would be shown on Tweets, and have found: +In our pilot test of Community Notes, we evaluated notes would be shown on posts, and have found: -- The majority of people surveyed on Twitter found notes helpful. -- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading Tweet after reading the note about it, compared to those who saw a Tweet without a note. +- The majority of people surveyed on X found notes helpful. +- People in surveys were 20-40% less likely to agree with the substance of a potentially misleading post after reading the note about it, compared to those who saw a post without a note. - Most notes have been rated highly on accuracy by professional reviewers; it has been rare to find a note that reviewers agree is inaccurate. -The people on Twitter span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. +The people on X span a wide gamut of backgrounds and experiences, and we believe that, working together, they can help create a better-informed world. {% /accordionItem %} -{% accordionItem title="I have a note on my Tweet. What can I do?" %} +{% accordionItem title="I have a note on my post. What can I do?" %} -As a Tweet author, if you disagree that a note provides important context about your Tweet, you can request [additional review](./contributing/additional-review.md). +As a post author, if you disagree that a note provides important context about your post, you can request [additional review](./contributing/additional-review.md). {% /accordionItem %} diff --git a/documentation/name-announcement.html b/documentation/name-announcement.html index d1e4ac10..d1dd2c09 100644 --- a/documentation/name-announcement.html +++ b/documentation/name-announcement.html @@ -1,12 +1,12 @@ --- -title: Redirecting to Tweet announcement... +title: Redirecting to post announcement... --- diff --git a/documentation/under-the-hood/download-data.md b/documentation/under-the-hood/download-data.md index 03a44051..e2cd41c1 100644 --- a/documentation/under-the-hood/download-data.md +++ b/documentation/under-the-hood/download-data.md @@ -7,9 +7,9 @@ navWeight: 1 We can't wait to learn with you! -All Community Notes contributions are publicly available on the [Download Data](https://twitter.com/i/communitynotes/download-data) page of the Community Notes site so that anyone has free access to analyze the data, identify problems, and spot opportunities to make Community Notes better. +All Community Notes contributions are publicly available on the [Download Data](https://x.com/i/communitynotes/download-data) page of the Community Notes site so that anyone has free access to analyze the data, identify problems, and spot opportunities to make Community Notes better. -If you have questions or feedback about the Community Notes public data or would like to share your analyses of this data with us, please DM us at [@CommunityNotes](http://twitter.com/communitynotes). +If you have questions or feedback about the Community Notes public data or would like to share your analyses of this data with us, please DM us at [@CommunityNotes](http://x.com/communitynotes). --- @@ -17,7 +17,7 @@ If you have questions or feedback about the Community Notes public data or would ### Data snapshots -The [Community Notes data](https://twitter.com/i/communitynotes/download-data) is released as four separate files: +The [Community Notes data](https://x.com/i/communitynotes/download-data) is released as four separate files: - **Notes:** Contains a table representing all notes - **Ratings:** Contains a table representing all ratings @@ -30,7 +30,7 @@ Currently, we release one cumulative file each for notes, notes status history, A new snapshot of the Community Notes public data is released daily, on a best-effort basis, and technical difficulties may occur and delay the data release until the next day. We are not able to provide guarantees about when this may happen. The snapshots are cumulative files, but only contain notes and ratings that were created as of 48 hours before the dataset release time. When notes and ratings are deleted, they will no longer be released in any future versions of the data downloads, although the note status history dataset will continue to contain metadata about all scored notes even after they’ve been deleted, which includes noteId, creation time, the hashed participant ID of the note’s author, and a history of which statuses each notes received and when; however, all the content of the note itself e.g. the note’s text will no longer be available. -The [data download page in Community Notes](https://twitter.com/i/communitynotes/download-data) displays a date stamp indicating the most recent date of data included in the downloadable files. +The [data download page in Community Notes](https://x.com/i/communitynotes/download-data) displays a date stamp indicating the most recent date of data included in the downloadable files. ### File structure @@ -179,7 +179,7 @@ As we iterate and improve Community Notes, we will occasionally make changes to | `notHelpfulIrrelevantSources` | Int | User-entered checkbox in response to prompt “What was unhelpful about it?” (Check all that apply question type). New as of 2021-06-30 | 1 if “Sources do not support note” is selected, else 0. | | `notHelpfulOpinionSpeculation` | Int | User-entered checkbox in response to prompt “What was unhelpful about it?” (Check all that apply question type). New as of 2021-12-15 | 1 if “Opinion or speculation” is selected, else 0. | | `notHelpfulNoteNotNeeded` | Int | User-entered checkbox in response to prompt “What was unhelpful about it?” (Check all that apply question type). New as of 2021-12-15 | 1 if “Note not needed on this Tweet” is selected, else 0. | -| `ratedOnTweetId` | Long | The unique ID of the Tweet that the note was rated on (which, in the case of media notes, may not be the same Tweet as the note was originally written on). _New as of 2023-05-24_. | +| `ratedOnTweetId` | Long | The unique ID of the Tweet that the note was rated on (which, in the case of media notes, may not be the same Tweet as the note was originally written on). _New as of 2023-05-24_. | ### User Enrollment diff --git a/documentation/under-the-hood/guardrails.md b/documentation/under-the-hood/guardrails.md index 97091980..49516f8d 100644 --- a/documentation/under-the-hood/guardrails.md +++ b/documentation/under-the-hood/guardrails.md @@ -5,7 +5,7 @@ navWeight: 6 --- # Evaluation -Community Notes aims to create a better-informed world, by empowering people on Twitter to collaboratively add helpful context to Tweets that might be misleading or missing important context. +Community Notes aims to create a better-informed world, by empowering people on X to collaboratively add helpful context to posts that might be misleading or missing important context. It’s important that the context that is [elevated to viewers from Community Notes](../contributing/notes-on-twitter.md) be helpful, informative, and accurate. @@ -17,29 +17,29 @@ To identify potential problems, Community Notes tracks a number of quality measu ### Quality Measures -Community Notes measures and monitors three top-line metrics to understand the quality of notes and identify potential issues to mitigate along the way. These measures are used for monitoring purposes, and do not impact the note status outcomes or visibility for individual notes. At present, these measures focus on notes that earn the status of Helpful (i.e., those that are [shown to viewers on Twitter](../contributing/notes-on-twitter.md), beyond enrolled contributors). +Community Notes measures and monitors three top-line metrics to understand the quality of notes and identify potential issues to mitigate along the way. These measures are used for monitoring purposes, and do not impact the note status outcomes or visibility for individual notes. At present, these measures focus on notes that earn the status of Helpful (i.e., those that are [shown to viewers on X](../contributing/notes-on-twitter.md), beyond enrolled contributors). Evidence of consistent or systematic problems in top rated notes that earn the status of Helpful on any of the following measures can trigger a guardrail or circuit breaker procedure: 1. **Accuracy: Whether notes that earn the status of Helpful contain accurate, high-quality information** - This is measured via partnerships with [professional reviewers](https://blog.twitter.com/en_us/topics/company/2021/bringing-more-reliable-context-to-conversations-on-twitter) who provide evaluations of note accuracy. These evaluations provide a tracking measure of how often Helpful notes that are rated as accurate vs. inaccurate by reviewers over time; they don't impact notes’ ratings/status. + This is measured via partnerships with [professional reviewers](https://blog.x.com/en_us/topics/company/2021/bringing-more-reliable-context-to-conversations-on-twitter) who provide evaluations of note accuracy. These evaluations provide a tracking measure of how often Helpful notes that are rated as accurate vs. inaccurate by reviewers over time; they don't impact notes’ ratings/status. We don't expect perfect consensus among expert reviewers that all notes are accurate, nor do professionals always [agree with one another on accuracy ratings](https://www.science.org/doi/10.1126/sciadv.abf4393). However, this gives us an indicator of whether there might be broad or consistent issues emerging with note accuracy. -2. **Informativeness: Whether notes that earn the status of Helpful help inform people’s understanding of the subject matter in Tweets** +2. **Informativeness: Whether notes that earn the status of Helpful help inform people’s understanding of the subject matter in posts** - This is measured via survey experiments of random samples of Twitter users in the US, comparing whether people who see potentially misleading Tweets with notes tend to have a different view of the Tweet’s main claim vs those who see Tweets without notes. + This is measured via survey experiments of random samples of users in the US, comparing whether people who see potentially misleading posts with notes tend to have a different view of the post’s main claim vs those who see posts without notes. - While notes may vary in their ability to inform, we want to ensure that on average notes that earn the status of Helpful are effective at helping people understand the topics in Tweets (vs having no impact or an adverse effect). + While notes may vary in their ability to inform, we want to ensure that on average notes that earn the status of Helpful are effective at helping people understand the topics in posts (vs having no impact or an adverse effect). -3. **Helpfulness: Whether Twitter users beyond the contributor pool tend to find notes that earn the status of Helpful to be helpful** +3. **Helpfulness: Whether users beyond the contributor pool tend to find notes that earn the status of Helpful to be helpful** - This is measured via surveys of random samples of Twitter users in the US. The surveys allow these Twitter users to give feedback on the helpfulness of Community Notes. + This is measured via surveys of random samples of users in the US. The surveys allow these users to give feedback on the helpfulness of Community Notes. We don’t expect all notes to be perceived as helpful by all people all the time. Instead, the goal is to ensure that on average notes that earn the status of Helpful are likely to be seen as helpful by a wide range of people from different points of view, and not only be seen as helpful by people from one viewpoint. -In addition to the three top-line metrics listed above, Community Notes monitors additional operational quality metrics, like whether any notes violate Twitter Rules. +In addition to the three top-line metrics listed above, Community Notes monitors additional operational quality metrics, like whether any notes violate X's [Rules](https://help.x.com/rules-and-policies/twitter-rules). ### Thresholds @@ -51,19 +51,19 @@ For each of these measures and metrics, there are thresholds to trigger ‘guard ### Remediations -Community Notes context is intended to be community driven. Notes are written and selected by people on Twitter, for people on Twitter. Except in the case of a Twitter Rule violation, Twitter employees do not make decisions about which individual notes do or do not display on Twitter, even in cases when a guardrail or circuit breaker condition is triggered. +Community Notes context is intended to be community driven. Notes are written and selected by people on X, for people on X. Except in the case of a Rule violation, X employees do not make decisions about which individual notes do or do not display on X, even in cases when a guardrail or circuit breaker condition is triggered. Instead, the goal is to build a _system_ that consistently elevates helpful, informative, and accurate information. In case of a serious note quality problem, the policy and operating procedure is to take system-wide actions, fix the problem, then resume normal service. Examples of system-wide actions Community Notes may take in the case of a guardrail or circuit breaker situation may include: -- Temporarily raising the [threshold](./note-ranking-code.md) required for notes to be publicly visible on a Tweet +- Temporarily raising the [threshold](./note-ranking-code.md) required for notes to be publicly visible on a post - Temporarily pausing the scoring of newly created notes -- Temporarily pausing the display of all notes on Tweets for viewers outside enrolled Community Notes contributors +- Temporarily pausing the display of all notes on posts for viewers outside enrolled Community Notes contributors -These actions allow Community Notes to mitigate potential risk of inaccurate or low quality notes, while the internal team conducts an investigation and builds an appropriate fix, if needed. When selecting which remediation to use, we aim to take the smallest corrective action possible to balance mitigating the risk of showing low quality notes with the risk of failing to show helpful, informative notes on potentially misleading Tweets. +These actions allow Community Notes to mitigate potential risk of inaccurate or low quality notes, while the internal team conducts an investigation and builds an appropriate fix, if needed. When selecting which remediation to use, we aim to take the smallest corrective action possible to balance mitigating the risk of showing low quality notes with the risk of failing to show helpful, informative notes on potentially misleading posts. -_Note: Community Notes contributors and notes are subject to the Twitter Rules. Failure to abide by rules can result in removal from the Community Notes program._ +_Note: Community Notes contributors and notes are subject to X's [Rules](https://help.x.com/rules-and-policies/twitter-rules). Failure to abide by rules can result in removal from the Community Notes program._ ## Understanding helpfulness across political viewpoints @@ -71,13 +71,13 @@ The goal of community notes is to provide context that people from different poi To better understand if notes are found widely helpful or only helpful to people from a given political perspective, we conduct additional analyses. These are solely for comparative purposes and do not influence the algorithm's decisions about each note, but they do provide valuable information about note quality that aids in system development. -Prior to the roll-out of Community Notes in the United States, we employed representative surveys involving Twitter users to understand the helpfulness of notes to diverse political viewpoints. In these surveys, participants optionally declared their political leanings before viewing and providing evaluations of tweets with and without Community Notes. Details of our methodology and past findings can be found in [our paper](https://github.com/twitter/communitynotes/blob/main/birdwatch_paper_2022_10_27.pdf). +Prior to the roll-out of Community Notes in the United States, we employed representative surveys involving users to understand the helpfulness of notes to diverse political viewpoints. In these surveys, participants optionally declared their political leanings before viewing and providing evaluations of posts with and without Community Notes. Details of our methodology and past findings can be found in [our paper](https://github.com/twitter/communitynotes/blob/main/birdwatch_paper_2022_10_27.pdf). -Additionally, we utilize a widely-used technique for viewpoint estimation. This approach makes use of Twitter’s follow, like, and retweet graphs to estimate political leanings based on an account’s proximity to and interaction with political accounts and content within the network. Our calculations can be approximated by the methods described in [this paper](http://pablobarbera.com/static/barbera_twitter_ideal_points.pdf). Note that calculation of viewpoint estimations are anonymized and used on an aggregate basis only to help evaluate perceived note quality. +Additionally, we utilize a widely-used technique for viewpoint estimation. This approach makes use of our follow, like, and repost graphs to estimate political leanings based on an account’s proximity to and interaction with political accounts and content within the network. Our calculations can be approximated by the methods described in [this paper](http://pablobarbera.com/static/barbera_twitter_ideal_points.pdf). Note that calculation of viewpoint estimations are anonymized and used on an aggregate basis only to help evaluate perceived note quality. ## Feedback? Ideas? -We will continue to evolve our approach as we grow the program, conduct additional analyses, and learn. We welcome feedback and ideas to improve. Please DM us at [@CommunityNotes](https://twitter.com/communitynotes). +We will continue to evolve our approach as we grow the program, conduct additional analyses, and learn. We welcome feedback and ideas to improve. Please DM us at [@CommunityNotes](https://x.com/communitynotes). ## Current Status diff --git a/documentation/under-the-hood/note-ranking-code.md b/documentation/under-the-hood/note-ranking-code.md index 9ae350a2..a1153818 100644 --- a/documentation/under-the-hood/note-ranking-code.md +++ b/documentation/under-the-hood/note-ranking-code.md @@ -1,10 +1,10 @@ --- title: Open-source code -description: Download the open-source files that power Community Notes on Twitter. +description: Download the open-source files that power Community Notes on X. navWeight: 2 --- # Open-source code -Here you can find links to code that reproduces the note scoring/ranking code that Twitter runs in production: [code on Github](https://github.com/twitter/communitynotes/tree/main/sourcecode). +Here you can find links to code that reproduces the note scoring/ranking code that X runs in production: [code on Github](https://github.com/twitter/communitynotes/tree/main/sourcecode). -If you download the data files made available on the [Data Download](https://twitter.com/i/communitynotes/download-data) page and put them in the same directory as the following code files, you can then run `python main.py` to produce a `scoredNotes.tsv` file that contains note scores, statuses, and explanation tags that will match what’s running in production (as of the time the data was from). +If you download the data files made available on the [Data Download](https://x.com/i/communitynotes/download-data) page and put them in the same directory as the following code files, you can then run `python main.py` to produce a `scoredNotes.tsv` file that contains note scores, statuses, and explanation tags that will match what’s running in production (as of the time the data was from). diff --git a/documentation/under-the-hood/ranking-notes.md b/documentation/under-the-hood/ranking-notes.md index 2e7df8af..e03f17b5 100644 --- a/documentation/under-the-hood/ranking-notes.md +++ b/documentation/under-the-hood/ranking-notes.md @@ -14,18 +14,18 @@ The algorithm used to rank Community Notes and compute their statuses is open-so ![Three Community notes in different statuses](../images/note-statuses.png) -Community Notes are submitted and rated by contributors. Ratings are used to determine note statuses (“Helpful”, “Not Helpful”, or “Needs More Ratings”). Note statuses determine which notes are displayed on each of the [Community Notes Site’s timelines](./timeline-tabs.md), and which notes are displayed [on Tweets](../contributing/notes-on-twitter.md). +Community Notes are submitted and rated by contributors. Ratings are used to determine note statuses (“Helpful”, “Not Helpful”, or “Needs More Ratings”). Note statuses determine which notes are displayed on each of the [Community Notes Site’s timelines](./timeline-tabs.md), and which notes are displayed [on posts](../contributing/notes-on-twitter.md). All Community Notes start with the Needs More Ratings status until they receive at least 5 total ratings. Notes with 5 or more ratings may be assigned a status of Helpful or Not Helpful according to the algorithm described below. -If a note is deleted, the algorithm will still score it (using all non-deleted ratings of that note) and the note will receive a status if it’s been rated more than 5 times, although since it is deleted it will not be shown on Twitter even if its status is Helpful. +If a note is deleted, the algorithm will still score it (using all non-deleted ratings of that note) and the note will receive a status if it’s been rated more than 5 times, although since it is deleted it will not be shown on X even if its status is Helpful. -Notes marking Tweets as "potentially misleading" with a Note Helpfulness Score of 0.40 and above earn the status of Helpful. At this time, only notes that indicate a Tweet is “potentially misleading” and earn the status of Helpful are eligible to be displayed on Tweets. +Notes marking posts as "potentially misleading" with a Note Helpfulness Score of 0.40 and above earn the status of Helpful. At this time, only notes that indicate a post is “potentially misleading” and earn the status of Helpful are eligible to be displayed on posts. Notes with a Note Helpfulness Score less than -0.05 -0.8 \* abs(noteFactorScore) are assigned Not Helpful, where noteFactorScore is described in [Matrix Factorization](#matrix-factorization). Additionally, notes with an upper confidence bound estimate of their Note Helpfulness Score (as computed via pseudo-raters) less than -0.04 are assigned Not Helpful, as described in [Modeling Uncertainty](#modeling-uncertainty). Notes with scores in between remain with a status of Needs more Ratings. Identifying notes as Not Helpful improves contributor helpfulness scoring and reduces time contributors spend reviewing low quality notes. -We plan to enable Helpful statuses for notes marking Tweets as "not misleading" as we continue to evaluate ranking quality and utility to users. +We plan to enable Helpful statuses for notes marking posts as "not misleading" as we continue to evaluate ranking quality and utility to users. When a note reaches a status of Helpful / Not Helpful, they're shown alongside the two most commonly chosen explanation tags which describe the reason the note was rated helpful or unhelpful. Notes with the status Needs More Ratings remain sorted by recency (newest first), and notes with a Helpful or Not Helpful status are sorted by their Helpfulness Score. @@ -86,7 +86,7 @@ Additionally, because the matrix factorization is re-trained from scratch every While the matrix factorization approach above has many nice properties, it doesn't give us a natural built-in way to estimate the uncertainty of its parameters. One approach that we use to help quantify the uncertainty in our parameter estimates is by adding in "extreme" ratings from "pseudo-raters", and measuring the maximum and minimum possible values that each note's intercept and factor parameters take on after all possible pseudo-ratings are adding. We add both helpful and not-helpful ratings, from pseudo-raters with the max and min possible rater intercepts, and with the max and min possible factors (as well as 0, since 0-factor raters can often have outsized impact on note intercepts). This approach is similar in spirtit to the idea of pseudocounts in Bayesian modeling, or to Shapley values. -We currently assign notes a "Not Helpful" status if the max (upper confidence bound) of their intercept is less than -0.05, in addition to the rules on the raw intercept values defined in the previous section. We also currently assign notes a "Helpful" status if the min (lower confidence bound) of their intercept is at least 0.31. +We currently assign notes a "Not Helpful" status if the max (upper confidence bound) of their intercept is less than -0.05, in addition to the rules on the raw intercept values defined in the previous section. We also currently assign notes a "Helpful" status if the min (lower confidence bound) of their intercept is at least 0.32. ## Tag Outlier Filtering @@ -137,23 +137,23 @@ Similarly, if a note was impacted by tag outlier filter and required note interc Multi-Model ranking allows Community Notes to run multiple ranking algorithms before reconciling the results to assign final note status. We use this ability to test new models, refine current approaches and support expanding the Community Notes contributor base. -We currently run three note ranking models: +We currently run several variations of the matrix facgtorizaiton approach. +Each variation uses the same modeling logic and parameters, but applies the model to different slices of the ratings data. -- The _Core_ model runs the matrix factorization approach described above to determine status for notes with most ratings from geographical areas where Community Notes is well established (e.g. the US, where Community Notes has been available for multiple years). We refer to established areas as _Core_ areas and areas where Community Notes has recently launched as _Expansion_ areas. The Core model includes ratings from users in Core areas on notes where the majority of ratings also came from users in Core areas. +- The _Core_ model determines status for notes with most ratings from geographical areas where Community Notes is well established (e.g. the US, where Community Notes has been available for multiple years). We refer to established areas as _Core_ areas and areas where Community Notes has recently launched as _Expansion_ areas. The Core model includes ratings from users in Core areas on notes where the majority of ratings also came from users in Core areas. - The _Expansion_ model runs the same ranking algorithm with the same parameters as the Core model, with the difference that the Expansion model includes all notes with all ratings across Core and Expansion areas. -- The _Coverage_ model runs the same ranking algorithm and processes the same notes and ratings as the Core model, except the intercept regularization $\lambda_i$ and Helpful note threshold have been [tuned differently](https://github.com/twitter/communitynotes/blob/main/sourcecode/scoring/mf_coverage_scorer.py) to increase the number of Helpful notes. +- The _Group_ models operate on smaller segments of the data to specifically improve note ranking in non-English speaking communities. Users are assigned to modeling groups (e.g. based on region, country or language) and then we run a separate matrix factorization for each group. The matrix factorization includes all ratings from users in the modeling group, but the scoring results only impact notes which were written by a member of the modeling group and have at least 80% of ratings from within the modeling group. We initially launched with 12 Group models and plan to monitor and adjust as Community Notes continues to grow. In cases where a note is ranked by both the Core and Expansion models the Core model is always authoritative. This approach allows us to grow Community Notes as quickly as possible in experimental Expansion areas without the risk of compromising quality in Core areas where Community Notes is well established. -In cases where the Core and Coverage models disagree, a Helpful rating from the Core model always takes precedence. -If a note is only rated as Helpful by the Coverage model, then the note must surpass a safeguard threshold for the Core model intercept to receive a final Helpful rating. -We have initialized the Core model safeguard threshold to 0.38, 0.02 below the Core model default Helpfulness threshold of 0.40, and will lower the safeguard threshold as the Coverage model continues to launch. +Like the Expansion model, the Group models increase Helpful note coverage beyond the Core model. +Group models only function to promote notes from Needs More Ratings to Helpful status, and only take effect when the Expansion or Core model intercept (as applicable) is between 0.3 and 0.4. -When using Twitter, you can see which model computed the status a given note by looking at the Note Details screen. +When using X, you can see which model computed the status a given note by looking at the Note Details screen. It might list one of the following models: - CoreModel (vX.X). The _Core_ model described above. - ExpansionModel (vX.X). The _Expansion_ model described above. -- CoverageModel (vX.X). The _Coverage_ model described above. +- GroupModelN (vX.X). The Nth instantiation of the _Group_ model described above. - ScoringDriftGuard. This is a scoring rule that locks note statuses after two weeks. See the [next section](#status-stabilization) for more details. ## Status Stabilization @@ -165,12 +165,12 @@ As older data comprise an increasingly small fraction of the dataset, ranking re To maintain Helpful note quality as Community Notes continues to grow, we are adding logic which stabilizes the status of a note once the note is two weeks old. This approach allows us to continue optimizing the ranking algorithm with a focus on the impact on current data while persisting helpful community contributions on older topics. Before a note is two weeks old, the helpfulness status will continue to be updated each time time the ranking algorithm is run. -After a note turns two weeks old we store the helpfulness status for that note and use the stored status in the future, including for displaying notes on Twitter and calculating user contribution statistics. +After a note turns two weeks old we store the helpfulness status for that note and use the stored status in the future, including for displaying notes on X and calculating user contribution statistics. -While a note may be scored by the Core, Expansion and Coverage models, we only finalize note status based on the Core model. +While a note may be scored by the Core, Expansion and Group models, we only finalize note status based on the Core model. Notes that are only ranked by the Expansion model are not eligible for stabilization since the Expansion model is under development and may be revised to improve quality at any time. -Similarly, if a note is rated Helpful by the Coverage model and Needs More Ratings by the Core model, we will allow the note status to remain at Helpful even after the note is two weeks old. -If at any point both models agree and the Core model scores the note as Helpful or the Coverage model scores the note as Needs More Ratings, then the status will be finalized in agreement with both models. +Similarly, if a note is rated Helpful by a Group model and Needs More Ratings by the Core model, we will allow the note status to remain at Helpful even after the note is two weeks old. +If at any point both models agree and the Core model scores the note as Helpful or a Group model scores the note as Needs More Ratings, then the status will be finalized in agreement with both models. ## Determining Note Status Explanation Tags @@ -215,12 +215,21 @@ For not-helpful notes: 3. Compute Author and Rater Helpfulness Scores based on the results of the first matrix factorization, then filter out raters with low helpfulness scores from the ratings data as described in [Filtering Ratings Based on Helpfulness Scores](./contributor-scores.md). 4. Re-fit the matrix factorization model on the ratings data that’s been filtered further in step 3. 5. Compute upper and lower confidence bounds on each note's intercept by adding pseudo-ratings and re-fitting the model with them. -6. Reconcile scoring results from the Core, Expansion and Coverage models to generate final status for each note. +6. Reconcile scoring results from the Core, Expansion and Group models to generate final status for each note. 7. Update status labels for any notes written within the last two weeks based the intercept terms (scores) and ratings tags. Stabilize helpfulness status for any notes older than two weeks. 8. Assign the top two explanation tags that match the note’s final status label as in [Determining Note Status Explanation Tags](#determining-note-status-explanation-tags), or if two such tags don’t exist, then revert the note status label to “Needs More Ratings”. ## What’s New? +**July 27, 2023** + +- Introduced modeling groups and associated Group models to improve Helpful note coverage. +- Improved model convergence logic to reduce scoring instability. +- Increased the scoring threshold for identifying Helpful notes based on confidence intervals. +- Added support for running a subset of scoring logic for development purposes. +- Added support for computing separate training and validation loss during development. +- Sunset the Coverage model, which trialed an expansion of Helpful notes through modified regularization. + **April 20, 2023** - Add additional rule to label more notes "Helpful", by computing a lower confidence bound on each note's intercept by adding pseudo-ratings, and marking notes "Helpful" if their intercept's lower confidence bound is at least 0.31. @@ -270,24 +279,24 @@ For not-helpful notes: **November 25, 2022** -- Resumed assigning statuses to notes that indicate the Tweet is “not misleading.” Only such notes written after October 3, 2022 will be eligible to receive statuses, as on that date we [updated the rating form](https://twitter.com/CommunityNotes/status/1576981914296102912) to better capture the helpfulness of notes indicating the Tweet is not misleading. +- Resumed assigning statuses to notes that indicate the Tweet is “not misleading.” Only such notes written after October 3, 2022 will be eligible to receive statuses, as on that date we [updated the rating form](https://x.com/CommunityNotes/status/1576981914296102912) to better capture the helpfulness of notes indicating the Tweet is not misleading. **November 10, 2022** - Launched scoring logic adjusting standards for "Helpful" notes based on tags assigned in reviews labeling the note as "Not Helpful." **October 3, 2022** -- Updated the rating form to better capture the strengths of notes which add context without indicating the Tweet is misleading. We have resumed assigning status to notes marking Tweets as "not misleading" in select circumstances as we evaluate ranking quality and utility to users. +- Updated the rating form to better capture the strengths of notes which add context without indicating the Tweet is misleading. We have resumed assigning status to notes marking posts as "not misleading" in select circumstances as we evaluate ranking quality and utility to users. **July 13, 2022** - To prevent manipulation of helpfulness scores through deletion of notes, notes that are deleted will continue to be assigned note statuses based on the ratings they received. These statuses are factored into author helpfulness scores. - Valid Ratings Definition Update: instead of just the first 5 ratings on a note, all ratings will be valid if they are within the first 48 hours after note creation and were created before the note first received its status of Helpful or Not Helpful (or if its status flipped between Helpful and Not Helpful, then all ratings will be valid up until that flip occurred). -- To make the above two changes possible, we are releasing a new dataset, note status history, which contains timestamps for when each note received statuses, and the timestamp and hashed participant ID of the author of a note. This data file is being populated now and will be available on the Community Notes [Data Download](https://twitter.com/i/communitynotes/download-data) page beginning Monday July 18, 2022. +- To make the above two changes possible, we are releasing a new dataset, note status history, which contains timestamps for when each note received statuses, and the timestamp and hashed participant ID of the author of a note. This data file is being populated now and will be available on the Community Notes [Data Download](https://x.com/i/communitynotes/download-data) page beginning Monday July 18, 2022. **Mar 09, 2022** -- Temporarily paused assigning statuses to notes that indicate the Tweet is “not misleading”. We observed that notes marking a Tweet as "not misleading" were often rated as “Unhelpful - Tweet doesn’t need a note”, so we paused assigning status to notes marking Tweets as "not misleading" pending improvements to the rating form. +- Temporarily paused assigning statuses to notes that indicate the Tweet is “not misleading”. We observed that notes marking a post as "not misleading" were often rated as “Unhelpful - Tweet doesn’t need a note”, so we paused assigning status to notes marking posts as "not misleading" pending improvements to the rating form. - Adjusted thresholds for notes statuses **Feb 28, 2022** diff --git a/documentation/under-the-hood/timeline-tabs.md b/documentation/under-the-hood/timeline-tabs.md index 67f03b7c..5ef93662 100644 --- a/documentation/under-the-hood/timeline-tabs.md +++ b/documentation/under-the-hood/timeline-tabs.md @@ -7,32 +7,32 @@ navWeight: 3 Community Notes participants are able to see a tab of notes that could use their ratings, and receive notifications about notes that need help. -![Community Notes home page, showing Tweets with notes to be rated](../images/home.png) +![Community Notes home page, showing posts with notes to be rated](../images/home.png) ### Needs Your Help tab This tab is only visible to contributors. It is designed to increase the likelihood that people from diverse perspectives rate each note, so that Community Notes can elevate notes that people from a wide range of perspectives will find helpful. It gives contributors an easy way to have immediate impact. -It contains a set of 10 Tweets that have notes that need more ratings (although there may be fewer than 10 Tweets if one of the Tweets was recently deleted by the author, or if not enough Tweets have new notes that meet the criteria to appear in the NYH tab). Tweets in this tab are filtered to those that the contributor hasn’t rated any of the notes on, and Tweets with notes from the past day, unless no Tweets pass those filters for you (that will only happen if you’re a very active rater!). The tab is updated roughly every hour or two, so when the contributor has rated notes in the tab, they can come back later to see fresh Tweets. +It contains a set of 10 posts that have notes that need more ratings (although there may be fewer than 10 posts if one of the posts was recently deleted by the author, or if not enough posts have new notes that meet the criteria to appear in the NYH tab). posts in this tab are filtered to those that the contributor hasn’t rated any of the notes on, and posts with notes from the past day, unless no posts pass those filters for you (that will only happen if you’re a very active rater!). The tab is updated roughly every hour or two, so when the contributor has rated notes in the tab, they can come back later to see fresh posts. -In order to appear in any of the tabs in the Community Notes site, a Tweet must have received at least 100 total likes plus Retweets. +In order to appear in any of the tabs in the Community Notes site, a post must have received at least 100 total likes plus reposts. -Additionally, Community Notes offers a way for Tweet authors to [request additional review](../contributing/additional-review.md) on notes on their Tweets. If an author requests additional review, the relevant Tweet will appear in all contributors’ Needs Your Help tabs. If there are more than 10 active requests for additional review, Tweets will be sorted by by the ranking score described below, which incorporates the viewpoints of raters. +Additionally, Community Notes offers a way for post authors to [request additional review](../contributing/additional-review.md) on notes on their posts. If an author requests additional review, the relevant post will appear in all contributors’ Needs Your Help tabs. If there are more than 10 active requests for additional review, posts will be sorted by by the ranking score described below, which incorporates the viewpoints of raters. ### Needs your help alerts -To ensure contributors don't miss the opportunity to rate notes on Tweets getting a lot of attention, Community Notes sends alerts requesting help from time to time. +To ensure contributors don't miss the opportunity to rate notes on posts getting a lot of attention, Community Notes sends alerts requesting help from time to time. ![One screenshot showing a Community Notes Contributor profile page, highlighting the settings button on the top right. Another screenshot showing the settings screen where contributors can edit their alert frequency](../images/alerts-settings.png) ### How does Community Notes decide which notes trigger alerts for help? -Alerts are optimized to increase the chance that potentially helpful notes on Tweets with high predicted visibility get rated by enough people that they have the chance to earn a status of Helpful quickly. Here's our current approach: +Alerts are optimized to increase the chance that potentially helpful notes on posts with high predicted visibility get rated by enough people that they have the chance to earn a status of Helpful quickly. Here's our current approach: -**Tweets are chosen based on:** +**Posts are chosen based on:** -- Projected future Likes and Retweets the Tweet will receive. -- The rater can see the Tweet (for example, excludes Tweets from authors you've blocked) +- Projected future Likes and Reposts the post will receive. +- The rater can see the post (for example, excludes posts from authors you've blocked) **Notes are chosen based on meeting some of the following:** @@ -47,4 +47,4 @@ We will continue to experiment with this logic to help ensure Needs Your Help no ### Rater similarity score for Needs Your Help -Tweets on the Needs Your Help tab are sorted by a ranking score, where Tweets are ranked higher based on the proportion of notes on the Tweet that are labeled Needs More Rating. Tweets are also ranked higher if they have notes with moderately high intercept scores (>=0.25) but which don’t have at least 3 raters with a similar viewpoint as yours (measured by whether your viewpoint factor is positive or negative). This is a mechanism to increase the likelihood that people from diverse perspectives rate each note. +posts on the Needs Your Help tab are sorted by a ranking score, where posts are ranked higher based on the proportion of notes on the post that are labeled Needs More Rating. Posts are also ranked higher if they have notes with moderately high intercept scores (>=0.25) but which don’t have at least 3 raters with a similar viewpoint as yours (measured by whether your viewpoint factor is positive or negative). This is a mechanism to increase the likelihood that people from diverse perspectives rate each note. diff --git a/documentation/video-transcript-pile-ons.md b/documentation/video-transcript-pile-ons.md index 35cf8897..61996dce 100644 --- a/documentation/video-transcript-pile-ons.md +++ b/documentation/video-transcript-pile-ons.md @@ -4,7 +4,7 @@ title: Video transcript How do Community Notes (formerly called Birdwatch) avoid one-sided bias? Pile-ons? Brigading? Manipulation? Abuse? -Community Notes are rated by contributors of multiple perspectives, and only notes that are widely found helpful are shown on Tweets. So one group alone can't determine what notes get shown, and pile-ons aren't effective. +Community Notes are rated by contributors of multiple perspectives, and only notes that are widely found helpful are shown on posts. So one group alone can't determine what notes get shown, and pile-ons aren't effective. This helps ensure that notes are helpful to a wide range of people. @@ -14,6 +14,6 @@ See the code on Github: [twitter.github.io/communitynotes](https://twitter.githu Community Notes (formerly called Birdwatch) -Context on Tweets. By the people, for the people. +Context on posts. By the people, for the people. Sign up to become a contributor. diff --git a/documentation/video-transcript.md b/documentation/video-transcript.md index 80b87278..f4ee5af8 100644 --- a/documentation/video-transcript.md +++ b/documentation/video-transcript.md @@ -4,22 +4,22 @@ title: Video transcript Introducing Community Notes (formerly called Birdwatch). -Sometimes, Tweets can be misleading or miss important context. +Sometimes, posts can be misleading or miss important context. -[Cleve @All_the_sportz 1hr ago tweeted, “Police are reporting that 3 ostriches are loose in Central Station 😲”] +[Cleve @All_the_sportz 1hr ago posted, “Police are reporting that 3 ostriches are loose in Central Station 😲”] -And you aren't sure whether to trust the Tweet… +And you aren't sure whether to trust the post… Community Notes empower people to share helpful context. Here's how it works: -Contributors propose notes that add context to Tweets. Notes are then rated by other contributors. +Contributors propose notes that add context to posts. Notes are then rated by other contributors. -Decisions aren't made by majority rules or popularity…So one group alone can't determine what notes get shown. Instead, Community Notes finds notes that are helpful to people with different points of view, and adds them as context to the Tweet. +Decisions aren't made by majority rules or popularity…So one group alone can't determine what notes get shown. Instead, Community Notes finds notes that are helpful to people with different points of view, and adds them as context to the post. -[Cleve @All_the_sportz 1hr ago tweeted, “Police are reporting that 3 ostriches are loose in Central Station 😲. Readers added more context they thought people might want to know. Is this note helpful? Rate it. Context is written by people who use Twitter, and appears when rated helpful by others. Find out more”] +[Cleve @All_the_sportz 1hr ago posted, “Police are reporting that 3 ostriches are loose in Central Station 😲. Readers added more context they thought people might want to know. Is this note helpful? Rate it. Context is written by people who use X, and appears when rated helpful by others. Find out more”] Community Notes (formerly called Birdwatch) -Context on Tweets. By the people, for the people. +Context on posts. By the people, for the people. Sign up to become a contributor. diff --git a/sourcecode/scoring/constants.py b/sourcecode/scoring/constants.py index a3f16649..51f213bc 100644 --- a/sourcecode/scoring/constants.py +++ b/sourcecode/scoring/constants.py @@ -16,6 +16,7 @@ maxTrainError = 0.09 +coreFlipPct = 0.15 expansionFlipPct = 0.19 maxReruns = 5 @@ -41,6 +42,7 @@ summaryKey = "summary" authorTopNotHelpfulTagValues = "authorTopNotHelpfulTagValues" modelingPopulationKey = "modelingPopulation" +modelingGroupKey = "modelingGroup" # TSV Values notHelpfulValueTsv = "NOT_HELPFUL" @@ -127,6 +129,14 @@ def rater_factor_key(i): coverageRatingStatusKey = "coverageRatingStatus" coverageNoteInterceptMaxKey = "coverageNoteInterceptMax" coverageNoteInterceptMinKey = "coverageNoteInterceptMin" +# Group Model +groupNoteInterceptKey = "groupNoteIntercept" +groupNoteFactor1Key = "groupNoteFactor1" +groupRatingStatusKey = "groupRatingStatus" +groupNoteInterceptMaxKey = "groupNoteInterceptMax" +groupNoteInterceptMinKey = "groupNoteInterceptMin" +groupRaterInterceptKey = "groupRaterIntercept" +groupRaterFactor1Key = "groupRaterFactor1" # Ids and Indexes noteIdKey = "noteId" @@ -364,6 +374,16 @@ def rater_factor_key(i): userEnrollmentTSVTypes = [dtype for (_, dtype) in userEnrollmentTSVColumnsAndTypes] userEnrollmentTSVTypeMapping = {col: dtype for (col, dtype) in userEnrollmentTSVColumnsAndTypes} +# TODO: delete expanded user enrollment definition once modeling group is fully rolled out +userEnrollmentExpandedTSVColumnsAndTypes = userEnrollmentTSVColumnsAndTypes + [ + (modelingGroupKey, np.float64) +] +userEnrollmentExpandedTSVColumns = [col for (col, _) in userEnrollmentExpandedTSVColumnsAndTypes] +userEnrollmentExpandedTSVTypes = [dtype for (_, dtype) in userEnrollmentExpandedTSVColumnsAndTypes] +userEnrollmentExpandedTSVTypeMapping = { + col: dtype for (col, dtype) in userEnrollmentExpandedTSVColumnsAndTypes +} + noteInterceptMaxKey = "internalNoteIntercept_max" noteInterceptMinKey = "internalNoteIntercept_min" noteParameterUncertaintyTSVMainColumnsAndTypes = [ @@ -421,6 +441,16 @@ def rater_factor_key(i): + incorrectFilterColumns ) +deprecatedNoteModelOutputColumns = frozenset( + { + coverageNoteInterceptKey, + coverageNoteFactor1Key, + coverageRatingStatusKey, + coverageNoteInterceptMinKey, + coverageNoteInterceptMaxKey, + } +) + noteModelOutputTSVColumnsAndTypes = [ (noteIdKey, np.int64), (coreNoteInterceptKey, np.double), @@ -451,9 +481,20 @@ def rater_factor_key(i): (expansionNoteInterceptMaxKey, np.double), (coverageNoteInterceptMinKey, np.double), (coverageNoteInterceptMaxKey, np.double), + (groupNoteInterceptKey, np.double), + (groupNoteFactor1Key, np.double), + (groupRatingStatusKey, np.str), + (groupNoteInterceptMaxKey, np.double), + (groupNoteInterceptMinKey, np.double), + (modelingGroupKey, np.float64), ] noteModelOutputTSVColumns = [col for (col, dtype) in noteModelOutputTSVColumnsAndTypes] noteModelOutputTSVTypeMapping = {col: dtype for (col, dtype) in noteModelOutputTSVColumnsAndTypes} +deprecatedNoteModelOutputTSVColumnsAndTypes = [ + (col, dtype) + for (col, dtype) in noteModelOutputTSVColumnsAndTypes + if col in deprecatedNoteModelOutputColumns +] raterModelOutputTSVColumnsAndTypes = [ (raterParticipantIdKey, np.int64), @@ -481,6 +522,9 @@ def rater_factor_key(i): (isEmergingWriterKey, np.bool_), (aggregateRatingReceivedTotal, pd.Int64Dtype()), (timestampOfLastEarnOut, np.double), + (groupRaterInterceptKey, np.double), + (groupRaterFactor1Key, np.double), + (modelingGroupKey, np.float64), ] raterModelOutputTSVColumns = [col for (col, dtype) in raterModelOutputTSVColumnsAndTypes] raterModelOutputTSVTypeMapping = {col: dtype for (col, dtype) in raterModelOutputTSVColumnsAndTypes} diff --git a/sourcecode/scoring/enums.py b/sourcecode/scoring/enums.py index d3ff9bd5..2e92df1f 100644 --- a/sourcecode/scoring/enums.py +++ b/sourcecode/scoring/enums.py @@ -6,8 +6,10 @@ class Scorers(Enum): """Exhaustive list of all scorers to simplify setting enabled/disabled scorers.""" MFCoreScorer = auto() - MFCoverageScorer = auto() MFExpansionScorer = auto() + # Note that the MFGroupScorer value controls whether *all* group scorers are instantiated, + # not just a single MFGroupScorer instance. + MFGroupScorer = auto() def scorers_from_csv(csv: str) -> Set[Scorers]: diff --git a/sourcecode/scoring/mf_base_scorer.py b/sourcecode/scoring/mf_base_scorer.py index 4acd52d1..d301ddab 100644 --- a/sourcecode/scoring/mf_base_scorer.py +++ b/sourcecode/scoring/mf_base_scorer.py @@ -87,6 +87,10 @@ def __init__( self._weightedTotalVotes = weightedTotalVotes self._mfRanker = matrix_factorization.MatrixFactorization() + def get_crh_threshold(self) -> float: + """Return CRH threshold for general scoring logic.""" + return self._crhThreshold + def get_scored_notes_cols(self) -> List[str]: """Returns a list of columns which should be present in the scoredNotes output.""" return [ @@ -160,6 +164,7 @@ def _score_notes_and_users( userScores pd.DataFrame: one row per user containing a column for each helpfulness score. """ if self._seed is not None: + print(f"seeding with {self._seed}") torch.manual_seed(self._seed) # Removes ratings where either (1) the note did not receive enough ratings, or diff --git a/sourcecode/scoring/mf_group_scorer.py b/sourcecode/scoring/mf_group_scorer.py new file mode 100644 index 00000000..756c2047 --- /dev/null +++ b/sourcecode/scoring/mf_group_scorer.py @@ -0,0 +1,285 @@ +from typing import Dict, List, Optional, Tuple + +from . import constants as c +from .mf_base_scorer import MFBaseScorer + +import numpy as np +import pandas as pd + + +# Number of MFGroupScorer objects we expect to instantiate +groupScorerCount = 12 + + +def _coalesce_columns(df: pd.DataFrame, columnPrefix: str) -> pd.DataFrame: + """Condense all columns beginning with columnPrefix into a single column. + + With each row there must be at most one column with a non-NaN value in the set of + columns beginning with columnPrefix. If a non-NaN value is present that will + become the value in the condensed column, otherwise the value will be NaN. After + column values are condensed the original (prefixed) columns will be dropped. + + Args: + df: DataFrame containing columns to condense + collumnPrefix: Prefix used to detect columns to coalesce, and the name for + the output column. + + Returns: + DataFrame with all columns prefixed by columnPrefix dropped and replaced by + a single column named columnPrefix + + Raises: + AssertionError if multiple columns prefixed by columnPrefix have non-NaN values + for any row. + """ + # Identify columns to coalesce + columns = [col for col in df.columns if col.startswith(f"{columnPrefix}_")] + if not columns: + return df + # Validate that at most one column is set, and store which rows have a column set + rowResults = np.invert(df[columns].isna()).sum(axis=1) + assert all(rowResults <= 1), "each row should only be in one modeling group" + # Coalesce results + def _get_value(row): + idx = row.first_valid_index() + return row[idx] if idx is not None else np.nan + + coalesced = df[columns].apply(_get_value, axis=1) + # Drop old columns and replace with new + df = df.drop(columns=columns) + df[columnPrefix] = coalesced + return df + + +def coalesce_group_models( + scoredNotes: pd.DataFrame, helpfulnessScores: pd.DataFrame +) -> Tuple[pd.DataFrame, pd.DataFrame]: + """Coalesce all group modeling columns across note and user scoring. + + Since each Scorer must have distinct output columns, we use coalescing to run + multiple instances of MFGroupScorer objects and then condense the results into + a single set of columns. This approach works because each note will be scored + by at most one MFGroupScorer instance. + + Args: + scoredNotes: scoring output for notes. + helpfulnessScores: scoring output for users. + + Returns: + tuple containing coalesced scoring results for notes and users. + """ + for col in [ + c.groupNoteInterceptKey, + c.groupNoteFactor1Key, + c.groupRatingStatusKey, + c.groupNoteInterceptMaxKey, + c.groupNoteInterceptMinKey, + c.modelingGroupKey, + ]: + scoredNotes = _coalesce_columns(scoredNotes, col) + + for col in [c.groupRaterInterceptKey, c.groupRaterFactor1Key, c.modelingGroupKey]: + helpfulnessScores = _coalesce_columns(helpfulnessScores, col) + + return scoredNotes, helpfulnessScores + + +class MFGroupScorer(MFBaseScorer): + def __init__( + self, + groupNumber: int, + seed: Optional[int] = None, + pseudoraters: Optional[bool] = False, + groupThreshold: float = 0.8, + ) -> None: + """Configure MFGroupScorer object. + + Notice that each MFGroupScorer defines column names by appending the groupNumber to + column prefixes which are constant. Dynamically defining the column names allows the + group scorer to be instantiated multiple times while maintaining the property that + the columns attached by each scorer remain unique. Once all scorers have ran, we + (will) validate that each note was scored by at most one group scorer and then coalesce + all of the group scoring columns and remove the groupNumber suffix. + + Args: + groupNumber: int indicating which group this scorer instance should filter for. + seed: if not None, seed value to ensure deterministic execution + pseudoraters: if True, compute optional pseudorater confidence intervals + groupThreshold: float indicating what fraction of ratings must be from within a group + for the model to be active + """ + super().__init__(seed, pseudoraters) + assert groupNumber > 0, "groupNumber must be positive. 0 is reserved for unassigned." + assert groupNumber <= groupScorerCount, "groupNumber exceeds maximum expected groups." + self._groupNumber = groupNumber + self._groupThreshold = groupThreshold + self._groupNoteInterceptKey = f"{c.groupNoteInterceptKey}_{self._groupNumber}" + self._groupNoteFactor1Key = f"{c.groupNoteFactor1Key}_{self._groupNumber}" + self._groupRatingStatusKey = f"{c.groupRatingStatusKey}_{self._groupNumber}" + self._groupNoteInterceptMaxKey = f"{c.groupNoteInterceptMaxKey}_{self._groupNumber}" + self._groupNoteInterceptMinKey = f"{c.groupNoteInterceptMinKey}_{self._groupNumber}" + self._groupRaterInterceptKey = f"{c.groupRaterInterceptKey}_{self._groupNumber}" + self._groupRaterFactor1Key = f"{c.groupRaterFactor1Key}_{self._groupNumber}" + self._modelingGroupKey = f"{c.modelingGroupKey}_{self._groupNumber}" + + def _get_note_col_mapping(self) -> Dict[str, str]: + """Returns a dict mapping default note column names to custom names for a specific model.""" + return { + c.internalNoteInterceptKey: self._groupNoteInterceptKey, + c.internalNoteFactor1Key: self._groupNoteFactor1Key, + c.internalRatingStatusKey: self._groupRatingStatusKey, + c.noteInterceptMinKey: self._groupNoteInterceptMinKey, + c.noteInterceptMaxKey: self._groupNoteInterceptMaxKey, + } + + def _get_user_col_mapping(self) -> Dict[str, str]: + """Returns a dict mapping default user column names to custom names for a specific model.""" + return { + c.internalRaterInterceptKey: self._groupRaterInterceptKey, + c.internalRaterFactor1Key: self._groupRaterFactor1Key, + } + + def get_scored_notes_cols(self) -> List[str]: + """Returns a list of columns which should be present in the scoredNotes output.""" + return [ + c.noteIdKey, + self._groupNoteInterceptKey, + self._groupNoteFactor1Key, + self._groupRatingStatusKey, + self._groupNoteInterceptMaxKey, + self._groupNoteInterceptMinKey, + self._modelingGroupKey, + ] + + def get_helpfulness_scores_cols(self) -> List[str]: + """Returns a list of columns which should be present in the helpfulnessScores output.""" + return [ + c.raterParticipantIdKey, + self._groupRaterInterceptKey, + self._groupRaterFactor1Key, + self._modelingGroupKey, + ] + + def get_auxiliary_note_info_cols(self) -> List[str]: + """Returns a list of columns which should be present in the auxiliaryNoteInfo output.""" + return [] + + def _get_dropped_note_cols(self) -> List[str]: + """Returns a list of columns which should be excluded from scoredNotes and auxiliaryNoteInfo.""" + return super()._get_dropped_note_cols() + ( + [ + c.internalActiveRulesKey, + c.activeFilterTagsKey, + c.ratingWeightKey, + ] + + c.notHelpfulTagsAdjustedColumns + + c.notHelpfulTagsAdjustedRatioColumns + + c.incorrectFilterColumns + + c.noteParameterUncertaintyTSVAuxColumns + ) + + def _get_dropped_user_cols(self) -> List[str]: + """Returns a list of columns which should be excluded from helpfulnessScores output.""" + return super()._get_dropped_user_cols() + [ + c.crhCrnhRatioDifferenceKey, + c.meanNoteScoreKey, + c.raterAgreeRatioKey, + c.aboveHelpfulnessThresholdKey, + ] + + def _filter_input( + self, ratings: pd.DataFrame, noteStatusHistory: pd.DataFrame, userEnrollment: pd.DataFrame + ) -> Tuple[pd.DataFrame, pd.DataFrame]: + """Prune the contents of ratings to only include ratings from users in the modeling group. + + This function identifies the subset of ratings to include in group model scoring. + To improve modeling within the group, we only include ratings from users in the modeling + group. However, we place no restriction on which notes to include in the model and instead + include ratings on any note. Including ratings on any note increases the amount of data + available during training about each user, in effect also increasing the number of users + and notes we are able to include in the model. + + Including notes by users outside of the modeling group means that the model will issue + scores for notes which do not meet group modeling criteria (i.e. >80% of ratings are + from users in the modeling group, and the author is also from the modeling group). We + enforce these criteria *after* scoring in _postprocess_output so that the maximum amount + of ratings are available during scoring. + + Args: + ratings (pd.DataFrame): preprocessed ratings + noteStatusHistory (pd.DataFrame): one row per note; history of when note had each status + userEnrollment (pd.DataFrame): one row per user specifying enrollment properties + + Returns: + Tuple[pd.DataFrame, pd.DataFrame]: + ratings: ratings filtered to only contain rows of interest + noteStatusHistory: noteStatusHistory filtered to only contain rows of interest + """ + userEnrollment = userEnrollment.rename(columns={c.participantIdKey: c.raterParticipantIdKey}) + userEnrollment = userEnrollment[userEnrollment[c.modelingGroupKey] == self._groupNumber] + ratings = ratings.merge(userEnrollment[[c.raterParticipantIdKey]].drop_duplicates()) + return ratings, noteStatusHistory + + def _postprocess_output( + self, + noteScores: pd.DataFrame, + userScores: pd.DataFrame, + ratings: pd.DataFrame, + noteStatusHistory: pd.DataFrame, + userEnrollment: pd.DataFrame, + ) -> Tuple[pd.DataFrame, pd.DataFrame]: + """Filter noteScores and userScores and add column containing modeling group. + + Enforce the requirements that: + - Notes scored by this group model have >=80% of ratings from the modeling group. + - Notes scored by this group model were authored by a member of the modeling group. + - Users assigned a factor / intercept are in the modeling group. + + Args: + noteScores: note outputs from scoring + userScores: user outputs from scoring + ratings (pd.DataFrame): preprocessed ratings + noteStatusHistory (pd.DataFrame): one row per note; history of when note had each status + userEnrollment (pd.DataFrame): one row per user specifying enrollment properties + + Returns: + Tuple[pd.DataFrame, pd.DataFrame]: + noteScores: filtered and updated note scoring output + userScores: filtered and updated user scoring output + """ + # Prune notes according to authorship filter. + noteScores = noteScores.merge( + userEnrollment[[c.participantIdKey, c.modelingGroupKey]].rename( + columns={c.participantIdKey: c.noteAuthorParticipantIdKey} + ), + how="left", + ) + noteScores = noteScores[noteScores[c.modelingGroupKey] == self._groupNumber] + noteScores = noteScores.drop(columns=c.modelingGroupKey) + # Identify notes with enough ratings from within the modeling group. + ratings = ratings.merge( + userEnrollment[[c.participantIdKey, c.modelingGroupKey]].rename( + columns={c.participantIdKey: c.raterParticipantIdKey} + ), + how="left", + ) + ratings["inGroup"] = ratings[c.modelingGroupKey] == self._groupNumber + ratios = ratings[[c.noteIdKey, "inGroup"]].groupby(c.noteIdKey).mean().reset_index() + notesAboveThreshold = ratios[ratios["inGroup"] >= self._groupThreshold][[c.noteIdKey]] + noteScores = noteScores.merge(notesAboveThreshold) + # Note that even though ratings were restricted to the modeling group, users outside of + # the modeling group may still have authored a note which was rated and may consequently + # appear in the userScores. Accordingly, we drop any user which was outside of the + # modeling group from the user scores. + userScores = userScores.merge( + userEnrollment[[c.participantIdKey, c.modelingGroupKey]].rename( + columns={c.participantIdKey: c.raterParticipantIdKey} + ), + how="left", + ) + userScores = userScores[userScores[c.modelingGroupKey] == self._groupNumber] + userScores = userScores.drop(columns=c.modelingGroupKey) + # Set the modelingGroupKey column in each output + noteScores[self._modelingGroupKey] = self._groupNumber + userScores[self._modelingGroupKey] = self._groupNumber + return noteScores, userScores diff --git a/sourcecode/scoring/process_data.py b/sourcecode/scoring/process_data.py index 20c923b4..d62cd9c6 100644 --- a/sourcecode/scoring/process_data.py +++ b/sourcecode/scoring/process_data.py @@ -100,9 +100,34 @@ def tsv_parser( raise ValueError("invalid input") -def tsv_reader(path: str, mapping, columns, header=False): +# TODO: remove this function once modelingGroup column is fully launched +def user_enrollment_parser(rawTSV: str, header: bool) -> pd.DataFrame: + """Parse user enrollment TSV and optinoally tolerate the modelingGroup column. + + Args: + rawTSV: str contianing entire TSV input + header: bool indicating whether the input will have a header + + Returns: + pd.DataFrame containing parsed data + """ + try: + df = tsv_parser(rawTSV, c.userEnrollmentTSVTypeMapping, c.userEnrollmentTSVColumns, header) + df[c.modelingGroupKey] = 0 + except ValueError: + df = tsv_parser( + rawTSV, c.userEnrollmentExpandedTSVTypeMapping, c.userEnrollmentExpandedTSVColumns, header + ) + return df + + +# TODO: remove support for specifying a custom parser once modelingGroup is fully rolled out +def tsv_reader(path: str, mapping, columns, header=False, parser=tsv_parser): with open(path, "r") as handle: - return tsv_parser(handle.read(), mapping, columns, header) + if parser == tsv_parser: + return parser(handle.read(), mapping, columns, header) + else: + return parser(handle.read(), header) def read_from_tsv( @@ -163,13 +188,13 @@ def read_from_tsv( userEnrollment = None else: userEnrollment = tsv_reader( - userEnrollmentPath, c.userEnrollmentTSVTypeMapping, c.userEnrollmentTSVColumns, header=headers + userEnrollmentPath, None, None, header=headers, parser=user_enrollment_parser ) - assert len(userEnrollment.columns.values) == len(c.userEnrollmentTSVColumns) and all( - userEnrollment.columns == c.userEnrollmentTSVColumns + assert len(userEnrollment.columns.values) <= len(c.userEnrollmentExpandedTSVColumns) and ( + len(set(userEnrollment.columns) - set(c.userEnrollmentExpandedTSVColumns)) == 0 ), ( - f"userEnrollment columns don't match: \n{[col for col in userEnrollment.columns if not col in c.userEnrollmentTSVColumns]} are extra columns, " - + f"\n{[col for col in c.userEnrollmentTSVColumns if not col in userEnrollment.columns]} are missing." + f"userEnrollment columns don't match: \n{[col for col in userEnrollment.columns if not col in c.userEnrollmentExpandedTSVColumns]} are extra columns, " + + f"\n{[col for col in c.userEnrollmentExpandedTSVColumns if not col in userEnrollment.columns]} are missing." ) return notes, ratings, noteStatusHistory, userEnrollment diff --git a/sourcecode/scoring/run_scoring.py b/sourcecode/scoring/run_scoring.py index 622fb9f5..07c258a0 100644 --- a/sourcecode/scoring/run_scoring.py +++ b/sourcecode/scoring/run_scoring.py @@ -7,16 +7,17 @@ from collections import namedtuple import concurrent.futures +from itertools import chain import multiprocessing import time -from typing import List, Optional, Set, Tuple +from typing import Dict, List, Optional, Set, Tuple from . import constants as c, contributor_state, note_ratings, note_status_history, scoring_rules from .enums import Scorers from .mf_base_scorer import MFBaseScorer from .mf_core_scorer import MFCoreScorer -from .mf_coverage_scorer import MFCoverageScorer, MFDummyCoverageScorer from .mf_expansion_scorer import MFExpansionScorer +from .mf_group_scorer import coalesce_group_models, groupScorerCount, MFGroupScorer from .scorer import Scorer from .scoring_rules import RuleID @@ -76,7 +77,6 @@ def _check_flips( prevCrh = max( 1, len(unlockedPopNsh.loc[unlockedPopNsh[c.currentLabelKey] == c.currentlyRatedHelpful]) ) - print(f"Percentage of previous CRH flipping status: {(statusFlips / prevCrh)}") return (statusFlips / prevCrh) > pctFlips @@ -84,7 +84,7 @@ def _check_flips( def _get_scorers( seed: Optional[int], pseudoraters: Optional[bool], enabledScorers: Optional[Set[Scorers]] -) -> List[Scorer]: +) -> Dict[Scorers, List[Scorer]]: """Instantiate all Scorer objects which should be used for note ranking. Args: @@ -93,18 +93,20 @@ def _get_scorers( enabledScorers: if not None, set of which scorers should be instantiated and enabled Returns: - List[Scorer] containing instantiated Scorer objects for note ranking. + Dict[Scorers, List[Scorer]] containing instantiated Scorer objects for note ranking. """ - scorers: List[Scorer] = [] + scorers: Dict[Scorers, List[Scorer]] = dict() if enabledScorers is None or Scorers.MFCoreScorer in enabledScorers: - scorers.append(MFCoreScorer(seed, pseudoraters)) + scorers[Scorers.MFCoreScorer] = [MFCoreScorer(seed, pseudoraters)] if enabledScorers is None or Scorers.MFExpansionScorer in enabledScorers: - scorers.append(MFExpansionScorer(seed)) - if enabledScorers is not None and Scorers.MFCoverageScorer in enabledScorers: - scorers.append(MFCoverageScorer(seed)) - else: - scorers.append(MFDummyCoverageScorer()) + scorers[Scorers.MFExpansionScorer] = [MFExpansionScorer(seed)] + if enabledScorers is None or Scorers.MFGroupScorer in enabledScorers: + # Note that index 0 is reserved, corresponding to no group assigned, so scoring group + # numbers begin with index 1. + scorers[Scorers.MFGroupScorer] = [ + MFGroupScorer(groupNumber=i) for i in range(1, groupScorerCount + 1) + ] return scorers @@ -166,9 +168,9 @@ def _run_scorer_parallelizable(scorer, ratings, noteStatusHistory, userEnrollmen result = None flips = False - if isinstance( - scorer, (MFBaseScorer, MFCoreScorer, MFCoverageScorer, MFExpansionScorer) - ) and not isinstance(scorer, MFDummyCoverageScorer): + # TODO: encapsulate this at a lower level so we're not accessing private state and can deal + # with the case where there are no ratings for a scorer to run on after filtering. + if isinstance(scorer, (MFBaseScorer, MFCoreScorer, MFExpansionScorer)): while ( (len(scorer._mfRanker.train_errors) == 0) or (scorer._mfRanker.train_errors[-1] > c.maxTrainError) @@ -176,6 +178,8 @@ def _run_scorer_parallelizable(scorer, ratings, noteStatusHistory, userEnrollmen ): runCounter += 1 result = ModelResult(*scorer.score(ratings, noteStatusHistory, userEnrollment)) + if runCounter > c.maxReruns: + break # check for notes createdat > lock time, not more than x% have changed status from nsh if isinstance(scorer, MFExpansionScorer): flips = _check_flips( @@ -186,6 +190,18 @@ def _run_scorer_parallelizable(scorer, ratings, noteStatusHistory, userEnrollmen c.expansionFlipPct, c.expansionRatingStatusKey, ) + elif isinstance(scorer, MFCoreScorer): + flips = _check_flips( + result.scoredNotes, + noteStatusHistory, + userEnrollment, + [c.core], + c.coreFlipPct, + c.coreRatingStatusKey, + ) + if flips: + if scorer._seed is not None: + scorer._seed = scorer._seed + 1 else: result = ModelResult(*scorer.score(ratings, noteStatusHistory, userEnrollment)) runCounter += 1 @@ -263,11 +279,13 @@ def _run_scorers( modelResult.helpfulnessScores, modelResult.auxiliaryNoteInfo, ) + scoredNotes, helpfulnessScores = coalesce_group_models(scoredNotes, helpfulnessScores) return scoredNotes, helpfulnessScores, auxiliaryNoteInfo def meta_score( + scorers: Dict[Scorers, List[Scorer]], scoredNotes: pd.DataFrame, auxiliaryNoteInfo: pd.DataFrame, lockedStatus: pd.DataFrame, @@ -314,12 +332,26 @@ def meta_score( RuleID.CORE_MODEL, {RuleID.META_INITIAL_NMR}, c.coreRatingStatusKey ) ) - if enabledScorers is not None and Scorers.MFCoverageScorer in enabledScorers: - rules.append( - scoring_rules.ApplyAdditiveModelResult( - RuleID.COVERAGE_MODEL, {RuleID.META_INITIAL_NMR}, c.coverageRatingStatusKey, 0.38 + if enabledScorers is None or Scorers.MFGroupScorer in enabledScorers: + # TODO: modify this code to work when MFExpansionScorer is disabled by the system test + assert len(scorers[Scorers.MFCoreScorer]) == 1 + assert len(scorers[Scorers.MFExpansionScorer]) == 1 + coreScorer = scorers[Scorers.MFCoreScorer][0] + assert isinstance(coreScorer, MFCoreScorer) + expnasionScorer = scorers[Scorers.MFExpansionScorer][0] + assert isinstance(expnasionScorer, MFExpansionScorer) + coreCrhThreshold = coreScorer.get_crh_threshold() + expansionCrhThreshold = expnasionScorer.get_crh_threshold() + for i in range(1, groupScorerCount + 1): + rules.append( + scoring_rules.ApplyGroupModelResult( + RuleID[f"GROUP_MODEL_{i}"], + {RuleID.EXPANSION_MODEL, RuleID.CORE_MODEL}, + i, + coreCrhThreshold, + expansionCrhThreshold, + ) ) - ) rules.extend( [ scoring_rules.ScoringDriftGuard( @@ -510,6 +542,26 @@ def _compute_helpfulness_scores( return helpfulnessScores +def _add_deprecated_columns(scoredNotes: pd.DataFrame) -> pd.DataFrame: + """Impute columns which are no longer used but must be maintained in output. + + Args: + scoredNotes: DataFrame containing note scoring output + + Returns: + scoredNotes augmented to include deprecated columns filled with dummy values + """ + for column, columnType in c.deprecatedNoteModelOutputTSVColumnsAndTypes: + assert column not in scoredNotes.columns + if columnType == np.double: + scoredNotes[column] = np.nan + elif columnType == np.str: + scoredNotes[column] = "" + else: + assert False, f"column type {columnType} unsupported" + return scoredNotes + + def _validate( scoredNotes: pd.DataFrame, helpfulnessScores: pd.DataFrame, @@ -576,7 +628,7 @@ def run_scoring( # Apply individual scoring models and obtained merged result. scorers = _get_scorers(seed, pseudoraters, enabledScorers) scoredNotes, helpfulnessScores, auxiliaryNoteInfo = _run_scorers( - scorers, ratings, noteStatusHistory, userEnrollment + list(chain(*scorers.values())), ratings, noteStatusHistory, userEnrollment ) # Augment scoredNotes and auxiliaryNoteInfo with additional attributes for each note @@ -588,6 +640,7 @@ def run_scoring( # Assign final status to notes based on individual model scores and note attributes. scoredNotesCols, auxiliaryNoteInfoCols = meta_score( + scorers, scoredNotes, auxiliaryNoteInfo, noteStatusHistory[[c.noteIdKey, c.lockedStatusKey]], @@ -618,6 +671,7 @@ def run_scoring( ), "noteStatusHistory should contain all notes after preprocessing" # Skip validation and selection out output columns if the set of scorers is overridden. + scoredNotes = _add_deprecated_columns(scoredNotes) if strictColumns: scoredNotes, helpfulnessScores, newNoteStatusHistory, auxiliaryNoteInfo = _validate( scoredNotes, helpfulnessScores, newNoteStatusHistory, auxiliaryNoteInfo diff --git a/sourcecode/scoring/scorer.py b/sourcecode/scoring/scorer.py index 5353c72c..b01c42ba 100644 --- a/sourcecode/scoring/scorer.py +++ b/sourcecode/scoring/scorer.py @@ -58,6 +58,35 @@ def _filter_input( """ return ratings, noteStatusHistory + def _postprocess_output( + self, + noteScores: pd.DataFrame, + userScores: pd.DataFrame, + ratings: pd.DataFrame, + noteStatusHistory: pd.DataFrame, + userEnrollment: pd.DataFrame, + ) -> Tuple[pd.DataFrame, pd.DataFrame]: + """Prune noteScores and userScores and augment with any additional columns as necessary. + + Note that ratings, noteStatusHistory and userEnrollment are expected to be the *raw* + versions which were supplied to "score", not the version output after filtering. + Operating on the raw versions allows accurately computing statistics over the entire dataset + (e.g. fraction of users from a modeling group). + + Args: + noteScores (pd.DataFrame): scoring output for notes + userScores (pd.DataFrame): scoirng output for users + ratings (pd.DataFrame): preprocessed ratings + noteStatusHistory (pd.DataFrame): one row per note; history of when note had each status + userEnrollment (pd.DataFrame): one row per user specifying enrollment properties + + Returns: + Tuple[pd.DataFrame, pd.DataFrame]: + noteScores: note scoring output from _score_notes_and_users + userScores: user scoring output from _score_notes_and_users + """ + return noteScores, userScores + def _get_note_col_mapping(self) -> Dict[str, str]: """Returns a dict mapping default note column names to custom names for a specific model.""" return {} @@ -83,14 +112,17 @@ def _score_notes_and_users( """ def score( - self, ratings: pd.DataFrame, noteStatusHistory: pd.DataFrame, userEnrollment: pd.DataFrame + self, + ratingsRaw: pd.DataFrame, + noteStatusHistoryRaw: pd.DataFrame, + userEnrollmentRaw: pd.DataFrame, ) -> Tuple[pd.DataFrame, Optional[pd.DataFrame], Optional[pd.DataFrame]]: """Process ratings to assign status to notes and optionally compute rater properties. Args: - ratings (pd.DataFrame): preprocessed ratings - noteStatusHistory (pd.DataFrame): one row per note; history of when note had each status - userEnrollment (pd.DataFrame): one row per user specifying enrollment properties + ratingsRaw (pd.DataFrame): preprocessed ratings + noteStatusHistoryRaw (pd.DataFrame): one row per note; history of when note had each status + userEnrollmentRaw (pd.DataFrame): one row per user specifying enrollment properties Returns: Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]: @@ -99,8 +131,24 @@ def score( auxiliaryNoteInfo: one row per note containing adjusted and ratio tag values """ # Transform input, run core scoring algorithm, transform output. - ratings, noteStatusHistory = self._filter_input(ratings, noteStatusHistory, userEnrollment) + ratings, noteStatusHistory = self._filter_input( + ratingsRaw, noteStatusHistoryRaw, userEnrollmentRaw + ) + # If there are no ratings left after filtering, then return empty dataframes. + if len(ratings) == 0: + return ( + pd.DataFrame(columns=self.get_scored_notes_cols()), + pd.DataFrame(columns=self.get_helpfulness_scores_cols()) + if self.get_helpfulness_scores_cols() + else None, + pd.DataFrame(columns=self.get_auxiliary_note_info_cols()) + if self.get_auxiliary_note_info_cols() + else None, + ) noteScores, userScores = self._score_notes_and_users(ratings, noteStatusHistory) + noteScores, userScores = self._postprocess_output( + noteScores, userScores, ratingsRaw, noteStatusHistoryRaw, userEnrollmentRaw + ) noteScores = noteScores.rename(columns=self._get_note_col_mapping()) userScores = userScores.rename(columns=self._get_user_col_mapping()) # TODO: Tolerate unexpcted columns if --nostrict-columns is set. diff --git a/sourcecode/scoring/scoring_rules.py b/sourcecode/scoring/scoring_rules.py index 319c8fe7..ea9bda94 100644 --- a/sourcecode/scoring/scoring_rules.py +++ b/sourcecode/scoring/scoring_rules.py @@ -36,6 +36,18 @@ class RuleID(Enum): EXPANSION_MODEL = RuleAndVersion("ExpansionModel", "1.1", False) CORE_MODEL = RuleAndVersion("CoreModel", "1.1", True) COVERAGE_MODEL = RuleAndVersion("CoverageModel", "1.1", False) + GROUP_MODEL_1 = RuleAndVersion("GroupModel01", "1.1", False) + GROUP_MODEL_2 = RuleAndVersion("GroupModel02", "1.1", False) + GROUP_MODEL_3 = RuleAndVersion("GroupModel03", "1.1", False) + GROUP_MODEL_4 = RuleAndVersion("GroupModel04", "1.1", False) + GROUP_MODEL_5 = RuleAndVersion("GroupModel05", "1.1", False) + GROUP_MODEL_6 = RuleAndVersion("GroupModel06", "1.1", False) + GROUP_MODEL_7 = RuleAndVersion("GroupModel07", "1.1", False) + GROUP_MODEL_8 = RuleAndVersion("GroupModel08", "1.1", False) + GROUP_MODEL_9 = RuleAndVersion("GroupModel09", "1.1", False) + GROUP_MODEL_10 = RuleAndVersion("GroupModel10", "1.1", False) + GROUP_MODEL_11 = RuleAndVersion("GroupModel11", "1.1", False) + GROUP_MODEL_12 = RuleAndVersion("GroupModel12", "1.1", False) INSUFFICIENT_EXPLANATION = RuleAndVersion("InsufficientExplanation", "1.0", True) SCORING_DRIFT_GUARD = RuleAndVersion("ScoringDriftGuard", "1.0", False) @@ -186,56 +198,6 @@ def score_notes( return (noteStatusUpdates, None) -class ApplyAdditiveModelResult(ScoringRule): - def __init__( - self, - ruleID: RuleID, - dependencies: Set[RuleID], - sourceColumn: str, - coreThreshold: float, - ): - """Set CRH status based on probationary model subject to safeguard threshold on core model. - - This rule sets CRH note status based on probationary models subject to several criteria: - * The note must be have CRH status from the probationary model. - * The note must currently be scored as NMR. This criteria guarantees that (1) probationary - models strictly expand coverage and (2) notes which were rated CRH by the core - model never have the decidedBy field overwritten by a less confident model. - * The note must score above a defined threshold on the core model. This criteria acts as - safeguard against dangerous detections from probationary models. - - Args: - rule: enum corresponding to a namedtuple defining a rule name and version string for the ScoringRule. - dependencies: Rules which must run before this rule can run. - sourceColumn: column containing note status (CRH, CRNH, NMR) to propagate to output. - coreThreshold: minimum score which notes must receive from the core model. - """ - super().__init__(ruleID, dependencies) - self._sourceColumn = sourceColumn - self._coreThreshold = coreThreshold - - def score_notes( - self, noteStats: pd.DataFrame, currentLabels: pd.DataFrame, statusColumn: str - ) -> (Tuple[pd.DataFrame, Optional[pd.DataFrame]]): - """Flip notes from NMR to CRH based on probationary model and subject to core model safeguard.""" - # Identify notes which meet each of the 3 criteria. - probationaryCRHNotes = noteStats[noteStats[self._sourceColumn] == c.currentlyRatedHelpful][ - [c.noteIdKey] - ] - currentNMRNotes = currentLabels[currentLabels[statusColumn] == c.needsMoreRatings][ - [c.noteIdKey] - ] - aboveSafeGuard = noteStats[noteStats[c.coreNoteInterceptKey] > self._coreThreshold][ - [c.noteIdKey] - ] - # Identify overlap and return status update. - noteStatusUpdates = probationaryCRHNotes.merge( - currentNMRNotes, on=c.noteIdKey, how="inner" - ).merge(aboveSafeGuard, on=c.noteIdKey, how="inner") - noteStatusUpdates[statusColumn] = c.currentlyRatedHelpful - return (noteStatusUpdates, None) - - class FilterTagOutliers(ScoringRule): def __init__( self, @@ -358,6 +320,92 @@ def score_notes( return (noteStatusUpdates, None) +class ApplyGroupModelResult(ScoringRule): + def __init__( + self, + ruleID: RuleID, + dependencies: Set[RuleID], + groupNumber: int, + coreCrhThreshold: float, + expansionCrhThreshold: float, + minSafeguardThreshold: float = 0.3, + ): + """Set CRH status based on a modeling group result. + + This rule sets CRH note status based on group models subject to several criteria: + * The note must be have CRH status from the group model. + * The note must currently be scored as NMR. This criteria guarantees that (1) group + models strictly expand coverage and (2) notes which were rated CRH by the core + model never have the decidedBy field overwritten by a less confident model. + * The note must have an intercept from either the core or expansion models, and the intercept + of the most confident model must fall within a defined range. We construct the range + to guarantee we can avoid CRHing notes which substantially lacked broad appeal, and + to guarantee that we will not CRH a note which was blocked by tag or inaccuracy filtering + from either the core or expansion models, as applicable. + + Args: + ruleID: enum corresponding to a namedtuple defining a rule name and version string for the ScoringRule. + dependencies: Rules which must run before this rule can run. + groupNumber: modeling group index which this instance of ApplyGroupModelResult should act on. + coreCrhThreshold: maximum intercept allowed on core model for group model CRH notes. + expansionCrhThreshold: maximum intercept allowed on expansion model for group model CRH notes. + minSafeguardThreshold: minimum intercept for core or expansion model. + """ + super().__init__(ruleID, dependencies) + self._groupNumber = groupNumber + self._minSafeguardThreshold = minSafeguardThreshold + self._coreCrhThreshold = coreCrhThreshold + self._expansionCrhThreshold = expansionCrhThreshold + + def score_notes( + self, noteStats: pd.DataFrame, currentLabels: pd.DataFrame, statusColumn: str + ) -> (Tuple[pd.DataFrame, Optional[pd.DataFrame]]): + """Flip notes from NMR to CRH based on group models and subject to core/expansion model safeguards.""" + # Identify notes which were CRH from the applicable group model. + probationaryCRHNotes = noteStats[ + (noteStats[c.groupRatingStatusKey] == c.currentlyRatedHelpful) + & (noteStats[c.modelingGroupKey] == self._groupNumber) + ][[c.noteIdKey]] + # Identify notes which are currently NMR. + currentNMRNotes = currentLabels[currentLabels[statusColumn] == c.needsMoreRatings][ + [c.noteIdKey] + ] + # Identify notes which pass score bound checks for expansion and core models. + noteStats = noteStats[[c.noteIdKey, c.coreNoteInterceptKey, c.expansionNoteInterceptKey]].copy() + noteStats["core"] = (noteStats[c.coreNoteInterceptKey] < self._coreCrhThreshold) & ( + noteStats[c.coreNoteInterceptKey] > self._minSafeguardThreshold + ) + noteStats.loc[noteStats[c.coreNoteInterceptKey].isna(), "core"] = np.nan + noteStats["expansion"] = ( + noteStats[c.expansionNoteInterceptKey] < self._expansionCrhThreshold + ) & (noteStats[c.expansionNoteInterceptKey] > self._minSafeguardThreshold) + noteStats.loc[noteStats[c.expansionNoteInterceptKey].isna(), "expansion"] = np.nan + + def _get_value(row): + idx = row.first_valid_index() + # If either core or expansion had an intercept then return whether it was in the valid + # range. If neither had an intercept, return False. Preference is given to core due + # to the ordering when selecting columns from noteStats below. + if idx is None: + return False + elif row[idx] == 1.0: + return True + elif row[idx] == 0.0: + return False + else: + assert False, f"unexpected value: {row[idx]}" + + noteStats["actionable"] = noteStats[["core", "expansion"]].apply(_get_value, axis=1) + actionableNotes = noteStats[noteStats["actionable"]][[c.noteIdKey]] + + # Identify overlap and return status update. + noteStatusUpdates = probationaryCRHNotes.merge( + currentNMRNotes, on=c.noteIdKey, how="inner" + ).merge(actionableNotes, on=c.noteIdKey, how="inner") + noteStatusUpdates[statusColumn] = c.currentlyRatedHelpful + return (noteStatusUpdates, None) + + class InsufficientExplanation(ScoringRule): def __init__( self,