Skip to content

Commit

Permalink
Document expanded incorrect tag filter
Browse files Browse the repository at this point in the history
  • Loading branch information
bradmiller authored Oct 23, 2023
1 parent 4f64737 commit 5b4ca8b
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion documentation/under-the-hood/ranking-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,10 @@ Given the quantities defined above, we modify scoring as follows:

## Additional Filtering for Incorrect Tags

Because surfacing high-quality information is the primary goal of Community Notes, the ranking algorithm employs extra checks around Currently Rated Helpful notes that ratings indicate might contain incorrect information. For any given note-rater pair, properties including the note and rater factors (See [Matrix Factorization](#matrix-factorization)), a rater's propensity to assign the "Incorrect" tag, and the overall polarization of assigned "Not Helpful" tags predict, at baseline, how likely a rater is to rate a note as "Incorrect". When "Incorrect" ratings on a given note are "surprisingly popular" among raters who would be expected to have a low probability of rating the note "Incorrect", the note will not earn a Currently Rated Helpful status.
Because surfacing high-quality information is the primary goal of Community Notes, the ranking algorithm employs extra checks around Currently Rated Helpful notes that ratings indicate might contain incorrect information.
For any given note-rater pair, properties including the note and rater factors (See [Matrix Factorization](#matrix-factorization)), a rater's propensity to assign the "Incorrect" tag, and the overall polarization of assigned "Not Helpful" tags predict, at baseline, how likely a rater is to rate a note as "Incorrect".
Notes that attract higher levels of "Incorrect" tags, in particular from raters who use the "Incorrect" tag sparingly or who had a low probability of rating the note as "Incorrect" based on the assigned note and rater factors, are held to a higher threshold to achieve Currently Rated Helpful status.
Depedning on the strength of the signal, a note may be blocked from Currently Rated Helpful status entirely.

## CRH Inertia

Expand Down Expand Up @@ -221,6 +224,9 @@ For not-helpful notes:

## What’s New?

**October 20, 2023**
- Expand filtering applied to "Incorrect" tags to raise the standard for Helpfulness and reduce risk of potentially incorrect Helpful notes.

**October 3, 2023**
- Rearchitect the backend (batch=>streaming) to dramatically speed up how long it takes between when notes are scored and when they are shown to users (we eliminate an approximately 1hr wait between when the Python scorer had finished and when updated statuses are shown to all users). Because timestamps are written by the Python scorer, they were typically around 1hr (with wide variance) before the new status was actually shown. Going forward, there will be more typically a delay of only minutes between the status timestamps in note status history and when statuses are propagated to be visible to all users.

Expand Down

0 comments on commit 5b4ca8b

Please sign in to comment.