Unexpected behavior in beeswarm (consensus vs divisive) visualization

[Simon-Dirks]

Expected behavior:
On the far right side of the visualization ("divisive statements") I'd expect to see voting behavior that differs greatly between groups/participants. E.g., group A votes mostly agree for a statement, group B votes mostly disagree. Or: Overall there's a (more or less) 50/50 split between participants on agree/disagree.

Actual behavior:
The most divisive statement according to this visualization shows fairly similar voting behavior across groups (in this case: most people seem to disagree with the statement), as well as overall.

I found statements further to the "consensus" side of the graph that showed (at least to my eye) much more divisive voting behavior (see screenshots below).

In fact, the most divisive statement (statement 1) is even listed in the "Majority" view, showing what most people agreed with.

To Reproduce:
We simply ran a Polis conversation, I understand that the exact voting behavior might be hard to reproduce.

Screenshots:

^ A statement that should be the "most" divisive, but shows fairly similar voting behavior across groups.

^ The most divisive statement being listed in the "Majority" view.

^ A statement that seems much more divisive to me, but is further to the left on the scale.

Device information:
Across all devices/platforms.

Additional context:
193 participants voted
4512 votes were cast
23.38 votes per participant on average
9 commented
66 comments submitted

Note that I removed the statement texts (I'm not sure if I'm allowed to publish them at this point).

[Simon-Dirks]

It took me a while, but after diving deeper into the math worker I found out that "divisiveness" seems to be based on the "extremity" of the PCA output.

As an example, in this conversation, statement #110 is the most divisive.

In the PCA graph, this "extremity" can be seen in the distance from statement #110 to the center:

In the explanation text of Polis, however, it says that:
"Statements (here as little circles) to the left were voted on the same way—either everyone agreed or everyone disagreed. Statements to the right were divisive—participants were split between agreement and disagreement."

This is, in fact, not always the case, as can be seen in my original post above. This explanation seems to suggest that voting "variance" is the metric used (e.g., everyone being or not being on the same page in voting behavior for this specific statement), but PCA covers more complex and multidimensional relationships between participants and their voting behavior, right?

My colleagues and I were confused about this for quite a while, and I can imagine users are too. Perhaps this "divisiveness" metric was originally calculated as variance if I look at the Bowling Blue calculations? In fact, in the "Analysis" page, it still says that the beeswarm is a metric of variance:

Some clarification would be great!

[metasoarous]

Hi @Simon-Dirks. Thanks for bringing this question up, and my apologies for not responding sooner. I had written up a long explanation when I moved the issue over to a discussion, but it seems that it never actually got sent 🤦

You are correct; The beeswarm is based on the extremity of the comment, as measured by the distance from the center of the PCA plot (aka "bullseye" figure) from the report. This is further explained in our methods paper, which is the canonical place to go for the Polis methodology as it stands.

In the particular instance you have pointed out, the reason the comment has such high extremity is that the (very) few agree votes all fall on one side of the conversation, and correlate very strongly with the primary dimensions of the PCA dimension reduction (so much so that agreeing with the comment is a perfect predictor of falling in group B). In contrast, comment 28 has dissenting views from within each group, and would therefore seem to be less correlated with the primary dimensions of the conversation.

Perhaps this "divisiveness" metric was originally calculated as variance if I look at the Bowling Blue calculations?

No; These notebooks were put together after the fact to reproduce the basic analysis, and as a place to tinker with new methods or analyses, so not reflective of what is going on in the math worker or report.

In fact, in the "Analysis" page, it still says that the beeswarm is a metric of variance:

That page is a sort of analysis roadmap for things we wanted to explore in custom (and potentially/eventually the automated report). As you can see, there are a lot of items there which are not at all reflected in the current manifestation of the automated report.

The behavior you're observing is intentional, and shows something we find is often interesting to look at. However, you are correct to point out that the explanation text for the beeswarm is not sufficiently clear about what is being displayed, and you're not the first person to find it confusing. Unfortunately, its rather hard to describe exactly what is being presented here concisely and in simple terms, without getting rather technical. We've considered whether it would be more appropriate to just display variance in this part of the report; One can get a sense for the extremity by looking at the bullseye figure, but for similarly extreme comments on different sides of the conversation, it can be hard to compare which is more extreme (part of the impetus for the beeswarm).

We appreciate you sharing your thoughts on what might be most helpful, now that you better understand the situation.

Thanks again