Confused user experience with Polis -- why can't I explore groups in more detail? #1368
Replies: 1 comment 4 replies
-
|
Hi @jackson-wxyz. Thanks for sharing your experience here, and raising these issues. The first thing I'll say is that Polis is first and foremost a data collection tool. While the report does offer some preliminary analyses, it's not meant as a be-all end-all. Typically, for projects we are working on directly, we take the data export and do additional analyses, based on domain context, demographic metadata, etc. We have some analysis notebooks available here and here which you can take a look at for examples of this, and which can serve as a starting point for custom analyses. For example, regarding your interest in how groups A & B break down, you could either run the K-means clustering algorithm with a specific K value. Or you could subset the data to each of these groups, and run the K-means + Silhouette (for K-selection) algorithm to see how each group breaks into subgroups (though if you do this, you might want to project via PCA to more than 2 dimensions, since some of the group substructure may be in tertiary dimensions). Or you could try an entirely different clustering algorithm which is better suited for finding more fine grained (or explicitly hierarchical/nested) clusters, such as the Leiden clustering algorithm. You could also do a 3 dimensional plot (as you suggest) and plug it into a VR headset! The sky's really the limit here, once you see Polis as primarily a data collection tool. The reason that we don't make it possible for those running the conversation to manually control the number of groups is that it might be possible to bias the results towards a desired outcome by selecting a parameter that best fits a "desired" narrative. We've thought about ways that we might get around this, and may eventually add additional detail like subgroups to the report, but all of this takes work, and for the moment our main focus is on improving the development & deployment infrastructure of the platform (especially important since our deployment is based in the US, and thus not viable for gov use in the EU, due to GDPR & Schems II ruling; also because data sovereignty is just important in some cases).
But it does! That's the "bulls-eye" chart in the report is all about:
The numbers there correspond to comments, and are plotted where a participant who only voted on that comment (in agreement) would be projected. Turning on the group hulls, this lets you see how each comment relates to the group structure (you can click on a comment id to see the associated comment and voting patterns above the chart, as in the snapshot above, where comment 7 very clearly emerges as a strong indicator of which side of the opinion space one falls on). If you wanted to get a better sense of what the y-axis (PC2) corresponds to, you might look at comments 25, 22 and 16.
The reason we don't do this explicitly is that it could potentially be a bit confusing. In this case, there are very clear comments that stand out for each projected dimension, but sometimes, it's less clear, and the comments which best explain (e.g.) the y-axis, also correlate somewhat strongly with one side or the other of the x-axis, so simply plotting the way we do is the best way of being honest and not misleading about the situation. Of course, if it helps to build a narrative to do this, there's nothing stopping you from putting it together as part of a custom analysis (as long as you do your best to explain the results honestly and faithfully).
While the "majority" view on the participation view (and section in the report) are often a decent place to start looking for common ground, the comments there are selected based on a simple global-majority calculation. I'd highly recommend looking at the "All Statements" section of the report, sorted by "Group-informed Consensus". This attempts to compensate for possible tyranny of the majority by placing each group on equal footing in the metric in their ability to dissent.
We often see that these end up being some of the most interesting and useful comments to emerge out of our engagements, and frequently express very nuanced or surprising positions. The main reason we don't highlight these more in the participation experience is that it's a little harder to explain how it's computed (and sometimes the results can bias towards comments with low vote counts).
This is indeed interesting, but also not entirely out of line with the kind of results we've seen in the past. It's also possible that some of your within-group structure is falling within a 3rd or 4th PCA dimension. Another reason why running a custom analysis may be helpful here. I'd highly recommend taking a look at our paper if you're interested in digging into some of these methods and analyses further. This explains exactly how we compute our projections and metrics, and the rationale behind why do various things. Thanks again |
Beta Was this translation helpful? Give feedback.
-
|
Ah, thank you for explaining how the question numbers are plotted on the graph! For some reason I thought those represented individual people. Silly me. I appreciate the concern that allowing people to mess with the K-means dial could let people to bias the results in various ways. But I also think it would be very informative. Often, in a discussion, it's pretty obvious what the main axis of disagreement is -- getting a sense of subgroups and secondary factors is precisely what I find interesting about Polis. Thanks for such an in-depth and helpful response. |
Beta Was this translation helpful? Give feedback.
-
|
Losing groups is extremely frustrating since they are the main unit of conveying information. A clear subpopulation that gets absorbed can really feel like a bit of a tragedy. Wanted to validate that before responding more in depth later! Solution space for this is both exciting and fraught 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for taking so much time to both experiment and post! |
Beta Was this translation helpful? Give feedback.
-
|
You're quite welcome @jackson-wxyz!
Absolutely; One thing I meant to clarify in relation to this is that It's not that manually messing with the K value is necessarily bad, but that it needs to be done thoughtfully. And our position to date has been that the best way to ensure that is to have a data scientist in the loop who can make reasoned decisions about these things (and thus why it's currently only possible as a post-processing option, where it's implied that there's an analyst in the loop). This is obviously not perfect, and there are a number of ways this could be addressed more openly (e.g. allowing participants to explore all of the K-values for example, or as mentioned previously, providing sub- or hierarchical clusters as a standard part of the report/analysis, etc.). Our general approach has been "move thoughtfully and don't break things", but this is something we're actively thinking about, and it's really useful to have your feedback here as an example we can point to for context as we do that.
There is also a button which lets you display individual "participant buckets" (basically, finer grained clusters, to keep the viz from breaking down). This can help you get a sense of the actual participant distribution if you're interested in that. By default though, we hide this, as it's actually a lot less informative than the comment projection (for the exact reasons you point out in your initial post). Please let us know if you have any further questions; Again, this is all very useful context for us to have, both for us, and for other users. Thanks again |
Beta Was this translation helpful? Give feedback.
-
Hi! I recently experimented with Polis by creating some Yimby-vs-Nimby questions and posting the resulting poll to both Nextdoor (I live in the extremely Nimby neighborhood of Palo Alto) and a couple of bay-area subreddits. The resulting conversation was interesting, but I came away disappointed in the Polis platform. Here is the report from the Polis conversation: (https://pol.is/report/r3ybbdmeeftimca9rmnjz)
And here are my thoughts about Polis as a platform and what insight it does vs doesn’t offer (responding to someone else on Nextdoor complaining that Polis was being too divisive by splitting people into two groups, A and B, when really there was a spectrum of thought):
Yeah, to be honest I’m a bit disappointed as well, especially after hearing it hyped up so much by Aubrey Tang on the 80,000 Hours podcast. (https://80000hours.org/podcast/effective-altruism-ten-global-problems/)
The summary is a bit of a disappointment now. Earlier, it had three groups -- homeowner/slow-growth types, libertarian-yimby types (like myself), and lefty types who want rent control and more money for government housing assistance programs, but are worried about gentrification and don't believe "luxury" construction will help anyone. At the time I thought it was pretty cool to see the three groups mapped out in relation to each other. But unfortunately with more votes the platform has since smushed the lefties and yimbys together. I wish the platform had some more dials so that I could control how many subgroups I want to split people into (even if 2 is the algorithmically natural number of groups based on people's answers to these specific questions)... it would be interesting to turn up the dial to 4 or 5 groups and try to visualize some variation within the main camps (ie, homeowners who mostly worry about impacts to themselves like traffic and property values, versus slow-growthers who are concerned about cultural/ideological things like limits to growth or architecture styles. Or on the Yimby side, you might see a spectrum from hardcore property-rights libertarians to moderates who are interested in technocratic stuff like public transit investment and congestion pricing tolls to reduce traffic impacts.)
As an alternative to "turning up the dial" and seeing how the algorithm would split the responses if it was forced to create 3, 4, 5, or more groups, I would also like to see a factor analysis of the two main axes of disagreement. Polis creates a cool 2-D chart, but it doesn't tell us anything about what the axes are!! When there were three groups visible, you could tell that one axis was sort of "yimby vs nimby" and the other was sort of "egalitarian leftiness" versus more individualism, or something. But we shouldn't have to guess! Polis should tell us what these axes are by telling us what questions map best onto those axes. (It would also be cool to see a 1-dimensional projection that emphasizes the main dimension of political disagreement, and a 3-dimensional projection to bring out yet more nuance than the 2D picture).
Anyways, you can see the statements where there's the most agreement by clicking to see "majority view" rather than "group A" or "group B".
I also think the platform was designed to be used in combination with Facebook, and then you can see where your friends are plotted on the two axes, which would be informative and fun. Here is another polis site that a friend of mine created to map out feelings about the terrible recent invasion of Ukraine -- you can see that a couple of people are logged in with facebook so their pictures show up on their exact location in the graph:
https://pol.is/5ejvhhzrdw
It's kind of interesting that the shape of the yimby "Group B" is smaller than Group A, representing less diversity of opinion, even though the yimby side has 2x more votes AND we've been squished together with lefties who believe in rent control! I think this reflects the fact that us yimbys are more ideologically driven -- we see ourselves as activists with a coherent political position. (I don't mean that as self-praise -- I don't think of myself as a political person and I don't like activism in general; this kind of occasional Nextdoor writing is by far the most political thing I do in life. I mostly think of myself as an engineer and a fan of "effective altruism".) Versus Group A is mostly actual homehowners who have lots of different reactions to hearing yimby arguments -- some have detailed and well-researched opposition (but often in different directions... for example your position of "SF should cap jobs in order to avoid outgrowing our geographical/infrastructure limits" isn't something I've seen other people bring up in the same way), others have a random mix of concerns about traffic, architecture, crime, greedy developers, etc. So their views are more spread out.
(In the Ukraine poll, you can see that the pro-western-intervention group is smaller and more compact -- they just think we should intervene. Versus the noninterventionist group is more triangular because most of them just think western powers should stay out of it, but grouped in with them are also a handful of crazy Russian nationalists or something who actually seem to think that the invasion was a good thing.)
I think for the Yimby/Nimby debate, plotting out the spectrum of opinions is not super informative (although it's cool to see how different groups break down on different questions -- a good reminder that people's opinions are always very diverse even in polarized debates.) Obviously the main axis is a pro- vs anti-construction debate, and then (this was more visible earlier) I think the second axis is kind of a free-market/pro-corporate/libertarian vs social-justice/government/lefty debate. Same goes for the Ukraine debate, where the contrast between hawks vs doves is not too surprising. But I think polis could be really interesting to use among small groups or decisions/issues that don't fit into an existing political landscape, where it might be harder to guess what the primary axis of contention might be.
Beta Was this translation helpful? Give feedback.
All reactions