Adjusting the rating system
News | Gustavo Ferreira | September 11th, 2009
We get a lot of positive feedback from FontStructors through email, on FontStruct threads and specially on Twitter. It feels great to give people something that makes them feel happy and creative. Thousands of FontStructors who voluntarily choose to share their work with the world are discovering this feeling too.
Of course, among all praises there are also feature requests, occasional bug reports and complaints. Interestingly, most complaints we get are targeted at one particular feature in FontStruct: the rating system.
The rating feature allows users to give grades – from 1 to 10 – to FontStructions created by other users. The idea is to have a parallel, user-driven quality control system complementary to Top Picks and Featured FontStructions (which are distributed by the FontStruct staff). We find this a useful and important feature to have; it invites participation and increases the self-organization powers of the community.
Why FontStructors complain about rating
FontStructors complaints are directed mainly to the great influence of “FontTrolls” – nasty creatures who appear to have great fun in giving very low grades to others – in the ratings. Several suggestions for improvements have been made by FontStructors (see the the FontTroll and FontMoot threads for examples), from the total removal of the rating feature to the acceptance of the rating system as it is.
Some FontStructors have also raised their voice against the recent inflation of 10s, which is as damaging to the system as the low ratings distributed by trolls – both distort the scale and take away the fine grading of intermediary values.
Looking at the rating data
We agreed that the rating system could be improved, and decided to work on it. Our first action was to look at the rating data to get a better idea of what was happening. The result of this research was very interesting, and gave us a solid foundation to plan and implement the necessary adjustments.
The following is a brief summary of our findings:
FontStructions – What gets voted?
The chart shows that Featured FontStructions get more and higher votes than Top Picks, which get more and higher votes than regular public FontStructions.
The more exposure a FontStruction gets, the more votes and the highest the ratings it receives.
FontStructors – Who votes?
Among all registered FontStructors, only a small minority (1,6%) makes use of the rating feature. Among the voters, FontStructors with Top Picks vote more than regular FontStructors, who vote more than users who registered but never created a FontStruction.
FontStructors with Top Picks are more demanding and vote in average lower than regular FontStructors, who vote in average lower than users who never created a FontStruction.
How do trolls behave?
Looking at the voting data we were able to identify three main “species” of trolls:
1. Down-voting trolls – The classic sort. We identified 9 “mini-trolls” (who have cast 10 or more votes with an average rating of less than 2), and 4 “mega-trolls” (who have cast 1000 or more votes with an average rating of less than 2).
2. Positrolls – Users who always and only award ratings of 10. We’ve found several “Robin Hood” accounts which appear to have been created solely for the purpose of spreading love and counterbalancing trollery.
3. Naughty Positrolls – Accounts used to boost the ratings of individual FontStructors. There were several of them!
* * *
Based on the data we gathered, we began to draw conclusions and think about what could be done to improve the system.
In Rob Meek’s words:
Some people get very upset by the ratings system, in particular by low votes (from “trolls”) which can seem unfair. Some people are more sensitive than others, and there are also plenty of people who are happy with the ratings system as it is. Many people have also invested a lot of time and thought in rating hundreds of FontStructions so I don’t think we should throw the current system out the window.
Looking at how people vote, it is clear that people have completely different standards and frames of reference for voting and understand the voting system differently: Some try and rate a FontStruction in terms of its value in the context of FontStruct while some rate it for its value in a broad typographic context. Some people obviously dislike the trend for high-resolution designs and vote accordingly. Others see voting solely as a way to encourage and spread positive karma. One person’s 7 is another’s 5. Many people may vote correctively i.e. they don’t just give a design the rating they feel is right, they try and “correct” the existing rating with a more extreme vote – I think this may explain some trolling. (…)
Looking at the data I think the best way to improve the system, and get meaningful ratings, is to pursue these two main goals:
- encourage and help people to vote more often. it’s simply low vote counts that make trolling significant.
- relativize the dominance of the linear gallery listings which make ratings so important and represent a catch-22: You need a high rating to get to the top, and you need to be at the top to get a high rating.
It was decided to invalidate votes from “Naughty Positrolls” (accounts used to boost the ratings of individual FontStructors). Votes from mega-trolls were left untouched, since they vote differently from each other, and most importantly they do sometimes vote high (they just seem to have a very strict criteria for quality). We also decided not to filter out statistically aberrant votes (i.e. the one low voter when everyone else votes high), as this would be plain censorship.
* * *
Ladies and gentleman — Without further delay, here are the long awaited changes to the rating system:
Vote indicator
The first visible change for voters is a grey check mark next to the ratings bar (the stars visible on FontStruction pages and in the gallery), meaning that the FontStruction has already received a vote from you. This helps to avoid repeating votes and stimulates users to rate different FontStructions.
New balancing formula
Balanced rating is probably the most important new feature being introduced today. The average rating values of a FontStruction are now calculated with a new balancing formula, which “pulls” ratings towards the system wide average. FontStructions with only a few votes can have neither very high nor very low ratings. As they get more votes, the balanced rating gets closer and closer to the true average. In general, for high-rated FontStructions, the “balanced rating” will be lower than the true average. For low-rated fonts with only a few votes, the “balanced” rating will be higher than the true average.
The old average rating is still there for comparison — it is displayed via mouse-over on the ratings, and appears as an alternative sorting parameter in the gallery.
Vote breakdown
The new vote breakdown is an alternative representation of the votes received by a FontStruction, showing the distribution of grades in graph. It can be accessed by clicking on the rating of a FontStruction.
The sample image shows the typical distribution of votes in a Top Pick – an overwhelming amount of 10s, but also several low votes from trolls or critics.
Hidden gems
Last but not least, the gallery now has a new category: Hidden Gems.
Hidden Gems are special FontStructions which are asking for your attention and your votes. Accessible from the main Gallery menu, Hidden Gems are a random selection of FontStructions which (A) you have not voted on and (B) seem to have been neglected in some way – maybe they’ve been downloaded frequently but have few received votes, or they are a Top Pick with a low rating.
Like FreeTags, Hidden Gems bring an element of surprise to the browsing of FontStructions.
* * *
Conclusions
We are excited to launch the new features and to pass the result of all this work to you.
We hope that these small adjustments will make it more fun to vote and increase the percentage of voting users. With more users voting and the new balanced formula, the quality of the rating data will only get better.
Finally, we have plans to add further ways of celebrating good designs, and involving the community in that process. Have fun & stay tuned!
/0)
