I looked at a bunch of the top played maps on the NA Battle.net, then recorded their average rating and the number of ratings. Then I went into the reviews section for each map and counted up an average of each review score, weighted by the number of helpfuls per review. I only counted reviews with 5 or more "helpfuls" and then only counted maps with around 200 helpfuls or more. I noticed that each of these maps had around 300 ratings or more, so I set the minimums at 200 helpfuls and 300 ratings.
I then used a Bayesian algorithm to determine a more accurate score for each map, based on the minimum number of helpfuls/ratings needed to qualify and the average rating a person is likely to give (based on the data of the 36 maps I analyzed). I did this both for Ratings and Reviews, then averaged the two scores together.
My results:
Rank
Map
Score
1
Special Forces
4.78
2
TD Tycoon
4.70
3
1v1 Obs
4.69
4
Squadron TD
4.68
5
Night of the Dead
4.67
6
Undead Assault 3
4.66
7
Photon Discs
4.65
8
HOTS Custom
4.64
9
MageCraft
4.63
10
Bx Monobattle
4.62
11
Civilization Sapphire
4.60
12
Star Battle
4.59
13
Marine Arena
4.58
14
Nexus Wars
4.55
15
Eras Zombie Invasion
4.55
16
StarJeweled
4.54
17
Desert Strike
4.52
18
A Song of Ice and Fire
4.52
19
The Troll Tribes
4.49
20
Galaxy Vampirism
4.47
21
The Star Strikers
4.41
22
Left 2 Die
4.40
23
Cortex Roleplay
4.39
24
Pobes vs Zeelot
4.34
25
New Established Order
4.30
26
Mafia
4.24
27
Aeon of Storms
4.14
28
Hero Attack
4.14
29
Mineralz Evolution
4.11
30
Auir Chef
3.98
31
Runling Run
3.91
32
Build and Fight
3.79
33
God's Land
3.71
34
Cruiser Command
3.69
35
Heaven Besieged
3.67
36
Survival Man
3.38
As you can see, although the ratings are a bit better than the completely nonsensical crap we have on battle.net right now, it still doesn't make much sense. Special Forces ranked 1? Mafia ranked lower than Pobes vs Zeelot?
Interesting data. In general I've found that people say a review is helpful if they agree with it, and not if its actually a good review. As a result of this, most "helpful" reviews are just positive reviews if the map is played a lot. Thats why I don't think the data, while differing from straight-up ratings, is really any better.
While I think what you did is cool, I'm not sure what the point is. (Besides the fact that its fun to mess around with data) I don't think anyone thought that the review\rating system would cause "good" maps to have a better rating then "bad" maps - these maps were popular for a reason. People like them, even if they suck, and people review maps they like highly, even if they suck. I don't think you'll be able to manipulate the data to better reflect the quality no matter what you do, sadly.
The Troll Tribes over Mafia!!! Haha, but really I agree with TheSkunk, the reviews can't perfectly reflect a map. Almost anyone who has played Island Troll Tribes from Wc3 almost instantly gives my map a 5 even though it is still rather underdeveloped and somewhat buggy for my liking. I'm very critical of my work and at this point could hardly see my map making over a 3 or so, but the fans of the Wc3 mod have bloated my score.
A note on "helpfuls" too, I have personally marked several reviews as "helpful" simply because they gave the map 5 stars, used proper grammar, and made at least one actual comment relating to the map, even if they would not really be "helpful" to a new player when deciding to play the map or not.
On the other hand, I have actually marked a couple reviews helpful that were slightly critical and pointed out a problem with the map or suggested an improvement, which gave me something to work towards.
Again, interesting post, but it won't really change my opinion of anything, other than that I need to work harder to get further in development of my map so that it actually deserves the rating it got. And what is this Pobes vs Zeelot, and how is it so highly rated?
Yeah, what my data proves is that there is no way to salvage the 5-star system. Even with statistical analysis I can't get accurate ratings out of the system. Besides, it wasn't like I had much data to go on. Helpfuls, review ratings, the map rating and the number of ratings are literally all that is available to us.
I wish Blizzard had a YouTube style like/dislike bar. At least it would be a more accurate portrayal of a map's quality.
Fun fact: Mafia had the 2nd highest number of helpfuls on its reviews, next to Squadron TD. Mafia also has the distinction of having the most polarized reviews of any map on battle.net. Seriously, go look at the reviews. In the top 9 reviews alone, 5 of them are 5-star and 4 of them are 1-star. Of course, the people who wrote those particular 1-star reviews are mostly guys who were banlisted.
Just WOW. That cant be a dedicated troll right? :D
I dont really care about the star rating system. It kinda works against niche maps, but oh well I dont think bnet has that many "non-mainstream" players.
I just want the open games list fixed so I can test my map better...currently it doesnt feel like the open games list works too well (since it doesnt recycle). Allthou would be kinda interesting to have "disable rating" option too.
To have someone so interested in your creation that they are petitioning blizzard for you to have to change your map is quite an accomplishment. I've never even heard of anyone pulling something like that. Kudos to you DarkRevenantX.
Thus, the top two reviews for Mafia (the only two you can see without scrolling) are 1/5. Four of the top seven reviews are 1/5.
Too lazy to update the whole list but now Mafia is ranked 30/36 with a score of 3.94. Now, even Mineralz Evolution is allegedly better than Mafia.
By definition, Mafia is worse than a large majority of all regularly played maps on Battle.net. Mafia also has the distinction of having the worst quality-to-popularity ratio of any StarCraft 2 map.
Logically, this means that Battle.net is filled with amazing maps of such a radiance that they utterly overshadow even a well-made game like Mafia
You seem defensive or insecure about your map, which you shouldn't be. Mafia is a fantastic and very well done game and has already risen above 99% of the other maps. You've been on the front page for many months. It doesn't really matter what the reviews reflect if your map is already beyond successful, and I think we can safely say that the majority of the sc2 custom map scene has played it by now and knows what to expect rather than depending on reviews to learn about it. At any rate, some of those top reviews just sound trolly and maybe you can get blue to remove them.
I use them as an example of why a 5-star system will never work. Scores will be inflated to an artificial high or dragged to an artificial low no matter what you do. Mafia, being a deep-rooted mod, will see very little disadvantage from having lower ratings. People will play it anyway.
I refuse to consider a map whose name is so horribly and deliberately misspelled to be a better-quality game than mine. Call me biased. It's like a guy called Captin Poopiepants beating you at chess.
Also, the mean score is around 4.2 and the lowest is about 3.4. Plus, individual scores and rankings fluctuate every day. There is also little correlation between ratings and objective quality of the map. How can Special Forces be ranked 1? The map is simplistic and almost sloppy.
I refuse to consider a map whose name is so horribly and deliberately misspelled to be a better-quality game than mine. Call me biased. It's like a guy called Captin Poopiepants beating you at chess.
Haha that would suck. But I did not say it was better. Just that it has more mass appeal. You won't find many haters for PvZ compared to Mafia.
Also, the mean score is around 4.2 and the lowest is about 3.4. Plus, individual scores and rankings fluctuate every day. There is also little correlation between ratings and objective quality of the map. How can Special Forces be ranked 1? The map is simplistic and almost sloppy.
How can Special Forces be on the first page? It boggles the mind. There is no such thing as objective quality of a map by the way.
Im with you. Its just plain painful to watch people rate maps like that. I tend not to overstress it, even though it does anoy me.
But they community is what it is...they do not have the ability to judge map from other point than if they enjoy it or not (and that is decided in 5 seconds).
I havent even bothered to whine about the star ratings before, since most people seem to be fine with them. But I think it does more harm than good...but hey, thats just me.
Trying to understand what the community likes is challenging for sure, but for me isn't a driving force in how I am designing my project. It has always boggled my mind as well as to why shooting down hoardes of zerg is such a popular recurring theme to so many maps. Yet back in the BW days I would play through a ton of Special Forces maps myself, concept was simple as hell, make units, upgrade them, try not to find any of WTF kill boss mobs without being prepared.
I guess what matters is your goal with your project......
If it is popularity and glory, best be thinking extremely dumbed down gameplay/no learning involved.
Innovation, things the starcraft world has never seen? Well, you might get a few dozen who actually like your project.
Ah yes, good point I forgot to mention there, these simple dumbed down maps also tend to take a lot of time to play/beat, seems to be another drawing feature. If you need more examples..... Any open RPG or Dragons RPG, lol.
I guess just the way you said what you did and how you did it made it sound like it would take quite a bit longer to do, going through every review on those maps and such. I would assume it to take quite a bit longer then 2 hours. But it would see I am mistaken.
Sorry Dark revenant no matter what, you're map will never be better then Mini Chop Farms....
it has over 1600 TREES!!!!!
Also I wouldnt call your info that unreliable, Mafias a decent map, but its hard to learn, and its not for that many people, you have to remember that SC2 is a game you have to think in, and analytical thinkers dont tend to be imaginative so the majority I would say wouldn't like your map just because the general public would be the opposite of what your audience is.
not saying your map is bad, just that its not for the type that plays RTS games.
Edit: Title changed to appeal to the dumb masses.
I looked at a bunch of the top played maps on the NA Battle.net, then recorded their average rating and the number of ratings. Then I went into the reviews section for each map and counted up an average of each review score, weighted by the number of helpfuls per review. I only counted reviews with 5 or more "helpfuls" and then only counted maps with around 200 helpfuls or more. I noticed that each of these maps had around 300 ratings or more, so I set the minimums at 200 helpfuls and 300 ratings.
I then used a Bayesian algorithm to determine a more accurate score for each map, based on the minimum number of helpfuls/ratings needed to qualify and the average rating a person is likely to give (based on the data of the 36 maps I analyzed). I did this both for Ratings and Reviews, then averaged the two scores together.
My results:
As you can see, although the ratings are a bit better than the completely nonsensical crap we have on battle.net right now, it still doesn't make much sense. Special Forces ranked 1? Mafia ranked lower than Pobes vs Zeelot?
Interesting data. In general I've found that people say a review is helpful if they agree with it, and not if its actually a good review. As a result of this, most "helpful" reviews are just positive reviews if the map is played a lot. Thats why I don't think the data, while differing from straight-up ratings, is really any better.
While I think what you did is cool, I'm not sure what the point is. (Besides the fact that its fun to mess around with data) I don't think anyone thought that the review\rating system would cause "good" maps to have a better rating then "bad" maps - these maps were popular for a reason. People like them, even if they suck, and people review maps they like highly, even if they suck. I don't think you'll be able to manipulate the data to better reflect the quality no matter what you do, sadly.
@DarkRevenantX: Go
The Troll Tribes over Mafia!!! Haha, but really I agree with TheSkunk, the reviews can't perfectly reflect a map. Almost anyone who has played Island Troll Tribes from Wc3 almost instantly gives my map a 5 even though it is still rather underdeveloped and somewhat buggy for my liking. I'm very critical of my work and at this point could hardly see my map making over a 3 or so, but the fans of the Wc3 mod have bloated my score.
A note on "helpfuls" too, I have personally marked several reviews as "helpful" simply because they gave the map 5 stars, used proper grammar, and made at least one actual comment relating to the map, even if they would not really be "helpful" to a new player when deciding to play the map or not. On the other hand, I have actually marked a couple reviews helpful that were slightly critical and pointed out a problem with the map or suggested an improvement, which gave me something to work towards.
Again, interesting post, but it won't really change my opinion of anything, other than that I need to work harder to get further in development of my map so that it actually deserves the rating it got. And what is this Pobes vs Zeelot, and how is it so highly rated?
Yeah, what my data proves is that there is no way to salvage the 5-star system. Even with statistical analysis I can't get accurate ratings out of the system. Besides, it wasn't like I had much data to go on. Helpfuls, review ratings, the map rating and the number of ratings are literally all that is available to us.
I wish Blizzard had a YouTube style like/dislike bar. At least it would be a more accurate portrayal of a map's quality.
Fun fact: Mafia had the 2nd highest number of helpfuls on its reviews, next to Squadron TD. Mafia also has the distinction of having the most polarized reviews of any map on battle.net. Seriously, go look at the reviews. In the top 9 reviews alone, 5 of them are 5-star and 4 of them are 1-star. Of course, the people who wrote those particular 1-star reviews are mostly guys who were banlisted.
http://us.battle.net/sc2/en/forum/topic/6307801231
Your maps not good until someone is trying to get it banned.
@Taintedwisp: Go
OMFG :DDD
Just WOW. That cant be a dedicated troll right? :D
I dont really care about the star rating system. It kinda works against niche maps, but oh well I dont think bnet has that many "non-mainstream" players.
I just want the open games list fixed so I can test my map better...currently it doesnt feel like the open games list works too well (since it doesnt recycle). Allthou would be kinda interesting to have "disable rating" option too.
To have someone so interested in your creation that they are petitioning blizzard for you to have to change your map is quite an accomplishment. I've never even heard of anyone pulling something like that. Kudos to you DarkRevenantX.
Oh, Blizzard deleted the top review for Mafia.
Thus, the top two reviews for Mafia (the only two you can see without scrolling) are 1/5. Four of the top seven reviews are 1/5.
Too lazy to update the whole list but now Mafia is ranked 30/36 with a score of 3.94. Now, even Mineralz Evolution is allegedly better than Mafia.
By definition, Mafia is worse than a large majority of all regularly played maps on Battle.net. Mafia also has the distinction of having the worst quality-to-popularity ratio of any StarCraft 2 map.
Logically, this means that Battle.net is filled with amazing maps of such a radiance that they utterly overshadow even a well-made game like Mafia
@DarkRevenantX: Go
You seem defensive or insecure about your map, which you shouldn't be. Mafia is a fantastic and very well done game and has already risen above 99% of the other maps. You've been on the front page for many months. It doesn't really matter what the reviews reflect if your map is already beyond successful, and I think we can safely say that the majority of the sc2 custom map scene has played it by now and knows what to expect rather than depending on reviews to learn about it. At any rate, some of those top reviews just sound trolly and maybe you can get blue to remove them.
Heh, I never cared about the reviews.
I use them as an example of why a 5-star system will never work. Scores will be inflated to an artificial high or dragged to an artificial low no matter what you do. Mafia, being a deep-rooted mod, will see very little disadvantage from having lower ratings. People will play it anyway.
I don't see what your point of contention is on this one. PvZ definitely has more mass appeal than Mafia. This should not be surprising.
Holy shit...biased much? Do you have some objective definition of accurate?
I refuse to consider a map whose name is so horribly and deliberately misspelled to be a better-quality game than mine. Call me biased. It's like a guy called Captin Poopiepants beating you at chess.
Also, the mean score is around 4.2 and the lowest is about 3.4. Plus, individual scores and rankings fluctuate every day. There is also little correlation between ratings and objective quality of the map. How can Special Forces be ranked 1? The map is simplistic and almost sloppy.
Haha that would suck. But I did not say it was better. Just that it has more mass appeal. You won't find many haters for PvZ compared to Mafia.
How can Special Forces be on the first page? It boggles the mind. There is no such thing as objective quality of a map by the way.
@DarkRevenantX: Go
Im with you. Its just plain painful to watch people rate maps like that. I tend not to overstress it, even though it does anoy me.
But they community is what it is...they do not have the ability to judge map from other point than if they enjoy it or not (and that is decided in 5 seconds).
I havent even bothered to whine about the star ratings before, since most people seem to be fine with them. But I think it does more harm than good...but hey, thats just me.
Trying to understand what the community likes is challenging for sure, but for me isn't a driving force in how I am designing my project. It has always boggled my mind as well as to why shooting down hoardes of zerg is such a popular recurring theme to so many maps. Yet back in the BW days I would play through a ton of Special Forces maps myself, concept was simple as hell, make units, upgrade them, try not to find any of WTF kill boss mobs without being prepared.
I guess what matters is your goal with your project......
If it is popularity and glory, best be thinking extremely dumbed down gameplay/no learning involved.
Innovation, things the starcraft world has never seen? Well, you might get a few dozen who actually like your project.
I suspect my project will get the latter.
You sir, have too much time on your hands.
@ZeroAme: Go
Ah yes, good point I forgot to mention there, these simple dumbed down maps also tend to take a lot of time to play/beat, seems to be another drawing feature. If you need more examples..... Any open RPG or Dragons RPG, lol.
Two hours is too much time on my hands?
@DarkRevenantX: Go
I guess just the way you said what you did and how you did it made it sound like it would take quite a bit longer to do, going through every review on those maps and such. I would assume it to take quite a bit longer then 2 hours. But it would see I am mistaken.
Sorry Dark revenant no matter what, you're map will never be better then Mini Chop Farms....
it has over 1600 TREES!!!!!
Also I wouldnt call your info that unreliable, Mafias a decent map, but its hard to learn, and its not for that many people, you have to remember that SC2 is a game you have to think in, and analytical thinkers dont tend to be imaginative so the majority I would say wouldn't like your map just because the general public would be the opposite of what your audience is.
not saying your map is bad, just that its not for the type that plays RTS games.