Arabia civ balance continued: what are the worst match-ups?

In my last couple of posts I looked at overall civ performance and the impact of the most recent balance changes. It is undeniable from the stats that some civs perform better than others on Arabia, but looking at overall civ performance is quite a crude measure - it doesn't account for the variability in civ performance between match-ups. This is particularly important if you are picking civs: it matters more if a civ is strong against other strong civs, rather than how convincingly they can wreck Khmer (poor Khmer).

The original analysis was on overall performance and didn't look at individual match-ups. So this post is an attempt to investigate that further. The chart below is the output from individual match-up binomial tests, controlling for the Elo of the players. The values are the difference from the expected probability if the civs were equally matched. For Franks vs Khmer, for example, the expected probability would be 0.5 but Franks significantly over-perform (p = 3.2e-08) with a difference of 0.39 (CI@0.95 = 0.27 - 0.45), which indicates that, between equally skilled players, Franks overall have a ~0.8-0.9 chance of beating Khmer. So Franks would win 8 or 9 out of 10 games.

Note on accuracy: Most of these match-ups have enough games for the results to be considered fairly confidently taken as overall trends, but the accuracy of the values is generally not great - confidence intervals will be in the range of +/-0.1 in the best cases. Also, at this level of analysis, it is worth bearing in mind that I can't guarantee that these were all random picks, and the chance of confounds at the level of an individual match-up is much higher than taking the data overall.

​

https://i.redd.it/657w6u9jexv11.png

I have made the following charts: 1700-2000 (this one) and a version with misleadingly exact numbers; 2000+ with circles; 2000+ with numbers (2000+ charts are a lot noisier and less reliable due to lack of data but I made them anyway).

There is a lot going on in this chart, but because of the lack of accuracy in any one cell it is better to look at the overall patterns than the marginal difference in a match-up between, say, Aztecs and Slavs. That said I can say fairly confidently that some of the worst match-ups are Franks vs Khmer, Slavs vs Khmer, Aztecs vs Saracens. No surprises there I think! More generally:

Firstly, it agrees pretty well with the estimates in my previous posts (which were calculated with a different model), which is nice.

Secondly, it is clear from looking at the chart that the weakest civs, as a rule, are weaker than the strongest civs are strong. Almost all of the larger effects involve the bottom four civs, and Khmer in particular gets consistently wrecked by almost all the other civs. This means that the easiest steps towards civ balance in Arabia 1v1 (if that is considered important) would not be nerfing Franks, but buffing Khmer, Vietnamese and Saracens.

Thirdly, it gives us a context for saying something like "Franks are the top performers in Arabia". They are, but that is partially because they rack up big numbers against weak civs (Khmer, Portuguese, Saracens, Vietnamese) while being pretty solid against every other civ as well, rather than because they are consistently winning at 60% against every civ. Having said that, I have played around with a model without the weakest four civs, and it seems that Franks still come out on top.

Final side note: if a civ you like doesn't look very good in these charts then don't get salty - get on Voobly, play as many matches as you can with them and prove your point by winning with them! That way you get to confound the data next time somebody runs some stats, and we can all wonder at why Malians or Burmese are suddenly killing all!