View Single Post

finelinebob's Avatar

01.02.2013 , 02:07 AM | #1
For the TLDR crowd:

Otherwise, my apologies in advance for the long post, but this is not a simple issue....

Let me begin with a phrase often blamed on Mark Twain, who in turn blamed it on Benjamin Disraeli (though there is no documented proof that he said it): "There are three types of lies: lies, damned lies, and statistics." I prefer to say that there are liars, damned liars and statisticians because the numbers don't lie; it's the interpretation that does it. The reason I am bringing this up is because my stats are limited in this discussion. I could not find anyone capable and willing to help with the more complex testing that needed to be done, so what I have to say begins with some fairly certain, rigorous basic testing that identifies what must be a built-in, intentional, systematic bias for certain types of Underworld Trading (UWT) missions. The rest is simply an argument based on the logic of probability that relies on a "fair coin", an unbiased randomization method that we know from what I just said is not entirely true. Any interactions between the bias I can demonstrate and other random determinations within the system that would help or hurt matters more than they should are things I cannot measure with the tests I can conduct.

Still, if you know your stuff about Two-Way Analysis of Variance, maybe we can work together and clear this issue even further.

To put things plainly, there is a statistical bias in how UWT missions are assigned when you randomize your list. The bias may not fall along lines you would suspect, though. In UWT missions, there are "factional" and "neutral" missions. Factional missions are ones with story lines that would not make sense for the opposite faction. For instance, the Republic mission "Behind Imperial Lines" would not make sense for an Imperial player. Factional missions roughly mirror one another in terms of yields and products. Neutral missions, on the other hand, have story lines that have no factional basis and so can be offered to both sides. I imagine this is to provide a sense of "story" to Crew Skills, since BioWare values story so much, but as I will show shortly this intent on introducing story to Crew Skills alters the distribution of how Gifts, Fabrics and Metals missions are assigned and therefore disrupts any chance for an even distribution between the three.

Note: I'm not posting much hard data because SWTOR's forums do not allow for tables. It's available to anyone who wants to collaborate with me; otherwise, summaries of the most important statistics are given as clearly as they can be. Also, when there are any critical statistical terms or definitions I need to point out, I will hide them in Spoiler tags for the well-informed or the uninterested to ignore as they wish.

First, a word needs to be said about data collection. I used the same level 50 character with Skill 400 UWT and the same companion out, one who had no specials to UWT, to eliminate any possible sources of bias there. I randomized my lists by stepping out of my ship into the Carrick Station Fleet hanger, recorded what missions were available without taking any, then stepped back into my ship to randomize the list again. I did this a total of 100 times, which produce 500 randomly chosen missions per Grade or a total of 3000 data points. I kept tally of missions by keeping a list of all Moderate Yield to Rare Yield UWT missions for a Republic character (the ones you would normally expect to see without using Mission Discoveries) in separate columns for each set of ten randomizations. By keeping track in groups of ten, I was able to use some simple Excel formulas to add up the total number of missions recorded at any given time as a means of checking to see if I forgot one or added too many. By saving the document after recording each grade, it made it possible to backtrack quickly and fix whatever error the check sum counter indicated.

Now, on to what I can say with some certainty.

Perhaps the greatest problem in SWTOR's system for UWT missions is that while it maintains a general ratio of 2:1:1 Gifts to Fabrics to Metals missions, the missions are not spread evenly between the various Yields available and, possibly much worse, they are not distributed evenly between the factional and neutral types of missions. The actual number of available missions per Grade is just as erratic, with a variance of almost two times the amount of missions available for the most "populated" Grade as opposed to the least populated. Even given the 2:1:1 ratio BioWare tries to maintain, the actual availability -- from the perspective of at least one player with characters who rely on UWT missions -- seems haphazard at best and intentionally biased at the worst. But let's look at the statistics first.

To keep the geek speak as brief as possible, I'll say what must be said in the open and toss the rest in Spoiler boxes. Selection of missions at each grade is independent with respect to each other, so the chance of reporting multiple T-Test statistics keeping these populations separate should minimize any increased chance for a Type I error for reporting multiple tests. Since the primary comparison being made here is a difference of means between two groups -- factional and neutral within a Grade -- a T-Test should be sufficient in examining whether the differences in distribution of missions to each category is sufficiently random or not.


In all six Grades, testing the difference between the mean frequency of selection for factional versus neutral missions was found to be significant at the p<0.001 level for Grades 2, 3 and 4 and at p<0.01 for all other Grades. In other words, it is extremely unlikely that these six tests' results occurred due to simultaneous highly improbably random fluctuations -- it makes more sense to assume the differences are due to some systematic cause built into the random assignment process of missions to a mission list, with an average bias of approximately two-to-one in favor of factional missions being chosen over neutral missions. Running these tests independently of each other means I cannot say that they are varying due to the same reason, but this appears to be a case of where logic prevails when statistical rigor cannot.


So, basically, yes: if you thought UWT mission assignments were biased, you were correct, but maybe not for the reason you thought.

So much for what I can say definitively with statistical testing, unless someone with some expertise in Two-Way ANOVA wants to step up and share credit with some more in-depth analysis. What follows is just based on the logic of probability and what is expected when results converge to expected outcomes. I would like to stress here strongly, though, that the uneven allocation of factional missions between the different mission products (gifts, fabrics, metals) means that there may be "interaction affects" that could significantly influence the assigment of missions. Without knowing what these interactions are, or whether they are statistically significant, certainly calls into question any precise interpretation of what I have to say below if not the sum of what I say, if the interaction is large.

First and foremost, BioWare has designed a system for UWT missions where Gift missions occur basically twice as often as either Fabric missions or Metal missions. There are precisely 42 Companion Gift missions, 21 Luxury Fabric missions and 21 Underworld Metals missions available to a character with a Skill of 400 UWT, regardless of faction (there is a sight difference between factions in the number of Gift missions at Grades 5 and 6, but they essentially balance one another out). This number includes only missions that are normally available through your Crew Skills window, and does not include the Prosperous or Wealthy Yield one-time missions you can learn from Mission Discoveries. More specifically, there are 3 non-metals missions for every 1 metals mission thus indicating a bias-by-design of 3:1 against the resource that is most needed by users of UWT: both Cybertechs and ArmorMECHs require only Underworld Metals for their Prototype and Artifact quality items, and Synthweavers require Underworld Metals for approximately two-thirds of their schematics. If the number of missions were balanced at the 2:1:1 ratio for all grades, the item needed the most from UWT missions is by design more difficult to get and for no better apparent reason than to maintain some arbitrary ratio of mission types.

However, we have not begun to discuss the distribution of missions which looks, to the best of my ability to discern any pattern, haphazard if unintentional and vindictive, perhaps, if by intentional design because I can see no meaningful positive distribution of the total number of missions available and any Grade nor the Yields associated with the missions at each grade. Below is a list of the total number of missions available per grade and a breakdown of how many of those are Underworld Metals are available and at what Yield:
  • Grade 1 -- 9 total missions -- 1 Moderate Yield Metals mission, 1 Abundant Yield Metals mission
  • Grade 2 -- 13 total missions -- 1 Moderate, 1 Abundant and 1 Rare Yield mission
  • Grade 3 -- 14 total missions -- 1 Moderate, 1 Abundant and 1 Bountiful Yield mission
  • Grade 4 -- 16 total missions -- 1 Moderate, 1 Abundant, 1 Bountiful and 1 Rare Yield mission
  • Grade 5 -- 17 total missions -- 1 Moderate, 3 Bountiful and 1 Rare Yield mission
I am leaving Grade 6 out because missions are limited to Grade 6 Moderate or Abundant Fabrics or Metals missions and Grade 5 Rare Gifts missions, making it difficult to compare this Grade to the others on equal footing. I would also like to note that there are 15 Grade 6 missions for the Republic, while there are 16 Grade 5 and 16 Grade 6 missions for Imperials. That slight difference in 1 mission at each level comes in terms of Companion Gift mission differences.

As you can see from the list above, there seems to be no rhyme or reason to the assignment of Yield level to the missions that are available and that the number of Metals missions available at Grade 3 and lower is less than the overall 2:1:1 ratio established for overall distribution ratios. There also appears to be no rhyme or reason why there are inconsistent numbers of missions overall at each grade when BioWare has taken pains to maintain a strict 2:1:1 ration of missions overall. Worst of all, if you really needed one particular grade of metals for what you are crafting, only Grade 5 presents 5 metals missions for you to send your entire crew out on. What seems inconceivable is that the lowest level missions -- taking "story" into account, the metals that should be easiest to obtain because they are more common, less valuable than higher grades, and your crew should arguably have been able to establish suppliers for these -- are the ones that are statistically and actually more difficult to obtain. There are fewer of them available and they are weighted towards low Yield missions. Shouldn't your crew, as it gains expertise, actually improve its ability to obtain these sorts of metals?

As I mentioned above, this whole scheme seems to be haphazard: not methodical, not balanced, not systematic. If it was done intentionally, I cannot for the life of me imagine what reasoning went into the distribution of missions by Yield or by Grade. Furthermore, it is my opinion that the bias of 3 missions to 1 against the possibility of seeing a metals mission is a slap in the face to ArmorMECHs, Cybertechs, and even Synthweavers as your chance of getting an Underworld Metals missions are weighted to approximately the same level of failure as your chance to learn a new schematic from RE or in your chance to crit on a mission and get those materials needed to craft artifact-level items. The odds are stacked largely against you by design, if not by intent.

When it comes to design decisions, if the 2:1:1 ration of Gifts to Fabrics to Metals missions was so sacrosanct, then why are there not 16 missions per Grade with a distribution of 2 to 1 to 1 Gifts to Fabrics to Metals at each possible Yield level?

Again, I wish to point out that possible interaction effects may tilt the score in favor of metals missions or against them, but until I can find a partner to do some more advanced statistical testing to determine this, absolutely nothing can be said for or against this issue.

So, what's an ArmorMECH, Cybertech or Synthweaver to do?

It seems most of the community knows about stepping into then out of your ship to force a load screen which randomizes your mission availability. Two questions remain, however: (1) If you do not find missions that you want, should you take any missions anyway, and (2) Does it matter which missions you take. The answer to both is Yes.

Taking Grade 3 as an example and assuming that you had drawn no metals missions when you first check: by assigning 5 unwanted missions (since they were all that were available) to your crew (for a combat level 48 toon and higher) from the total of 14 possible missions, you have narrowed it down to 9 missions that can be selected for repopulating your list. Furthermore, it removes the missions you do not want to see again, thereby increasing the odds that a mission you need will appear in the newly randomized list. You chose a companion, you click on the choice of the mission it is running, and you get your list of 5 newly randomized different missions. Should you see a mission that you want, you can then abandon the mission the comp already has WITHOUT AFFECTING the list that is there. You will then be able to select the mission you really want.

So, again, you are facing a list of 5 missions you do not want. Does it make a difference which ones to choose? It certainly does. The only thing I've been able to state as a result of statistical hypothesis testing, not just the logic of how probability works, is that FACTIONAL missions appear about twice as frequently as NEUTRAL missions. This can only be explained by a bias within the mission selection system that favors factional missions. So, don't just pick any mission on the list; be aware of which missions are factional missions regardless of their product and choose those before choosing any neutral missions. If you are on the hunt for metals missions and you see a factional companion gifts and a factional luxury fabrics mission among the five available, choose those two to remove them from the possible replacements. I should note that the design bias that makes gift missions twice as common really has no effect here since, ignoring faction for the moment, they probably have an equal probability of the other types of missions for being chosen so after picking out the unwanted factional missions, any other unwanted mission will do. One category being twice as common does not equate to those missions individually occurring twice as frequently.

You should also note, for those of you hunting down metals missions, that only Grade 5 actually offers 5 metals missions. Know what limits you face, and know when you have selected all for the grade you want. Send any free companions off on other chores, such as running metals missions of different Grades.

Given all this, it sure would be handy to have a list of what the factional missions are and what they produce, wouldn't it? Anything not on the list would be a neutral mission and so of lower priority with respect to knowing what to look for. To make your lives easier, here is that list. Republic mission names are blue, Imperial names are red, and yields/products are white:

Grade 1
  • Overthrowing The Tyrants - Dangerous Pastimes - Bountiful Companion Gifts
  • Disruptive Behavior - Fueling Dissent - Bountiful Luxury Fabrics
  • Shady Deal - Organ Transplant - Rare Companion Gifts

Grade 2
  • Sord Trunnel - New Deal - Rare Companion Gifts
  • Double-Cross - Pirate Queen - Rare Luxury Fabrics
  • Forging Alliances - Mandalorian Mercenaries - Rare Underworld Metals

Grade 3
  • Neimoidian Cowardice - Major Straken - Moderate Companion Gifts
  • Supplying the Resistance - Rodian Greed - Moderate Underworld Metals
  • Undercutting The Empire - Quashing the Rebels - Rare Companion Gifts
  • Taking Down A Rival - Think Again - Rare Luxury Fabrics

Grade 4
  • Banned Beasts - A Winning Hand - Moderate Companion Gifts
  • Access Allowed - A Simple Job - Moderate Underworld Metals
  • Rebel Ordnance - Ransom Demands - Abundant Companion Gifts
  • An Unhealthy Loan - Sith Fodder - Rare Companion Gifts
  • Tribute - Stolen Plans - Rare Companion Gifts
  • Double Exposure - Unauthorized R & R - Rare Luxury Fabrics

Grade 5
  • Imperial Joyride - Silencing the Snitch - Moderate Luxury Fabrics
  • A New Home - Military Takeover - Moderate Underworld Metals
  • Behind Imperial Lines - A Convincing Argument - Rare Companion Gifts
  • Incriminating Footage - Special Delivery - Rare Companion Gifts
  • Secret Vices - (No Competition: Appears in Grade 6 Imperial) - Rare Companion Gifts
  • For The Troops - Shakedown - Rare Luxury Fabrics
  • Blockade Run - Cherished Memento - Rare Underworld Metals

Grade 6
  • Meds For Vets - The Sinews of War - Abundant Luxury Fabrics
  • Well-Earned Rest - A Volatile Gift - Abundant Underworld Metals
  • A Better Life - Discreet Exit - Rare Companion Gifts
  • Shattering Their Chains - Exchange Envy - Rare Companion Gifts
  • Vital Intel - Gaining Favor - Rare Companion Gifts
  • (Secret Vices: Appears in Grade 5 Republic) - No Competition - Rare Companion Gifts

[Note: Without the resources of Darth Hater and the Darth Hater Database for Underworld Trading Missions, I would not have been able to fill in the Imperial counterparts to the Republic Missions. Thank you very much, folks!]


You might want to ask, after seeing all of this: What about Treasure Hunting? Slicing? Diplomacy? Investigation, etc. etc.? Well, if the same design process was used, then I would lay heavy odds that those systems are biased by design as well. Just off the top of my head, I can say with certainty that there are no Rare Yield Lockbox missions for Grade 4 Slicing. What it means practically would vary by the Crew Skill. What it means in sum is that the system of mission assignment in SWTOR Crew Skills is biased-by-design, if not actually biased-by-intent against success, and that it needs to be fixed to the extent that anyone out there believes it is wrong to have an unmodified chance for getting a mission for what you want that is 3 to 1 against, and that in 5 cases out of 6 you will not be able to send all your companions out to get the materials you need, even though you and your crew are as skilled as the game allows.

Yes, as usual, I have some ideas on how to fix all this. That will have to wait for another thread coming in a few days. For those who know me, that may mean ignoring me, TLDRing me, or maybe taking what I have to say seriously and having a chuckle along the way. I can say that this new thread is going to incorporate proposals and suggestions for improving the RE and Crew Skills games without replacing it completely (or at all in some cases) and these ideas will be from many people, not just my own. The case against the atrocious mechanics behind reverse engineering have been made elsewhere; I'm just taking a shot at pointing out that Crew Skill Missions are just as flawed, but in much more complex ways. Just as with the problems inherent in the Reverse Engineering process, Crew Skill mission distribution gives all the signs of having a design/development team that grasps the fundamentals of statistics and probability but takes no care whatsoever as to the larger implications their choices in what to randomize and how to randomize it affect the player community in both the short and long term. I'd like to think of that as several large instances of oversights, and not a lack of understanding of the higher statistical issues involved. The only other assumption available -- that the design/development team understood how the statistics of their minigame would scale to a system designed for failure en masse for the player population -- is one I would not prefer to suggest or entertain.

Jedi Guardian JonBonJovi'wan
"Wanted: Dead or Alive"
Defenders of Monkeys - Prophecy of the Five
DatacronHunter's Video Guides to Datacrons | Datacron Spreadsheet