Jump to content

Reverse Engineering is not 20%


Darth_Sweets

Recommended Posts

The question is, what is "enough times" to prove the 20% out? It's not going to be measured in 100's or even 1,000's. Any statisticians care to answer that?

It depends on how certain you want to be. The uncertainty depends not on the number of tries, but on the number of successes. With 100 successes, one standard deviation is about 10 (or 10%). With 1000 successes, it is around 31.6 (3%). More generally, it goes as roughly the square roof of the number of successes. If you know how many standard deviations away you are from the expected value, you can calculate the probability of the expectation being valid.

 

This is in fact exactly what Darth Sweets did. He had 400-odd tries (i.e. with a 20% chance, the expectation is 80) and 53 successes so he is a little over 3 standard deviations away. He did the calculation and it works out to about a 3 in a 1000 chance of the RE probability being 20%. Like a lot of people have said earlier in the thread, this is improbable, but hardly impossible.

 

By the way, I tried to record my RE attempts and what I got at the point when I stopped was 26 successes out of 125 attempts (i.e. it's basically dead-on 20%, even with a relatively large uncertainty).

Link to comment
Share on other sites

  • Replies 160
  • Created
  • Last Reply

Top Posters In This Topic

First off, to anybody reading through this thread who doesn't have a strong background in probability and statistics, please do not assume anything written in this thread is correct. Some (a minority) of the people posting here know what they're talking about (like for instance the post just above mine), but there's a lot of junk here. So your best bet is really not to trust anything but read about this stuff somewhere else if it interests you.

 

The testing technique in the original post is legitimate. I've seen tons of posts claiming that the sample size just isn't large enough, without actually responding to the numbers the OP put up. Most of those posts are simply incorrect. There is no fundamental sample size that you need. To find a discrepancy between a claimed hypothesis and reality, the size of the sample required depends on the size of the discrepancy and how certain you want to be. Sometimes 100 samples is enough.

 

Suppose I have a coin that comes up heads 100% of the time. How long will it take you to determine my coin is rigged? Not very long. After it comes up heads even just 20 times in a row, you will be very suspicious. After 50 times it's virtually a certainty. If, on the other hand, I have a coin that comes up heads 51% of the time, it will take a very large number of samples to prove anything.

 

To work out what these sample sizes are, there is no alternative except to crunch the numbers, which the original poster did. You start with the hypothesis that 20% is the probability of RE. You do the experiment. You see how likely it is that you would get the outcome you got or one more extreme, if the probability really were 20%. If this probability is low, you justify discarding the hypothesis that 20% is the true probability.

 

Here is what (in my opinion) is missing from this discussion: Bayesian statistics. Some of the posters are correct in not criticizing the original methodology, but saying that it simply doesn't support the conclusions adequately. Why not? It seems like the confidence interval is pretty convincing. The reason is that 20% isn't just another number. It's the number given to us by the game. Since it's really easy to generate heads randomly (pseudo-randomly, technically) at a 20% level, I tend to suspect that Bioware did not screw this up. In other words I have prior beliefs about the likelihood that 20% is the RE rate, as opposed to other values.

 

Suppose I flip a coin I find on the street 100 times. I get 75 heads. This is a wildly improbably result for a fair coin. Yet, I will not conclude that the coin I found on the street is unfair. Why? Because if you find a random coin on the street, it is many many many times more likely to be fair than unfair (note that when I say fair, I mean within a small tolerance of 50%, as real coins are). When you work out the math, it results in the final conclusion still being that the coin is likely fair. Because it is more likely that I found a fair coin and had an unusual sequence of flips, then that I found an unfair coin and had a usual sequence of flips.

 

The same applies here. The evidence would be convincing if there was nothing special about 20%. But I have pretty strong prior beliefs about Bioware programmers being able to do something so simple correctly. In other words, despite the evidence, I think it is more likely that Bioware got this right and that your test was a fluke, then that Bioware screwed this up and your test is representative. So I will require much, much stronger evidence before I believe that 20% is not the true rate.

Edited by quickNir
clarification
Link to comment
Share on other sites

@quickNir

 

After taking a quick refresher in statistics (using wikipedia) and running the OP's numbers, I want to apologize to the OP. His analysis is spot on. His results fall outside the predictable margin of error. I focused on the sample size of his test rather than the analysis of that sample and for that I apologize.

 

But as you said, his is only one test. And to conclude that the system is broken based on that one test is irresponsible. More testing is required to come to a viable conclusion. So, at the end of the day, to an extent, we are right back where we started...the "sample size" argument in fact still applies :confused::D.

Link to comment
Share on other sites

@quickNir, Being one of the main posters who keeps harping on the sample size being inadequate, I feel the need to defend my position.

 

Early on (close to a year ago now) my anecdotal experience was suggesting to me that I was getting too few successes for each RE. So I began logging every attempt at REing a green at 20% and every attempt at REing a blue at 10% (I was mainly focusing on Armormech and Synthweaving at the time).

 

After 200 tries, I was way off of expected results

After 400 tries, it was still off, but better

After 600 tries, I was starting to reach the 20% and 10% respectively

 

I stopped tracking shortly thereafter, as the more I did the closer I reached 20%/10%. This was adequate for my purposes.

 

Please notice the title of this thread is: "Reverse Engineering is not 20%".

 

The OPs sample size is adequate to prove that HE wasn't getting 20%. It is NOT adequate to prove that everyone else isn't getting 20% either.

 

This statement that he made:

With the mean sample is out of the 99.7 boundary that mean that it is almost impossible that the 20 percent is the true rate of getting a new plan.

 

At only 400-odd samples in his raw data, it wouldn't take very many successes in a row to bring his data above the 99.7. Cleet_Xia (post 29) pointed out that 3 more successes would bring the results within the boundary.

 

-------------------------------------------

 

Now, my most significant objection with the sample size is a purely practical one:

 

If Bioware has, in fact, incorrectly implemented RNG and it is not actually working at the stated 20%, they will need a lot more data than the OP has provided to look into it.

 

Does anyone really believe that those 400 points of data is "proof" that the 20% is wrong? Really? And that based on this alone the devs should believe it is broken? Really really? Seriously?

 

If the OP instead was asking for others to log their RE results and add them to this thread, and collectively a broader picture started to form, THAT would be useful. But that's not what happened. Data was gotten; a conclusion was formed; it was briefly defended; and that was it.

Edited by Khevar
Link to comment
Share on other sites

So, ultimately I agree that the sample size is inadequate; I wrote as much in my post above. It's just the reasoning being provided that is incorrect. Why is the sample size too small? Because of our prior beliefs. This issue is being danced around (in Khevar's post as well) but not being made explicit.

 

Khevar, the 99.7 is an arbitrary boundary. In fact, it's common in many areas of science to use 95% as the cut-off for significance. Pointing out that 3 more events would have brought it within the boundary is irrelevant. The fact is, the chance of getting such a low result given a rate of 20% is extremely small. The whole idea of the confidence interval is that it builds in information both from the size of the discrepancy to the proposed model, and the sample size. If a measurement lies outside the 99.7 percent confidence interval for a given hypothesis, that carries the same weight in disproving our hypothesis regardless of the sample size.

 

I also do not know what you mean by saying that it proves that "HE" isn't getting 20%. Are you claiming that the OP somehow has RE generated at different probabilities? Clearly his results are universally applicable. That they don't support the conclusion is another story, and it's because of what you describe as "practical issues", which is in fact a very good name for it. Those practical issues amount to our beliefs about Bioware and their ability to program something in line with their claims.

 

Let's get quantitative: if we use the Bayesian framework, we can estimate the relative likelihood of different success rates. Let's assume that initially we have no bias about what the success rate is. We can calculate the relative probability of any given success rate given the data. I assumed an experiment with 400 trials and 52 successes. The most likely success rate according to this is 13%. It turns out a success rate of 13% is about 1000 times more likely than a success rate of 20%. The question is this: before you saw this experiment, how many times more likely did you think a success rate of 20% (the quoted value) was than 13%? If you think it's more than 1000 times higher (I do), then this experiment should not change your mind. If you think that people at Bioware are incompetent and only had a 50-50 chance to get the correct rate, with the other 50 percent being distributed equally at every possible success rate, you would have thought that 20% is only about 100 times more likely than 13%, so you would find this experiment persuasive.

Link to comment
Share on other sites

Just as a comment, I do appreciate this level of discourse. :)

 

I think I should clarify the point I made about HE vs EVERYONE getting the 20%.

 

When I did my own personal logging of tests, my end result after >600 samples was, as I recall, ~19% for the green-blue and ~11% for the blue-purple.

 

So my tests came up with a very different result than Darth_Sweets. Here are some of the possible conclusions that can be drawn:

 

1. Between the time of my test and Darth_Sweets' test, the underlying RNG code changed.

2. My test contained innacurate raw data.

3. His test contained innacurate raw data.

4. One test or the other (or both) contained insufficient data to draw a conclusion.

Link to comment
Share on other sites

Everybody in this thread - "Math math math math math, I'm smart, math math math, you're wrong, math math"

 

The only answer that matters:

 

Whenever you have RNG, people "feel" like they come out on the wrong side of the equation. People being mad at your game all the time = bad. Implement a system where you have a specific # of attempts to achieve a schematic. You have to RE 5 greens to get a blue, and 10 blues to get a purple. I'll gladly trade the "joy" of getting a schem on my 3rd attempt, to rid myself of the frustration of not getting one by the 37th, and then we are still making the investment in creds/mats/time etc.

 

I don't isn't that the fun of it all seeing those big green letters saying you know it even after you spend 6 mill for one Schematic to me it was worth it ...

Edited by tanktest
Link to comment
Share on other sites

I'm trying to dig out the info, but I'm pretty sure the devs have stated that RE chance is modified by your level of crafting skill VS the item's actual level. Which means that you simply wouldn't have enough info to claim anything at all, since we don't know how the initial 20% base rating would be affected.

 

For all we know, you could be at 20% base - 5% mod. Meaning your 14ish% would be right within target.

 

I'll post link of the Dev post ASAP.

Link to comment
Share on other sites

I'm trying to dig out the info, but I'm pretty sure the devs have stated that RE chance is modified by your level of crafting skill VS the item's actual level. Which means that you simply wouldn't have enough info to claim anything at all, since we don't know how the initial 20% base rating would be affected.

 

For all we know, you could be at 20% base - 5% mod. Meaning your 14ish% would be right within target.

 

I'll post link of the Dev post ASAP.

Hmm. I've been following crafting posts for a long time and have never seen any dev post like that.

 

Are you sure you aren't thinking of the green/yellow/orange gathering missions? Those have different success percentages.

Link to comment
Share on other sites

It is sarcasm. Excuse me. I am made 22 implants and at their destruction not give the recipe of an artefact. To me it is very sad, that such decisions are accepted BioWare. The matter is that to lift a level of a speciality was the interesting - ambassador of 10 % not. It became very sad. :( So my Eng simple, Excuse me. I am understand you. Edited by Hrolf_Masters
Link to comment
Share on other sites

I'm trying to dig out the info, but I'm pretty sure the devs have stated that RE chance is modified by your level of crafting skill VS the item's actual level. Which means that you simply wouldn't have enough info to claim anything at all, since we don't know how the initial 20% base rating would be affected.

 

For all we know, you could be at 20% base - 5% mod. Meaning your 14ish% would be right within target.

 

I'll post link of the Dev post ASAP.

 

I don't know for certain...but I feel like this is a statement I could believe. Whenever I RE a green item on par with my crafting skill level, it takes a lot of RE'ing to get the blue item....but if I RE an item very far below my current skill level, I only have to RE a couple or three at most to get the blue item. This would suggest a curvature in the RE possibility function being used as it relates to crafting skill level.

 

Which would explain why it's so difficult to obtain blue/purple endgame crafting schematics from RE.

Link to comment
Share on other sites

quickNir, Khevar; I'm not sure whether I should be grateful to your two for your raising the level of discourse here, or disappointed because it's just not as funny as the earlier comments.

 

Intentionally funny, or just bad and not realizing it funny? In any case, you're welcome/I apologize.

Link to comment
Share on other sites

I don't know for certain...but I feel like this is a statement I could believe. Whenever I RE a green item on par with my crafting skill level, it takes a lot of RE'ing to get the blue item....but if I RE an item very far below my current skill level, I only have to RE a couple or three at most to get the blue item. This would suggest a curvature in the RE possibility function being used as it relates to crafting skill level.

 

Which would explain why it's so difficult to obtain blue/purple endgame crafting schematics from RE.

 

In my opinion, it seems more likely we are all falling into the "watched pot never boils" syndrome: if you work crafting and missions when you do not need anything, the desired goal seems to happen more readily. But as soon as you feel you do need something that something just does not happen.

 

Take the situation I ran into two nights ago. I was trying to craft a belt for an alt. On my JK level 50, 400 synthweavver, it took 15 tries to get a level 25 green belt to RE into ONE schematic. I got lucky that it was the type I wanted (redoubt) and I stopped REing. I was also trying to make armoring and mods on my BH level 50, 400 cybertech for the same alt and it took 10 REs to get the resolve armoring and 9 REs to get the mod to generate blue schematics.

 

BUT then on that same cybertech, I started making grade 22 armoring and mod to RE (because I had a ton of material and felt like getting those schematics) and got the blue might armoring 22 schematic in one try.

 

My point is that the frustration of not getting what you want (whether it be a schematic from RE or a specific mission for a specific material) stems more from not getting it when you feel you need it, not from the system being designed a certain way.

Link to comment
Share on other sites

  • 11 months later...
Unnecessary bump. No way in hell it's 20%. There's a bug, no question about it. Cheers!

 

How many attempts have you made?

Where's your tracked data?

 

I recently made 20 of each of two types of green level 52 droid parts (for the purpose of farming MK-9 augmentation components) and got all six blue schematics in 37 attempts. That is a little worse than 20% but well within the margin of error for such a tiny sample.

Link to comment
Share on other sites

Unnecessary bump. No way in hell it's 20%. There's a bug, no question about it. Cheers!

It's easy to remember the loosing streaks and harder to remember the winning streaks.

 

I recently had about 56 failures in a row, and was starting to get grumbly about it. I had to remind myself that two weeks ago I had 8 successes in 10 tries.

 

If you're not just resurrecting this thread to whine, and you actually think that it is bugged, the only thing you can do is put together a spreadsheet and log every RE attempt. When you have enough data, present it. Otherwise it just sounds as if you've fallen for some variation on the Gambler's fallacy.

Link to comment
Share on other sites

It's easy to remember the loosing streaks and harder to remember the winning streaks.

 

I recently had about 56 failures in a row, and was starting to get grumbly about it. I had to remind myself that two weeks ago I had 8 successes in 10 tries.

 

If you're not just resurrecting this thread to whine, and you actually think that it is bugged, the only thing you can do is put together a spreadsheet and log every RE attempt. When you have enough data, present it. Otherwise it just sounds as if you've fallen for some variation on the Gambler's fallacy.

What's the point? People will say the sample size is not big enough no matter how many I do. You really got more than 50 failures in a row and you think it's not bugged? 50 failures? The odds against getting 50 failures in a row are 99.99858% ... worse than 1 in 50,000.

 

There's a bug, no question about it.

Link to comment
Share on other sites

You've just had bad luck.

 

The number of rolls has NO bearing on the outcome, as each instance is it's OWN. Think of it like rolling a pair of dice at a craps table. You have a certain % to roll a 6. This % does not change no matter how many times you rolls. Murphy's law makes you feel like if you roll 100 times, and you have a 20% chance of rolling, means you'll see that 6 - 20 times but it doesn't work that way.

 

RE 1 - you have a 1/5 chance of RE'ing

RE 2 - you have a 1/5 chance of RE'ing

RE 3 - you have a 1/5 chance of RE'ing

....and so on.

 

 

It doesn't matter if you RE 1000 things, you still have a 1/5 chance on each INDIVIDUAL RE, not as a whole.

Link to comment
Share on other sites

What's the point? People will say the sample size is not big enough no matter how many I do. You really got more than 50 failures in a row and you think it's not bugged? 50 failures? The odds against getting 50 failures in a row are 99.99858% ... worse than 1 in 50,000.

 

There's a bug, no question about it.

I haven't kept a precise spreadsheet in over a year now, so I'm not going to try and convince you it's not bugged.

 

I will say this. I have 16 alts with crafting professions at 450. I've been playing this game for over two years. I have every available low endurance 9-49 armoring, mod, barrel and hilt reversed to purple, as well many enhancements, and a number of 63, 69 and 72 items. So, let's just say I've spent some time doing reverse engineering.

 

Every single losing streak has balanced out with a winning streak. The first time I ran into an extensive losing streak I tried to "prove" it was broken. So I started my spreadsheet. The longer I went, the closer the average came out to 20%

 

But frankly, it sounds like you're just here to vent. I have no doubt that you don't want to hear any of this.

Edited by Khevar
Link to comment
Share on other sites

I haven't kept a precise spreadsheet in over a year now, so I'm not going to try and convince you it's not bugged.

 

I will say this. I have 16 alts with crafting professions at 450. I've been playing this game for over two years. I have every available low endurance 9-49 armoring, mod, barrel and hilt reversed to purple, as well many enhancements, and a number of 63, 69 and 72 items. So, let's just say I've spent some time doing reverse engineering.

 

Every single loosing streak has balanced out with a winning streak. The first time I ran into an extensive loosing streak I tried to "prove" it was broken. So I started my spreadsheet. The longer I went, the closer the average came out to 20%

 

But frankly, it sounds like you're just here to vent. I have no doubt that you don't want to hear any of this.

 

The OP (or the person who resurrected this necro thread) is just here to vent, and does not understand RNG theory. Every 6 months someone complains that RE is bugged and RNG theory must be explained to them.

 

As a side point, this:

LOSE: (verb) to misplace something or come out on the bad end of a gambling match; example: "I'm on a losing streak."

LOOSE: (adjective) floppy; example: "That dude is crazy--I think he has a screw loose."

Link to comment
Share on other sites

after going through too many materials i decided to check the reverse engineering rate of going greens to blues which is stated to be at 20% according to the tool tip in game. I have collected the number of reverse engineering attempt and success that i have had going from a green to a blue, I have not counted the times where I RE'ed and there was no chance of a plan to me gained.

 

Reverse Reverse

Engineering Engineering

Attempts Successes

9 - 1

15 - 2

5 - 1

6 - 1

1 - 1

5 - 0

5 - 1

5 - 0

10 - 1

19 - 5

15 - 4

8 - 2

10 - 2

10 - 2

1 - 1

24 - 7

10 - 0

7 - 2

5 - 0

9 - 2

10 - 1

5 - 1

5 - 0

10 - 1

10 - 0

10 - 1

10 - 1

10 - 0

4 - 1

15 - 0

9 - 2

10 - 1

5 - 0

5 - 0

5 - 0

10 - 1

5 - 1

10 - 1

10 - 0

5 - 0

2 - 1

5 - 0

10 - 0

7 - 1

13 - 2

8 - 1

15 - 2

 

This gives a 13.4 percent mean. Using a standard confidence calculation with 99.7 percent boundary if 20 percent is the true mean as defined in the tool tip the average for the sample above should be between 14.3 to 25.7 percent. With the mean sample is out of the 99.7 boundary that mean that it is almost impossible that the 20 percent is the true rate of getting a new plan.

 

/5char

Edited by Thubb
Link to comment
Share on other sites


×
×
  • Create New...