In Defense of EA/Bioware

He said "software aint easy" too. Ya I read it. The jiff of his defense was "poop happens". My kids have better defenses for non sense.

As paying customers of a product that is currently broken, many of you are mad at EA/Bioware. You should be. They let us down today. However, as an online software developer myself, I'd have you consider the following:

1. Software ain't easy.

2. Online software is even harder.

3. To-date EA has been VERY quick to address our concerns.

4. Bad things happen to good people (see Murphy's Law).

If you feel a tl;dr coming on, it's ok to stop reading now. The rest is just more detail to support my assertions above.

I Know What I'm Saying

I have written, upgraded, and deployed many online enterprise applications whose complexity pales in comparison to that of an MMO, so please try to understand that my assertions come from a place of long and painful experience.

The More Moving Parts, The Greater The Chance For Failure

SW:TOR is a complex system of systems. I know that sounds odd, but it's true. What we think of as SW:TOR is composed of *at least* the following:

1. A software client (the game you execute on your desktop and play).
2. A server (not the physical machine(s) but the software at EA that coordinates the interactions of all of us with the game and with each other).
3. A database (what the "server" mentioned above uses to remember what has gone on in the game such as items in your bag, your stats, your toon name, your lockouts, etc...)
4. Networking "stuff" such as routers, load balancers, firewalls, switches, fiber routers for database storage.

Each of the above can be viewed as a system of it's own. Together they combine to form Voltron...er....SW:TOR as you know it. Each of them are comprised of even more "parts" or sub systems. Each sub system can break.

Systems Are Made By People

Each of the aforementioned systems undoubtedly has an "owner" at EA/Bioware. By "owner" I mean they are responsible for developing and/or maintaining each sub system in some way. The more people involved in the development or deployment of an online "system" such as SW:TOR (or any online system for that matter), the greater the risk for miss-communication. It may come down to a single person mistyping a single character. You never know. People are fallible and will mess up from time-to-time. The margin for error in such a complex system is very very slim. Judging by previous patches, EA has a solid dev/deployment team. If human error is to blame, it's simply a mistake.

Developing An Update Is A Monumental Undertaking

Midnight oil was burned. Spouses patience was tested. Microsoft Project was ridden hard and put away wet. I have no doubt that there were all-nighters pulled in order to bring us patch 1.2. There were fights between project managers and developers. Hair was lost. Some people probably had to re-evaluate their decision to go into software development. Considering the time from original release to now, I'd say EA performed a minor miracle with patch 1.2. My point here is that we may want to take the time to appreciate the personal sacrifice of EA devs and their supporting business staff. These guys just ran a 6 month gauntlet that would make a grown man cry.

Development Is Not Deployment

I've been both a developer and an application hosting engineer. The guy who writes the software isn't usually the guy who puts it on the live server. To deploy is to place the software on the live server. This usually means running some upgrade routine on the "server" software or replacing it. Maybe adding some new physical hardware. Finally, running routines against the database to bring it in line with the new changes installed on the "server" software.

The database part is usually the most tricky. Don't ask why. It just is. One slip of the keyboard can ruin the whole thing and cause you to have to do it all over again (I recently did this to a development database and I'll now lose an entire day restoring the backup).

Murphy Is Our Co-Pilot

Murphy's Law states: Anything that can go wrong will go wrong. Let's assume just for the sake of argument that the Bioware team is infallible. Even a perfect man cannot account for a bad line of code in a piece of supporting software. If your software depends on that software and that software suddenly misbehaves, it could ruin your software without warning. Nobody is to blame. Bad things just happen.

What I Think Happened

I'm going to go out on a limb here and say that Murphy caused the rollback which is resulting in 8 hours of sudden downtime. Consider if you will:

1. The patch went out this morning as planned. Bioware had no indication that the update failed.

2. They told us right away that the maintenance would take 8 hours, which is the normal maintenance window. This means they are restoring the backup they took last night and rerunning the upgrade routines.

3. Bioware has been so attentive to our needs and bug reports in the past that I just don't buy negligence or incompetence as the cause for the down time. They patch us like once a week in order to keep the amount of changes they deploy at one time very low; thus reducing potential for catastrophe.

In Conclusion

I don't expect anyone to read this and suddenly not be mad at EA/Bioware. As someone who has gone through catastrophic upgrades himself (a client was once down for 4 days, during which I didn't sleep at all, and we lost millions) through no fault of his own, I just wanted you guys to get a glimpse of Bioware's point of view. They cannot provide that kind of insight themselves, so I figured I'd do it for them.

Why would anyone defend EA



It's like saying Bobby Kotick is doing great things for video gamers everywhere. The very idea is so absurd it's making my head spin.

And software isn't easy. If you've done any of it before, you'd realize that even the most skilled of programmers make errors. Finding and squashing a bug can sometimes be like finding a piece of hay in a needlestack. Chances are actually pretty good that fixing one bug might cause a bigger one to manifest.

Don't start something that you can't finish and if it isn't ready don't release it.

I could name a long list of software that has minor bugs but is within the perfectly acceptable to use class.

SWTOR, unacceptable.