Here's a quick reference table:

http://i.imgur.com/1MSauDP.png
Two principles guide armor penetration calculation

1) boss base armor

*rating* is approximately 7538, and Damage Reduction = ArmorRating / ( ArmorRating + 240 * 55 + 800 ) * 100 (note that this equation for DR applies to both PCs and NPCs)

2) incoming effects are calculated multiplicatively with outgoing effects. But within each class {incoming, outgoing} effects are summed. The only consistent incoming effect is armor reduction; consistent outgoing effects include both penetration and ignorance

Here's a full walkthrough:

- calculate total outgoing effects; 1 - sum armor penetration/ignorance effects of your character. For example if your character has 0.1 armor penetration from spec and an extra 0.2 armor penetration on one specific attack that you want to look at, your character has 1 - 0.1 - 0.2 = 0.7 penetration/ignorance effects total on that attack, and 1 - 0.1 = 0.9 on most kinetic/energy damage

- calculate total outgoing effects on rating; the effect of that reduction/penetration coefficient on base armor rating, 7538. So if we're working with a coefficient of 0.7, 7538*0.7 = 5276. If we're working with a coefficient of 1-0.1 = 0.9, 7538*0.9 = 6784.2

- calculate total incoming effects on rating; we'll almost always just have to worry about whether there's an armor debuff (-20% armor rating) or not (-0% armor rating) on the target. Building off our previous example, if we have 0.3 armor penetration/ignorance and a 20% rating debuff, now the target effectively has 7538*(1-0.3)*(1-0.2) = 5276*(1-0.2) = 4221.28 armor
*rating*. If we have 0.1 armor penetration/ignorance and no 20% reduction debuff, the target still has 7538*(1-0.1)*(1-0) = 6784.2 armor *rating*

- calculate target final damage reduction as function of effective armor rating; whether 4221 armor rating or 6784, we feed it into the damage reduction formula Damage Reduction = ArmorRating / ( ArmorRating + 240 * 55 + 800 ) * 100. In the case of 4221 rating we'd see 23.16 DR, and in the case of 6784 rating we'd see 32.64 DR

- optional calculation: take 1 - DR to find tooltip coefficient. In the case of 4221 rating, we'd see 1 - 0.2316 = 0.768, and in the case of 6784 rating we'd see 1 - 0.3264 = 0.676. Now you can take coefficient*(attack min + attack max)/2 to estimate the noncrit average damage you'd see from that attack.

In PVP, where targets have a variety of armor ratings, the general effect is that picking on a lightly armored target is still almost always going to allow you to do more damage than picking on a heavily armored target when you have high armor penetration. Really the only two exceptions to this are

1) the Assault Vanguard; since almost all their damage is either elemental (ignores DR) or HIB (has a 0.903 coefficient) you're doing basically the same damage to a heavily armored target as you would do to a lightly armored target. The light target will still take harder hits from HIB, but the difference is quite small

2) the Combat Sentinel; during your burst window, you could probably hurt a heavily armored target much more than anyone else nearby

I'll just do one example, that in no way applies to all enemies...

Suppose you have 0.3 outgoing armor penetration/ignorance from spec, your target doesn't have a 0.2 damage reduction effect, your attack has 7500 damage before crits and damage reduction, and your target is wearing light armor from full 78s (3212). You'd see (1-0.3)*3212 = 2248 effective rating, or 2248/ ( 2248+ 240 * 55 + 800 ) * 100 = 13.83 damage reduction, so we'd expect a non-crit hit to hit for 7500*(1-0.1383) = 6462 damage.

Another fun note, the Sharpshooter Gunslinger's Aimed Shot with Illegal Mods is the only attack in the game that enjoys 4 sources of armor penetration/ignorance/reduction: 0.1 from spec for kinetic damage, 0.2 from spec on aimed shot, 0.3 from Illegal Mods, and 0.2 from the incoming reduction, giving its targets 2412 effective rating or a coefficient of 0.853. Unfortunately that's not enough to beat Assault Vanguard HIB, at 1507 rating or a coefficient 0.903.