Thanks for all the responses people.
New RAM is something I will definitely do, 4GB doesn't leave a lot of margin, so I'll upgrade that to 8GB.
Some of you suggested my temps may be getting too high, which results in downthrottling of CU or GPU. I specifically monitored that, and my clock speeds are at max when measuring. Temperature is not really an issue, since GPU and CPU utilization are both at 50-60%, so they're not really getting hot at all.
But when I hear some of you mention dropping down to 40-50FPS, that something I would be really glad with actually, because it's almost double of my minimum right now. I think the GPU is powerful enough to run at the resolution and detail that I run it on, so I guess I could stick a core i5 in there, that doubles the amount of physical cores I have available. But I'm still a bit weary to do so, since CPU utilization is fairly low. (Though any excuse to get new hardware is good of course :P)