Nuts. Was hoping to narrow down on a specific type of function on the site to look at. We're narrowing down at least.
I got 2 things I want you to check.
Next time you get a Internal Server error, note the time it happens again and also right click the page and go to "View Page Source". That will give you a whole mess of info. Send that to us by PM and we'll see if it can point us in the right direction for a cause. Also send the link to the page you got the error on.
Next time you get general site slowness I want you to see if there is any message along the bottom of the screen. Something that might detail what the site is waiting on. There should be several messages when you load, but you want one that lingers on the page. Take a screenshot of that. Should look something like this: