No I haven't figured out what is going on, it doesn't happen frequently enough, for me to prioritize. I did, sort of, recreate this problem by flooding the server with requests using Apache Bench. There appeared to be some sort of staircase pattern going on with slowdowns, that got worse over time. It the server was given a chance to catch up, the slowdown went away.
I wasn't convinced my testing was actually chasing the problem, because I don't think my servers were getting flooded with requests. Also the server responded to all my testing requests, it just got slower and slower.