The best guess at this point is that maybe the ec is throttling the gpu.
Have you monitored the gpu frequency and the cpu frequency when this happens?
Have you monitored the gpu frequency and the cpu frequency when this happens?