Internal Error CUDA status out of memory

This is a hardware error. See this thread.

First, try to reset a server connection to get a new GPU.
If this fails, then, you may need to wait for a while…

Resetting a kernel does not work. And, just waiting for does not work. Need to disconnect GPU as soon as possible, and try to see the chance to get another one.

Sorry for your inconvenience.