Users of the Nvidia GeForce GTX 970 tell there are problems when using more than 3.5G of memory. The GeForce GTX 970 has a total of 4G vram but according to users this cannot be fully used. One of the users told:

“Going beyond 3.5GB vram usage … severely degrades the performance, as if the last 512mb is actually being swapped from the RAM.”

980 works fine in comparison.

The computation power quickly declines when this much memory is used, which is visible while playing games.

It is unknown where the memory bug is. Nvidia has said it will look into the issue but it is unknown if it will be resolved.

Sources:
http://www.reddit.com/r/pcgaming/comments/2s2968/gtx970_memoryvram_allocation_bug/
https://www.reddit.com/r/hardware/comments/2s333r/gtx_970_vram_allocation_bug_xpost_rpcgaming/