Problem with Hashcat and large hashlists

Whenever I try to crack very large hashlists (50-100k unique hashes and larger) I get massive lag when I start the attack and for the first 30 seconds I cant even update the status screen. The cracking speed also starts extremely slow and builds up slowly to about full speed after 1-2 minutes.

I have tried using an output file, disabling/enabling the potfile and tried a few different hashlists & dictionaries.

I am using the Nvidia version of oclHashcat with 2 x GTX 970s.
Any suggestions are much appreicated.