One of six GPUs has 99% Util and 0 H/s, Progress gets stuck at 99%

Arch: 64bit
OS: Linux
GPU: 7 x AMD Sapphire 7970 HD
oclHashcat: 1.31
Catalyst: fglrx 14.30.4

The run starts fine. Status looks good. At some point one of the GPU's starts to show 99% Util and 0 H/s, Progress gets stuck at 99% and the job never finishes because it seems to be waiting for the last GPU to finish. I'm already skipping one GPU due to this issue, and now a second is acting up.

Command for testing, run should complete in about 25 min. Notice I'm already skipping GPU 6 due to the same issue.

oclHashcat64.bin --gpu-devices=1,2,3,4,5,7 --gpu-temp-retain=65 -m 0 notahash -a 3 -1 ?l?u?s?d ?1?1?1?1?1?1?1

Three status are shown below
1. Beginning, all good
2. GPU 4 wedged, other GPUs still working at progress 7%
3. GPU 4 wedged, other GPUs finished with work, progress 99%, will never finish.


1. Beginning, all good

Session.Name...: oclHashcat
Status.........: Running
Input.Mode.....: Mask (?1?1?1?1?1?1?1) [7]
Hash.Target....: ffd1ca720c067fadb7b57ee4ab07db05
Hash.Type......: MD5
Time.Started...: Mon Jan 19 15:15:16 2015 (26 secs)
Time.Estimated.: Mon Jan 19 15:42:13 2015 (26 mins, 29 secs)
Speed.GPU.#1...: 7377.6 MH/s
Speed.GPU.#2...: 7374.1 MH/s
Speed.GPU.#3...: 7388.7 MH/s
Speed.GPU.#4...: 7370.7 MH/s
Speed.GPU.#5...: 7392.8 MH/s
Speed.GPU.#6...: 7369.9 MH/s
Speed.GPU.#*...: 44273.8 MH/s
Recovered......: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.......: 1150984126464/69833729609375 (1.65%)
Skipped........: 0/1150984126464 (0.00%)
Rejected.......: 0/1150984126464 (0.00%)
HWMon.GPU.#1...: 94% Util, 45c Temp, 20% Fan
HWMon.GPU.#2...: 94% Util, 48c Temp, 20% Fan
HWMon.GPU.#3...: 94% Util, 45c Temp, 20% Fan
HWMon.GPU.#4...: 94% Util, 55c Temp, 15% Fan
HWMon.GPU.#5...: 94% Util, 44c Temp, 20% Fan
HWMon.GPU.#6...: 94% Util, 45c Temp, 20% Fan


2. GPU 4 wedged at progress 7%, other GPUs still normal

Session.Name...: oclHashcat
Status.........: Running
Input.Mode.....: Mask (?1?1?1?1?1?1?1) [7]
Hash.Target....: ffd1ca720c067fadb7b57ee4ab07db05
Hash.Type......: MD5
Time.Started...: Mon Jan 19 15:15:16 2015 (2 mins, 1 sec)
Time.Estimated.: Mon Jan 19 15:42:05 2015 (24 mins, 46 secs)
Speed.GPU.#1...: 7354.0 MH/s
Speed.GPU.#2...: 7454.7 MH/s
Speed.GPU.#3...: 7327.9 MH/s
Speed.GPU.#4...: 0 H/s
Speed.GPU.#5...: 7385.6 MH/s
Speed.GPU.#6...: 7400.3 MH/s
Speed.GPU.#*...: 36922.5 MH/s
Recovered......: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.......: 5287989215232/69833729609375 (7.57%)
Skipped........: 0/5287989215232 (0.00%)
Rejected.......: 0/5287989215232 (0.00%)
HWMon.GPU.#1...: 94% Util, 66c Temp, 15% Fan
HWMon.GPU.#2...: 94% Util, 72c Temp, 40% Fan
HWMon.GPU.#3...: 94% Util, 64c Temp, 15% Fan
HWMon.GPU.#4...: 99% Util, 62c Temp, 15% Fan
HWMon.GPU.#5...: 94% Util, 67c Temp, 15% Fan
HWMon.GPU.#6...: 93% Util, 66c Temp, 15% Fan



3. GPU 4 wedged, other cards finished with work, progress 99%, will never finish.

Session.Name...: oclHashcat
Status.........: Running
Input.Mode.....: Mask (?1?1?1?1?1?1?1) [7]
Hash.Target....: ffd1ca720c067fadb7b57ee4ab07db05
Hash.Type......: MD5
Time.Started...: Mon Jan 19 15:15:16 2015 (31 mins, 44 secs)
Time.Estimated.: Mon Jan 19 15:47:09 2015 (7 secs)
Speed.GPU.#1...: 0 H/s
Speed.GPU.#2...: 0 H/s
Speed.GPU.#3...: 0 H/s
Speed.GPU.#4...: 0 H/s
Speed.GPU.#5...: 0 H/s
Speed.GPU.#6...: 0 H/s
Speed.GPU.#*...: 0 H/s
Recovered......: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.......: 69544608370335/69833729609375 (99.59%)
Skipped........: 0/69544608370335 (0.00%)
Rejected.......: 0/69544608370335 (0.00%)
HWMon.GPU.#1...: 0% Util, 51c Temp, 15% Fan
HWMon.GPU.#2...: 0% Util, 51c Temp, 15% Fan
HWMon.GPU.#3...: 0% Util, 57c Temp, 15% Fan
HWMon.GPU.#4...: 99% Util, 60c Temp, 15% Fan
HWMon.GPU.#5...: 0% Util, 51c Temp, 15% Fan
HWMon.GPU.#6...: 0% Util, 50c Temp, 15% Fan