Wordlist optimizing program?

Is there any program out there that will merge big files and remove duplicates and optimize the wordlist?

I have google the web and find nothing.

I have tried to merge files and optimizing them but all my word programs hang or crash so no point in doing it.

MD5