PROBLEM VERY LONG LISTS

[eried]

As Han said, I think you could improve everything thinking on a different way to access the lists. It does not need to be fancy, you can take the first N letters of the word and have several files containing smaller and more manageable chunks of data.

Example, just use lang_firstletter.txt, so you need about 30 files per language:
en_a.txt
en_b.tct
...