I have an algorithm, it works on very large data sets. In it's initial version it simply reads in all the data before creating data-structures so that queries can be made on it, I'm realising that I ...

How are you reading this XML file? If you're using DOM, you're not going to get this to work on 32-bits, ever. And on 64-bits, you're not going to want to do it without lots of RAM. If you're using ...