In some cases GDX compression can make a lot of difference. Here are the results for a large CSV file:
AccPsdData_2009-11-19Su.gdx 4,318,222 (with compression turned on)
The CSV file was read using SQL2GMS:
--- Job readcsv.gms Start 04/28/10 15:50:32 WEX-WEI 23.3.3 x86_64/MS Windows
It is a little bit slow as we make different passes over the data. Probably this would be faster if it was stored in a database.
Note: the sets were created to make it easier to read the data in a GAMS model. This way we have sets that can be read during compile time and thus can be used as domain. It is easy to calculate those sets in GAMS at run time, but then they cannot be used as domains.
The compression was turned on by the statement:
* compress gdx file