Tuesday, November 30, 2010

Reorganizing a large parameter in GAMS

To export some fairly substantial data in a certain format I wanted to form:

   DownLoadData(downloadfiles,SP_Varname,SP_SubVarName,SP_SubSubVarName,SP_Indicator,
                               
SP_Description,SP_Source,SP_Unit,cty,Fwdy) =

     
sum
((mapdownloadfiles(downloadfiles,SP_control),SP_Map,SP_Graph,SP_Options,SP_Type,mapy(FwdY,SP_year)),
          cube(SP_Control,
               SP_VarName,
               SP_SubVarName,
               SP_SubSubVarName,
               SP_Year,
               SP_Indicator,
               SP_Source,
               SP_Description,
               SP_Unit,
               SP_Map,
               SP_Graph,
               SP_Options,
               SP_Type,
               cty
               ) );


The original Cube parameter has 326,755 elements, and this operation basically reshuffles things into a lower-dimensional parameter with some mappings applied. Upon completion the new parameter Download_Data has the same number of elements: 326,755. Unfortunately this operation was very expensive, so much so that I never saw it finish.

The following code does exactly the same in just 0.5 seconds:

loop((mapdownloadfiles(downloadfiles,SP_control),mapy(Download_Year,SP_year)),

   Download_Data(downloadfiles,SP_Varname,SP_SubVarName,SP_SubSubVarName,cty,SP_Indicator,
                             SP_Description,SP_Source,SP_Unit,Download_Year) =
 
     
sum
((SP_Map,SP_Graph,SP_Options,SP_Type),
          cube(SP_Control,
               SP_VarName,
               SP_SubVarName,
               SP_SubSubVarName,
               SP_Year,
               SP_Indicator,
               SP_Source,
               SP_Description,
               SP_Unit,
               SP_Map,
               SP_Graph,
               SP_Options,
               SP_Type,
               cty
               ) );

);

GAMS has some built-in facilities to automatically rearrange assignments like these to achieve better performance, but in this case it was not very effective. So human intervention was needed to optimize this assignment.