Move GAMS data into R: gdxrrw vs gdx2r

GAMS tool GDXRRW

Script:

$set script script.R
$set rpath  C:\Program Files\R\R-3.2.3\bin\x64\Rscript

set i /i1*i200/;
alias (i,j,k);
parameter p(i,j,k);
p(i,j,k) = uniform(0,1);
execute_unload "p",p;

$onecho > %script%
library(gdxrrw)
p<-rgdx.param("p.gdx","p")
head(p)
$offecho

execute '="%rpath%" "%system.fp%%script%"';

Log:

--- Job Untitled_5.gms Start 02/12/16 18:26:13 24.6.1 r55820 WEX-WEI x86 64bit/MS Windows
GAMS 24.6.1   Copyright (C) 1987-2016 GAMS Development. All rights reserved
Licensee: Erwin Kalvelagen                               G150803/0001CV-GEN
          Amsterdam Optimization Modeling Group                     DC10455
--- Starting compilation
--- Untitled_5.gms(17) 3 Mb
--- Starting execution: elapsed 0:00:00.007
--- Untitled_5.gms(8) 258 Mb  3 secs
--- Untitled_5.gms(9) 261 Mb
--- GDX File C:\tmp\p.gdx
--- Untitled_5.gms(17) 261 Mb
   i  j  k         p
1 i1 i1 i1 0.1717471
2 i1 i1 i2 0.8432667
3 i1 i1 i3 0.5503754
4 i1 i1 i4 0.3011379
5 i1 i1 i5 0.2922121
6 i1 i1 i6 0.2240529
*** Status: Normal completion
--- Job Untitled_5.gms Stop 02/12/16 18:26:41 elapsed 0:00:27.719

New tool gdx2r

Script:

$set script script.R
$set rpath  C:\Program Files\R\R-3.2.3\bin\x64\Rscript

set i /i1*i200/;
alias (i,j,k);
parameter p(i,j,k);
p(i,j,k) = uniform(0,1);
execute_unload "p",p;

$onecho > %script%
load("p.rdata")
head(p)
$offecho


execute '=gdx2r -i p.gdx -o p.rdata -compression no';
execute '="%rpath%" "%system.fp%%script%"';

Log:

--- Job Untitled_7.gms Start 02/12/16 18:35:14 24.6.1 r55820 WEX-WEI x86 64bit/MS Windows
GAMS 24.6.1   Copyright (C) 1987-2016 GAMS Development. All rights reserved
Licensee: Erwin Kalvelagen                               G150803/0001CV-GEN
          Amsterdam Optimization Modeling Group                     DC10455
--- Starting compilation
--- Untitled_7.gms(18) 3 Mb
--- Starting execution: elapsed 0:00:00.007
--- Untitled_7.gms(9) 261 Mb
--- GDX File C:\tmp\p.gdx
--- Untitled_7.gms(17) 261 Mb
GDX2R v 0.1 Copyright (c) 2016 Amsterdam Optimization Modeling Group LLC
64 bit version

   Input file:p.gdx
   Output file:p.rdata
   File format:Uncompressed
   Buffer size:4096
   Strings as factors:True

   Uels:200 (unique strings in input data)
   Symbols:1
   Exporting symbols:
      p (Converting 8000000 records from a 3 dimensional parameter to a data frame)
   Time:3.75 seconds
--- Untitled_7.gms(18) 261 Mb
   i  j  k     value
1 i1 i1 i1 0.1717471
2 i1 i1 i2 0.8432667
3 i1 i1 i3 0.5503754
4 i1 i1 i4 0.3011379
5 i1 i1 i5 0.2922121
6 i1 i1 i6 0.2240529
*** Status: Normal completion
--- Job Untitled_7.gms Stop 02/12/16 18:35:23 elapsed 0:00:09.487

Discussion

It is somewhat surprising that gdx2r is so much faster on this task. Here are the steps in the first GDXRRW script:

1. Generate data and write GDX file
2. Start R
3. Read GDX file

The gdx2r script does actually much more I/O:

1. Generate data and wite GDX file
2. Read GDX file
3. Write .Rdata file
4. Start R
5. Read .Rdata file

Looking at these steps I would expect the first GDXRRW script to be twice as fast as the second gdx2r script while actually it is three times as slow (so we are a factor six off).