Thursday, October 20, 2016

Excel and very large data files

Tables with many rows (much more than the one million row limit in Excel) can be loaded in Excel’s Power Pivot Data Model. Here is an example from Statistics New Zealand with a CSV file of more than 23 million records.

image 

Actually Excel stores this data quite efficiently:

image

Note: wc reports one extra row because of the headers in line 1.

References
  1. http://yetanothermathprogrammingconsultant.blogspot.com/2016/07/excel-pivot-table-and-large-data-sets.html