Yet Another Math Programming Consultant: Bostock's command-line cartography tutorial under Windows

In [1], an excellent tutorial is presented about the process of making maps. It is a little bit dated, so here I develop some Windows-based scripts that make it possible to follow these tutorials. The goal is two-fold:

Make things work for Windows. I am comfortable with the Unix command line, but, by far, most of my colleagues are not. To allow for easier sharing, I used Windows CMD.
Update the commands so they all function again. Some URLs have changed a little bit, and we need to use a specific version of d3. If you want to use the original Unix commands, and you encounter some issues, you may want to check how I repaired them.

The output format of these scripts is svg. SVG files are plain text, represent vectors (instead of bitmaps), and can be read directly by a standard web browser.

Part 1

The first thing to do is to install node.js if you don't have this available already. Using [2] this process is quite straightforward. Our first batch file will follow the steps in [1].

::---------------------------------------------------------------------------
::
::   Mike Bostock
::   Command-Line Cartography, Part 1
::   A tour of d3-geo’s new command-line interface.
:: 
::   https://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c
::
::   Needs node.js. Install from: https://nodejs.org/en/download/
::
::   we translate UNIX command line to Windows CMD
::---------------------------------------------------------------------------

:: download shape file from Census Bureau (region 06 = California)
::
:: curl 'http://www2.census.gov/geo/tiger/GENZ2014/shp/cb_2014_06_tract_500k.zip' -o cb_2014_06_tract_500k.zip
:: use https instead, curl is part of windows
curl -O https://www2.census.gov/geo/tiger/GENZ2014/shp/cb_2014_06_tract_500k.zip 

:: unzip -o cb_2014_06_tract_500k.zip
:: use tar instead (part of windows)
tar -xf cb_2014_06_tract_500k.zip

:: install node.js package
:: and extract geojson
call npm install -g shapefile
call shp2json cb_2014_06_tract_500k.shp -o ca.json

:: perform projection 
call npm install -g d3-geo-projection
call geoproject "d3.geoConicEqualArea().parallels([34, 40.5]).rotate([120, 0]).fitSize([960, 960], d)" < ca.json > ca-albers.json

:: create svg
call geo2svg -w 960 -h 960 < ca-albers.json > ca-albers.svg

:: launch browser
start ca-albers.svg

Notes:

The curl command is available on newer versions of Windows.
Windows does not have unzip, but it has tar which can unzip files.
Use https instead of HTTP.
The use of quotes is different under Windows compared to Unix.

The result should be the following vector image displayed in your browser:

Part 2

::---------------------------------------------------------------------------
::
::   Mike Bostock
::   Command-Line Cartography, Part 2
::   A tour of d3-geo’s new command-line interface.
:: 
::   https://medium.com/@mbostock/command-line-cartography-part-2-c3a82c5c0f3
::
::---------------------------------------------------------------------------

:: local environment variables
setlocal 

:: split json by adding newlines for readability
call npm install -g ndjson-cli
call ndjson-split d.features < ca-albers.json  > ca-albers.ndjson

:: set the id of each feature
call ndjson-map "d.id = d.properties.GEOID.slice(2), d" < ca-albers.ndjson > ca-albers-id.ndjson

:: get API key from https://api.census.gov/data/key_signup.html
set key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

::curl 'http://api.census.gov/data/2014/acs5?get=B01003_001E&for=tract:*&in=state:06' -o cb_2014_06_tract_B01003.json
:: https, slightly different path, add key, and use double quotes
curl "https://api.census.gov/data/2014/acs/acs5?get=B01003_001E&for=tract:*&in=state:06&key=%key%" -o cb_2014_06_tract_B01003.json


::  1. remove the newlines (ndjson-cat)
::  2. separate the array into multiple lines (ndjson-split)
::  3. reformat each line as an object (ndjson-map)
call ndjson-cat cb_2014_06_tract_B01003.json | ndjson-split "d.slice(1)" | ndjson-map "{id: d[2] + d[3], B01003: +d[0]}" > cb_2014_06_tract_B01003.ndjson


:: Join the population data to the geometry using ndjson-join
call ndjson-join "d.id" ca-albers-id.ndjson cb_2014_06_tract_B01003.ndjson > ca-albers-join.ndjson

:: compute the population density
call ndjson-map "d[0].properties = {density: Math.floor(d[1].B01003 / d[0].properties.ALAND * 2589975.2356)}, d[0]" < ca-albers-join.ndjson > ca-albers-density.ndjson

:: convert back to json
:: call ndjson-reduce < ca-albers-density.ndjson | ndjson-map "{type: ""FeatureCollection"", features: d}" > ca-albers-density.json
:: or directly:
call ndjson-reduce "p.features.push(d), p" "{type: ""FeatureCollection"", features: []}"  < ca-albers-density.ndjson > ca-albers-density.json


:: install d3.6
call npm install -g d3@6

:: create map
call ndjson-map -r d3 "(d.properties.fill = d3.scaleSequential(d3.interpolateViridis).domain([0, 4000])(d.properties.density), d)"  < ca-albers-density.ndjson  > ca-albers-color.ndjson

:: create svg
call geo2svg -n --stroke none -p 1 -w 960 -h 960  < ca-albers-color.ndjson > ca-albers-color.svg

:: launch browser
start ca-albers-color.svg

Notes:

A key for using the Census Bureau data-API can be had from [3]. You need to update the script with your key before running it.
We have to use v6 of d3 to be compatible with these command-line tools.

You should see something like:

Part 3

::---------------------------------------------------------------------------
::
::   Mike Bostock
::   Command-Line Cartography, Part 3
::   A tour of d3-geo’s new command-line interface.
:: 
::   https://medium.com/@mbostock/command-line-cartography-part-3-1158e4c55a1e
::
::---------------------------------------------------------------------------

:: for simplification topojson is a better representation
:: "TopoJSON facilitates topology-preserving simplification"
call npm install -g topojson

:: convert to TopoJSON (should reduce the size)
call geo2topo -n tracts=ca-albers-density.ndjson > ca-tracts-topo.json

:: simplify (should reduce the size even further)
call toposimplify -p 1 -f  < ca-tracts-topo.json  > ca-simple-topo.json

:: further reduction in size
call topoquantize 1e5 < ca-simple-topo.json > ca-quantized-topo.json

:: create county map by merging to 3 digit ids
call topomerge -k "d.id.slice(0, 3)" counties=tracts < ca-quantized-topo.json > ca-merge-topo.json

:: only keep internal boundaries
call topomerge --mesh -f "a !== b" counties=counties < ca-merge-topo.json > ca-topo.json

:: show layers
call topo2geo -l < ca-topo.json    
:: check counties
call topo2geo counties=ca-check-geo.json < ca-topo.json  
call geo2svg -n -p 1 -w 960 -h 960  < ca-check-geo.json > ca-check.svg
start ca-check.svg

Notes:

Added commands to check the results.

Results:

part4

::---------------------------------------------------------------------------
::
::   Mike Bostock
::   Command-Line Cartography, Part 4
::   A tour of d3-geo’s new command-line interface.
:: 
::   https://medium.com/@mbostock/command-line-cartography-part-4-82d0d26df0cf
::
::---------------------------------------------------------------------------


:: create map 
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3 "z = d3.scaleSequential(d3.interpolateViridis).domain([0, 4000]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d"  |^
ndjson-split "d.features"  | geo2svg -n --stroke none -p 1 -w 960 -h 960 > ca-tracts-color.svg
start ca-tracts-color.svg

:: sqrt transform
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3 "z = d3.scaleSequential(d3.interpolateViridis).domain([0, 100]), d.features.forEach(f => f.properties.fill = z(Math.sqrt(f.properties.density))), d" |^
ndjson-split "d.features" | geo2svg -n --stroke none -p 1 -w 960 -h 960 > ca-tracts-sqrt.svg
start ca-tracts-sqrt.svg

:: log transform
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3 "z = d3.scaleLog().domain(d3.extent(d.features.filter(f => f.properties.density), f => f.properties.density)).interpolate(() => d3.interpolateViridis), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d" |^
ndjson-split "d.features" | geo2svg -n --stroke none -p 1 -w 960 -h 960  > ca-tracts-log.svg
start ca-tracts-log.svg

:: color quantiles
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3 "z = d3.scaleQuantile().domain(d.features.map(f => f.properties.density)).range(d3.quantize(d3.interpolateViridis, 256)), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d"  |^
ndjson-split "d.features" | geo2svg -n --stroke none -p 1 -w 960 -h 960 > ca-tracts-quantile.svg
start ca-tracts-quantile.svg

:: other colors
:: call npm install -g d3-scale-chromatic  (not needed, also change the map command a bit)
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3  "z = d3.scaleThreshold().domain([1, 10, 50, 200, 500, 1000, 2000, 4000]).range(d3.schemeOrRd[9]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d" |^
ndjson-split "d.features" | geo2svg -n --stroke none -p 1 -w 960 -h 960 > ca-tracts-threshold.svg
start ca-tracts-threshold.svg


:: combine with county borders
call topo2geo tracts=- < ca-topo.json |^
ndjson-map -r d3 "z = d3.scaleThreshold().domain([1, 10, 50, 200, 500, 1000, 2000, 4000]).range(d3.schemeOrRd[9]), d.features.forEach(f => f.properties.fill = z(f.properties.density)), d" |^
ndjson-split "d.features" > geo.json
call topo2geo counties=- < ca-topo.json | ndjson-map "d.properties = {""stroke"": ""#000"", ""stroke-opacity"": 0.3}, d" >> geo.json
call geo2svg -n --stroke none -p 1 -w 960 -h 960 < geo.json > ca.svg
start ca.svg