Category Archives: All Posts

Creating static data visualizations with D3.js and Node.js

D3.js is known for its great data viz capabilities for the web, including dynamic and interactive capability. But D3.js can also be used to generate static data visualizations via the command line, including CSS styling and/or stylesheets.

For example, if you want a command line solution for automated visualization creation, if you want to output an SVG into your make workflow, or if you need to prepare a visualization for print (for a research paper, journal article, a newspaper or magazine), a plain old non-interactive SVG can be very useful. The output we’re aiming for is an SVG you can load into Inkscape, Illustrator, or one that we can convert to an EPS, a PDF, render to a PNG or any other image file format you can think of.

Alternatives before diving in

Of course if you’ve already created your visualization on the web, you can directly save the SVG via copying the source code and CSS, or using a handy bookmarklet like the NYTimes SVG-crowbar. So the below is a bit overkill for a case where you just want to download your already-web-based data visualization.

Using Node.js to execute our JavaScript code

Let’s jump in. Luckily D3.js and Node.js make it possible to output an SVG from a JS source file, your data, and the command line. Node.js internally uses the Google V8 JS engine to execute JS code, and is often used as a server or for networking applications. Using Node.js to output SVG is a bit of a hack vs. creating D3 visualizations for the web. Regardless, we’ll be doing so due to the advantages mentioned above — automated generation of static SVGs is our goal. As a working example, I’ll be altering Mike Bostock’s area choropleth example to be used via Node.js to output the resulting SVG.

Tools needed (any platform, but Windows is a bit tougher):

  • Node.js and npm (the node package manager)
  • D3.js for Node (install via: “npm install d3”). Note that installing D3.js on Windows via npm can be a bit tough, you may or may not run into some dependency roadblocks that will require some googling, not recommended for the faint of heart
  • Terminal/Command Prompt depending on your platform (OSX/Linux/Windows)

First things first, for your Node.js script, you’ll need to require/import D3. This is similar to including d3.v3.min.js at the top of your HTML file.

d3 = require("d3");

To load json from our local filesystem, we’ll need the fs package, since d3.json won’t work as we’re not loading json via a HTTP request (see further down).

var fs = require("fs")

Next, we’ll need to include the client-side Topojson, Topojson.js. Download the latest Topojson.js here. Often Topojson is used via Node.js to simpify Geojson data (Geojson to Topojson for a smaller file size) via the server side API. However, it also can be used to convert the Topojson back to Geojson, via the client side API when rendering a map. Because the geo data we’re using here is Topojson, we’ll need the client side Topojson API. More information on the Topojson APIs is available here.

var vm = require('vm');

var includeInThisContext = function(path) {
    var code = fs.readFileSync(path);
    vm.runInThisContext(code, path);
}.bind(this);
includeInThisContext("topojson.js");

VM is a Node.js virtual machine. It imports the topojson.js in the context of this JS file, so that we can use the topojson client side API methods. Credit goes to this stackoverflow discussion here on including external .js files in Node.js.

Next, almost all the rest of the JS code is the same. However, one difference is that d3.json will need to be replaced by the node JSON parser. d3.json loads JSON via a HTTP request, however, when we’re running via the command line won’t be necessary (or possible, in our case).

var us = JSON.parse(fs.readFileSync("us.json", 'utf8'));

To include the CSS styling, we need to include the CSS within defs tags, within the SVG block.  In addition, for compatibility with Adobe Illustrator, or if our CSS includes > or < characters, we need to enclose the CSS within a CDATA block. This ensures that when we output the SVG, the CSS is attached, but not accidentally parsed.

var css_text = "

Because JavaScript isn’t a fan of multi-line strings,  we need to backslash (escape) the end of each line. Multiline strings in JS need escape characters at the end of each line, since there is no native support for multiline strings. Note that there are alternatives to escaping the end of each line, so feel free to explore other options if you have a larger style sheet.

Last but not least, we will output the svg to stdout, which we can write to an svg file.

console.log(d3.select('body').html());

This selects the html inside the body tag, which in our case is just the SVG block. From there, we can run our JS code (I saved the source file as “area_choropleth.js”) from the command line via:

node area_choropleth.js > area_choropleth.svg

which will direct the SVG text output from stdout into the file “area_choropleth.svg”. And we’re done!

All together now

Here’s the resulting SVG, rendered as a PNG:

area_choropleth

Here is the code in full below, which is also hosted with the us.json data from Mike Bostock as a gist on github:

//require modules
var d3 = require("d3");
var fs = require("fs");
var vm = require('vm');

//import topojson.js client side API
var includeInThisContext = function(path) {
 var code = fs.readFileSync(path);
 vm.runInThisContext(code, path);
}.bind(this);
includeInThisContext("topojson.js");

//SVG dimensions
var width = 960,
 height = 500;

//scale for county-population-based fill
var fill = d3.scale.log()
 .domain([10, 500])
 .range(["brown", "steelblue"]);

var path = d3.geo.path();

var svg = d3.select("body").append("svg")
 .attr('xmlns', 'http://www.w3.org/2000/svg')
 .attr("width", width)
 .attr("height", height);

//parse the US county population data JSON file
var us = JSON.parse(fs.readFileSync("us.json", 'utf8'));

//enter the counties
svg.append("g")
 .attr("class", "counties")
 .selectAll("path")
 .data(topojson.feature(us, us.objects.counties).features)
 .enter().append("path")
 .attr("d", path)
 .style("fill", function(d) { return fill(path.area(d)); });

//outline the states
svg.append("path")
 .datum(topojson.mesh(us, us.objects.states, function(a, b) { return a.id !== b.id; }))
 .attr("class", "states")
 .attr("d", path);

//add css stylesheet
var svg_style = svg.append("defs")
 .append('style')
 .attr('type','text/css');

//text of the CSS stylesheet below -- note the multi-line JS requires 
//escape characters "\" at the end of each line
var css_text = "

Summary

  • D3.js via node.js on the command line is certainly possible, although needs some extra massaging to ensure we have stylesheets embedded within the SVG, and that JSON/CSV/TSV data loads without issue.
  • CDATA blocks need to enclose the CSS stylesheet for proper rendering within Illustrator, and multiline CSS style sheets loaded as JS strings need end of lines to be escaped with “\”
  • For Topojson.js, the client side API source needs to be included separately via vm (or the Topojson.js methods need to be embeded directly in our JS code)
  • d3.json, d3.csv, d3.tsv etc… need to be converted to a local file reader via fs, since the former require HTTP requests to retrieve data

Questions, comments, feedback, have a better way?

Post a comment below, or message me on twitter @pykerl. Also thanks to Eric Dodge for some helpful suggestions.

Quick and easy lactose free dark chocolate hot cocoa

Just finished off a can of this great (albeit expensive) hot cocoa and decided to reverse engineer the recipe…

Turns out it’s just two ingredients: cocoa and powdered sugar (great for the lactose intolerants). Can’t really mess that up. And yeah alright, most of the world has hopefully realized that hot cocoa is pretty damn simple, but if not…

Dark Hot Cocoa recipe for the, ahem, majority of us lactose intolerant folk.

  • 30% (by weight) cocoa (Hershey’s cocoa or generic works fine) (Example: 90 grams)
  • 70% (by weight) powdered sugar (Example: 210 grams)
  • Mix thoroughly in a ziplock bag (Example yields 300 grams for those that cannot add two numbers).
  • Change proportions to adjust dark chocolate-ness.
  • Yeah, these ingredients cost next to nothing.

STEPS TO GLORY: Warm up an 8oz-12oz glass (13oz is just too much, c’mon) of milk (soy milk, almond milk, “regular” milk… others are untested). Warming can be done via microwave or via the stove if that’s your style. Mix in 4 Tbsps of the above mixture. Cocoa should sort of melt into the milk. Toughest recipe you’ve seen this year, I know.

Predictions of the seemingly mundane: the bathroom

These are my predictions for changes to the majority of household bathrooms in the US. Yep, I have not posted in over a year, and this indeed is what I’m starting off new posts with.

In my lifetime:

  • Bidets within homes in the US
  • Faucets and shower heads with user preferences
  • Automatic faucets in bathrooms (and kitchens)
  • In home hand air dryers
  • Daily body tracking tool usage is common place (not just for diabetics and fitness fanatics)

Beyond my lifetime:

  • Non-mint toothpaste taking hold of a significant % of the US market
  • Replaceable organic teeth grown exactly to specification at a cheap price (<$1000)
  • Water conservation plumbing for recycling non-potable in-home water

happy new year

Great to see everyone over the break… I brought this site up since I haven’t updated in forever. So as for this blog, I’m going to try to trend towards something new for a change… I’m going to aim to post more content (it’s plausible, right?) with useful things — for instance, trends I see, technology that’s promising, maybe even products I enjoy. We’ll see.

For now and a few links to places where I already post content, in approximate descending order of posting regularity:
me on google reader
me on twitter
me on hypem

and as a taste, here is a sweet (but not really short) documentary on statistics:

Change is on the way

I will be updating this site to accommodate a interesting “feed” (links essentially) items I’m reading which will run parallel to the posts on this blog. I’m pretty sure wordpress can handle this, but it will require a redesign. We’ll see how it turns out. It will be structured something like, “short blog (slog)”, “long blog (blog),” and “permanent blog (plog).”

In addition I will be making bacon chocolate chip cookies soon. Mmmmm…