Headless D3

When I talk about my interest in data-vis with other programmers, the most frequent follow-up questions concern my preferred framework or tooling. The answer, as always, is that it's complicated. Those of us who do data-vis in JavaScript usually say that "we use d3", but more and more often what this means is that we use parts of d3. The reason is that d3 wants to handle the DOM, but the programmatic model it has for doing so was developed before the current generation of view libraries.

It's become a bit fashionable in the last year or so to throw shade on React, and oftentimes I'm inclined to agree. But even when modern React specifically seems to be well on its way to becoming bloatware, the mental model it has created for frontend developers is still really, really useful. What React does well (and what pretty much everyone else has copied) is to map data into elements, and update those elements in the most efficient way possible when the data change. (React has been referred to derisively as a "templating library with benefits." I also agree with this, because the benefits are kind of a game-changer).

This is a really great fit for data-vis! Data points, represented somewhere as arrays or objects, are transformed into SVG elements. We can model this transformation/mapping as a pure-functional relationship, even it is isn't really. When the data change, so does the chart.

Yet often I come across attempts to "combine d3 with React" the code looks something like this:

const Chart = ({ data }) => {
  const ref = useRef();

  useEffect(() => {
    select(ref.current);
    // hundreds of lines of d3 select/chaining
  }, [data]);

  return <svg ref={ref}></svg>;
};

In other words, we've tossed out the useful part of React. We're back to imperitively building the DOM.

The good news is, d3 is not a really a framework, but a library. It's actually a more like a suite of libraries, and we can pick and choose the parts we want. We can make use of the "headless parts" to manipulate data in a pure-functional manner. We leave out the DOM-adjacent bits and let React (or Vue or Svelte or Solid or Preact) manage the view layer.

This post is inspired, to some degree, by previous writing from Elijah Meeks and Amelia Wattenberger. However, they are primarily writing for a d3 audience wanting to incorporate React. My intention here is break down the parts of d3 that are most useful to incorporate "data vis" into an existing view layer. It assumes no prior knowledge of d3.

Therefore, everything below can be applied to React, Vue, Svelte, Solid, Angular, Preact, or any other reactive, component-based UI library.

d3-dsv

It all starts with data. The most common at-rest data format for spreadsheet-like data is the CSV (comma separated values) format. d3-dsv is an extemely useful package that parses CSV (or tab-separated or other-delimiter-separated) strings into arrays of arrays or arrays of objects.

import { csvParse } from "d3-dsv";

console.log(csvParse(text));

Of course, in a real web application, we have the problem of getting the original file from the web or from disk. d3-dsv may be most useful on the server (if the server is based on JavaScript), to parse a file from disk and send it to the frontend as JSON.

More about ds-dsv.

d3-scale

d3-scale converts a domain (the starting scale, or the scale of the actual data) to a range.

The most common use I have for d3-scale is transform Cartesian data (how I think about it) to pixel coordinates (which are always defined with the origin at the top left). This provides a seemless interface for creating a Cartesian plane:

In the example above, the x scale is created with:

import { scaleLinear } from "d3-scale";

const scaleX = scaleLinear().domain([xMin, xMax]).range([0, clientWidth]);

And the y scale:

const scaleY = scaleLinear().domain([yMin, yMax]).range([clientHeight, 0]);

These return a function that transforms a coordinate from one system to another.

Scales can also be inverted, which is really useful for event handling. In the example above, try hovering over any part of the chart and you will see the real coordinates of your pointer.

scaleX.invert(coordinate);

Other scales

Conversion from one linear scale to another is essential for this work, but d3-scale can also handle a huge variety of "non-linear" scales. One common need in data-vis is plotting on a logarithmic scale. There's an app for that.

More about d3-scale.

d3-shape

Just because we aren't using d3 to change the DOM doesn't it doesn't have useful parts for drawing.

Lines

The most free-form drawing tool for SVG is the d attribute of the path element - it's the programmatic equivalent of the pencil tool, but with a syntax literally no one can read or write.

d3-line converts an array of x,y coordinates to the syntax of the d attribute. This means we can draw nearly any shape we like as a series of coordinates. We can also draw curved lines.

The use of d3-line looks like this:

import { line as d3Line } from 'd3-shape';

const line = d3Line().curve(curveFactory)(path);

return `<path d="${line}" />`

Just remember that the path needs to be re-scaled to pixel space first!

More about lines in d3.;

Symbols

While all shapes are fundamentally made out of lines, for complex symbols there is usually a simpler method. d3 provides a rich set of symbol "generators" to use for scatterplots.

More about symbols in d3.

d3-ticks

Determining where to show axis ticks is not a pure programming problem. Ideally, we want ticks that are nicely rounded and fit the range of our data, and this is challenging to determine if we don't know the content ahead of time. d3-ticks creates "nicely formatted" tick marks for a given range, even if you don't know the range ahead of time.

More about ticks in d3.

d3-ease

I prefer to address animation after almost everything else in a chart has been done. The best designed charts do not rely on animation, since a chart that is in motion cannot be properly read and understood by the reader in real-time.

Nevertheless, animation can be a great way of fluidly moving from one view to another, while allowing the eye to follow the identity of the data. d3 has several options for easing animation.

In the demo below, the circles move to a new random position every second.

The one I use most commonly is easeCubicInOut, which seems to provide the most "natural" experience.

Read more about d3-ease.

d3-delaunay

Scatterplots with tooltips often make use of Delaunay triangulation to find the nearest data point to the mouse pointer. A Delaunay triangulation can be calculated as follows:

import { Delaunay } from "d3-delaunay";

const delaunay = Delaunay.from(points);

And the nearest data point found by applying that:

delaunay.find(...pointerPosition);

The key thing to note is that the first step is the computationally expensive process (which is good, because the pointer will move often, while the Delaunay only needs to be calculated when the data updates). Depending on the frontend framework, you'll want to memoize or otherwise cache the function itself while providing its find function to the event handler for the element. Doing the calculation within requestIdleCallback can be useful to ensure it doesn't interfere with rendering itself.

Read more about d3-delaunay.

What about color, datetime, fetch?

d3 also has numerous functions for manipulating color, datetimes, fetching files, and other things that don't touch the DOM. But just as we have better options for managing UI nowadays than we had when d3 was first released, so we have more community options for those things. There are more directed packages for lots of these general needs, that may be a better fit for your project.

Conclusion

When it comes to data manipulation, d3 stands alone within the JavaScript ecosystem. Although rendering the DOM is easier now than it's ever been (and certainly easier than with d3 itself), the manipulation of data to get to what can be rendered is still a difficult problem. Knowing how to use d3 with a modern framework is an essential part of the data vis workflow.