![\"QR](\"/image/qrcode?__frsh_c=94250ce1da5bae1ce228dec270b68266b0cb2491\")
Recently, the Hard Fork and Search Engine podcasts teamed up to release their own social network. Well, not really a new social network, but rather an instance of Mastodon, which is just one of many software packages that run on the ActivityPub protocol. The collection of ActivityPub-enabled servers is commonly known as the "fediverse", and the new instance was called The Forkiverse.
This is just the latest in several rounds of migration to Mastodon, the previous having coincided with Elon Musk's takeover of that other microblogging site (I forget the name). But how big is the Forkiverse? Where does it fit into the larger picture?
Fortunately, since ActivityPub is by nature public, it's relatively to get this data. I was able to download a complete set of data on ActivityPub servers from a service called FediDB.
The fediverse, plotted on a log scale, has a peloton-like shape, with a few leaders followed by a large group. (the "default" server) is by far the largest, followed by just a few servers with over 1000 monthly active users. This category includes (near the front), and my own home server .
What I found really interesting in the data, though, were the huge number of servers with between 10 and 100 monthly active users. These servers must represent some kind of stable community: remember, they are monthly active users. Yet I wondered why they've stayed away from the big players: are they groups of people brought together by some specific purpose or pre-existing social ties?
Alas, most of these small servers don't have description fields, a property of ActivityPub that allows a server to state its intended purpose. But since a few do, I built a quick API to query descriptions of mastodon servers with between 10 and 100 monthly active users. You can see a random sample of those here:
I really hope that the fediverse has a bright future. It feels like (the best of) what the web used to be like: quirky, non-conforming communities of people who've found a home outside of the big platforms. I hope that more people with existing followings will create instances like The Forkiverse to bring people to the fediverse, but I also love that small servers of friends or colleagues are using ActivityPub to carve out their own online spaces.
","summary":"Awestruck by an ever-expanding fediverse","image":"fedi-scrn.png","date_published":"2026-02-10T00:00:00.000Z"},{"id":"divorce-rates-stats-can","url":"http://localhost:8000/blog/divorce-rates-stats-can","title":"divorce-rates-stats-can","content_text":"On Tuesday, April 30, the CBC published an article claiming that the rate of divorce in Canada is declining, and contrasting that hard data with the common misconception that half of marriages end in divorce. The article is a summary of a recent report published by the Vanier Institute on the Family , which is itself drawn from data published by Statistics Canada (references therein). The CBC article embeds the above chart , created with the ever-cromulent DataWrapper. The punchline of the chart is pretty clear: the divorce rate in Canada peaked in 1987 (the chart points out that the Divorce Act was amended in 1986). Since then, the divorce rate has been declining in lock step with the declining marriage rate. Both rates plummeted in 2020 for obvious reasons, which is the last year for which we have available data. While neither the CBC article nor the Vanier institute are factually incorrect, it isn't apparent to me that these data have anything whatsoever to do with divorce. If fewer people are married overall, then we would expect a lower rate of divorce. Even analyzing the rate of divorce as a percentage of Canadians who are married doesn't tell us all that much, because an increasingly large share of the population are older people who are less likely to divorce at all (see below). Fortunately, Statistics Canada tracks the yearly divorce rate broken up by the year in which the couple was first wed . We can use this to track a \"marriage survivorship curve\". This isn't a true survivorship curve, because is doesn't tell us how many of these marriages are still intact (the alternative way a marriage can end isn't represented), but rather the cumulative likelihood that the marriage resulted in divorce as a function of time. The following chart shows the divorce rate as a function of years of marriage for a couple married in 1971 (the earliest year for which data are available). The slider allows you to change to view couples married in later years, but we have less and less data for more recently married couples as we don't know the eventual fate of younger marriages. The basic shape of this curve shouldn't come as a surprise. Newlyweds rarely divorce, but a rapid acceleration is underway by the end of the first decade. This has begun to level off by the end of the second decade. For couples married in 1971, the curve has almost fully leveled out by the time the couple has been married for 30 years (partly because they've made it that far, partly because one member of the couple has died, making it impossible for the marriage to result in divorce at that point). As we examine couples married later than 1971, the curve pulls up and to the left. More marriages result in divorce, and a greater percentage of couples are divorced earlier in their marriages. As we get to younger couples, this trend may be leveling out or even reversing; however, it's difficult to know for sure given the relatively short timespan we have to work with: these young couples simply aren't yet through the years when divorce is most likely to occur. To think of this slightly differently: let's ask how long it is until some fraction of married couples are divorced. The \"half-life\" of divorce is technically infinite, because it is not true and has never been true that half of marriages end in divorce (the curve never reaches 500 out of 1000 marriages). However, the period right around 140 out of 1000 is quite dynamic. How long does it take for 14% of couples to get divorced? Note: click on the left-hand graph above to change the examined rate. Again, higher rates mean we have less data to work with! This shows approximately the same thing. A big drop the expected shelf-life of a marriage occurred throughout the 1970s. Also, the \"leveling-out\" point of all these curves is higher, so a greater percentage of couples overall are divorced. The drop-off was pretty much done by the time the 80s were underway: a couple married in 1981 has approximately the same \"divorce curve\" as a couple married in 2001. This trend may have rebound in recent years: for couples married around 2011, the curve seems a little wonky and may be accelerating more slowly than for couples married in the 80s, 90s, or noughts. Someone with different skills than me could probably use this to tell a story about the Great Recession or the cost of housing in Canada. It is worth saying again that 2020 is the last year for which we have available data: a couple married in 2011 was right in the divorce sweet spot when the pandemic hit. Time will tell whether the declining rate is a trend that will continue. Conclusion The CBC (and Vanier Institute) reports on the declining divorce rate are interesting, but are not about divorce. Couples are most likely to divorce in their first 20 years or so of marriage. They became somewhat more likely to do so the later in the 1970s they were married; a trend that has been stable ever since then. The fact that a Canadian in 2020 is relatively unlikely to divorce is primarily because that Canadian is: (1) less likely to be married in the first place, and (2) if they are married, likely to have been married for a relatively long time.","content_html":"On Tuesday, April 30, the CBC published an article claiming that the rate of divorce in Canada is declining, and contrasting that hard data with the common misconception that half of marriages end in divorce. The article is a summary of a recent report published by the Vanier Institute on the Family, which is itself drawn from data published by Statistics Canada (references therein).
The CBC article embeds the above chart, created with the ever-cromulent DataWrapper. The punchline of the chart is pretty clear: the divorce rate in Canada peaked in 1987 (the chart points out that the Divorce Act was amended in 1986). Since then, the divorce rate has been declining in lock step with the declining marriage rate. Both rates plummeted in 2020 for obvious reasons, which is the last year for which we have available data.
While neither the CBC article nor the Vanier institute are factually incorrect, it isn't apparent to me that these data have anything whatsoever to do with divorce. If fewer people are married overall, then we would expect a lower rate of divorce. Even analyzing the rate of divorce as a percentage of Canadians who are married doesn't tell us all that much, because an increasingly large share of the population are older people who are less likely to divorce at all (see below).
Fortunately, Statistics Canada tracks the yearly divorce rate broken up by the year in which the couple was first wed. We can use this to track a "marriage survivorship curve". This isn't a true survivorship curve, because is doesn't tell us how many of these marriages are still intact (the alternative way a marriage can end isn't represented), but rather the cumulative likelihood that the marriage resulted in divorce as a function of time.
The following chart shows the divorce rate as a function of years of marriage for a couple married in 1971 (the earliest year for which data are available). The slider allows you to change to view couples married in later years, but we have less and less data for more recently married couples as we don't know the eventual fate of younger marriages.
The basic shape of this curve shouldn't come as a surprise. Newlyweds rarely divorce, but a rapid acceleration is underway by the end of the first decade. This has begun to level off by the end of the second decade. For couples married in 1971, the curve has almost fully leveled out by the time the couple has been married for 30 years (partly because they've made it that far, partly because one member of the couple has died, making it impossible for the marriage to result in divorce at that point).
As we examine couples married later than 1971, the curve pulls up and to the left. More marriages result in divorce, and a greater percentage of couples are divorced earlier in their marriages. As we get to younger couples, this trend may be leveling out or even reversing; however, it's difficult to know for sure given the relatively short timespan we have to work with: these young couples simply aren't yet through the years when divorce is most likely to occur.
To think of this slightly differently: let's ask how long it is until some fraction of married couples are divorced. The "half-life" of divorce is technically infinite, because it is not true and has never been true that half of marriages end in divorce (the curve never reaches 500 out of 1000 marriages). However, the period right around 140 out of 1000 is quite dynamic. How long does it take for 14% of couples to get divorced?
Note: click on the left-hand graph above to change the examined rate. Again, higher rates mean we have less data to work with!
This shows approximately the same thing. A big drop the expected shelf-life of a marriage occurred throughout the 1970s. Also, the "leveling-out" point of all these curves is higher, so a greater percentage of couples overall are divorced. The drop-off was pretty much done by the time the 80s were underway: a couple married in 1981 has approximately the same "divorce curve" as a couple married in 2001.
This trend may have rebound in recent years: for couples married around 2011, the curve seems a little wonky and may be accelerating more slowly than for couples married in the 80s, 90s, or noughts. Someone with different skills than me could probably use this to tell a story about the Great Recession or the cost of housing in Canada. It is worth saying again that 2020 is the last year for which we have available data: a couple married in 2011 was right in the divorce sweet spot when the pandemic hit. Time will tell whether the declining rate is a trend that will continue.
The CBC (and Vanier Institute) reports on the declining divorce rate are interesting, but are not about divorce. Couples are most likely to divorce in their first 20 years or so of marriage. They became somewhat more likely to do so the later in the 1970s they were married; a trend that has been stable ever since then. The fact that a Canadian in 2020 is relatively unlikely to divorce is primarily because that Canadian is: (1) less likely to be married in the first place, and (2) if they are married, likely to have been married for a relatively long time.
","summary":"A commentary on Tuesday's CBC article","image":"graphic/divorce-rate.png","date_published":"2024-05-02T00:00:00.000Z"},{"id":"headless-d3","url":"http://localhost:8000/blog/headless-d3","title":"headless-d3","content_text":"When I talk about my interest in data-vis with other programmers, the most frequent follow-up questions concern my preferred framework or tooling. The answer, as always, is that it's complicated. Those of us who do data-vis in JavaScript usually say that \"we use d3\", but more and more often what this means is that we use parts of d3. The reason is that d3 wants to handle the DOM, but the programmatic model it has for doing so was developed before the current generation of view libraries. It's become a bit fashionable in the last year or so to throw shade on React, and oftentimes I'm inclined to agree. But even when modern React specifically seems to be well on its way to becoming bloatware, the mental model it has created for frontend developers is still really, really useful. What React does well (and what pretty much everyone else has copied) is to map data into elements, and update those elements in the most efficient way possible when the data change. (React has been referred to derisively as a \"templating library with benefits.\" I also agree with this, because the benefits are kind of a game-changer). This is a really great fit for data-vis! Data points, represented somewhere as arrays or objects, are transformed into SVG elements. We can model this transformation/mapping as a pure-functional relationship, even it is isn't really. When the data change, so does the chart. Yet often I come across attempts to \"combine d3 with React\" the code looks something like this: In other words, we've tossed out the useful part of React. We're back to imperitively building the DOM. The good news is, d3 is not a really a framework, but a library. It's actually a more like a suite of libraries, and we can pick and choose the parts we want. We can make use of the \"headless parts\" to manipulate data in a pure-functional manner. We leave out the DOM-adjacent bits and let React (or Vue or Svelte or Solid or Preact) manage the view layer. This post is inspired, to some degree, by previous writing from Elijah Meeks and Amelia Wattenberger . However, they are primarily writing for a d3 audience wanting to incorporate React. My intention here is break down the parts of d3 that are most useful to incorporate \"data vis\" into an existing view layer. It assumes no prior knowledge of d3. Therefore, everything below can be applied to React, Vue, Svelte, Solid, Angular, Preact, or any other reactive, component-based UI library. d3-dsv It all starts with data. The most common at-rest data format for spreadsheet-like data is the CSV (comma separated values) format. d3-dsv is an extemely useful package that parses CSV (or tab-separated or other-delimiter-separated) strings into arrays of arrays or arrays of objects. Of course, in a real web application, we have the problem of getting the original file from the web or from disk. d3-dsv may be most useful on the server (if the server is based on JavaScript), to parse a file from disk and send it to the frontend as JSON. More about ds-dsv . d3-scale d3-scale converts a domain (the starting scale, or the scale of the actual data) to a range . The most common use I have for d3-scale is transform Cartesian data (how I think about it) to pixel coordinates (which are always defined with the origin at the top left). This provides a seemless interface for creating a Cartesian plane: In the example above, the x scale is created with: And the y scale: These return a function that transforms a coordinate from one system to another. Scales can also be inverted, which is really useful for event handling. In the example above, try hovering over any part of the chart and you will see the real coordinates of your pointer. Other scales Conversion from one linear scale to another is essential for this work, but can also handle a huge variety of \"non-linear\" scales. One common need in data-vis is plotting on a logarithmic scale. There's an app for that. More about d3-scale . d3-shape Just because we aren't using d3 to change the DOM doesn't it doesn't have useful parts for drawing. Lines The most free-form drawing tool for SVG is the attribute of the element - it's the programmatic equivalent of the pencil tool, but with a syntax literally no one can read or write. d3-line converts an array of x,y coordinates to the syntax of the attribute. This means we can draw nearly any shape we like as a series of coordinates. We can also draw curved lines. The use of looks like this: Just remember that the needs to be re-scaled to pixel space first! More about lines in d3 .; Symbols While all shapes are fundamentally made out of lines, for complex symbols there is usually a simpler method. d3 provides a rich set of symbol \"generators\" to use for scatterplots. More about symbols in d3 . d3-ticks Determining where to show axis ticks is not a pure programming problem. Ideally, we want ticks that are nicely rounded and fit the range of our data, and this is challenging to determine if we don't know the content ahead of time. creates \"nicely formatted\" tick marks for a given range, even if you don't know the range ahead of time. More about ticks in d3 . d3-ease I prefer to address animation after almost everything else in a chart has been done. The best designed charts do not rely on animation, since a chart that is in motion cannot be properly read and understood by the reader in real-time. Nevertheless, animation can be a great way of fluidly moving from one view to another, while allowing the eye to follow the identity of the data. d3 has several options for easing animation. In the demo below, the circles move to a new random position every second. The one I use most commonly is , which seems to provide the most \"natural\" experience. Read more about d3-ease . d3-delaunay Scatterplots with tooltips often make use of Delaunay triangulation to find the nearest data point to the mouse pointer. A Delaunay triangulation can be calculated as follows: And the nearest data point found by applying that: The key thing to note is that the first step is the computationally expensive process (which is good, because the pointer will move often, while the Delaunay only needs to be calculated when the data updates). Depending on the frontend framework, you'll want to memoize or otherwise cache the function itself while providing its function to the event handler for the element. Doing the calculation within can be useful to ensure it doesn't interfere with rendering itself. Read more about d3-delaunay . What about color, datetime, fetch? d3 also has numerous functions for manipulating color, datetimes, fetching files, and other things that don't touch the DOM. But just as we have better options for managing UI nowadays than we had when d3 was first released, so we have more community options for those things. There are more directed packages for lots of these general needs, that may be a better fit for your project. Conclusion When it comes to data manipulation, d3 stands alone within the JavaScript ecosystem. Although rendering the DOM is easier now than it's ever been (and certainly easier than with d3 itself), the manipulation of data to get to what can be rendered is still a difficult problem. Knowing how to use d3 with a modern framework is an essential part of the data vis workflow.","content_html":"When I talk about my interest in data-vis with other programmers, the most frequent follow-up questions concern my preferred framework or tooling. The answer, as always, is that it's complicated. Those of us who do data-vis in JavaScript usually say that "we use d3", but more and more often what this means is that we use parts of d3. The reason is that d3 wants to handle the DOM, but the programmatic model it has for doing so was developed before the current generation of view libraries.
It's become a bit fashionable in the last year or so to throw shade on React, and oftentimes I'm inclined to agree. But even when modern React specifically seems to be well on its way to becoming bloatware, the mental model it has created for frontend developers is still really, really useful. What React does well (and what pretty much everyone else has copied) is to map data into elements, and update those elements in the most efficient way possible when the data change. (React has been referred to derisively as a "templating library with benefits." I also agree with this, because the benefits are kind of a game-changer).
This is a really great fit for data-vis! Data points, represented somewhere as arrays or objects, are transformed into SVG elements. We can model this transformation/mapping as a pure-functional relationship, even it is isn't really. When the data change, so does the chart.
Yet often I come across attempts to "combine d3 with React" the code looks something like this:
const Chart = ({ data }) => {\n const ref = useRef();\n\n useEffect(() => {\n select(ref.current);\n // hundreds of lines of d3 select/chaining\n }, [data]);\n\n return <svg ref={ref}></svg>;\n};In other words, we've tossed out the useful part of React. We're back to imperitively building the DOM.
The good news is, d3 is not a really a framework, but a library. It's actually a more like a suite of libraries, and we can pick and choose the parts we want. We can make use of the "headless parts" to manipulate data in a pure-functional manner. We leave out the DOM-adjacent bits and let React (or Vue or Svelte or Solid or Preact) manage the view layer.
This post is inspired, to some degree, by previous writing from Elijah Meeks and Amelia Wattenberger. However, they are primarily writing for a d3 audience wanting to incorporate React. My intention here is break down the parts of d3 that are most useful to incorporate "data vis" into an existing view layer. It assumes no prior knowledge of d3.
Therefore, everything below can be applied to React, Vue, Svelte, Solid, Angular, Preact, or any other reactive, component-based UI library.
It all starts with data. The most common at-rest data format for spreadsheet-like data is the CSV (comma separated values) format. d3-dsv is an extemely useful package that parses CSV (or tab-separated or other-delimiter-separated) strings into arrays of arrays or arrays of objects.
import { csvParse } from \"d3-dsv\";\n\nconsole.log(csvParse(text));Of course, in a real web application, we have the problem of getting the original file from the web or from disk. d3-dsv may be most useful on the server (if the server is based on JavaScript), to parse a file from disk and send it to the frontend as JSON.
More about ds-dsv.
d3-scale converts a domain (the starting scale, or the scale of the actual data) to a range.
The most common use I have for d3-scale is transform Cartesian data (how I think about it) to pixel coordinates (which are always defined with the origin at the top left). This provides a seemless interface for creating a Cartesian plane:
In the example above, the x scale is created with:
import { scaleLinear } from \"d3-scale\";\n\nconst scaleX = scaleLinear().domain([xMin, xMax]).range([0, clientWidth]);And the y scale:
const scaleY = scaleLinear().domain([yMin, yMax]).range([clientHeight, 0]);These return a function that transforms a coordinate from one system to another.
Scales can also be inverted, which is really useful for event handling. In the example above, try hovering over any part of the chart and you will see the real coordinates of your pointer.
scaleX.invert(coordinate);Conversion from one linear scale to another is essential for this work, but d3-scale can also handle a huge variety of "non-linear" scales. One common need in data-vis is plotting on a logarithmic scale. There's an app for that.
More about d3-scale.
Just because we aren't using d3 to change the DOM doesn't it doesn't have useful parts for drawing.
The most free-form drawing tool for SVG is the d attribute of the path element - it's the programmatic equivalent of the pencil tool, but with a syntax literally no one can read or write.
d3-line converts an array of x,y coordinates to the syntax of the d attribute. This means we can draw nearly any shape we like as a series of coordinates. We can also draw curved lines.
The use of d3-line looks like this:
import { line as d3Line } from 'd3-shape';\n\nconst line = d3Line().curve(curveFactory)(path);\n\nreturn `<path d=\"${line}\" />`Just remember that the path needs to be re-scaled to pixel space first!
More about lines in d3.;
While all shapes are fundamentally made out of lines, for complex symbols there is usually a simpler method. d3 provides a rich set of symbol "generators" to use for scatterplots.
More about symbols in d3.
Determining where to show axis ticks is not a pure programming problem. Ideally, we want ticks that are nicely rounded and fit the range of our data, and this is challenging to determine if we don't know the content ahead of time. d3-ticks creates "nicely formatted" tick marks for a given range, even if you don't know the range ahead of time.
More about ticks in d3.
I prefer to address animation after almost everything else in a chart has been done. The best designed charts do not rely on animation, since a chart that is in motion cannot be properly read and understood by the reader in real-time.
Nevertheless, animation can be a great way of fluidly moving from one view to another, while allowing the eye to follow the identity of the data. d3 has several options for easing animation.
In the demo below, the circles move to a new random position every second.
The one I use most commonly is easeCubicInOut, which seems to provide the most "natural" experience.
Read more about d3-ease.
Scatterplots with tooltips often make use of Delaunay triangulation to find the nearest data point to the mouse pointer. A Delaunay triangulation can be calculated as follows:
import { Delaunay } from \"d3-delaunay\";\n\nconst delaunay = Delaunay.from(points);And the nearest data point found by applying that:
delaunay.find(...pointerPosition);The key thing to note is that the first step is the computationally expensive process (which is good, because the pointer will move often, while the Delaunay only needs to be calculated when the data updates). Depending on the frontend framework, you'll want to memoize or otherwise cache the function itself while providing its find function to the event handler for the element. Doing the calculation within requestIdleCallback can be useful to ensure it doesn't interfere with rendering itself.
Read more about d3-delaunay.
d3 also has numerous functions for manipulating color, datetimes, fetching files, and other things that don't touch the DOM. But just as we have better options for managing UI nowadays than we had when d3 was first released, so we have more community options for those things. There are more directed packages for lots of these general needs, that may be a better fit for your project.
When it comes to data manipulation, d3 stands alone within the JavaScript ecosystem. Although rendering the DOM is easier now than it's ever been (and certainly easier than with d3 itself), the manipulation of data to get to what can be rendered is still a difficult problem. Knowing how to use d3 with a modern framework is an essential part of the data vis workflow.
","summary":"Using D3 with React, Preact, Vue, Svelte, Angular, Solid","image":"charts/scatterplot-graphpaper","date_published":"2024-03-06T00:00:00.000Z"},{"id":"improving-web-scatterplots","url":"http://localhost:8000/blog/improving-web-scatterplots","title":"improving-web-scatterplots","content_text":"Of all my regular stops for data-vis design, Our World in Data is probably my favourite. Unlike (for example) FiveThirtyEight , which keeps complex graphics and news articles on different parts of the site, and unlike news sites which embed data graphics as iframes, OWID has a unique approach to mixing graphics and prose. This is much how I once imagined scholarly articles working , and was to some extent my inspiration for how to build this website . In October, OWID announced a rebuild of Grapher , the engine behind their embedded graphics. This introduces several great features, including access to the raw data and easy-to-find information about data sources. This matters. As someone with a deep interest in data graphic design for the web, I decided to dig into this. There is particular type of graph I'm interested in, because it is ubiquitous on OWID and on the web at large. It is a kind of scatterplot where the sizes of the data points convey information. Often, the data points represent countries, where area is proportional to the country's population. This is useful in that bigger countries should play a bigger role in our understanding of overall world trends. This type of chart is a real challenge for the web medium, because large circles quickly overlap and obscure each other on small screens. In light of the OWID redesign, I wanted to see how they cracked that nut, and if their solution can be improved. My intention here is not to troll anyone . I'm well aware that OWID is operating with constraints - technical and otherwise - that I don't have. And as I've written before , I think that good UX for data graphics is a very hard problem. But I do think that data-vis should be a critical area for web developers and UX professionals to improve on, because a quantitative understanding of the world we live in will help societies make better decisions. Graphical representation of the Gini coefficient In this article , OWID breaks down the Gini coefficient, a measure of income inequality within a population. (I won't go through how it is calculated here, you can read the original article.) The second chart on the page plots how income tax changes inequality: by comparing the position of a country on the x-axis to the y-axis, we can see how much of an impact income re-distribution via taxes levels the playing field. Here is the original chart, albiet embedded in an iframe. A more \"accurate\" representation, including responsive features, may be obtained by following the links above. The chart is impressive in terms of how much information is conveyed. Let's break that down: Each data point on it's own conveys four pieces of essential information: the Gini coefficient before taxes (x-axis), (2) the Gini coefficient after taxes (y-axis), the population (size), and the continent/region (colour). Many of the points are labeled; for those that aren't, the name (and other information) is accessible in a tooltip that appears via hover. Historical information is available, too, accessible via the bottom range slider. Countries can be highlihted/selected in various ways - clicking/tapping on a country, selecting an entire region via the legend, or using the \"Select countries and regions\" button. We can go back in time, or we can view historical changes via a temporal trace (this view can be accessed by clicking where it says \"1999\" and then changing the range handles). All in all, a lot of useful information is communicated by this chart. Can we improve on it? Well, for one thing, the chart is much less usable on mobile/touch devices that it is on desktop browsers. Most obviously, the data points and labels overlap with each other too much to be visible. The legend remains next to the chart, taking up even more space. To get around this, I tried rotating my phone to landscape mode. This caused the y-axis to collapse completely. This wasn't a \"runtime\" bug of changing the window size; it happened even when I booted the page in landscape mode. ( This is a legit bug and is logged here .) Another issue was apparent when tapping on the legend, which seems to mess with the selection state in a way I can't make sense of. (To illustrate this, try tapping on the same continent twice, and then tap on a different continent. I can't tell what state I'm in at that point, but it's clearly unexpected.) I think what's going on is that the legend elements respond differently to both click and hover, and get confused which is which when responding to a tap. ( This is another bug, logged here. ) There are a few other UX annoyances and inconsistencies which are apparent on all devices. The behaviour of the year selection slider is mysterious. It appears to be possible to convert the chart to a comet-tail view, where changes in the Gini coefficient are tracked in two dimensions over time. Unexpectedly, clicking the \"1999\" button on the slider converts to this view, while placing the slider handles on top of one another converts back. As a user, it isn't obvious to expect this behaviour. Improving the crowded scatterplot This type of chart is clearly a challenge to design well for the web. When we don't know how much horizontal space is available, how can we make effective use of the \"size\" of each circle, and how can we make it possible for users to get the most out of it, regardless of what device they are using to read? I think the particular problem of this kind of chart is that the exact identity of each data point matters quite a bit . We can't count on there being enough space to label all of them, yet we want to make it easy for the user to find what they're looking for. The most likely thing a user will want to know is how their own country compares to the big players (I'm Canadian, and share my nation's tendency to constantly compare ourselves to our better-known neighbour.) A user also might want to know what's going on with some of the obvious outliers (Like, WTF South Africa?) Here is my reimagining of the OWID chart. In this case, my goal is not a verbatim reproduction, but an attempt to demonstrate how the usability can be improved to benefit more users. Zeroing in on a data point Finding \"Canada\" on the OWID chart required at least 3 clicks and probably a few keystrokes. This can be significantly improved with a live search. There are few enough countries represented in the chart that the implementation can be done entirely on the frontend, through string matching. The component is just an input box with a datalist: To help users find a particular country, we can highlight text matches as they type: But when they choose an exact match (probably by selecting from the dropdown list) the tooltip pops up as well: The user can also tap on any data point to bring up the tooltip, which contains name of the country and other information. To improve usability on touch devices, I chose to make this happen only on an actual tap or click, and ignore hover events. Selecting groups Following which countries/regions are selected in the OWID chart is complicated. There seem to be at least three different states a data point can be in, and the interplay between them is not obvious. Additionally, there seems to be a bug where taps are interpreted as hover events on touch devices, meaning that it's easy to get into a weird state. Tap around on the legend and individual data points for a bit and it quickly becomes hard to follow where you are and how to get back to where you started. This set off my mutable-state spidey senses and I decided there needed to be a central store to track which countries are highlighted and which aren't. The \"active countries\" list behaves like a finite state machine: we start with no active countries when we search, any countries matching the search query are activated. All others are deactivated. given that ALL countries in a continent are inactive, clicking on the continent in the legend activates all of them given that ANY country in a continent is active, click on the continent in the legend deactivates all of them This state then maps to the user interface according to the following: if there are zero active countries, all of the data is shown in full colour. The 10 biggest countries are also labelled, if the chart is at least 768px wide. Otherwise, no labels are shown. if ANY country is active, that country is shown in full colour with its label. Otherwise, it is ghosted (shown in grey). if there are zero active countries, all items in the legend are shown in full colour. if ANY country outside of the continent is active, but no countries within that continent are active, the legend item is ghosted. if ALL countries in a continent are active, the legend item is shown in full colour. if SOME countries in a continent are active, the legend is \"partially ghosted\". Finally, the easiest way to ensure users don't get lost in weird places is to give them the option to reset the chart to its original state at any point. Clicking Reset puts everything back to the way it was at page load. Improving the experience on small screens We've already discussed some aspects of responsive design: removing (all) data labels on small screens so that they don't obscure the view when there's too little space available. A further improvement we can make is to scale the size of the circles so that they don't take up overlap with each other too much. In my previous post on responsive chart design , I discussed how to measure the chart size in pixels so that we can scale things to the space available. In this case, let's scale each data point like so: where is the population (in billions), is the chart width in pixels, and is the radius of the circle. There are a couple of things to note about this expression. Firstly, we use Math.max to ensure each circle has a minimum radius of 2 pixels, otherwise it might not be rendered at all. Secondly, we are scaling the radius based on the square root of population. This means that the area of the circle is directly proportional to the population. Since people tend to perceive area as \"size\" this is a more appropriate relationship than directly relating radius to population. However, the biggest improvement we can make in this area concerns how the chart stacks with its controls, not the chart area itself. The OWID chart legend is rendered as part of the SVG, making control of stacking behaviour hard or impossible. I just moved the legend into HTML-land, and used a combination of flexbox and CSS columns to change the layout when the screen becomes too small. The best way to see how this chart would look on a phone, is to look at it on a phone. Here's a QR code for this page: Introducing the dimension of time One detail of the data we've glossed over so far is that data isn't available for all countries in a single year. OWID is showing data for the \"closest\" available data point; I simply decided to show the latest available data. Nevertheless, how inequality changes over time is likely to be something users are interested in. (One surprise I had while doing this analysis was how little change there is over time, at least according to the Gini coefficient. At least, the differences between countries seem to be much greater than the change within a country over time. I'm wondering if the Gini coefficient is a reasonable way to measure the type of inequality that gets frequently discussed in the news, and plan to address that in a future post.) Let's discuss some features that will help users see a temporal view of this data. Temporal charts within the tooltip The most obvious way to show change over time is a simple line plot showing time on x-axis and the Gini coefficient on the y-axis. This has the additional advantage that it clearly shows the years in which data was collected for a given country (the plot points). I simply embedded a small line chart within the tooltips, so they can be called up for a given country when one wants more information. The one downside to presenting the data this way is that comparisons cannot be made between countries - however the main chart shows that pretty clearly. The tooltips therefore continue to act as a way to call up more information about each data point - and even a small chart can convey a lot of data visually. Improving the year selection UX OWID's year selector slider is hard to use. It converts between two different modes: one where\na single year is selected and a different mode where a range of years can be selected. The only way to access the range mode is to click on either the \"1999\" or \"2021\" button, after which a second slider handle appears. One can then convert back to the single-year mode by setting the slider handles to the same point, which \"collapses\" them back to a single handle. None of these actions have the consequence users expect. Moreover, the elements in the slider design are all s, meaning screenreaders won't make sense of them, nor can they be controlled by the keyboard. Given all of this, I decided to throw out the slider entirely and use explicitly labelled form elements. The data cover a range of only 22 years: enough for a dropdown to be an acceptable solution. The user can swap modes with a radio switch, making it entirely apparent what they are trying to look at. Conclusion Making data graphics work well across a range of device capabilities is challenging even for simple plots, and scatterplots with this amount of information in them are not simple. There really is a unifying principle behind UX design and chart design: empathy for the user , followed by consideration of all the myriad ways a user might interact with your work. While a great experience across every device might not be possible if we bind ourselves to the chart itself, we can also externalize the legend, controls, and tooltips, giving us all the usual responsive design tricks, while picking sematically correct HTML elements.","content_html":"Of all my regular stops for data-vis design, Our World in Data is probably my favourite. Unlike (for example) FiveThirtyEight, which keeps complex graphics and news articles on different parts of the site, and unlike news sites which embed data graphics as iframes, OWID has a unique approach to mixing graphics and prose. This is much how I once imagined scholarly articles working, and was to some extent my inspiration for how to build this website.
In October, OWID announced a rebuild of Grapher, the engine behind their embedded graphics. This introduces several great features, including access to the raw data and easy-to-find information about data sources. This matters. As someone with a deep interest in data graphic design for the web, I decided to dig into this.
There is particular type of graph I'm interested in, because it is ubiquitous on OWID and on the web at large. It is a kind of scatterplot where the sizes of the data points convey information. Often, the data points represent countries, where area is proportional to the country's population. This is useful in that bigger countries should play a bigger role in our understanding of overall world trends.
This type of chart is a real challenge for the web medium, because large circles quickly overlap and obscure each other on small screens. In light of the OWID redesign, I wanted to see how they cracked that nut, and if their solution can be improved. My intention here is not to troll anyone. I'm well aware that OWID is operating with constraints - technical and otherwise - that I don't have. And as I've written before, I think that good UX for data graphics is a very hard problem. But I do think that data-vis should be a critical area for web developers and UX professionals to improve on, because a quantitative understanding of the world we live in will help societies make better decisions.
In this article, OWID breaks down the Gini coefficient, a measure of income inequality within a population. (I won't go through how it is calculated here, you can read the original article.) The second chart on the page plots how income tax changes inequality: by comparing the position of a country on the x-axis to the y-axis, we can see how much of an impact income re-distribution via taxes levels the playing field.
Here is the original chart, albiet embedded in an iframe. A more "accurate" representation, including responsive features, may be obtained by following the links above.
The chart is impressive in terms of how much information is conveyed. Let's break that down:
Each data point on it's own conveys four pieces of essential information: the Gini coefficient before taxes (x-axis), (2) the Gini coefficient after taxes (y-axis), the population (size), and the continent/region (colour).
Many of the points are labeled; for those that aren't, the name (and other information) is accessible in a tooltip that appears via hover.
Historical information is available, too, accessible via the bottom range slider.
Countries can be highlihted/selected in various ways - clicking/tapping on a country, selecting an entire region via the legend, or using the "Select countries and regions" button. We can go back in time, or we can view historical changes via a temporal trace (this view can be accessed by clicking where it says "1999" and then changing the range handles).
All in all, a lot of useful information is communicated by this chart. Can we improve on it?
Well, for one thing, the chart is much less usable on mobile/touch devices that it is on desktop browsers. Most obviously, the data points and labels overlap with each other too much to be visible. The legend remains next to the chart, taking up even more space. To get around this, I tried rotating my phone to landscape mode. This caused the y-axis to collapse completely. This wasn't a "runtime" bug of changing the window size; it happened even when I booted the page in landscape mode. (This is a legit bug and is logged here.)
Another issue was apparent when tapping on the legend, which seems to mess with the selection state in a way I can't make sense of. (To illustrate this, try tapping on the same continent twice, and then tap on a different continent. I can't tell what state I'm in at that point, but it's clearly unexpected.) I think what's going on is that the legend elements respond differently to both click and hover, and get confused which is which when responding to a tap. (This is another bug, logged here.)
There are a few other UX annoyances and inconsistencies which are apparent on all devices. The behaviour of the year selection slider is mysterious. It appears to be possible to convert the chart to a comet-tail view, where changes in the Gini coefficient are tracked in two dimensions over time. Unexpectedly, clicking the "1999" button on the slider converts to this view, while placing the slider handles on top of one another converts back. As a user, it isn't obvious to expect this behaviour.
This type of chart is clearly a challenge to design well for the web. When we don't know how much horizontal space is available, how can we make effective use of the "size" of each circle, and how can we make it possible for users to get the most out of it, regardless of what device they are using to read?
I think the particular problem of this kind of chart is that the exact identity of each data point matters quite a bit. We can't count on there being enough space to label all of them, yet we want to make it easy for the user to find what they're looking for. The most likely thing a user will want to know is how their own country compares to the big players (I'm Canadian, and share my nation's tendency to constantly compare ourselves to our better-known neighbour.) A user also might want to know what's going on with some of the obvious outliers (Like, WTF South Africa?)
Here is my reimagining of the OWID chart. In this case, my goal is not a verbatim reproduction, but an attempt to demonstrate how the usability can be improved to benefit more users.
Finding "Canada" on the OWID chart required at least 3 clicks and probably a few keystrokes. This can be significantly improved with a live search. There are few enough countries represented in the chart that the implementation can be done entirely on the frontend, through string matching. The component is just an input box with a datalist:
<div>\n <label for=\"search-input\">Search</label>\n <input\n type=\"text\"\n value=\"\"\n id=\"search-input\"\n list=\"search-datalist\"\n />\n <datalist id=\"search-datalist\">\n <option key=\"IND\" value=\"India\" />\n <!-- etc. -->\n </datalist>\n</div>To help users find a particular country, we can highlight text matches as they type:
const matches = data.filter((point) =>\n point.country.toLowerCase().includes(searchString.toLowerCase())\n);But when they choose an exact match (probably by selecting from the dropdown list) the tooltip pops up as well:
const exactMatch = data.find((point) => point.country === searchString);\nif (exactMatch) {\n // show tooltip\n}The user can also tap on any data point to bring up the tooltip, which contains name of the country and other information. To improve usability on touch devices, I chose to make this happen only on an actual tap or click, and ignore hover events.
Following which countries/regions are selected in the OWID chart is complicated. There seem to be at least three different states a data point can be in, and the interplay between them is not obvious. Additionally, there seems to be a bug where taps are interpreted as hover events on touch devices, meaning that it's easy to get into a weird state. Tap around on the legend and individual data points for a bit and it quickly becomes hard to follow where you are and how to get back to where you started.
This set off my mutable-state spidey senses and I decided there needed to be a central store to track which countries are highlighted and which aren't. The "active countries" list behaves like a finite state machine:
we start with no active countries
when we search, any countries matching the search query are activated. All others are deactivated.
given that ALL countries in a continent are inactive, clicking on the continent in the legend activates all of them
given that ANY country in a continent is active, click on the continent in the legend deactivates all of them
This state then maps to the user interface according to the following:
if there are zero active countries, all of the data is shown in full colour. The 10 biggest countries are also labelled, if the chart is at least 768px wide. Otherwise, no labels are shown.
if ANY country is active, that country is shown in full colour with its label. Otherwise, it is ghosted (shown in grey).
if there are zero active countries, all items in the legend are shown in full colour.
if ANY country outside of the continent is active, but no countries within that continent are active, the legend item is ghosted.
if ALL countries in a continent are active, the legend item is shown in full colour.
if SOME countries in a continent are active, the legend is "partially ghosted".
Finally, the easiest way to ensure users don't get lost in weird places is to give them the option to reset the chart to its original state at any point. Clicking Reset puts everything back to the way it was at page load.
We've already discussed some aspects of responsive design: removing (all) data labels on small screens so that they don't obscure the view when there's too little space available. A further improvement we can make is to scale the size of the circles so that they don't take up overlap with each other too much.
In my previous post on responsive chart design, I discussed how to measure the chart size in pixels so that we can scale things to the space available. In this case, let's scale each data point like so:
const r = Math.max(\n Math.sqrt(width * pop),\n 2,\n);where pop is the population (in billions), width is the chart width in pixels, and r is the radius of the circle. There are a couple of things to note about this expression. Firstly, we use Math.max to ensure each circle has a minimum radius of 2 pixels, otherwise it might not be rendered at all. Secondly, we are scaling the radius based on the square root of population. This means that the area of the circle is directly proportional to the population. Since people tend to perceive area as "size" this is a more appropriate relationship than directly relating radius to population.
However, the biggest improvement we can make in this area concerns how the chart stacks with its controls, not the chart area itself. The OWID chart legend is rendered as part of the SVG, making control of stacking behaviour hard or impossible. I just moved the legend into HTML-land, and used a combination of flexbox and CSS columns to change the layout when the screen becomes too small.
The best way to see how this chart would look on a phone, is to look at it on a phone. Here's a QR code for this page:
One detail of the data we've glossed over so far is that data isn't available for all countries in a single year. OWID is showing data for the "closest" available data point; I simply decided to show the latest available data. Nevertheless, how inequality changes over time is likely to be something users are interested in.
(One surprise I had while doing this analysis was how little change there is over time, at least according to the Gini coefficient. At least, the differences between countries seem to be much greater than the change within a country over time. I'm wondering if the Gini coefficient is a reasonable way to measure the type of inequality that gets frequently discussed in the news, and plan to address that in a future post.)
Let's discuss some features that will help users see a temporal view of this data.
The most obvious way to show change over time is a simple line plot showing time on x-axis and the Gini coefficient on the y-axis. This has the additional advantage that it clearly shows the years in which data was collected for a given country (the plot points). I simply embedded a small line chart within the tooltips, so they can be called up for a given country when one wants more information. The one downside to presenting the data this way is that comparisons cannot be made between countries - however the main chart shows that pretty clearly. The tooltips therefore continue to act as a way to call up more information about each data point - and even a small chart can convey a lot of data visually.
OWID's year selector slider is hard to use. It converts between two different modes: one where\na single year is selected and a different mode where a range of years can be selected. The only way to access the range mode is to click on either the "1999" or "2021" button, after which a second slider handle appears. One can then convert back to the single-year mode by setting the slider handles to the same point, which "collapses" them back to a single handle.
None of these actions have the consequence users expect. Moreover, the elements in the slider design are all divs, meaning screenreaders won't make sense of them, nor can they be controlled by the keyboard.
Given all of this, I decided to throw out the slider entirely and use explicitly labelled form elements. The data cover a range of only 22 years: enough for a dropdown to be an acceptable solution. The user can swap modes with a radio switch, making it entirely apparent what they are trying to look at.
Making data graphics work well across a range of device capabilities is challenging even for simple plots, and scatterplots with this amount of information in them are not simple. There really is a unifying principle behind UX design and chart design: empathy for the user, followed by consideration of all the myriad ways a user might interact with your work. While a great experience across every device might not be possible if we bind ourselves to the chart itself, we can also externalize the legend, controls, and tooltips, giving us all the usual responsive design tricks, while picking sematically correct HTML elements.
","summary":"Responsive design of the crowded Cartesian planes","image":"charts/gini-coefficient-scatterplot","date_published":"2024-01-30T00:00:00.000Z"},{"id":"responsive-charts","url":"http://localhost:8000/blog/responsive-charts","title":"responsive-charts","content_text":"A very common topic of questions on data visualization forums concerns responsive charts. In this post I will use an example of responsive chart design to illustrate the key issues when confronting this problem, outline the framework of a general solution, and show a bit of general-purpose code. The principles should be applicable to any frontend framework (React, Vue, Svelte). I'll use d3 throughout, but no prior d3 expertise is necessary. I will also use TypeScript, because it more clearly illustrates what data we are working with, but JavaScript should work just as well. Finally, although this particular chart is rendered in HTML5 canvas, the same principles would be applicable to a chart built with SVG. Basic familiarity with JavaScript, how to draw with canvas , and DOM event handlers are needed. What we'll be building The inspiration for this post came from this question on Observable Talk . A user named hiroyukikumazawa asked how to build a responsive chart with a mouse-hover tooltip and linked to this example . The chart in question shows Bitcoin price tracked over the last ~24hrs and has several features that are interesting from a web developer's perspective: There is both a line chart and box chart view, which can be toggled via the \"icon buttons\" at the top. The data resizes to fit its container. (Caveat: there's a narrow set of viewport sizes just above 1000px where it breaks out of the container in Firefox. It worked smoothly at all sizes in Chrome. I'm not sure what the difference is.) In the box chart view, the width of each box scales so that it fits the space available. The axis tick labels remain legible at all sizes. Some of the labels on the x-axis are hidden at smaller screen sizes so they don't obscure each other. The x-axis labels are based on local time, not server time or GMT. The data point for midnight shows the day of the month instead of the hour, and is bold . On mouse hover a tooltip appears corresponding to the nearest data point on the x-axis. A vertical line indicates which datapoint is being shown. (Caveat: this doesn't work on devices without a mouse, even if you poke at the chart with your finger). Here is my attempt at reverse-engineering what they did: It doesn't do everything, but it does illustrate the responsive features I'm going to talk about here (and adds/fixes a few more). I wrote it in Deno/Fresh/Preact (the stack for this website), but the \"pure functions\", and the d3 bits I will show below can be applied to any framework. Their chart is based on canvas, so I'll stick to that, as well. The full solution is available here . What do we mean by \"responsive\", anyway? Charts present a particular challenge for responsive design because they do not fit neatly into the categories of other element types that we are used to dealing with. Simple text can reflow to fit its container. Raster images can resize to maintain a consistent aspect ratio (though even so, HTML5 gives us mechanisms to load different images for different devices). Charts are not just images: maintaining a constant aspect ratio almost never works well outside of the context in which it was designed, while text quickly becomes illegible as it shrinks down to mobile sizes. (This is most easily observed for charts that are displayed as raster images, but SVGs don't improve the situation much, without extensive cajoling). How can we scale the bits we need to? The answer is to add a layer of indirection between the \"data scale\" and the actual pixels that we draw on the canvas. In the example, the heights (and vertical positions) of each box are given as properties of the data themselves, and must be scaled prior to drawing. Other dimensions (such as the width of each box), are detemined after scaling, and are drawn without reference to the data. I will refer to these two different layers as data space and pixel space throughout. Obtaining and processing the raw data If I had more time, I would figure out how to use the Coin Market Cap API to show up-to-date Bitcoin prices. However, large parts of their API don't seem to work as advertised (at least on the free tier). So, the chart data show the data for the date I downloaded the data: Dec 13, 2023. The data can be downloaded as a CSV via the button to the top-right of the chart. The CSV data are in one hour intervals, instead of 15 min intervals as in the original chart. Therefore, my chart has sparser data points than the original. Furthermore, although the file has a extension, the data are actually \"semicolon-separated\", not comma separated. They can be imported into JavaScript with d3's handy package: The data are objects consisting of a timestamp and a value for , , , and prices for each hour. (There's some other values in there, too, but we'll ignore those for this charting project): D3 also has several modules for working with and plotting datetimes, but for reasons which should become clear below, I find it easier to just convert these to simple numbers: aka UTC timecodes. It is useful to define a function that extracts a numeric x value from a data row: Getting the space available Usually, responsive design depends on media queries that apply different styles depending on the width of the overall viewport. In our case, it is more useful to respond to the width of the chart itself. (In principle, this is related to the CSS container query, but as our charts are based on TypeScript we'll get the container width and height there): where is the canvas (chart) element. Data space and pixel space Having determined our pixel space, we'll now turn our attention to the \"data space\" that make up the Cartesian plane upon which we usually think about graphs. We need to know the minimum and maximum values we'll have to worry about for both the x- and y-axis. Since the data starts out sorted by timestamp (x-axis) values, the minimum and maximum are just defined by the first and last values: The y-axis is more complicated. To simplify things, we'll use the same scale for both the line and box plot, which means the lowest possible y-value in the series is given by: And the highest value by: We'll give that some breathing room by \"zooming out\" by 20%: We should also make sure the chart has sufficient \"padding\" to ensure the axes have enough space regardless of the overall width. Let's give the following padding values in pixels: Now we can define a function for interconverting between data space and pixel space. D3 provides a facade which creates a function to return any value on a given \"domain\" (in our case, data space) to a given \"range\" (in our case, pixel space). Our overall scaling function for the x-axis looks like this: The y-axis is similar, but also reverses the up/down direction. We do this by putting the desired range in backwards: These two functions together can be used to take any data point and convert it to \"pixel space\" for plotting. Plotting the actual line in our line chart, for example, depends on converting them first and then using to do the actual drawing: where is the canvas context. There are some more details here, like how to draw a line that changes color depending on its value. However, I want to stay focused on how to implement responsive features, so I will skip over that. Responsive box width The width of each data box scales so that the boxes fill the available space at any given width, without overlapping each other (in fact, there are gaps in between them). Here we see how to use data space and pixel space together. The top and bottom of each box are determined by the data: However , the box width can be determined by the actual width of the chart in pixel space: Drawing the boxes is given by combining both of these concepts of \"space\": This path can be passed to to draw boxes. Placing ticks on the y-axis Our y-axis is not responsive. We have 300px of height to deal with, regardless of the width. However, we don't (in theory) know the range of values that will be charted and therefore can't determine a priori what ticks to draw. D3 makes this quite easy, though the relevant function is hidden within the sub-package. does the following according to the documentation : \"Returns an array of approximately count + 1 uniformly-spaced, nicely-rounded values between start and stop (inclusive)\". I love the words \"approximate\" and \"nicely\" in this description. This isn't a situation where we are looking for total predictability: we need a bunch of ticks that fit within the range of our axis, and they should be round numbers, within whatever range we are talking about. The first two arguments to this function are the minimum and maximum values of the axis, the third is the approximate number of ticks we want. The original chart had about ten: What about the tick label? Let's define a function that takes a y value as argument and returns the label: Responsive x-axis ticks The x-axis shrinks as the graph container shrinks, and it has a couple nice features that makes it legible on small screens. To keep the labels concise, they do not show the full date and time. Only the time is shown, except at midnight, when the day of the month is shown in bold. As the width decreases, some of the ticks disappear, preventing the text from overlapping and obscuring one another. Combining these two features is a challenge: we need to decide which labels to hide, but the midnight/new day timepoint should always be shown (or else the user will lose track of which day it is). It's also important to keep in mind that, so far, we have converted ISO datetimes (which are all in GMT), to UTC numbers for plotting. The labels should represent local time, including showing the new day of the month at midnight, local. Let's \"thin out\" our tick marks, by defining a parameters that takes every 2nd, 3rd, or n th tick mark, counting midnight as 0: Calling will create a function that returns only even hours: midnight, 2:00 am, 4:00 am, 6:00 am, etc.. We have to play with the chart a little bit to see what value is appropriate at each chart width. I came up with: Although we've used the datetime constructor a couple of times already, I still think it's simplest to think of tick formatting as a pure function that produces a label from the position of the tick mark: The midnight/day of month label is also shown in bold. I'm going to skip over that implementation, as it again depends on the interface with canvas (or the DOM). In principle, it uses the same strategy. Reminder: All of this occurs in data space. These values need to be converted to pixel space for drawing. Using reverse scaling to get tooltip data So far, we've been using d3's scale function to convert data values to pixels. When the user hovers over the chart, we want to perform this function in reverse, and convert the mouse position back to data so that we can select the correct data point. Fortunately, d3 allows us to easily reverse our scaling function. Firstly, we need to determine the mouse position in pixels. We can construct an event handler like this: Then, we can round this to the nearest discrete x-value by stepping through our data series and finding the first data point that has a lower x value than the pointer position: Here, we assume that will handle setting some state with our tooltip data. The tooltips themselves are HTML, not canvas drawing, and so they are best handled in React or whatever framework is used for rendering. To position them, we surround our canvas element with a relatively-positioned and render the tooltip with absolute positioning. Final remarks My intention here has been to highlight \"pure functions\" that can be used to implement responsive chart design. I've avoided going in the details of how to tie this to an SVG or canvas implementation because that depends quite a bit on how the website in question is built. The full code is available here, however, if you'd like more detail. It is important to note that responsive design is about more than more than \"making it work on mobile\" - the goal is to provide the best experience we can across a wide range of device capabilities. By separating the concerns of content (data) and presentation (pixels), we can create better data-vis experiences for more users.","content_html":"A very common topic of questions on data visualization forums concerns responsive charts. In this post I will use an example of responsive chart design to illustrate the key issues when confronting this problem, outline the framework of a general solution, and show a bit of general-purpose code. The principles should be applicable to any frontend framework (React, Vue, Svelte). I'll use d3 throughout, but no prior d3 expertise is necessary. I will also use TypeScript, because it more clearly illustrates what data we are working with, but JavaScript should work just as well. Finally, although this particular chart is rendered in HTML5 canvas, the same principles would be applicable to a chart built with SVG.
Basic familiarity with JavaScript, how to draw with canvas, and DOM event handlers are needed.
The inspiration for this post came from this question on Observable Talk. A user named hiroyukikumazawa asked how to build a responsive chart with a mouse-hover tooltip and linked to this example. The chart in question shows Bitcoin price tracked over the last ~24hrs and has several features that are interesting from a web developer's perspective:
There is both a line chart and box chart view, which can be toggled via the "icon buttons" at the top.
The data resizes to fit its container. (Caveat: there's a narrow set of viewport sizes just above 1000px where it breaks out of the container in Firefox. It worked smoothly at all sizes in Chrome. I'm not sure what the difference is.)
In the box chart view, the width of each box scales so that it fits the space available.
The axis tick labels remain legible at all sizes. Some of the labels on the x-axis are hidden at smaller screen sizes so they don't obscure each other.
The x-axis labels are based on local time, not server time or GMT. The data point for midnight shows the day of the month instead of the hour, and is bold.
On mouse hover a tooltip appears corresponding to the nearest data point on the x-axis. A vertical line indicates which datapoint is being shown. (Caveat: this doesn't work on devices without a mouse, even if you poke at the chart with your finger).
Here is my attempt at reverse-engineering what they did:
It doesn't do everything, but it does illustrate the responsive features I'm going to talk about here (and adds/fixes a few more). I wrote it in Deno/Fresh/Preact (the stack for this website), but the "pure functions", and the d3 bits I will show below can be applied to any framework. Their chart is based on canvas, so I'll stick to that, as well. The full solution is available here.
Charts present a particular challenge for responsive design because they do not fit neatly into the categories of other element types that we are used to dealing with. Simple text can reflow to fit its container. Raster images can resize to maintain a consistent aspect ratio (though even so, HTML5 gives us mechanisms to load different images for different devices). Charts are not just images: maintaining a constant aspect ratio almost never works well outside of the context in which it was designed, while text quickly becomes illegible as it shrinks down to mobile sizes. (This is most easily observed for charts that are displayed as raster images, but SVGs don't improve the situation much, without extensive cajoling).
How can we scale the bits we need to? The answer is to add a layer of indirection between the "data scale" and the actual pixels that we draw on the canvas. In the example, the heights (and vertical positions) of each box are given as properties of the data themselves, and must be scaled prior to drawing. Other dimensions (such as the width of each box), are detemined after scaling, and are drawn without reference to the data. I will refer to these two different layers as data space and pixel space throughout.
If I had more time, I would figure out how to use the Coin Market Cap API to show up-to-date Bitcoin prices. However, large parts of their API don't seem to work as advertised (at least on the free tier). So, the chart data show the data for the date I downloaded the data: Dec 13, 2023.
The data can be downloaded as a CSV via the button to the top-right of the chart. The CSV data are in one hour intervals, instead of 15 min intervals as in the original chart. Therefore, my chart has sparser data points than the original.
Furthermore, although the file has a .csv extension, the data are actually "semicolon-separated", not comma separated. They can be imported into JavaScript with d3's handy dsv package:
import { dsvFormat } from \"https://cdn.skypack.dev/d3-dsv@3\";\nconst { parse } = dsvFormat(\";\");\nconst data = parse(text);The data are objects consisting of a timestamp and a value for high, low, open, and close prices for each hour. (There's some other values in there, too, but we'll ignore those for this charting project):
interface Row {\n timestamp: string;\n open: number;\n close: number;\n high: number;\n low: number;\n}D3 also has several modules for working with and plotting datetimes, but for reasons which should become clear below, I find it easier to just convert these to simple numbers: aka UTC timecodes. It is useful to define a function that extracts a numeric x value from a data row:
import { datetime } from \"https://deno.land/x/ptera@v1.0.2\";\nimport { dateToTS } from \"https://deno.land/x/ptera@v1.0.2/convert.ts\";\n\nconst xValue = (row: Row): number => dateToTS(datetime(row.timestamp));Usually, responsive design depends on media queries that apply different styles depending on the width of the overall viewport. In our case, it is more useful to respond to the width of the chart itself. (In principle, this is related to the CSS container query, but as our charts are based on TypeScript we'll get the container width and height there):
const pxWidth = el.clientWidth;\nconst pxHeight = el.clientHeight;where el is the canvas (chart) element.
Having determined our pixel space, we'll now turn our attention to the "data space" that make up the Cartesian plane upon which we usually think about graphs. We need to know the minimum and maximum values we'll have to worry about for both the x- and y-axis. Since the data starts out sorted by timestamp (x-axis) values, the minimum and maximum are just defined by the first and last values:
const xMin = xValue(data[0]);\nconst xMax = xValue(data[data.length - 1]);The y-axis is more complicated. To simplify things, we'll use the same scale for both the line and box plot, which means the lowest possible y-value in the series is given by:
let yMin = Math.min(...data.map((p) => p.low));And the highest value by:
let yMax = Math.max(...data.map((p) => p.high));We'll give that some breathing room by "zooming out" by 20%:
const Y_ZOOM = 1.2;\n\nconst yLength = (yMax - yMin) * Y_ZOOM;\nyMin = yMin - (yLength - yMax + yMin) / 2;\nyMax = yMin + yLength;We should also make sure the chart has sufficient "padding" to ensure the axes have enough space regardless of the overall width. Let's give the following padding values in pixels:
const PAD_LEFT = 40;\nconst PAD_RIGHT = 65;\nconst PAD_BOTTOM = 50;Now we can define a function for interconverting between data space and pixel space. D3 provides a scaleLinear facade which creates a function to return any value on a given "domain" (in our case, data space) to a given "range" (in our case, pixel space). Our overall scaling function for the x-axis looks like this:
import { scaleLinear } from \"https://cdn.skypack.dev/d3-scale@3\";\n\nconst scaleX = (dataValue: number): number => \n scaleLinear()\n .domain([xMin, xMax])\n .range([PAD_LEFT, pxWidth - PAD_RIGHT]);The y-axis is similar, but also reverses the up/down direction. We do this by putting the desired range in backwards:
const scaleY = (dataValue: number): number =>\n scaleLinear()\n .domain([yMin, yMax])\n .range([pxHeight - PAD_BOTTOM, 0]);These two functions together can be used to take any data point and convert it to "pixel space" for plotting. Plotting the actual line in our line chart, for example, depends on converting them first and then using d3-shape to do the actual drawing:
import { line as d3Line } from \"https://cdn.skypack.dev/d3-shape@3\";\n\nconst path: Array<[number, number]> = rows.map((row: Row) => [\n scaleX(xValue(row)),\n scaleY(row.close),\n]);\n\nd3Line().context(ctx)(path);\n\nctx.stroke();where ctx is the canvas context.
There are some more details here, like how to draw a line that changes color depending on its value. However, I want to stay focused on how to implement responsive features, so I will skip over that.
The width of each data box scales so that the boxes fill the available space at any given width, without overlapping each other (in fact, there are gaps in between them). Here we see how to use data space and pixel space together. The top and bottom of each box are determined by the data:
const drawBox = (datapoint: Row) => {\n const y0 = scaleY(datapoint.open);\n const y1 = scaleY(datapoint.close); \n}However, the box width can be determined by the actual width of the chart in pixel space:
const BOX_GAP = 10;\n\nconst barWidth = (pxWidth / data.length) - BOX_GAP;Drawing the boxes is given by combining both of these concepts of "space":
const BOX_SPACING = 10;\n\nconst boxPath = (datapoint: Row) => {\n const y0 = scaleY(datapoint.open);\n const y1 = scaleY(datapoint.close);\n const x = scaleX(xValue(datapoint));\n \n const barWidth = (pxWidth / data.length) - BOX_SPACING;\n const halfWidth = barWidth / 2;\n const path = [\n [x - halfWidth, y1],\n [x + halfWidth, y1],\n [x + halfWidth, y0],\n [x - halfWidth, y0],\n [x - halfWidth, y1], // close the path\n ];\n};This path can be passed to d3Line to draw boxes.
Our y-axis is not responsive. We have 300px of height to deal with, regardless of the width. However, we don't (in theory) know the range of values that will be charted and therefore can't determine a priori what ticks to draw.
D3 makes this quite easy, though the relevant function is hidden within the array sub-package. d3.array.ticks does the following according to the documentation: "Returns an array of approximately count + 1 uniformly-spaced, nicely-rounded values between start and stop (inclusive)". I love the words "approximate" and "nicely" in this description. This isn't a situation where we are looking for total predictability: we need a bunch of ticks that fit within the range of our axis, and they should be round numbers, within whatever range we are talking about.
The first two arguments to this function are the minimum and maximum values of the axis, the third is the approximate number of ticks we want. The original chart had about ten:
import { ticks } from \"https://cdn.skypack.dev/d3-array@3\";\n\nconst yTicks = ticks(yMin, yMax, 10);What about the tick label? Let's define a function that takes a y value as argument and returns the label:
const formatYTick = (position: number) => `${(position / 1000).toFixed(2)}K`;The x-axis shrinks as the graph container shrinks, and it has a couple nice features that makes it legible on small screens.
To keep the labels concise, they do not show the full date and time. Only the time is shown, except at midnight, when the day of the month is shown in bold.
As the width decreases, some of the ticks disappear, preventing the text from overlapping and obscuring one another.
Combining these two features is a challenge: we need to decide which labels to hide, but the midnight/new day timepoint should always be shown (or else the user will lose track of which day it is).
It's also important to keep in mind that, so far, we have converted ISO datetimes (which are all in GMT), to UTC numbers for plotting. The labels should represent local time, including showing the new day of the month at midnight, local.
Let's "thin out" our tick marks, by defining a parameters skip that takes every 2nd, 3rd, or nth tick mark, counting midnight as 0:
const xTicks = (skip: number) => (rows: Row): number[] =>\n rows.map(row => datetime(xValue(row)))\n .filter(dt => dt.hour % skip)\n .map(dt => dateToTS(dt));Calling xTicks(2) will create a function that returns only even hours: midnight, 2:00 am, 4:00 am, 6:00 am, etc.. We have to play with the chart a little bit to see what skip value is appropriate at each chart width. I came up with:
const skip = width < 600 ? 6 : width < 992 ? 3 : 2;Although we've used the datetime constructor a couple of times already, I still think it's simplest to think of tick formatting as a pure function that produces a label from the position of the tick mark:
const formatXLabel = (position: number) => {\n const dt = datetime(position);\n return formatDate(dt, dt.hours === 0 ? \"d\" : \"hh:mm a\")\n}The midnight/day of month label is also shown in bold. I'm going to skip over that implementation, as it again depends on the interface with canvas (or the DOM). In principle, it uses the same strategy.
Reminder: All of this occurs in data space. These values need to be converted to pixel space for drawing.
So far, we've been using d3's scale function to convert data values to pixels. When the user hovers over the chart, we want to perform this function in reverse, and convert the mouse position back to data so that we can select the correct data point. Fortunately, d3 allows us to easily reverse our scaling function.
Firstly, we need to determine the mouse position in pixels. We can construct an event handler like this:
const handleMouseMove = (ev: MouseEvent) => {\n const el = ev.target as HTMLCanvasElement;\n const offsets = el.getBoundingClientRect();\n const x = scaleX.invert(ev.clientX - offsets.x);\n const y = scaleY.invert(ev.clientY - offsets.y);\n}Then, we can round this to the nearest discrete x-value by stepping through our data series and finding the first data point that has a lower x value than the pointer position:
const handleMouseMove = (ev: MouseEvent) => {\n const el = ev.target as HTMLCanvasElement;\n const offsets = el.getBoundingClientRect();\n const x = scaleX.invert(ev.clientX - offsets.x);\n const y = scaleY.invert(ev.clientY - offsets.y);\n \n const indexToRight = data.findIndex((p) =>\n xValue(p) > x\n );\n if (indexToRight === 0) return cb(data[0]);\n if (indexToRight === -1) return cb(data[data.length - 1]);\n \n const dToRight = Math.abs(\n x - xValue(data[indexToRight]),\n );\n const dToLeft = Math.abs(\n x - xValue(data[indexToRight - 1]),\n );\n if (dToRight < dToLeft) {\n return cb(data[indexToRight]);\n }\n return cb(data[indexToRight - 1]);\n}Here, we assume that cb will handle setting some state with our tooltip data. The tooltips themselves are HTML, not canvas drawing, and so they are best handled in React or whatever framework is used for rendering. To position them, we surround our canvas element with a relatively-positioned div and render the tooltip with absolute positioning.
My intention here has been to highlight "pure functions" that can be used to implement responsive chart design. I've avoided going in the details of how to tie this to an SVG or canvas implementation because that depends quite a bit on how the website in question is built.
The full code is available here, however, if you'd like more detail. It is important to note that responsive design is about more than more than "making it work on mobile" - the goal is to provide the best experience we can across a wide range of device capabilities. By separating the concerns of content (data) and presentation (pixels), we can create better data-vis experiences for more users.
","summary":"A real-world example highlighting a common challenge","image":"charts/coin-base","date_published":"2024-01-09T00:00:00.000Z"},{"id":"canada-food-inflation","url":"http://localhost:8000/blog/canada-food-inflation","title":"canada-food-inflation","content_text":"Overall inflation seems to be cooling, but the costs of food and housing continue to soar across Canada. In October 2023, food inflation was still nearly double the overall inflation rate, and the federal governement began steps to stabilize prices through regulation of the major grocery chains . Exactly what items are behind the increased price at the register depends on where you live. In they were in 2017, while in that has nearly doubled. and seem to be up across the board, while are up overall but bounce faster than bitcoin. A select handful of items have actually since 2017, though the difference is highly dependent on where you live. The data come from Stats Canada , which has been tracking the average price of 110 food items across every province since 2017. The table below shows a heatmap of price increases over the last six years: the shading of each point represents the overall change in price compared to the same month the previous year. The profile of nearly every product (and region) is unique. One common theme that emerges however is that the most significant increases happened in late 2021 and early 2022. That is, for items that increased the most, that increase more often than not came in the later stages of the pandemic.","content_html":"Overall inflation seems to be cooling, but the costs of food and housing continue to soar across Canada. In October 2023, food inflation was still nearly double the overall inflation rate, and the federal governement began steps to stabilize prices through regulation of the major grocery chains.
Exactly what items are behind the increased price at the register depends on where you live. In they were in 2017, while in that has nearly doubled. and seem to be up across the board, while are up overall but bounce faster than bitcoin. A select handful of items have actually since 2017, though the difference is highly dependent on where you live.
The data come from Stats Canada, which has been tracking the average price of 110 food items across every province since 2017. The table below shows a heatmap of price increases over the last six years: the shading of each point represents the overall change in price compared to the same month the previous year.
The profile of nearly every product (and region) is unique. One common theme that emerges however is that the most significant increases happened in late 2021 and early 2022. That is, for items that increased the most, that increase more often than not came in the later stages of the pandemic.
","summary":"Christmas is coming, but the squeeze continues","image":"food-price-scrn.png","date_published":"2023-12-04T00:00:00.000Z"},{"id":"banana-index-interactive","url":"http://localhost:8000/blog/banana-index-interactive","title":"banana-index-interactive","content_text":"In April, The Economist proposed a new metric for measuring the climate impact of food . The motivating factor was that while most plant-based foods are \"better\" for the environment that most meat-based ones, they also contain fewer calories and less protein, and therefore more of them are needed to make up a full meal or diet. The Economist thus weighted carbon emissions by kilocalorie, gram of protein or gram of fat produced by each kind of food. They then expressed this relative environmental impact in units of bananas - how much is emitted per kcal compared to the same ratio for bananas. Unsurprisingly, beef (and meat in general) produces a lot of no matter how you slice it. But are carbon emissions the only environmental impact worth considering? When I found the raw data , I discovered that several other impacts had been measured for the same foodstuffs, including land use, water use, and eutrophication (pollution from fertilizer run-off). My thanks for Agustín Formoso for his original riff and link to the raw data. Some of these other impacts don't look as bad for burger-lovers. When looking at , for example, some types of plant-crops, as well as some kinds of nuts and mushrooms, seem to rank pretty poorly.","content_html":"In April, The Economist proposed a new metric for measuring the climate impact of food. The motivating factor was that while most plant-based foods are "better" for the environment that most meat-based ones, they also contain fewer calories and less protein, and therefore more of them are needed to make up a full meal or diet.
The Economist thus weighted carbon emissions by kilocalorie, gram of protein or gram of fat produced by each kind of food. They then expressed this relative environmental impact in units of bananas - how much is emitted per kcal compared to the same ratio for bananas.
Unsurprisingly, beef (and meat in general) produces a lot of no matter how you slice it. But are carbon emissions the only environmental impact worth considering? When I found the raw data, I discovered that several other impacts had been measured for the same foodstuffs, including land use, water use, and eutrophication (pollution from fertilizer run-off).
My thanks for Agustín Formoso for his original riff and link to the raw data.
Some of these other impacts don't look as bad for burger-lovers. When looking at , for example, some types of plant-crops, as well as some kinds of nuts and mushrooms, seem to rank pretty poorly.
","summary":"Riffing on an idea from The Economist concerning the climate impact of food","image":"charts/food-impact","date_published":"2023-11-27T00:00:00.000Z"},{"id":"in-praise-of-fresh","url":"http://localhost:8000/blog/in-praise-of-fresh","title":"in-praise-of-fresh","content_text":"I gave a talk earlier this week about how much I liked working with Fresh, but I ran a bit short on time and didn't talk about what brought me to Fresh in the first place. This seems like a good place to go into it, because the motivation for trying out Fresh was building this website! This website is in part an attempt to explore technology solutions for data-rich blogging and journalism on the web. I have found that neither traditional server-side nor today's most popular client-side rendering solutions are a good fit for this task. The JavaScript libraries that couple well with server-rendered HTML are not up to the task of providing the level of interactivity I want, and websites built entirely in JavaScript are error-prone and reject the principles of progressive enhancement. Without the testing resources that only very large companies have access to, they usually provide a frustrating experience for users. JAM stack frameworks are currently the best idea the community has to bridge this gap, but they fall short of server-side frameworks in terms of programming HTTP responses in a flexible way. For a website that requests data and graphs it, dynamically rendered pages and a flexible API should be strict requirements. In my opinion, Fresh is one of the best things to happen to the web in a long time. The vast majority of the content on this side is rendered on the server. Where interactivity is needed, JavaScript and data are delivered and hydrated to existing components. Data graphics can be rendered server-side as SVG and seemlessly re-animated as canvas. Additional data can be requested from an API and appended to the graphic, or entire new pages rendered from the same template. Fresh is also written in Deno, so it provides TypeScript, JSX, linting, testing and package management with very little configuration. It uses Preact rather than React, drastically decreasing the amount of JavaScript delivered and improving time-to-glass performance. Other tech choices: I'm using CouchDB as a database which gives me neat way to sync my development and production environments. I'm using this to store raw data (for graphs), markdown, and even images. Mostly, I work with content locally on my computer and \"ingest\" it into the database using some custom Deno code. I may open-source this at some point in the future if I have time. Markdown is converted to an mdast tree at ingestion-time by unified , and then rendered via a custom Preact component. This actually gives me a way to render interactive elements (eg data graphics) within content by using markdown extensions. I use d3 for data manipulation, but stick to Preact for rendering. JSX is truly expressive for data-graphics, and going back to imperative programming for rendering is never going to be a realistic option. CSS is currently Fresh's biggest weakness. Although I ended up using Tailwind , I didn't use Fresh's own plugin because I don't like CSS-in-JS and felt it didn't fit with the other values Fresh was espousing. I'm also using ImageMagick to resize images, and it wasn't super-easy to get it to work with Deno. This sort of thing will hopefully become less common as the community matures and Deno gains wider acceptance. Despite these growing pains, I think Fresh's approach to unifying server-side and client-side rendering is the future. Building this site was a pleasure and I hope to create more like it.","content_html":"I gave a talk earlier this week about how much I liked working with Fresh, but I ran a bit short on time and didn't talk about what brought me to Fresh in the first place. This seems like a good place to go into it, because the motivation for trying out Fresh was building this website!
This website is in part an attempt to explore technology solutions for data-rich blogging and journalism on the web.
I have found that neither traditional server-side nor today's most popular client-side rendering solutions are a good fit for this task. The JavaScript libraries that couple well with server-rendered HTML are not up to the task of providing the level of interactivity I want, and websites built entirely in JavaScript are error-prone and reject the principles of progressive enhancement. Without the testing resources that only very large companies have access to, they usually provide a frustrating experience for users.
JAM stack frameworks are currently the best idea the community has to bridge this gap, but they fall short of server-side frameworks in terms of programming HTTP responses in a flexible way. For a website that requests data and graphs it, dynamically rendered pages and a flexible API should be strict requirements.
In my opinion, Fresh is one of the best things to happen to the web in a long time. The vast majority of the content on this side is rendered on the server. Where interactivity is needed, JavaScript and data are delivered and hydrated to existing components. Data graphics can be rendered server-side as SVG and seemlessly re-animated as canvas. Additional data can be requested from an API and appended to the graphic, or entire new pages rendered from the same template.
Fresh is also written in Deno, so it provides TypeScript, JSX, linting, testing and package management with very little configuration. It uses Preact rather than React, drastically decreasing the amount of JavaScript delivered and improving time-to-glass performance.
Other tech choices:
I'm using CouchDB as a database which gives me neat way to sync my development and production environments. I'm using this to store raw data (for graphs), markdown, and even images. Mostly, I work with content locally on my computer and "ingest" it into the database using some custom Deno code. I may open-source this at some point in the future if I have time.
Markdown is converted to an mdast tree at ingestion-time by unified, and then rendered via a custom Preact component. This actually gives me a way to render interactive elements (eg data graphics) within content by using markdown extensions.
I use d3 for data manipulation, but stick to Preact for rendering. JSX is truly expressive for data-graphics, and going back to imperative programming for rendering is never going to be a realistic option.
CSS is currently Fresh's biggest weakness. Although I ended up using Tailwind, I didn't use Fresh's own plugin because I don't like CSS-in-JS and felt it didn't fit with the other values Fresh was espousing.
I'm also using ImageMagick to resize images, and it wasn't super-easy to get it to work with Deno. This sort of thing will hopefully become less common as the community matures and Deno gains wider acceptance.
Despite these growing pains, I think Fresh's approach to unifying server-side and client-side rendering is the future. Building this site was a pleasure and I hope to create more like it.
","summary":"In my opinion, the best thing to happen to the web in a long time","date_published":"2023-11-23T00:00:00.000Z"},{"id":"world-demographics-in-2100","url":"http://localhost:8000/blog/world-demographics-in-2100","title":"world-demographics-in-2100","content_text":"One of the most predictable trends in the ecology of humans is that as countries become wealthier, fertility rates decline. This means that even as mortality rates plummet and most people can expect to live into old age, the population of a country stabilizes and may even begin to shrink. The population of the world has been increasing exponentially for centuries. This period is now at an end . It is possible that within the lifetime of many people reading this, the world population may actually peak and begin to decline. Exponential growth being what is is, there is also considerable uncertainty about how exactly this will play out. Note that the difference between these projections, by 2100, is close to the entire current population of the world. For those of us concerned about the finitude of the planet we live on, this seems an unaccountable good. We simply cannot continue adding more and more billions of humans, forever. It also means that the transitory period is going to open new challenges. As the population continues to age and doesn't fully replace itself, the fraction of people past retirement age will skyrocket. This can be seen in the interactive below. Red shaded areas indicate ages that will make up a lower share of the population in the target year than they do now, while green areas indicate a higher share than they do now. The small chart at the bottom shows how the dependency ratio (the ratio of people outside the workforce to inside it) will change over time. For comparison, present-day Florida (considered the retirement capital of the US) has a dependency ratio of 0.68. That's right: the world in 2100 will have a higher dependency ratio than present-day Florida , even under the highest fertility projection. You can click on or drag the bottom chart to change the target year. Higher income countries trend towards even higher demographic ratios. Japan presently has a famously elderly population. Its dependency ratio will continue to get even higher, before finally stabilizing at just above 1 (meaning more dependents than members of the workforce). The bright spot, of course, will be countries that are currently developing, which will see their dependency ratios decline as their fertility rates decrease as their pyramid-shaped demographic curves change to include many more workers. You can explore the data further and look at your country or region by using bottom search box. Data hosted by the United Nations and processed in this Observable Notebook . Inspired by Our World in Data .","content_html":"One of the most predictable trends in the ecology of humans is that as countries become wealthier, fertility rates decline. This means that even as mortality rates plummet and most people can expect to live into old age, the population of a country stabilizes and may even begin to shrink.
The population of the world has been increasing exponentially for centuries. This period is now at an end. It is possible that within the lifetime of many people reading this, the world population may actually peak and begin to decline. Exponential growth being what is is, there is also considerable uncertainty about how exactly this will play out.
Note that the difference between these projections, by 2100, is close to the entire current population of the world.
For those of us concerned about the finitude of the planet we live on, this seems an unaccountable good. We simply cannot continue adding more and more billions of humans, forever. It also means that the transitory period is going to open new challenges. As the population continues to age and doesn't fully replace itself, the fraction of people past retirement age will skyrocket.
This can be seen in the interactive below. Red shaded areas indicate ages that will make up a lower share of the population in the target year than they do now, while green areas indicate a higher share than they do now.
The small chart at the bottom shows how the dependency ratio (the ratio of people outside the workforce to inside it) will change over time. For comparison, present-day Florida (considered the retirement capital of the US) has a dependency ratio of 0.68. That's right: the world in 2100 will have a higher dependency ratio than present-day Florida, even under the highest fertility projection. You can click on or drag the bottom chart to change the target year.
Higher income countries trend towards even higher demographic ratios.
Japan presently has a famously elderly population. Its dependency ratio will continue to get even higher, before finally stabilizing at just above 1 (meaning more dependents than members of the workforce).
The bright spot, of course, will be countries that are currently developing, which will see their dependency ratios decline as their fertility rates decrease as their pyramid-shaped demographic curves change to include many more workers.
You can explore the data further and look at your country or region by using bottom search box.
Data hosted by the United Nations and processed in this Observable Notebook. Inspired by Our World in Data.
","summary":"The world's population structure will change massively","image":"graphic/world-demo.png","date_published":"2023-10-25T00:00:00.000Z"},{"id":"simpsons-imdb-ratings-by-season","url":"http://localhost:8000/blog/simpsons-imdb-ratings-by-season","title":"simpsons-imdb-ratings-by-season","content_text":"The wall came down, and about a month later of The Simpsons aired on the Fox network. For 34 years the show has been locked in a kind of strange timewarp, with characters who never age, while the world around them has changed to be almost unrecognizable. They have outlasted five US presidents (including and then still being around for the end of President Trump), and essentially pre-date the (widespread usage of) the internet. Mat Groening's cynical take on modern American life had a profound impact on the millenial generation, and many of us can effortlessly reference the first few seasons. The show is forever connected with the 90s, and the commonplace understanding is that it jumped the shark when that decade ended. The always felt like an ending, and while I kept hope alive for another year or so, my interest waned pretty rapidly after that. The numbers confirm it (people can come up with statistics to prove anything). The data show user-provided ratings, from IMDB , for each episode (in green). The dotted line shows the median rating for each season, while the grey box contains the middle two quartiles. In other words, half of each season's episodes fall within the grey zone, while the worst quarter of episodes fall below it and the best quarter fall above it. I took the raw data (stole, made up, what's the difference?) from this Observable notebook . They represent the first 600 episodes. A few noteworthy data points. The is my personal favourite, but although it tops season 4 it isn't the highest of all time. That honor is a tie between and the one where . The early seasons have a few notable stinkers: . Among later seasons, the 16th season episode (in 2005), caused quite a splash in the media when it aired, but it ranks distinctly mediocre in terms of what audiences thought. The one - the only episode I've made an effort to watch in at least 20 years - fares a little better but isn't a standout. Meanwhile, there are a handful of episodes from the past two decades that are clearly a cut above the common crowd. They are pretty easy to pick out, and I haven't seen them, so I won't call them out individually. Most popular TV shows last a few seasons past the point where audiences give up. This show seems exceptional: I don't know who exactly is still watching, but it's nice to know that science confirms that we gave up at the right time.","content_html":"The wall came down, and about a month later of The Simpsons aired on the Fox network. For 34 years the show has been locked in a kind of strange timewarp, with characters who never age, while the world around them has changed to be almost unrecognizable. They have outlasted five US presidents (including and then still being around for the end of President Trump), and essentially pre-date the (widespread usage of) the internet.
Mat Groening's cynical take on modern American life had a profound impact on the millenial generation, and many of us can effortlessly reference the first few seasons. The show is forever connected with the 90s, and the commonplace understanding is that it jumped the shark when that decade ended. The always felt like an ending, and while I kept hope alive for another year or so, my interest waned pretty rapidly after that.
The numbers confirm it (people can come up with statistics to prove anything).
The data show user-provided ratings, from IMDB, for each episode (in green). The dotted line shows the median rating for each season, while the grey box contains the middle two quartiles. In other words, half of each season's episodes fall within the grey zone, while the worst quarter of episodes fall below it and the best quarter fall above it.
I took the raw data (stole, made up, what's the difference?) from this Observable notebook. They represent the first 600 episodes.
A few noteworthy data points. The is my personal favourite, but although it tops season 4 it isn't the highest of all time. That honor is a tie between and the one where . The early seasons have a few notable stinkers: .
Among later seasons, the 16th season episode (in 2005), caused quite a splash in the media when it aired, but it ranks distinctly mediocre in terms of what audiences thought. The one - the only episode I've made an effort to watch in at least 20 years - fares a little better but isn't a standout.
Meanwhile, there are a handful of episodes from the past two decades that are clearly a cut above the common crowd. They are pretty easy to pick out, and I haven't seen them, so I won't call them out individually.
Most popular TV shows last a few seasons past the point where audiences give up. This show seems exceptional: I don't know who exactly is still watching, but it's nice to know that science confirms that we gave up at the right time.
","summary":"When did the longest-running series in history jump the shark?","image":"graphic/simpsons.png","date_published":"2023-10-13T00:00:00.000Z"}]}