Cross-Filter

Cross-Filter multi-variant analysis example - Server Statistics CPU & Mem Utilization

by Jason Normandin

HTML

<script src="http://square.github.io/crossfilter/crossfilter.v1.min.js"></script>
<script src="http://square.github.io/crossfilter/d3.v3.min.js"></script>
<script src="http://d3js.org/d3.v3.min.js"></script>
<title>Crossfilter</title>

@import url(http://fonts.googleapis.com/css?family=Yanone+Kaffeesatz:400,700);

<div id="body">

<a href="https://github.com/square"><img src="logotype.png" width="122" height="31"></a>

<h1>Crossfilter</h1>

<h2>Fast Multidimensional Filtering for Coordinated Views</h2>

<p><b>Crossfilter</b> is a <a href="https://github.com/square/crossfilter">JavaScript library</a> for exploring large multivariate datasets in the browser. Crossfilter supports extremely fast (&lt;30ms) interaction with coordinated views, even with datasets containing a million or more records; we built it to power analytics for <a href="https://squareup.com/register">Square Register</a>, allowing merchants to slice and dice their payment history fluidly.

<p>Since most interactions only involve a single dimension, and then only small adjustments are made to the filter values, incremental filtering and reducing is significantly faster than starting from scratch. Crossfilter uses sorted indexes (and a few bit-twiddling hacks) to make this possible, dramatically increasing the perfor&shy;mance of live histograms and top-<i>K</i> lists. For more details on how Crossfilter works, see the <a href="https://github.com/square/crossfilter/wiki/API-Reference">API reference</a>.

<h2>Example: Airline on-time performance</h2>

<p>The coordinated visualizations below (built with <a href="http://mbostock.github.com/d3/">D3</a>) show nearly a quarter-million flights from early 2001: part of the <a href="http://stat-computing.org/dataexpo/2009/">ASA Data Expo</a> dataset. The dataset is 5.3MB, so it might take a few seconds to download. Click and drag on any chart to filter by the associated dimension. The table beneath shows the eighty most recent flights that match the current...

CSS

<style>

@import url(http://fonts.googleapis.com/css?family=Yanone+Kaffeesatz:400,700);

body {
  font-family: "Helvetica Neue";
  margin: 40px auto;
  width: 960px;
  min-height: 2000px;
}

#body {
  position: relative;
}

footer {
  padding: 2em 0 1em 0;
  font-size: 12px;
}

h1 {
  font-size: 96px;
  margin-top: .3em;
  margin-bottom: 0;
}

h1 + h2 {
  margin-top: 0;
}

h2 {
  font-weight: 400;
  font-size: 28px;
}

h1, h2 {
  font-family: "Yanone Kaffeesatz";
  text-rendering: optimizeLegibility;
}

#body > p {
  line-height: 1.5em;
  width: 640px;
  text-rendering: optimizeLegibility;
}

#charts {
  padding: 10px 0;
}

.chart {
  display: inline-block;
  height: 151px;
  margin-bottom: 20px;
}

.reset {
  padding-left: 1em;
  font-size: smaller;
  color: #ccc;
}

.background.bar {
  fill: #ccc;
}

.foreground.bar {
  fill: steelblue;
}

.axis path, .axis line {
  fill: none;
  stroke: #000;
  shape-rendering: crispEdges;
}

.axis text {
  font: 10px sans-serif;
}

.brush rect.extent {
  fill: steelblue;
  fill-opacity: .125;
}

.brush .resize path {
  fill: #eee;
  stroke: #666;
}

#hour-chart {
  width: 260px;
}

#delay-chart {
  width: 230px;
}

#distance-chart {
  width: 420px;
}

#date-chart {
  width: 920px;
}

#flight-list {
  min-height: 1024px;
}

#flight-list .date,
#flight-list .day {
  margin-bottom: .4em;
}

#flight-list .flight {
  line-height: 1.5em;
  background: #eee;
  width: 640px;
  margin-bottom: 1px;
}

#flight-list .time {
  color: #999;
}

#flight-list .flight div {
  display: inline-block;
  width: 100px;
}

#flight-list div.distance,
#flight-list div.delay {
  width: 160px;
  padding-right: 10px;
  text-align: right;
}

#flight-list .early {
  color: green;
}

aside {
  position: absolute;
  left: 740px;
  font-size: smaller;
  width: 220px;
}

JavaScript

// (It's CSV, but GitHub Pages only gzip's JSON at the moment.)

//d3.csv("/gh/get/response.jason/jasonnorm/data/demo.response.json", function (flights, errors) {

d3.xhr("/gh/get/response.json/jasonnorm/D3/demo.response.json")
    .header("X-Requested-With", "XMLHttpRequest")
    .get("delay=1", function (flights, error) {
    
  // Various formatters.
  var formatNumber = d3.format(",d"),
      formatChange = d3.format("+,d"),
      formatDate = d3.time.format("%B %d, %Y"),
      formatTime = d3.time.format("%I:%M %p");

  // A nest operator, for grouping the flight list.
  var nestByDate = d3.nest()
      .key(function(d) { return d3.time.day(d.date); });

  // A little coercion, since the CSV is untyped.
  flights.forEach(function(d, i) {
    d.index = i;
    d.date = parseDate(d.date);
    d.delay = +d.delay;
    d.distance = +d.distance;
  });

  // Create the crossfilter for the relevant dimensions and groups.
  var flight = crossfilter(flights),
      all = flight.groupAll(),
      date = flight.dimension(function(d) { return d.date; }),
      dates = date.group(d3.time.day),
      hour = flight.dimension(function(d) { return d.date.getHours() + d.date.getMinutes() / 60; }),
      hours = hour.group(Math.floor),
      delay = flight.dimension(function(d) { return Math.max(-60, Math.min(149, d.delay)); }),
      delays = delay.group(function(d) { return Math.floor(d / 10) * 10; }),
      distance = flight.dimension(function(d) { return Math.min(1999, d.distance); }),
      distances = distance.group(function(d) { return Math.floor(d / 50) * 50; });

  var charts = [

    barChart()
        .dimension(hour)
        .group(hours)
      .x(d3.scale.linear()
        .domain([0, 24])
        .rangeRound([0, 10 * 24])),

    barChart()
        .dimension(delay)
        .group(delays)
      .x(d3.scale.linear()
        .domain([-60, 150])
        .rangeRound([0, 10 * 21])),

    barChart()
        .dimension(distance)
        .group(distances)
     ...