Klipfolio extends flexibility with Vega integration

Klipfolio is designed with the end-user in mind which is why it has many features to make their life easier. Klipfolio’s feature set is built on a solid platform with extension points that allow power users to go beyond the out of the box capabilities and create their own custom Klips and dashboards. In this post I will focus on helping power users and visualization champions to extend Klipfolio’s visualizations by integrating Vega (see Vega’s licensing agreement at the end of this article) into our HTML template. Although all the steps are explained here, I am targeting power users with the assumption that they are experienced with JavaScript and HTML concepts and are familiar with Klipfolio.

On the Trifacta web site, Vega is depicted as “a visualization grammar, a declarative format for creating, saving and sharing visualization designs.” Vega allows users to describe visualizations using JSON format to generate interactive visualizations. It is built on D3 and was named by Fast Company as one of the five best libraries for building data visualizations. You can read more about Vega on their web site.

Klipfolio’s HTML Template is designed to allow power users to create custom Klips (Klipfolio for visualizations) relying on our powerful data platform. Integrating visualisation libraries like Vega with Klipfolio’s HTML Template allows visualization experts to focus on building the visualization without also worrying about fetching the data from various cloud data sources.

In this article we cover step-by-step the tasks you need to accomplish in order to create a data source, integrate Vega in Klipfolio’s HTML Template and create a visualization. By the end of this article, you will be able to build the following Klip and add it to your dashboard:

Step 1 - Creating a data source

In this example we are going to use one of the data sources used by the Vega editor called population. To create the data source in Klipfolio, go to the Data Sources tab under Library and click on Create a New Data Source. Select the Web Accessible Resource as your data source type and point to the following sample data URL:

http://trifacta.github.io/vega/data/population.json

Tip: to learn more about web accessible data sources check out our knowledge base article.

---

Step 2 - Build a new Klip

To create a new HTML template Klip, click on Build a New Klip under the Klips tab in the Library. Choose “HTML template” and then use an existing data source and select the data source that you just created.

---

Step 3 - Create the data series

The data source is a JSON data source and provides four fields: year, age, sex, and people. You have to create a data series for each of these fields. The HTML Template has one data series by default. Add additional series, click on the + icon in the controls panel on the right side of the page.

After creating the four data series:

1. Select the first data series.
2. Go to the data panel at the bottom of the page.
3. Expand the JSON object hierarchy until you get to the first object in the array and assign the first fields in the data source to the selected data series by clicking on the field (e.g. year).
4. Switch to the properties pane and change the data series Data ID to be the same as the field in the data source (e.g. year).
5. Repeat steps 1 to 4 for all the fields and data series. When you are finished, you should see the following in the Klip editor component tree.

---

Step 4 - Add in the JavaScript and HTML scaffolding

In this step, we insert in the JavaScript that is required to load the JS libraries we are going to use. In this example we load D3, topojson (used by Vega) and Vega. All you need to do is to copy and paste the following code into the Script section of the HTML template. To expose the Script section, first click on HTML Template on the component tree and then click on “Use JavaScript with this HTML Template” in the properties pane at the button.

Note: If you are familiar with HTML template and don’t want to know the detail, feel free to jump to the last step and copy paste the code.

//Define the path configurations to the JS modules that you want to load
//In this example we are loading D3, topojson (used by Vega) and Vega  
//Note: URLs have to be HTTPS due to browser security restrictions  
var vizmodules = {
    d3:'https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js?noext',
    topojson:'https://cdnjs.cloudflare.com/ajax/libs/topojson/1.6.19/topojson.min.js?noext',
    vega:'https://cdnjs.cloudflare.com/ajax/libs/vega/1.5.0/vega.min.js?noext'
};

//Pass in the configurations to requirejs  
requirejs.config({
    paths: vizmodules
});

//Create a reference to the component 
var _component = this;
//Create a reference to the data model 
var _dataModel = this.dataModel;

//Load the libraries 
require([vizmodules.vega], 
    function(vg) {
    //You custom Klip code goes here.
    console.log(_dataModel);
    console.log(d3);
    console.log(vg);     
});

Tip 1: Use this approach to load other JS libraries to build your own custom Klips
Tip 2: Changes to the HTML Template property in properties pane, rerenders the Klip. Add and remove a space to enforce a rerender, for debugging purposes.

At this point when the Klip renders you should see the data, Vega, and D3 objects in the browser console (e.g. Chrome dev tools console). This is the validation that you need to know you have access to the data and the visualization libraries.

To be able to render the visualization you also need to add the following to the HTML section of your HTML Template Klip:

<div id="vis"></div>  

Tip 1: If you have more than one custom visualization on your dashboard, make sure you use a different ID for the HTML element. Otherwise they will conflict and don’t work properly.

---

Step 5 - Modify the data model

The data model can be used in two formats either as an Object model or an Array model. Read more about these formats here. Use the format that is most suitable for your scenario. However, if you still need to make some changes to the default model you have the opportunity to do that. In this case, the data values are coming back as strings and we need to change them to numbers before feeding them into the visualization spec. Therefore, we add the following code to our function that uses underscore to go through the data model and push the modified data objects into a new Array after converting the strings to numbers. Underscore is available in the runtime and is a very good library that you can use for data manipulation.

var _data = [];

_.each(_dataModel.data, function(element){
	_data.push({year: Number(element.year), age: Number(element.age), sex: Number(element.sex), people: Number(element.people)}); 
});

Tip: For performance reasons, when you are in the Klip Editor the dataModel may not contain all the data that you expect from your original data source. The complete data will be available when you add the Klip to the dashboard.

Step 6 - Get the width value of the component

In order for the visualization to dynamically resize in a similar fashion to the Klipfolio native visualizations when a dashboard is resized, you need to reference the HTML Template component width. Component width changes dynamically as the dashboard width changes based on the browser window or screen size. You can then use the component width to change the properties of the visualizations that need to be changed.

//Get the component width
var _width = _component.dimensions.w;

//Set the properties that needs to be changed as the width is changed
var _scales_range1 = _width / 2 - 10;
var _marks_properties_x =  _width / 2 - 20;
var _marks_properties_width = _width / 2 - 40;

Note: We don’t get into the details of what properties need to be updated in this article as these properties would be different depending on the visualization and the specification that is being used.

---

Step 6 - Feed in the data to the visualization spec and render

Vega uses a visualization specification to render the data. The visualization spec contains a section that defines the data. Use the Array that you have created above and pass it into the visualization specification as shown in the following block. Read more about Vega data here.

"data": [
    {
      "name": "pop",
       "values": _data,
      "transform": [
        {"type": "filter", "test": "d.data.year == 1850 || d.data.year == 2000"}
      ]
    }
  ]

Tip: Vega allows you to apply transformation to the data before rendering the visualization. One of these transformations is filtering. In the example above, the data is filtered to only show the data for years 1850 and 2000.

For this example, we are using a visualization spec based on the population sample available in Vega editor. We don’t explain the visualization specification in this article. Feel free to explore Vega documentation on your own. The final JavaScript code that you need to have in the script section of your Klip template is the following:

HTML Template:

<div id="vis"></div>  

Script:

//Define the path configurations to the JS modules that you want to load
//In this example we are loading D3, topojson (used by Vega) and Vega  
//Note: URLs have to be HTTPS due to browser security restrictions 

var vizmodules = {
    d3               : 'https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js?noext',
    topojson         : 'https://cdnjs.cloudflare.com/ajax/libs/topojson/1.6.19/topojson.min.js?noext',
    vega             : 'https://cdnjs.cloudflare.com/ajax/libs/vega/1.5.0/vega.min.js?noext'
};

//Pass in the configurations to requirejs  
requirejs.config({
    paths: vizmodules
});

//Create a reference to the component 
var _component = this;

//Create a reference to the data model 
var _dataModel = this.dataModel;


//Load the libraries 
require([vizmodules.vega], 
function(vg) { // the callback function that is called after the libraries are loaded

var _data = [];

// use the dataModel and convert the string to numbers 
_.each(_dataModel.data, function(element){
  _data.push({year: Number(element.year), age: Number(element.age), sex: Number(element.sex), people: Number(element.people)}); 
});

//Get the component width
var _width = _component.dimensions.w;

//Set the properties that needs to be updated as the width is updated
var _scales_range1 = _width / 2 - 10;
var _marks_properties_x =  _width / 2 - 20;
var _marks_properties_width = _width / 2 - 40;

var spec = {
  "width": _width, //reference the component width in the spec 
  "height": 400,
  "data": [
    {
      "name": "pop",
      values: _data, // add the data to the spec
      "transform": [
        {"type": "filter", "test": "d.data.year == 1850 || d.data.year == 2000"}
      ]
    }
  ],
  "scales": [
    {
      "name": "g",
      "domain": [0, 1],
      "range": [_scales_range1, 10]
    },
    {
      "name": "y",
      "type": "ordinal",
      "range": "height",
      "reverse": true,
      "domain": {"data": "pop", "field": "data.age"}
    },
    {
      "name": "c",
      "type": "ordinal",
      "domain": [1, 2],
      "range": ["#1f77b4", "#e377c2"]
    }
  ],
  "marks": [
    {
      "type": "text",
      "interactive": false,
      "from": {
        "data": "pop",
        "transform": [{"type":"unique", "field":"data.age", "as":"age"}]
      },
      "properties": {
        "enter": {
          "x": {"value": _marks_properties_x},
          "y": {"scale": "y", "field": "age", "offset": 11},
          "text": {"field": "age"},
          "baseline": {"value": "middle"},
          "align": {"value": "center"},
          "fill": {"value": "steelblue"}
        }
      }
    },
    {
      "type": "group",
      "from": {
        "data": "pop",
        "transform": [
          {"type":"facet", "keys":["data.sex"]}
        ]
      },
      "properties": {
        "update": {
          "x": {"scale": "g", "field": "index"},
          "y": {"value": 0},
          "width": {"value": _marks_properties_width},
          "height": {"group": "height"}
        }
      },
      "scales": [
        {
          "name": "x",
          "type": "linear",
          "range": "width",
          "reverse": {"field": "index"},
          "nice": true,
          "domain": {"data": "pop", "field": "data.people"}
        }
      ],
      "axes": [
         {
           "type": "x",
           "scale": "x",
           "format": "s",
           "properties": {
             "ticks": {
               "stroke": {"value": "steelblue"}
             },
             "majorTicks": {
               "strokeWidth": {"value": 2}
             },
             "labels": {
               "fill": {"value": "steelblue"},
               "angle": {"value": 50},
               "align": {"value": "left"},
               "baseline": {"value": "middle"},
             },
             "title": {
               "fontSize": {"value": 16}
             },
             "axis": {
               "stroke": {"value": "#333"},
               "strokeWidth": {"value": 1.5}
             }
           }
         }
      ],
      "marks": [
        {
          "type": "rect",
          "properties": {
            "enter": {
              "x": {"scale": "x", "field": "data.people"},
              "x2": {"scale": "x", "value": 0},
              "y": {"scale": "y", "field": "data.age"},
              "height": {"scale": "y", "band": true, "offset": -1},
              "fillOpacity": {"value": 0.6},
              "fill": {"scale": "c", "field": "data.sex"}
            }
          }
        }
      ]
    }
  ]
};

// pass in the spec to vega to render 
 vg.parse.spec(spec, function(chart) { chart({el:"#vis"}).update(); });

});

At this point, you should see the visualization render in the editor as follows. If it does not, you may need to make a minor change to the HTML code (e.g. add and then delete a space) for your changes to be picked up.

You can now save and add the visualization to your dashboard.

In summary, we used Vega, a powerful visualization library built on D3, to extend Klipfolio’s visualization capability and added a custom visualization based on Vega samples. This shows the flexibility and extensibility of Klipfolio. In addition, it shows how visualization experts can use Klipfolio as a platform to connect to various data sources and use their favourite JavaScript libraries to render the custom visualizations and controls that they like to use in their Klips.

Note: All Vega visualization library related specs, code and samples are under Vega license: https://github.com/vega/vega/blob/master/LICENSE

Copyright (c) 2013, Trifacta Inc.