Tag: dataviz

Download my travel-time map

I am a bit ner­vous about doing this, but since sev­er­al peo­ple asked, here goes: You can now down­load the trav­el-time map of the Nether­lands I made in Pro­cess­ing. I have export­ed appli­ca­tions for Lin­ux, Mac OS X and Win­dows. Each down­load includes the source files, but not the data file. For that, you will need to head to Alper’s site (he’s the guy who pulled the data from 9292 and ANWB). I hope you’ll enjoy play­ing around with this, or learn some­thing from the way it was put together.

Some notes, in no par­tic­u­lar order:

  • Please remem­ber I am not a pro­gram­mer. The vast major­i­ty of this sketch was put togeth­er from bits and pieces of code I found in books and online. I have tried to cred­it all the sources in the code. The full write-up I post­ed ear­li­er should point you to all the sources too. In short; all the good bits are by oth­er peo­ple, the bad code is mine. But who cares, it’s the end-result that counts (at least for me).
  • Relat­ed to the pre­vi­ous point is the fact that I can­not fig­ure out under which license (if any) to release this. So the usu­al CC by-nc-sa license applies, as far as I’m concerned.
  • If this breaks your com­put­er, offends you, makes you cry, or eats your kit­tens, do not come knock­ing. This is pro­vid­ed as is, no war­ranties what­so­ev­er, etc.
  • Why am I ner­vous? Prob­a­bly because for me the point of the whole exer­cise was the process, not the outcome.
  • I can’t think of any­thing else. Have fun.

The making of a travel-time map of the Netherlands

Sub­scribers to my Flickr stream have prob­a­bly noticed a num­ber of images of some kind of map flow­ing past late­ly. They were the result of me track­ing my progress on a pet project. I have more or less fin­ished work on it this week, so I thought I’d detail what I did over here.

Background

Fol­low­ing my Twit­ter dataviz sketch­es, I thought I’d take anoth­er stab at pro­to­typ­ing with Pro­cess­ing. On the one hand I want­ed to increase my famil­iar­i­ty with the envi­ron­ment. On the oth­er, I con­tin­ued to be fas­ci­nat­ed with data-visu­al­iza­tion, so I want­ed to do anoth­er design exer­cise in this domain. I was par­tic­u­lar­ly inter­est­ed in cre­at­ing dis­plays that assist in deci­sion mak­ing and present data in a way that allows peo­ple to ‘play’ with it — explore it and learn from it.

The seed for this thing was plant­ed when I saw Sta­men’s work on the mySo­ci­ety trav­el-time maps. I thought the idea of visu­al­ly over­lay­ing two datasets and allow­ing the inter­sec­tion to be manip­u­lat­ed by peo­ple was sim­ple but pow­er­ful. But, at that time, I saw no way to ‘eas­i­ly’ try my hand at some­thing sim­i­lar. I had no ready access to any poten­tial­ly inter­est­ing data, and my scrap­ing skills are lim­it­ed at best.

Luck­i­ly, I was not the only one whose curios­i­ty was piqued. After see­ing Ben Cer­ve­ny demo­ing the same maps at The Web and Beyond 2008, Alper won­dered how hard it would be to cre­ate some­thing sim­i­lar for the Nether­lands. He pre­sent­ed a way to do it with freely avail­able tools and data (to an extent) in a work­shop at a local uncon­fer­ence.1

I did not attend the event, but after see­ing his blog post, I sent him an email and asked if he was will­ing to part with the data he had col­lect­ed from the Dutch pub­lic trans­port trav­el plan­ning site 9292. Alper being the nice guy he is, he soon emailed me a JSON con­tain­ing of the data. 

So that’s the back­ground. I had an exam­ple, I had some data, and I had a lit­tle expe­ri­ence with mak­ing things in Processing.

JSON

The first step was to read the data in the JSON file from Pro­cess­ing. I fol­lowed the instruc­tions on how to get the JSON library into Pro­cess­ing from Ben Fry’s book (pages 315–316). On the Pro­cess­ing boards, a cur­so­ry search unearthed some code exam­ples. After a lit­tle fid­dling, I got it to work and could print the data to Processing’s console.

Plotting

Next up was to start visu­al­iz­ing it. I used the exam­ples of scat­ter­plot maps in Visu­al­iz­ing Data as a start­ing point, and plugged in the JSON data. Pret­ty soon, I had a nice plot of the postal codes that actu­al­ly resem­bled the Netherlands.

Playing with some data Alper gave me

Coloring

From there, it was rather easy to show each postal code’s trav­el time.2 I sim­ply mapped trav­el times to a hue in the HSB spec­trum. The result nice­ly shows col­ored bands of trav­el-time regions and also allows you to pick out some inter­est­ing out­liers (such as Gronin­gen in the north).

Second pass

Selecting

At this point, I want­ed to be able to select trav­el-time ranges and hide postal codes out­side of that range. Ini­tial­ly, I used the key­board for input. This was OK for this stage of the project, but of course it would need to be replaced with some­thing more intu­itive lat­er on. In any case, I could high­light select­ed points and dim oth­ers, which increased the display’s explorabil­i­ty considerably.

Pass 3

Coloring, again

The HSB spec­trum is quick and easy way of get­ting access to a full a range of col­ors. It served me well in my Twit­ter visu­al­iza­tions. How­ev­er in this case it left some­thing to be desired, aes­thet­i­cal­ly speak­ing. Via Tom Car­den I found the won­der­ful cpt-city, which cat­a­logues gra­di­ents for car­tog­ra­phy and the like. Ini­tial­ly I strug­gled with ways to get these col­ors into Pro­cess­ing, but then it turned out you could eas­i­ly read out the col­ors of pix­els from images. This allowed me to cycle through many palettes just by adding the files to my Pro­cess­ing sketch. I dis­cov­ered that a palette with a clear divi­sion in the mid­dle was best, because that pro­vides you with an extra ref­er­ence point besides the begin­ning and end.

Playing with palettes (pass 4)

Selecting, again

I next turned to the inter­ac­tion bits. I knew I want­ed a so-called dual slid­er that would allow peo­ple to select the upper and low­er lim­it of trav­el time. In the Pro­cess­ing book, there is code for plen­ty of inter­face wid­gets, but sad­ly no dual slid­er. I looked around on the Pro­cess­ing board and could find none either, to my sur­prise. Even in the UI libraries (such as controlP5 and Inter­fas­cia) I could not locate one. 

So I decid­ed to low­er the bar and first include two hor­i­zon­tal slid­ers, one for the upper and one for the low­er lim­it. These I made using the code on pages 448–452 of the Pro­cess­ing book. Not per­fect, but an improve­ment over the key­board controls.

Pass 5 – some basic interactivity

Selecting, yet again

Next, I decid­ed I’d see if I could mod­i­fy the code of the hor­i­zon­tal scroll­bar so that I would end up with a dual slid­er. After some mess­ing about (which did increase my under­stand­ing of the orig­i­nal code con­sid­er­ably) I man­aged to get it to work. This was an unex­pect­ed suc­cess. I now had a decent dual slider.

A proper dual slider

Exploring

So far there was no way of telling which point cor­re­spond­ed to which postal code. So, I added a rollover that dis­played the postal code’s name and trav­el time. At this point it became clear the data wasn’t per­fect — some postal codes were erro­neous­ly geocod­ed by GeoN­ames. For instance, code 9843 (which is Gri­jpskerk, 199 min­utes to the Dam) was placed on the map as Ams­ter­dam Noord-Oost!

Rollovers

Adding more data

Around this point I vis­it­ed Alper in Delft and we dis­cussed adding a sec­ond dataset. Although hous­ing prices à la mySo­ci­ety would have been inter­est­ing, we decid­ed to take a dif­fer­ent route and add a sec­ond trav­el-time set for cars.3 My first step in inte­grat­ing this was to sim­ply gen­er­ate a map each for the pub­lic trans­port and car trav­el data and man­u­al­ly jux­ta­pose them. What I liked about this was that even though you know intu­itive­ly that trav­el­ing by car is faster, the two maps next to each oth­er pro­vide a dra­mat­ic visu­al con­fir­ma­tion of this piece of knowledge.

Compare: travel by public transport or car

Representing

Mov­ing ahead with the extra data, I start­ed to strug­gle with how to rep­re­sent both trav­el times. My first effort was to draw two sets of dots on top of each oth­er (one for car trav­el times and one for pub­lic trans­port) and col­or each accord­ing­ly. For each set I intro­duced a sep­a­rate slid­er. I wasn’t very sat­is­fied with the result of this. It did not help in under­stand­ing what was going on that much.

The gap

Showing differences

After dis­cus­sions with Alper and sev­er­al oth­er peo­ple, I decid­ed it would make more sense to show the dif­fer­ence between trav­el times. So I cal­cu­lat­ed the per­cent­age dif­fer­ence between pub­lic trans­port and car trav­el time for each postal code. This val­ue I mapped to a col­or. Here, a sim­ple gra­di­ent worked bet­ter than the palettes used ear­li­er for trav­el times.

I also dis­card­ed the idea of hav­ing two dual slid­ers and sim­ply went with one trav­el time selec­tor. Although more user-friend­ly, it cre­at­ed a new prob­lem: for some points both trav­el times would fall with­in the select­ed range, and for oth­ers one or the oth­er. So I need­ed an extra visu­al dimen­sion to show this. This turned out to be the great­est challenge.

After try­ing many approach­es, I even­tu­al­ly set­tled on using the shape of the point to show which trav­el times fell with­in the range. A small dot meant that only the pub­lic trans­port trav­el time is with­in the range, a donut means only the car trav­el time is select­ed, and a big dot rep­re­sents selec­tion of both times.

Return of the map

Final tweaks

Around this point I felt that it was time to wrap up. I had learnt about all I could from the exer­cise and any extra time spent on the project would result in mar­gin­al improve­ments at best. I added a leg­end for both the shapes and col­or, improved the leg­i­bil­i­ty of the rollover and increased the visu­al affor­dance of the slid­er, and that was it.

It's hard to stop tweaking

Thoughts

It is becom­ing appar­ent to me that the act of build­ing dis­plays like this is play­ful in its own way. Through sketch­ing in code, you can have some­thing like a con­ver­sa­tion with the data and get a sense of what’s there. Per­haps the end result is mere­ly a byprod­uct of this process?

I’m amazed at how far a novice pro­gram­mer like myself, with a dra­mat­ic lack of affin­i­ty for any­thing relat­ed to math­e­mat­ics or physics, can get by sim­ply mod­i­fy­ing, aug­ment­ing and com­bin­ing code that is already out there. I have no ambi­tion what­so­ev­er of becom­ing a pro­fes­sion­al devel­op­er of pro­duc­tion-qual­i­ty code. But build­ing a col­lec­tion bits and pieces of code that can do use­ful and inter­est­ing things seems like a good strat­e­gy for any design­er. I am learn­ing to trust my innate reluc­tance to code stuff from scratch. 

Also, isn’t it cool that it is becom­ing increas­ing­ly fea­si­ble for reg­u­lar cit­i­zens to start ana­lyz­ing data that is — or at least should be — pub­licly avail­able? Gov­ern­ment still has a long way to go. Why do we need to go through the painstak­ing process of scrap­ing this data from sources such as 9292 which for all intents and pur­pos­es is a pub­lic ser­vice?4

I will prob­a­bly make the final pro­to­type avail­able online at some point in the future. For now, if you have any ques­tions or com­ments I would love to hear them here, or via email.

Update: Alper has released a JSON file con­tain­ing all the data I used to make this. Go on and grab it, and make some dis­plays of your own!

And anoth­er update: I’ve decid­ed to make this appli­ca­tion avail­able for down­load, includ­ing source files.

  1. Those of you who under­stand Dutch might enjoy his walk­through on Vimeo. [↩]
  2. Inci­den­tal­ly, all trav­el times in this project were from the Dam in Ams­ter­dam to all the postal codes in NL. [↩]
  3. This we retrieved from the ANWB site. The time of day was set to 12:00 noon. [↩]
  4. Tools like Mech­a­nize make this eas­i­er, but still. [↩]

Playing With Complexity — slides and notes for my NLGD Festival of Games talk

When the NLGD Foun­da­tion invit­ed me to speak at their anu­al Fes­ti­val of Games I asked them what they would like me to dis­cuss. “Any­thing you like,” was what they said, essen­tial­ly. I decid­ed to sub­mit an abstract deal­ing with data visu­al­iza­tion. I had been pay­ing more and more atten­tion to this field, but was unsuc­cess­ful in relat­ing it the oth­er themes run­ning through my work, most notably play. So I thought I’d force myself to tack­le this issue by promis­ing to speak about it. Often a good strat­e­gy, I’ve found. If it worked out this time I leave for you to judge.

In brief, in the pre­sen­ta­tion I argue two things: one — that the more sophis­ti­cat­ed appli­ca­tions of inter­ac­tive data visu­al­iza­tion resem­ble games and toys in many ways, and two — that game design can con­tribute to the solu­tions to sev­er­al design issues I have detect­ed in the field of data visualization.

Below are the notes for the talk, slight­ly edit­ed, and with ref­er­ences includ­ed. The full deck of slides, which includes cred­its for all the images used, is up on SlideShare.

Hel­lo every­one, my name is Kars Alfrink. I am a Dutch inter­ac­tion design­er and I work free­lance. At the moment I work in Copen­hagen, but pret­ty soon I will be back here in Utrecht, my love­ly hometown. 

In my work I focus on three areas: mobil­i­ty, social inter­ac­tions, and play. Here is an exam­ple of my work: These are sto­ry­boards that explore pos­si­ble appli­ca­tions of mul­ti­touch tech­nol­o­gy in a gat­ed com­mu­ni­ty. Using these tech­nolo­gies I tried to com­pen­sate for the neg­a­tive effects a gat­ed com­mu­ni­ty has on the build-up of social cap­i­tal. I also tried to bal­ance ‘being-in-the-screen’ with ‘being-in-the-world’ — mul­ti­touch tech­nolo­gies tend to be very atten­tion-absorb­ing, but in built envi­ron­ments this is often not desir­able.1

I am not going to talk about mul­ti­touch though. Today’s top­ic is data visu­al­iza­tion and what oppor­tu­ni­ties there are for game design­ers in that field. My talk is rough­ly divid­ed in three parts. First, I will briefly describe what I think data visu­al­iza­tion is. Next, I will look at some appli­ca­tions beyond the very obvi­ous. Third and last, I will dis­cuss some design issues involved with data visu­al­iza­tion. For each of these issues, I will show how game design can contribute.

Right, let’s get started.

Con­tin­ue read­ing Play­ing With Com­plex­i­ty — slides and notes for my NLGD Fes­ti­val of Games talk

  1. For more back­ground on this project please see this old­er blog post. More exam­ples of my recent work can be found in my port­fo­lio. [↩]

Sketching in code — Twitter, Processing, dataviz

Sketch­ing is the defin­ing activ­i­ty of design writes Bux­ton and I tend to agree. The genius of his book is that he shows sketch­ing can take on many forms. It is not lim­it­ed to work­ing with pen­cils and paper. You can sketch in 3D using wood or clay. You can sketch in time using video, etc. Bux­ton does not include many exam­ples of sketch­ing in code, though.1 Pro­gram­ming in any lan­guage tends to be a hard earned skill, he writes, and once you have achieved suf­fi­cient mas­tery in it, you tend to try and solve all prob­lems with this one tool. Good design­ers can draw on a broad range of sketch­ing tech­niques and pick the right one for a giv­en sit­u­a­tion. This might include pro­gram­ming, but then it would need to con­form to Bux­ton’s defin­ing char­ac­ter­is­tics of sketch­ing: quick, inex­pen­sive, dis­pos­able, plen­ti­ful, offer min­i­mal detail, and sug­gest and explore rather than confirm.

I have been spend­ing some time broad­en­ing my sketch­ing reper­toire as a design­er. Before I start­ed inter­ac­tion design I was most­ly into visu­al arts (draw­ing, paint­ing, comics) so I am quite com­fort­able sketch­ing in 2D, using sto­ry­boards, etc.2 Sketch­ing in code though, has always been a weak spot. I have start­ed to rem­e­dy this by look­ing into Pro­cess­ing.

As an exer­cise I took some data from Twit­ter — one data set was the 20 most recent tweets and the oth­er my friends list — and decid­ed to see how quick I could cre­ate a few dif­fer­ent visu­al­iza­tions of that data. The end results were: 

Today's start - timeline

one: a time­line that spa­tial­ly plots the lat­est tweets from my friends — show­ing den­si­ty at cer­tain points in time; or how ‘noisy’ it is on my Twit­ter stream, 

Neatly centred now

two: an order­ing of friends based on the per­cent­age of their tweets that take up my time­line — who’s the loud­est of my friends?,

Bugfix – made a mistake in the tick mark labels

three: a graph of my friends list, with num­ber of friends and fol­low­ers on the axes and their total num­ber of tweets mapped to the size of each point.

The aim was not to come up with ground­break­ing solu­tions, or fin­ished appli­ca­tions.3 The goal was to exer­cise this idea of sketch­ing in code and use it to get a feel for a ‘com­plex’ data set, iter­at­ing on many dif­fer­ent ways to show the data before com­mit­ting to one solu­tion. In a real-world project I could see myself as a design­er do this and then col­lab­o­rate with a ‘prop­er’ pro­gram­mer to devel­op the final solu­tion (which would most like­ly be inter­ac­tive). I would choose dif­fer­ent sketch­ing tech­niques to design the inter­ac­tive aspects of a data-visu­al­iza­tion. For now I am con­tent with Pro­cess­ing sketch­es that sim­ply out­put a sta­t­ic image.

Tools & resources used were:

If as a design­er you are con­front­ed with a project that involves mak­ing a large amount of data under­stand­able, sketch­ing in code can help. You can use it to ‘talk’ to the data, and get a sense of its ‘shape’.

  1. There is one involv­ing Phid­gets and Max/MSP, a visu­al pro­gram­ming solu­tion for phys­i­cal com­put­ing. [↩]
  2. Some exam­ples include a mul­ti-touch project I did for InUse and a recent pre­sen­ta­tion at TWAB 2008. [↩]
  3. I don’t think any of these visu­al­iza­tions are very pro­found, they’re inter­est­ing at best. [↩]