The making of a travel-time map of the Netherlands

Subscribers to my Flickr stream have probably noticed a number of images of some kind of map flowing past lately. They were the result of me tracking my progress on a pet project. I have more or less finished work on it this week, so I thought I’d detail what I did over here.

Background

Following my Twitter dataviz sketches, I thought I’d take another stab at prototyping with Processing. On the one hand I wanted to increase my familiarity with the environment. On the other, I continued to be fascinated with data-visualization, so I wanted to do another design exercise in this domain. I was particularly interested in creating displays that assist in decision making and present data in a way that allows people to ‘play’ with it — explore it and learn from it.

The seed for this thing was planted when I saw Stamen’s work on the mySociety travel-time maps. I thought the idea of visually overlaying two datasets and allowing the intersection to be manipulated by people was simple but powerful. But, at that time, I saw no way to ‘easily’ try my hand at something similar. I had no ready access to any potentially interesting data, and my scraping skills are limited at best.

Luckily, I was not the only one whose curiosity was piqued. After seeing Ben Cerveny demoing the same maps at The Web and Beyond 2008, Alper wondered how hard it would be to create something similar for the Netherlands. He presented a way to do it with freely available tools and data (to an extent) in a workshop at a local unconference.1

I did not attend the event, but after seeing his blog post, I sent him an email and asked if he was willing to part with the data he had collected from the Dutch public transport travel planning site 9292. Alper being the nice guy he is, he soon emailed me a JSON containing of the data.

So that’s the background. I had an example, I had some data, and I had a little experience with making things in Processing.

JSON

The first step was to read the data in the JSON file from Processing. I followed the instructions on how to get the JSON library into Processing from Ben Fry’s book (pages 315-316). On the Processing boards, a cursory search unearthed some code examples. After a little fiddling, I got it to work and could print the data to Processing’s console.

Plotting

Next up was to start visualizing it. I used the examples of scatterplot maps in Visualizing Data as a starting point, and plugged in the JSON data. Pretty soon, I had a nice plot of the postal codes that actually resembled the Netherlands.

Playing with some data Alper gave me

Coloring

From there, it was rather easy to show each postal code’s travel time.2 I simply mapped travel times to a hue in the HSB spectrum. The result nicely shows colored bands of travel-time regions and also allows you to pick out some interesting outliers (such as Groningen in the north).

Second pass

Selecting

At this point, I wanted to be able to select travel-time ranges and hide postal codes outside of that range. Initially, I used the keyboard for input. This was OK for this stage of the project, but of course it would need to be replaced with something more intuitive later on. In any case, I could highlight selected points and dim others, which increased the display’s explorability considerably.

Pass 3

Coloring, again

The HSB spectrum is quick and easy way of getting access to a full a range of colors. It served me well in my Twitter visualizations. However in this case it left something to be desired, aesthetically speaking. Via Tom Carden I found the wonderful cpt-city, which catalogues gradients for cartography and the like. Initially I struggled with ways to get these colors into Processing, but then it turned out you could easily read out the colors of pixels from images. This allowed me to cycle through many palettes just by adding the files to my Processing sketch. I discovered that a palette with a clear division in the middle was best, because that provides you with an extra reference point besides the beginning and end.

Playing with palettes (pass 4)

Selecting, again

I next turned to the interaction bits. I knew I wanted a so-called dual slider that would allow people to select the upper and lower limit of travel time. In the Processing book, there is code for plenty of interface widgets, but sadly no dual slider. I looked around on the Processing board and could find none either, to my surprise. Even in the UI libraries (such as controlP5 and Interfascia) I could not locate one.

So I decided to lower the bar and first include two horizontal sliders, one for the upper and one for the lower limit. These I made using the code on pages 448-452 of the Processing book. Not perfect, but an improvement over the keyboard controls.

Pass 5 – some basic interactivity

Selecting, yet again

Next, I decided I’d see if I could modify the code of the horizontal scrollbar so that I would end up with a dual slider. After some messing about (which did increase my understanding of the original code considerably) I managed to get it to work. This was an unexpected success. I now had a decent dual slider.

A proper dual slider

Exploring

So far there was no way of telling which point corresponded to which postal code. So, I added a rollover that displayed the postal code’s name and travel time. At this point it became clear the data wasn’t perfect — some postal codes were erroneously geocoded by GeoNames. For instance, code 9843 (which is Grijpskerk, 199 minutes to the Dam) was placed on the map as Amsterdam Noord-Oost!

Rollovers

Adding more data

Around this point I visited Alper in Delft and we discussed adding a second dataset. Although housing prices à la mySociety would have been interesting, we decided to take a different route and add a second travel-time set for cars.3 My first step in integrating this was to simply generate a map each for the public transport and car travel data and manually juxtapose them. What I liked about this was that even though you know intuitively that traveling by car is faster, the two maps next to each other provide a dramatic visual confirmation of this piece of knowledge.

Compare: travel by public transport or car

Representing

Moving ahead with the extra data, I started to struggle with how to represent both travel times. My first effort was to draw two sets of dots on top of each other (one for car travel times and one for public transport) and color each accordingly. For each set I introduced a separate slider. I wasn’t very satisfied with the result of this. It did not help in understanding what was going on that much.

The gap

Showing differences

After discussions with Alper and several other people, I decided it would make more sense to show the difference between travel times. So I calculated the percentage difference between public transport and car travel time for each postal code. This value I mapped to a color. Here, a simple gradient worked better than the palettes used earlier for travel times.

I also discarded the idea of having two dual sliders and simply went with one travel time selector. Although more user-friendly, it created a new problem: for some points both travel times would fall within the selected range, and for others one or the other. So I needed an extra visual dimension to show this. This turned out to be the greatest challenge.

After trying many approaches, I eventually settled on using the shape of the point to show which travel times fell within the range. A small dot meant that only the public transport travel time is within the range, a donut means only the car travel time is selected, and a big dot represents selection of both times.

Return of the map

Final tweaks

Around this point I felt that it was time to wrap up. I had learnt about all I could from the exercise and any extra time spent on the project would result in marginal improvements at best. I added a legend for both the shapes and color, improved the legibility of the rollover and increased the visual affordance of the slider, and that was it.

It's hard to stop tweaking

Thoughts

It is becoming apparent to me that the act of building displays like this is playful in its own way. Through sketching in code, you can have something like a conversation with the data and get a sense of what’s there. Perhaps the end result is merely a byproduct of this process?

I’m amazed at how far a novice programmer like myself, with a dramatic lack of affinity for anything related to mathematics or physics, can get by simply modifying, augmenting and combining code that is already out there. I have no ambition whatsoever of becoming a professional developer of production-quality code. But building a collection bits and pieces of code that can do useful and interesting things seems like a good strategy for any designer. I am learning to trust my innate reluctance to code stuff from scratch.

Also, isn’t it cool that it is becoming increasingly feasible for regular citizens to start analyzing data that is — or at least should be — publicly available? Government still has a long way to go. Why do we need to go through the painstaking process of scraping this data from sources such as 9292 which for all intents and purposes is a public service?4

I will probably make the final prototype available online at some point in the future. For now, if you have any questions or comments I would love to hear them here, or via email.

Update: Alper has released a JSON file containing all the data I used to make this. Go on and grab it, and make some displays of your own!

And another update: I’ve decided to make this application available for download, including source files.

  1. Those of you who understand Dutch might enjoy his walkthrough on Vimeo. []
  2. Incidentally, all travel times in this project were from the Dam in Amsterdam to all the postal codes in NL. []
  3. This we retrieved from the ANWB site. The time of day was set to 12:00 noon. []
  4. Tools like Mechanize make this easier, but still. []

Pollinator — a casual game prototype made with Mobile Processing

I wrote a game about a bee and flowers today

Last sunday I sat down and coded a prototype of a casual game in Mobile Processing. I got the idea for it the evening before: You’re a bee who needs to collect as much honey as possible in his hive while at the same time keeping a flower-bed blooming by pollinating… Play it and let me know what your high score is in the comments!

Thinking and making

I’ve been looking for an excuse to get some experience with Processing (particularly the variant suitable for developing mobile stuff) for a while. I also felt I needed to get back into the making part of the field I’ve been thinking about so much lately: Game Design. I agree with Saffer, Webb and others – making is an important part of the design practice, it cannot be replaced by lots of thinking. The things learnt from engaging with the actual stuff things are made of (which in the case of digital games is code) aren’t gained in any other way and very valuable.

Get the game

I’ve uploaded the first version of the game here. You can play it in the emulator in your browser or if your phone runs Java midlets, download the file and play it like you’re supposed to: While out and about. The source code is provided as well, if you feel like looking at it.1

Pollinator 0.1

How to play

You’re the yellow oval. The orange triangle in the top left corner is your hive. Green squares are grass, brown squares are seeds, red squares are flowers and pink squares are pollinated flowers. The field is updated in columns from left to right (indicated by the yellow marker in the bottom). A seed will turn into a flower (in rare cases a pollinated flower). A flower will die, a pollinated flower will die and spread seeds to grass around it. Move your bee with the directional keys, use the centre key to grab nectar from a flower. You can cary a maximum of 100 nectar. Drop your nectar off at the hive (again using the centre key) to up your score. When you first grab nectar from a pollinated flower and subsequently from a normal flower, the latter is pollinated. Try to keep the flower-bed in bloom while at the same time racking up a high-score!

You’ll get 10 nectar from a flower (in bloom or not). Pollinating a flower costs 5 nectar. If you try to take nectar more than once from the same flower, you’ll loose 10 nectar.2

Improvements

Stuff not in here that I might put into a next version (whenever I get around to it):

  • Animation — I need to get my feet wet with some scripted animation. Thing is I’ve always sucked at this. For now it’s all tile-based stuff.
  • Better feedback — For instance show the points you earn near the bee and the hive. I think that’ll make the game a lot easier to understand and therefore more fun.
  • Menus, pause, game over — It’s a prototype, so you get dumped into the action right away. (The game starts on the first key you press.) And there’s no actual game over message, the field just turns green and you’re left to wonder what to do.
  • Balance — I’m not sure if the game like it stands is balanced right, I will need to play it a lot to figure that out. Also there’s probably a dominant strategy that’ll let you rack up points easily.

The aim was to create a relatively casual game experience that will almost allow you to zone out while playing. I think it is far too twitchy now, so perhaps I really should sit down and do a second version sometime soon.

Mobile Processing

I enjoy working with Mobile Processing. I like the way it allows you to program in a very naive way but if you like structure things in a more sophisticated fashion. It really does allow you to sketch in code, which is exactly what I need. The emphasis on just code also prevents me from fiddling around with animations, graphics and so on (like I would in Flash for instance.) Perhaps the only thing that would be nice is an editor that is a bit more full-featured.3 Perhaps I should grab an external editor next time?

Feedback

If you played the game and liked it (or thought it was too hard, boring or whatever) I’d love to get your feedback in the comments. Anyone else out there prototyping games in Processing? Or using it to teach game design? I’d be very interested to hear about it.

  1. Not that it’s particularly good, I’m an amateur coder at best. []
  2. I’m not sure this is the right kind of negative reinforcement. []
  3. The automatic code formatting refused to work for me, requiring me to spend a bit too much effort on formatting by hand. []