Tag: sketching

Writing for conversational user interfaces

Last year at Hub­bub we worked on two projects fea­tur­ing a con­ver­sa­tion­al user inter­face. I thought I would share a few notes on how we did the writ­ing for them. Because for con­ver­sa­tion­al user inter­faces a large part of the design is in the writing. 

At the moment, there aren’t real­ly that many tools well suit­ed for doing this. Twine comes to mind but it is real­ly more focused on pub­lish­ing as opposed to author­ing. So while we were work­ing on these projects we just grabbed what­ev­er we were famil­iar with and felt would get the job done. 

I actu­al­ly think there is an oppor­tu­ni­ty here. If this con­ver­sa­tion­al ui thing takes off design­ers would ben­e­fit a lot from bet­ter tools to sketch and pro­to­type them. After all this is the only way to fig­ure out if a con­ver­sa­tion­al user inter­face is suit­able for a par­tic­u­lar project. In the words of Bill Bux­ton:

“Every­thing is best for some­thing and worst for some­thing else.”

Okay so below are my notes. The two projects are KOKORO (a code­name) and Free Birds. We have yet to pub­lish exten­sive­ly on both, so a quick descrip­tion is in order.

KOKORO is a dig­i­tal coach for teenagers to help them man­age and improve their men­tal health. It is cur­rent­ly a pro­to­type mobile web app not pub­licly avail­able. (The engine we built to dri­ve it is avail­able on GitHub, though.)

Free Birds (Vri­je Vogels in Dutch) is a game about civ­il lib­er­ties for fam­i­lies vis­it­ing a war and resis­tance muse­um in the Nether­lands. It is a loca­tion-based iOS app cur­rent­ly avail­able on the Dutch app store and playable in Air­borne Muse­um Harten­stein in Oosterbeek.

For KOKORO we used Gingko to write the con­ver­sa­tion branch­es. This is good enough for a pro­to­type but it becomes unwieldy at scale. And any­way you don’t want to be lim­it­ed to a tree struc­ture. You want to at least be able to loop back to a par­ent branch, some­thing that isn’t sup­port­ed by Gingko. And maybe you don’t want to use the branch­ing pat­tern at all.

Free Birds’s sto­ry has a very lin­ear struc­ture. So in this case we just wrote our con­ver­sa­tions in Quip with some basic rules for for­mat­ting, not unlike a screenplay. 

In Free Birds play­er choic­es ‘colour’ the events that come imme­di­ate­ly after, but the path stays the same.

This approach was inspired by the Walk­ing Dead games. Those are super clever at giv­ing play­ers a sense of agency with­out the need for sprawl­ing sto­ry trees. I remem­ber see­ing the cre­ators present this strat­e­gy at PRACTICE and some­thing clicked for me. The impor­tant point is, choic­es don’t have to branch out to dif­fer­ent direc­tions to feel meaningful.

KOKORO’s choic­es did have to lead to dif­fer­ent paths so we had to build a tree struc­ture. But we also kept track of things a user says. This allows the app to “learn” about the user. Sub­se­quent seg­ments of the con­ver­sa­tion are adapt­ed based on this learn­ing. This allows for more flex­i­bil­i­ty and it scales bet­ter. A sec­tion of a con­ver­sa­tion has var­i­ous states between which we switch depend­ing on what a user has said in the past. 

We did some­thing sim­i­lar in Free Birds but used it to a far more lim­it­ed degree, real­ly just to once again colour cer­tain pieces of dia­logue. This is already enough to give a play­er a sense of agency.

As you can see, it’s all far from rock­et surgery but you can get sur­pris­ing­ly good results just by stick­ing to these sim­ple pat­terns. If I were to inves­ti­gate more advanced strate­gies I would look into NLP for input and pro­ce­dur­al gen­er­a­tion for out­put. Who knows, maybe I will get to work on a project involv­ing those things some time in the future.

Design without touching the surface

I am prepar­ing two class­es at the moment. One is an intro­duc­tion to user expe­ri­ence design, the oth­er to user inter­face design. I did not come up with this divi­sion, it was part of the assign­ment. I thought it was odd at first. I wasn’t sure where one dis­ci­pline ends and the oth­er begins. I still am not sure. But I made a prag­mat­ic deci­sion to have the UX class focus on the high lev­el process of design­ing (soft­ware) prod­ucts, and the UI class focus on the visu­al aspects of a product’s inter­face. The UI class deals with a product’s sur­face, form, and to some extent also its behav­iour, but on a micro lev­el. Where­as the UX class focus­es on behav­iour on the macro lev­el. Sim­ply speaking—the UX class is about behav­iour across screens, the UI class is about behav­iour with­in screens.

The solu­tion is work­able. But I am still not entire­ly com­fort­able with it. I am not com­fort­able with the idea of being able to prac­tice UX with­out ‘touch­ing the sur­face’, so to speak. And it seems my two class­es are advo­cat­ing this. Also, I am pret­ty sure this is every­day real­i­ty for many UX prac­ti­tion­ers. Notice I say “prac­ti­tion­er”, because I am not sure ‘design­er’ is the right term in these cas­es. To be hon­est I do not think you can prac­tice design with­out doing sketch­ing and pro­to­typ­ing of some sort. (See Bill Buxton’s ‘Sketch­ing User Expe­ri­ences’ for an expand­ed argu­ment on why this is.) And when it comes to design­ing soft­ware prod­ucts this means touch­ing the sur­face, the form.

Again, the real­i­ty is, ‘UX design­er’ and ‘UI design­er’ are com­mon terms now. Cer­tain­ly here in Sin­ga­pore peo­ple know they need both to make good prod­ucts. Some prac­ti­tion­ers say they do both, oth­ers one or the oth­er. The lat­ter appears to be the most com­mon and expect­ed case. (By the way, in Sin­ga­pore no-one I’ve met talks about inter­ac­tion design.)

My con­cern is that by encour­ag­ing the prac­tice of doing UX design with­out touch­ing the sur­face of a prod­uct, we get shit­ty designs. In a process where UX and UI are seen as sep­a­rate things the risk is one comes before the oth­er. The UX design­er draws the wire­frames, the UI design­er gets to turn them into pret­ty pic­tures, with no back-and-forth between the two. An iter­a­tive process can mit­i­gate some of the dam­age such an arti­fi­cial divi­sion of labour pro­duces, but I think we still start out on the wrong foot. I think a bet­ter prac­tice might entail includ­ing visu­al con­sid­er­a­tions from the very begin­ning of the design process (as we are sketching).

Two things I came across as I was prepar­ing these class­es are some­how in sup­port of this idea. Both result­ed from a call I did for resources on user inter­face design. I asked for books about visu­al aspects, but I got a lot more.

  1. In ‘Mag­ic Ink’ Bret Vic­tor writes about how the design of infor­ma­tion soft­ware is huge­ly indebt­ed to graph­ic design and more specif­i­cal­ly infor­ma­tion design in the tra­di­tion of Tufte. (He also men­tions indus­tri­al design as an equal­ly big prog­en­i­tor of inter­ac­tion design, but for soft­ware that is main­ly about manip­u­la­tion, not infor­ma­tion.) The arti­cle is big, but the start of it is actu­al­ly a pret­ty good if unortho­dox gen­er­al intro­duc­tion to inter­ac­tion design. For soft­ware that is about learn­ing through look­ing at infor­ma­tion Vic­tor says inter­ac­tion should be a last resort. So that leaves us with a task that is 80% if not more visu­al design. Touch­ing the sur­face. Which makes me think you might as well get to it as quick­ly as pos­si­ble and start sketch­ing and pro­to­typ­ing aimed not just at struc­ture and behav­iour but also form. (Hat tip to Pieter Diepen­maat for this one.)

  2. In ‘Jump­ing to the End’ Matt Jones ram­bles enter­tain­ing­ly about design fic­tion. He argues for pay­ing atten­tion to details and that a lot of the design he prac­tices is about ‘sig­na­ture moments’ aka micro-inter­ac­tions. So yeah, again, I can’t imag­ine design­ing these effec­tive­ly with­out doing sketch­ing and pro­to­typ­ing of the sort that includes the visu­al. And in fact Matt men­tions this more or less at one point, when he talks about the fact that his team’s deliv­er­ables at Google are almost all visu­al. They are high fideli­ty mock­ups, ani­ma­tions, videos, and so on. These then become the start­ing points for fur­ther devel­op­ment. (Hat tip to Alexan­der Zeh for this one.)

In sum­ma­ry, I think dis­tin­guish­ing UX design from UI design is non­sense. Because you can­not prac­tice design with­out sketch­ing and pro­to­typ­ing. And you can­not sketch and pro­to­type a soft­ware prod­uct with­out touch­ing its sur­face. In stead of tak­ing visu­al design for grant­ed, or talk­ing about it like it is some innate tal­ent, some kind of mag­i­cal skill some peo­ple are born with and oth­ers aren’t, user expe­ri­ence prac­ti­tion­ers should con­sid­er being less enam­oured with acquir­ing more skills from busi­ness, mar­ket­ing and engi­neer­ing and in stead prac­tice at the skills that define the fields user expe­ri­ence design is indebt­ed to the most: graph­ic design and indus­tri­al design. In oth­er words, you can’t do user expe­ri­ence design with­out touch­ing the surface.

On sketching

Catch­ing up with this slight­ly neglect­ed blog (it’s been 6 weeks since the last prop­er post). I’d like to start by telling you about a small thing I helped out with last week. Peter Boers­ma1 asked me to help out with one of his UX Cock­tail Hours. He was inspired by a recent IxDA Stu­dio event where, in stead of just chat­ting and drink­ing, design­ers actu­al­ly made stuff. (Gasp!) Peter want­ed to do a work­shop where atten­dees col­lab­o­rat­ed on sketch­ing a solu­tion to a giv­en design problem.

Part of my con­tri­bu­tion to the evening was a short pre­sen­ta­tion on the the­o­ry and prac­tice of sketch­ing. On the the­o­ry side, I ref­er­enced Bill Bux­ton’s list of qual­i­ties that define what a sketch is2, and empha­sized that this means a sketch can be done in any mate­r­i­al, not nec­es­sar­i­ly pen­cil and paper. Fur­ther­more I dis­cussed why sketch­ing works, using part of an arti­cle on embod­ied inter­ac­tion3. The main point there, as far as I am con­cerned is that when sketch­ing, as design­ers we have the ben­e­fit of ‘back­talk’ from our mate­ri­als, which can pro­vide us with new insights. I wrapped up the pre­sen­ta­tion with a case study of a project I did a while back with the Ams­ter­dam-based agency Info.nl4 for a social web start-up aimed at inde­pen­dent pro­fes­sion­als. In the project I went quite far in using sketch­es to not only devel­op the design, but also col­lab­o­ra­tive­ly con­struct it with the client, tech­nol­o­gists and others.

The whole thing was record­ed; you can find a video of the talk at Vimeo (thanks to Iskan­der and Alper). I also uploaded the slides to SlideShare (sans notes).

The sec­ond, and most inter­est­ing part of the evening was the work­shop itself. This was set up as fol­lows: Peter and I had pre­pared a fic­tion­al case, con­cern­ing peer-to-peer ener­gy. We used the Dutch com­pa­ny Qur­rent as an exam­ple, and asked the par­tic­i­pants to con­cep­tu­alise a way to encour­age use of Qurrent’s prod­uct range. The aim was to have peo­ple be more ener­gy effi­cient, and share sur­plus ener­gy they had gen­er­at­ed with the Qur­rent com­mu­ni­ty. The par­tic­i­pants split up in teams of around ten peo­ple each, and went to work. We gave them around one hour to design a solu­tion, using only pen and paper. After­wards, they pre­sent­ed the out­come of their work to each oth­er. For each team, we asked one par­tic­i­pant to cri­tique the work by men­tion­ing one thing he or she liked, and one thing that could be improved. The team was then giv­en a chance to reply. We also asked each team to briefly reflect on their work­ing process. At the end of the evening every­one was giv­en a chance to vote for their favourite design. The win­ner received a prize.5

Wrap­ping up, I think what I liked most about the work­shop was see­ing the many dif­fer­ent ways the teams approached the prob­lem (many of the par­tic­i­pants did not know each oth­er before­hand). Group dynam­ics var­ied huge­ly. I think it was valu­able to have each team share their expe­ri­ences on this front with each oth­er. One thing that I think we could improve was the case itself; next time I would like to pro­vide par­tic­i­pants with a more focused, more rich­ly detailed brief­ing for them to sink their teeth in. That might result in an assign­ment that is more about struc­ture and behav­iour (or even inter­face) and less about con­cepts and val­ues. It would be good to see how sketch­ing func­tions in such a context.

  1. the Nether­lands’ tallest IA and one of sev­er­al famous Peters who work in UX [↩]
  2. tak­en from his won­der­ful book Sketch­ing User Expe­ri­ences [↩]
  3. titled How Bod­ies Mat­ter (PDF) by Kle­mer and Takaya­ma [↩]
  4. who were also the hosts of this event [↩]
  5. I think it’s inter­est­ing to note that the win­ner had a remark­able con­cept, but in my opin­ion was not the best exam­ple of the pow­er of sketch­ing. Appar­ent­ly the audi­ence val­ued prod­uct over process. [↩]

The making of a travel-time map of the Netherlands

Sub­scribers to my Flickr stream have prob­a­bly noticed a num­ber of images of some kind of map flow­ing past late­ly. They were the result of me track­ing my progress on a pet project. I have more or less fin­ished work on it this week, so I thought I’d detail what I did over here.

Background

Fol­low­ing my Twit­ter dataviz sketch­es, I thought I’d take anoth­er stab at pro­to­typ­ing with Pro­cess­ing. On the one hand I want­ed to increase my famil­iar­i­ty with the envi­ron­ment. On the oth­er, I con­tin­ued to be fas­ci­nat­ed with data-visu­al­iza­tion, so I want­ed to do anoth­er design exer­cise in this domain. I was par­tic­u­lar­ly inter­est­ed in cre­at­ing dis­plays that assist in deci­sion mak­ing and present data in a way that allows peo­ple to ‘play’ with it — explore it and learn from it.

The seed for this thing was plant­ed when I saw Sta­men’s work on the mySo­ci­ety trav­el-time maps. I thought the idea of visu­al­ly over­lay­ing two datasets and allow­ing the inter­sec­tion to be manip­u­lat­ed by peo­ple was sim­ple but pow­er­ful. But, at that time, I saw no way to ‘eas­i­ly’ try my hand at some­thing sim­i­lar. I had no ready access to any poten­tial­ly inter­est­ing data, and my scrap­ing skills are lim­it­ed at best.

Luck­i­ly, I was not the only one whose curios­i­ty was piqued. After see­ing Ben Cer­ve­ny demo­ing the same maps at The Web and Beyond 2008, Alper won­dered how hard it would be to cre­ate some­thing sim­i­lar for the Nether­lands. He pre­sent­ed a way to do it with freely avail­able tools and data (to an extent) in a work­shop at a local uncon­fer­ence.1

I did not attend the event, but after see­ing his blog post, I sent him an email and asked if he was will­ing to part with the data he had col­lect­ed from the Dutch pub­lic trans­port trav­el plan­ning site 9292. Alper being the nice guy he is, he soon emailed me a JSON con­tain­ing of the data. 

So that’s the back­ground. I had an exam­ple, I had some data, and I had a lit­tle expe­ri­ence with mak­ing things in Processing.

JSON

The first step was to read the data in the JSON file from Pro­cess­ing. I fol­lowed the instruc­tions on how to get the JSON library into Pro­cess­ing from Ben Fry’s book (pages 315–316). On the Pro­cess­ing boards, a cur­so­ry search unearthed some code exam­ples. After a lit­tle fid­dling, I got it to work and could print the data to Processing’s console.

Plotting

Next up was to start visu­al­iz­ing it. I used the exam­ples of scat­ter­plot maps in Visu­al­iz­ing Data as a start­ing point, and plugged in the JSON data. Pret­ty soon, I had a nice plot of the postal codes that actu­al­ly resem­bled the Netherlands.

Playing with some data Alper gave me

Coloring

From there, it was rather easy to show each postal code’s trav­el time.2 I sim­ply mapped trav­el times to a hue in the HSB spec­trum. The result nice­ly shows col­ored bands of trav­el-time regions and also allows you to pick out some inter­est­ing out­liers (such as Gronin­gen in the north).

Second pass

Selecting

At this point, I want­ed to be able to select trav­el-time ranges and hide postal codes out­side of that range. Ini­tial­ly, I used the key­board for input. This was OK for this stage of the project, but of course it would need to be replaced with some­thing more intu­itive lat­er on. In any case, I could high­light select­ed points and dim oth­ers, which increased the display’s explorabil­i­ty considerably.

Pass 3

Coloring, again

The HSB spec­trum is quick and easy way of get­ting access to a full a range of col­ors. It served me well in my Twit­ter visu­al­iza­tions. How­ev­er in this case it left some­thing to be desired, aes­thet­i­cal­ly speak­ing. Via Tom Car­den I found the won­der­ful cpt-city, which cat­a­logues gra­di­ents for car­tog­ra­phy and the like. Ini­tial­ly I strug­gled with ways to get these col­ors into Pro­cess­ing, but then it turned out you could eas­i­ly read out the col­ors of pix­els from images. This allowed me to cycle through many palettes just by adding the files to my Pro­cess­ing sketch. I dis­cov­ered that a palette with a clear divi­sion in the mid­dle was best, because that pro­vides you with an extra ref­er­ence point besides the begin­ning and end.

Playing with palettes (pass 4)

Selecting, again

I next turned to the inter­ac­tion bits. I knew I want­ed a so-called dual slid­er that would allow peo­ple to select the upper and low­er lim­it of trav­el time. In the Pro­cess­ing book, there is code for plen­ty of inter­face wid­gets, but sad­ly no dual slid­er. I looked around on the Pro­cess­ing board and could find none either, to my sur­prise. Even in the UI libraries (such as controlP5 and Inter­fas­cia) I could not locate one. 

So I decid­ed to low­er the bar and first include two hor­i­zon­tal slid­ers, one for the upper and one for the low­er lim­it. These I made using the code on pages 448–452 of the Pro­cess­ing book. Not per­fect, but an improve­ment over the key­board controls.

Pass 5 – some basic interactivity

Selecting, yet again

Next, I decid­ed I’d see if I could mod­i­fy the code of the hor­i­zon­tal scroll­bar so that I would end up with a dual slid­er. After some mess­ing about (which did increase my under­stand­ing of the orig­i­nal code con­sid­er­ably) I man­aged to get it to work. This was an unex­pect­ed suc­cess. I now had a decent dual slider.

A proper dual slider

Exploring

So far there was no way of telling which point cor­re­spond­ed to which postal code. So, I added a rollover that dis­played the postal code’s name and trav­el time. At this point it became clear the data wasn’t per­fect — some postal codes were erro­neous­ly geocod­ed by GeoN­ames. For instance, code 9843 (which is Gri­jpskerk, 199 min­utes to the Dam) was placed on the map as Ams­ter­dam Noord-Oost!

Rollovers

Adding more data

Around this point I vis­it­ed Alper in Delft and we dis­cussed adding a sec­ond dataset. Although hous­ing prices à la mySo­ci­ety would have been inter­est­ing, we decid­ed to take a dif­fer­ent route and add a sec­ond trav­el-time set for cars.3 My first step in inte­grat­ing this was to sim­ply gen­er­ate a map each for the pub­lic trans­port and car trav­el data and man­u­al­ly jux­ta­pose them. What I liked about this was that even though you know intu­itive­ly that trav­el­ing by car is faster, the two maps next to each oth­er pro­vide a dra­mat­ic visu­al con­fir­ma­tion of this piece of knowledge.

Compare: travel by public transport or car

Representing

Mov­ing ahead with the extra data, I start­ed to strug­gle with how to rep­re­sent both trav­el times. My first effort was to draw two sets of dots on top of each oth­er (one for car trav­el times and one for pub­lic trans­port) and col­or each accord­ing­ly. For each set I intro­duced a sep­a­rate slid­er. I wasn’t very sat­is­fied with the result of this. It did not help in under­stand­ing what was going on that much.

The gap

Showing differences

After dis­cus­sions with Alper and sev­er­al oth­er peo­ple, I decid­ed it would make more sense to show the dif­fer­ence between trav­el times. So I cal­cu­lat­ed the per­cent­age dif­fer­ence between pub­lic trans­port and car trav­el time for each postal code. This val­ue I mapped to a col­or. Here, a sim­ple gra­di­ent worked bet­ter than the palettes used ear­li­er for trav­el times.

I also dis­card­ed the idea of hav­ing two dual slid­ers and sim­ply went with one trav­el time selec­tor. Although more user-friend­ly, it cre­at­ed a new prob­lem: for some points both trav­el times would fall with­in the select­ed range, and for oth­ers one or the oth­er. So I need­ed an extra visu­al dimen­sion to show this. This turned out to be the great­est challenge.

After try­ing many approach­es, I even­tu­al­ly set­tled on using the shape of the point to show which trav­el times fell with­in the range. A small dot meant that only the pub­lic trans­port trav­el time is with­in the range, a donut means only the car trav­el time is select­ed, and a big dot rep­re­sents selec­tion of both times.

Return of the map

Final tweaks

Around this point I felt that it was time to wrap up. I had learnt about all I could from the exer­cise and any extra time spent on the project would result in mar­gin­al improve­ments at best. I added a leg­end for both the shapes and col­or, improved the leg­i­bil­i­ty of the rollover and increased the visu­al affor­dance of the slid­er, and that was it.

It's hard to stop tweaking

Thoughts

It is becom­ing appar­ent to me that the act of build­ing dis­plays like this is play­ful in its own way. Through sketch­ing in code, you can have some­thing like a con­ver­sa­tion with the data and get a sense of what’s there. Per­haps the end result is mere­ly a byprod­uct of this process?

I’m amazed at how far a novice pro­gram­mer like myself, with a dra­mat­ic lack of affin­i­ty for any­thing relat­ed to math­e­mat­ics or physics, can get by sim­ply mod­i­fy­ing, aug­ment­ing and com­bin­ing code that is already out there. I have no ambi­tion what­so­ev­er of becom­ing a pro­fes­sion­al devel­op­er of pro­duc­tion-qual­i­ty code. But build­ing a col­lec­tion bits and pieces of code that can do use­ful and inter­est­ing things seems like a good strat­e­gy for any design­er. I am learn­ing to trust my innate reluc­tance to code stuff from scratch. 

Also, isn’t it cool that it is becom­ing increas­ing­ly fea­si­ble for reg­u­lar cit­i­zens to start ana­lyz­ing data that is — or at least should be — pub­licly avail­able? Gov­ern­ment still has a long way to go. Why do we need to go through the painstak­ing process of scrap­ing this data from sources such as 9292 which for all intents and pur­pos­es is a pub­lic ser­vice?4

I will prob­a­bly make the final pro­to­type avail­able online at some point in the future. For now, if you have any ques­tions or com­ments I would love to hear them here, or via email.

Update: Alper has released a JSON file con­tain­ing all the data I used to make this. Go on and grab it, and make some dis­plays of your own!

And anoth­er update: I’ve decid­ed to make this appli­ca­tion avail­able for down­load, includ­ing source files.

  1. Those of you who under­stand Dutch might enjoy his walk­through on Vimeo. [↩]
  2. Inci­den­tal­ly, all trav­el times in this project were from the Dam in Ams­ter­dam to all the postal codes in NL. [↩]
  3. This we retrieved from the ANWB site. The time of day was set to 12:00 noon. [↩]
  4. Tools like Mech­a­nize make this eas­i­er, but still. [↩]

Sketching in code — Twitter, Processing, dataviz

Sketch­ing is the defin­ing activ­i­ty of design writes Bux­ton and I tend to agree. The genius of his book is that he shows sketch­ing can take on many forms. It is not lim­it­ed to work­ing with pen­cils and paper. You can sketch in 3D using wood or clay. You can sketch in time using video, etc. Bux­ton does not include many exam­ples of sketch­ing in code, though.1 Pro­gram­ming in any lan­guage tends to be a hard earned skill, he writes, and once you have achieved suf­fi­cient mas­tery in it, you tend to try and solve all prob­lems with this one tool. Good design­ers can draw on a broad range of sketch­ing tech­niques and pick the right one for a giv­en sit­u­a­tion. This might include pro­gram­ming, but then it would need to con­form to Bux­ton’s defin­ing char­ac­ter­is­tics of sketch­ing: quick, inex­pen­sive, dis­pos­able, plen­ti­ful, offer min­i­mal detail, and sug­gest and explore rather than confirm.

I have been spend­ing some time broad­en­ing my sketch­ing reper­toire as a design­er. Before I start­ed inter­ac­tion design I was most­ly into visu­al arts (draw­ing, paint­ing, comics) so I am quite com­fort­able sketch­ing in 2D, using sto­ry­boards, etc.2 Sketch­ing in code though, has always been a weak spot. I have start­ed to rem­e­dy this by look­ing into Pro­cess­ing.

As an exer­cise I took some data from Twit­ter — one data set was the 20 most recent tweets and the oth­er my friends list — and decid­ed to see how quick I could cre­ate a few dif­fer­ent visu­al­iza­tions of that data. The end results were: 

Today's start - timeline

one: a time­line that spa­tial­ly plots the lat­est tweets from my friends — show­ing den­si­ty at cer­tain points in time; or how ‘noisy’ it is on my Twit­ter stream, 

Neatly centred now

two: an order­ing of friends based on the per­cent­age of their tweets that take up my time­line — who’s the loud­est of my friends?,

Bugfix – made a mistake in the tick mark labels

three: a graph of my friends list, with num­ber of friends and fol­low­ers on the axes and their total num­ber of tweets mapped to the size of each point.

The aim was not to come up with ground­break­ing solu­tions, or fin­ished appli­ca­tions.3 The goal was to exer­cise this idea of sketch­ing in code and use it to get a feel for a ‘com­plex’ data set, iter­at­ing on many dif­fer­ent ways to show the data before com­mit­ting to one solu­tion. In a real-world project I could see myself as a design­er do this and then col­lab­o­rate with a ‘prop­er’ pro­gram­mer to devel­op the final solu­tion (which would most like­ly be inter­ac­tive). I would choose dif­fer­ent sketch­ing tech­niques to design the inter­ac­tive aspects of a data-visu­al­iza­tion. For now I am con­tent with Pro­cess­ing sketch­es that sim­ply out­put a sta­t­ic image.

Tools & resources used were:

If as a design­er you are con­front­ed with a project that involves mak­ing a large amount of data under­stand­able, sketch­ing in code can help. You can use it to ‘talk’ to the data, and get a sense of its ‘shape’.

  1. There is one involv­ing Phid­gets and Max/MSP, a visu­al pro­gram­ming solu­tion for phys­i­cal com­put­ing. [↩]
  2. Some exam­ples include a mul­ti-touch project I did for InUse and a recent pre­sen­ta­tion at TWAB 2008. [↩]
  3. I don’t think any of these visu­al­iza­tions are very pro­found, they’re inter­est­ing at best. [↩]

Sketching the experience of toys

A frame from the Sketch-A-Move video

“Play is the high­est form of research.”

—Albert Ein­stein1

That’s what I always say when I’m play­ing games, too. 

I real­ly liked Bill Bux­ton’s book Sketch­ing User Expe­ri­ences. I like it because Bux­ton defends design as a legit­i­mate pro­fes­sion sep­a­rate from oth­er disciplines—such as engineering—while at the same time show­ing that design­ers (no mat­ter how bril­liant) can only suc­ceed in the right ecosys­tem. I also like the fact that he iden­ti­fies sketch­ing (in its many forms) as a defin­ing activ­i­ty of the design pro­fes­sion. The many exam­ples he shows are very inspiring.

One in par­tic­u­lar stood out for me, which is the project Sketch-A-Move by Anab Jain and Louise Klink­er done in 2004 at the RCA in Lon­don. The image above is tak­en from the video they cre­at­ed to illus­trate their con­cept. It’s about cars auto-mag­i­cal­ly dri­ving along tra­jec­to­ries that you draw on their roof. You can watch the video over at the book’s com­pan­ion web­site. It’s a very good exam­ple of visu­al­iz­ing an inter­ac­tive prod­uct in a very com­pelling way with­out actu­al­ly build­ing it. This was all faked, if you want to find out how, buy the book.2

The great thing about the video is not only does it illus­trate how the con­cept works, it also gives you a sense of what the expe­ri­ence of using it would be like. As Bux­ton writes:3

“You see, toys are not about toys. Toys are about play and the expe­ri­ence of fun that they help fos­ter. And that is what this video real­ly shows. That, and the pow­er of video to go beyond sim­ply doc­u­ment­ing a con­cept to com­mu­ni­cat­ing some­thing about expe­ri­ence in a very vis­cer­al way.”

Not only does it com­mu­ni­cate the fun you would have play­ing with it, I think this way of sketch­ing actu­al­ly helped the design­ers get a sense them­selves of wether what they had come up with was fun. You can tell they are actu­al­ly play­ing, being sur­prised by unex­pect­ed out­comes, etc.

The role of play in design is dis­cussed by Bux­ton as well, although he admits he need­ed to be prompt­ed by a friend of his: Alex Manu, a teacher at OCAD in Toron­to writes in an email to Bux­ton:4

“With­out play imag­i­na­tion dies.”

“Chal­lenges to imag­i­na­tion are the keys to cre­ativ­i­ty. The skill of retriev­ing imag­i­na­tion resides in the mas­tery of play. The ecol­o­gy of play is the ecol­o­gy of the pos­si­ble. Pos­si­bil­i­ty incu­bates creativity.”

Which Bux­ton rephras­es in one of his own per­son­al mantras:5

“These things are far too impor­tant to take seriously.”

All of which has made me real­ize that if I’m not hav­ing some sort of fun while design­ing, I’m doing some­thing wrong. It might be worth con­sid­er­ing switch­ing from one sketch­ing tech­nique to anoth­er. It might help me get a dif­fer­ent per­spec­tive on the prob­lem, and yield new pos­si­ble solu­tions. Bux­ton’s book is a trea­sure trove of sketch­ing tech­niques. There is no excuse for being bored while design­ing anymore.

  1. Sketch­ing User Expe­ri­ences p.349 [↩]
  2. No, I’m not get­ting a com­mis­sion to say that. [↩]
  3. Ibid. 1, at 325 [↩]
  4. Ibid., at 263 [↩]
  5. Ibid. [↩]

Storyboarding multi-touch interactions

I think it was around half a year ago that I wrote “UX design­ers should get into every­ware”. Back then I did not expect to be part of a ubi­comp project any­time soon. But here I am now, writ­ing about work I did in the area of mul­ti-touch interfaces. 

Background

The peo­ple at InUse (Swe­den’s pre­mier inter­ac­tion design con­sul­tan­cy firm) asked me to assist them with visu­al­is­ing poten­tial uses of mul­ti-touch tech­nol­o­gy in the con­text of a gat­ed com­mu­ni­ty. That’s right—an actu­al real-world phys­i­cal real-estate devel­op­ment project. How cool is that?

InUse storyboard 1

This res­i­den­tial com­mu­ni­ty is aimed at well-to-do seniors. As with most gat­ed com­mu­ni­ties, it offers them con­ve­nience, secu­ri­ty and pres­tige. You might shud­der at the thought of liv­ing in one of these places (I know I have my reser­va­tions) but there’s not much use in judg­ing peo­ple want­i­ng to do so. Planned ameni­ties include sports facil­i­ties, fine din­ing, onsite med­ical care, a cin­e­ma and on and on…

Social capital

One of the known issues with these ‘com­mu­ni­ties’ is that there’s not much evi­dence of social cap­i­tal being high­er there than in any reg­u­lar neigh­bour­hood. In fact some have argued that the glob­al trend of gat­ed com­mu­ni­ties is detri­men­tal to the build-up of social cap­i­tal in their sur­round­ings. They throw up phys­i­cal bar­ri­ers that pre­vent free inter­ac­tion of peo­ple. These are some of the things I tried to address: To see if we could sup­port the emer­gence of com­mu­ni­ty inside the res­i­den­cy using social tools while at the same coun­ter­act­ing phys­i­cal bar­ri­ers to the out­side world with “vir­tu­al inroads” that allow for free inter­ac­tion between res­i­dents and peo­ple in the periphery.

Being in the world

Anoth­er con­cern I tried to address is the dif­fer­ent ways mul­ti-touch inter­faces can play a role in the lives of peo­ple. Recent­ly Matt Jones addressed this in a post on the iPhone and Noki­a’s upcom­ing mul­ti-touch phones. In a com­mu­ni­ty like the one I was design­ing for, the worst thing I could do is make every instance of mul­ti-touch tech­nol­o­gy an atten­tion-grab­bing pres­ence demand­ing full immer­sion from its user. In many cas­es ‘my’ users would be bet­ter served with them behav­ing in an unob­tru­sive way, allow­ing almost uncon­scious use. In oth­er words: I tried to bal­ance being in the world with being in the screen—apply­ing each par­a­digm based on how appro­pri­ate it was giv­en the user’s con­text. (After all, some­times peo­ple want or even need to be immersed.)

Process

InUse had already pre­pared sev­er­al per­sonas rep­re­sen­ta­tive of the future res­i­dents of the com­mu­ni­ty. We went through those togeth­er and exam­ined each for sce­nar­ios that would make good can­di­dates for sto­ry­board­ing. We want­ed to come up with a range of sce­nar­ios that not only showed how these per­sonas could be sup­port­ed with mul­ti-touch inter­faces, but also illus­trate the dif­fer­ent spaces the inter­ac­tions could take place in (pri­vate, semi­pri­vate and pub­lic) and the scales at which the tech­nol­o­gy can oper­ate (from small key-like tokens to full wall-screens). 

InUse storyboard 2

I draft­ed each sce­nario as a tex­tu­al out­line and sketched the poten­tial sto­ry­boards on thumb­nail size. We went over those in a sec­ond work­shop and refined them—making adjust­ments to bet­ter cov­er the con­cerns out­lined above as well as improv­ing clar­i­ty. We want­ed to end up with a set of sto­ry­boards that could be used in a pre­sen­ta­tion for the client (the real-estate devel­op­ment firm) so we need­ed to bal­ance user goals with busi­ness objec­tives. To that end we thought about and includ­ed exam­ples of API-like inte­gra­tion of the plat­form with ser­vice providers in the periph­ery of the com­mu­ni­ty. We also tried to cre­ate self-ser­vice expe­ri­ences that would feel like being wait­ed on by a per­son­al butler.

Outcome

I end­ed up draw­ing three sce­nar­ios of around 9 pan­els each, digi­tis­ing and clean­ing them up on my Mac. Each sce­nario intro­duces a per­sona, the phys­i­cal con­text of the inter­ac­tion and the per­son­a’s moti­va­tion that dri­ves him to engage with the tech­nol­o­gy. The inter­ac­tions visu­alised are a mix of ges­tures and engage­ments with mul­ti-touch screens of dif­fer­ent sizes. Usu­al­ly the per­sona is sup­port­ed in some way by a social dimension—fostering serendip­i­ty and emer­gence of real relations.

InUse storyboard 3

All in all I have to say I am pret­ty pleased with the result of this short but sweet engage­ment. Col­lab­o­ra­tion with the peo­ple of InUse was smooth (as was expect­ed, since we are very much the same kind of ani­mal) and there will be fol­low-up work­shops with the client. It remains to be seen how much of this mul­ti-touch stuff will find its way into the final gat­ed com­mu­ni­ty. That as always will depend on what makes busi­ness sense. 

In any case it was a great oppor­tu­ni­ty for me to immerse myself ful­ly in the inter­re­lat­ed top­ics of mul­ti-touch, ges­ture, urban­ism and social­i­ty. And final­ly, it gave me the per­fect excuse to sit down and do lots and lots of drawings.

Pollinator — a casual game prototype made with Mobile Processing

I wrote a game about a bee and flowers today

Last sun­day I sat down and cod­ed a pro­to­type of a casu­al game in Mobile Pro­cess­ing. I got the idea for it the evening before: You’re a bee who needs to col­lect as much hon­ey as pos­si­ble in his hive while at the same time keep­ing a flower-bed bloom­ing by pol­li­nat­ing… Play it and let me know what your high score is in the comments!

Thinking and making

I’ve been look­ing for an excuse to get some expe­ri­ence with Pro­cess­ing (par­tic­u­lar­ly the vari­ant suit­able for devel­op­ing mobile stuff) for a while. I also felt I need­ed to get back into the mak­ing part of the field I’ve been think­ing about so much late­ly: Game Design. I agree with Saf­fer, Webb and oth­ers — mak­ing is an impor­tant part of the design prac­tice, it can­not be replaced by lots of think­ing. The things learnt from engag­ing with the actu­al stuff things are made of (which in the case of dig­i­tal games is code) aren’t gained in any oth­er way and very valuable.

Get the game

I’ve uploaded the first ver­sion of the game here. You can play it in the emu­la­tor in your brows­er or if your phone runs Java midlets, down­load the file and play it like you’re sup­posed to: While out and about. The source code is pro­vid­ed as well, if you feel like look­ing at it.1

Pollinator 0.1

How to play

You’re the yel­low oval. The orange tri­an­gle in the top left cor­ner is your hive. Green squares are grass, brown squares are seeds, red squares are flow­ers and pink squares are pol­li­nat­ed flow­ers. The field is updat­ed in columns from left to right (indi­cat­ed by the yel­low mark­er in the bot­tom). A seed will turn into a flower (in rare cas­es a pol­li­nat­ed flower). A flower will die, a pol­li­nat­ed flower will die and spread seeds to grass around it. Move your bee with the direc­tion­al keys, use the cen­tre key to grab nec­tar from a flower. You can cary a max­i­mum of 100 nec­tar. Drop your nec­tar off at the hive (again using the cen­tre key) to up your score. When you first grab nec­tar from a pol­li­nat­ed flower and sub­se­quent­ly from a nor­mal flower, the lat­ter is pol­li­nat­ed. Try to keep the flower-bed in bloom while at the same time rack­ing up a high-score!

You’ll get 10 nec­tar from a flower (in bloom or not). Pol­li­nat­ing a flower costs 5 nec­tar. If you try to take nec­tar more than once from the same flower, you’ll loose 10 nec­tar.2

Improvements

Stuff not in here that I might put into a next ver­sion (when­ev­er I get around to it):

  • Ani­ma­tion — I need to get my feet wet with some script­ed ani­ma­tion. Thing is I’ve always sucked at this. For now it’s all tile-based stuff.
  • Bet­ter feed­back — For instance show the points you earn near the bee and the hive. I think that’ll make the game a lot eas­i­er to under­stand and there­fore more fun.
  • Menus, pause, game over — It’s a pro­to­type, so you get dumped into the action right away. (The game starts on the first key you press.) And there’s no actu­al game over mes­sage, the field just turns green and you’re left to won­der what to do.
  • Bal­ance — I’m not sure if the game like it stands is bal­anced right, I will need to play it a lot to fig­ure that out. Also there’s prob­a­bly a dom­i­nant strat­e­gy that’ll let you rack up points easily.

The aim was to cre­ate a rel­a­tive­ly casu­al game expe­ri­ence that will almost allow you to zone out while play­ing. I think it is far too twitchy now, so per­haps I real­ly should sit down and do a sec­ond ver­sion some­time soon.

Mobile Processing

I enjoy work­ing with Mobile Pro­cess­ing. I like the way it allows you to pro­gram in a very naive way but if you like struc­ture things in a more sophis­ti­cat­ed fash­ion. It real­ly does allow you to sketch in code, which is exact­ly what I need. The empha­sis on just code also pre­vents me from fid­dling around with ani­ma­tions, graph­ics and so on (like I would in Flash for instance.) Per­haps the only thing that would be nice is an edi­tor that is a bit more full-fea­tured.3 Per­haps I should grab an exter­nal edi­tor next time?

Feedback

If you played the game and liked it (or thought it was too hard, bor­ing or what­ev­er) I’d love to get your feed­back in the com­ments. Any­one else out there pro­to­typ­ing games in Pro­cess­ing? Or using it to teach game design? I’d be very inter­est­ed to hear about it.

  1. Not that it’s par­tic­u­lar­ly good, I’m an ama­teur coder at best. [↩]
  2. I’m not sure this is the right kind of neg­a­tive rein­force­ment. [↩]
  3. The auto­mat­ic code for­mat­ting refused to work for me, requir­ing me to spend a bit too much effort on for­mat­ting by hand. [↩]