Design and machine learning – an annotated reading list

Ear­li­er this year I coached Design for Inter­ac­tion mas­ter stu­dents at Delft Uni­ver­si­ty of Tech­nol­o­gy in the course Research Method­ol­o­gy. The stu­dents organ­ised three sem­i­nars for which I pro­vid­ed the claims and assigned read­ing. In the sem­i­nars they argued about my claims using the Toul­min Mod­el of Argu­men­ta­tion. The read­ings served as sources for back­ing and evidence.

The claims and read­ings were all relat­ed to my nascent research project about machine learn­ing. We delved into both design­ing for machine learn­ing, and using machine learn­ing as a design tool.

Below are the read­ings I assigned, with some notes on each, which should help you decide if you want to dive into them yourself.

Hebron, Patrick. 2016. Machine Learn­ing for Design­ers. Sebastopol: O’Reilly.

The only non-aca­d­e­m­ic piece in this list. This served the pur­pose of get­ting all stu­dents on the same page with regards to what machine learn­ing is, its appli­ca­tions of machine learn­ing in inter­ac­tion design, and com­mon chal­lenges encoun­tered. I still can’t think of any oth­er sin­gle resource that is as good a start­ing point for the sub­ject as this one.

Fiebrink, Rebec­ca. 2016. “Machine Learn­ing as Meta-Instru­ment: Human-Machine Part­ner­ships Shap­ing Expres­sive Instru­men­tal Cre­ation.” In Musi­cal Instru­ments in the 21st Cen­tu­ry, 14:137–51. Sin­ga­pore: Springer Sin­ga­pore. doi:10.1007/978–981–10–2951–6_10.

Fiebrink’s Wek­ina­tor is ground­break­ing, fun and inspir­ing so I had to include some of her writ­ing in this list. This is most­ly of inter­est for those look­ing into the use of machine learn­ing for design and oth­er cre­ative and artis­tic endeav­ours. An impor­tant idea explored here is that tools that make use of (inter­ac­tive, super­vised) machine learn­ing can be thought of as instru­ments. Using such a tool is like play­ing or per­form­ing, explor­ing a pos­si­bil­i­ty space, engag­ing in a dia­logue with the tool. For a tool to feel like an instru­ment requires a tight action-feed­back loop.

Dove, Gra­ham, Kim Hal­skov, Jodi For­l­izzi, and John Zim­mer­man. 2017. UX Design Inno­va­tion: Chal­lenges for Work­ing with Machine Learn­ing as a Design Mate­r­i­al. The 2017 CHI Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/3025453.3025739.

A real­ly good sur­vey of how design­ers cur­rent­ly deal with machine learn­ing. Key take­aways include that in most cas­es, the appli­ca­tion of machine learn­ing is still engi­neer­ing-led as opposed to design-led, which ham­pers the cre­ation of non-obvi­ous machine learn­ing appli­ca­tions. It also makes it hard for design­ers to con­sid­er eth­i­cal impli­ca­tions of design choic­es. A key rea­son for this is that at the moment, pro­to­typ­ing with machine learn­ing is pro­hib­i­tive­ly cumbersome.

Fiebrink, Rebec­ca, Per­ry R Cook, and Dan True­man. 2011. “Human Mod­el Eval­u­a­tion in Inter­ac­tive Super­vised Learn­ing.” In, 147. New York, New York, USA: ACM Press. doi:10.1145/1978942.1978965.

The sec­ond Fiebrink piece in this list, which is more of a deep dive into how peo­ple use Wek­ina­tor. As with the chap­ter list­ed above this is required read­ing for those work­ing on design tools which make use of inter­ac­tive machine learn­ing. An impor­tant find­ing here is that users of intel­li­gent design tools might have very dif­fer­ent cri­te­ria for eval­u­at­ing the ‘cor­rect­ness’ of a trained mod­el than engi­neers do. Such cri­te­ria are like­ly sub­jec­tive and eval­u­a­tion requires first-hand use of the mod­el in real time. 

Bostrom, Nick, and Eliez­er Yud­kowsky. 2014. “The Ethics of Arti­fi­cial Intel­li­gence.” In The Cam­bridge Hand­book of Arti­fi­cial Intel­li­gence, edit­ed by Kei­th Frank­ish and William M Ram­sey, 316–34. Cam­bridge: Cam­bridge Uni­ver­si­ty Press. doi:10.1017/CBO9781139046855.020.

Bostrom is known for his some­what crazy but thought­pro­vok­ing book on super­in­tel­li­gence and although a large part of this chap­ter is about the ethics of gen­er­al arti­fi­cial intel­li­gence (which at the very least is still a way out), the first sec­tion dis­cuss­es the ethics of cur­rent “nar­row” arti­fi­cial intel­li­gence. It makes for a good check­list of things design­ers should keep in mind when they cre­ate new appli­ca­tions of machine learn­ing. Key insight: when a machine learn­ing sys­tem takes on work with social dimensions—tasks pre­vi­ous­ly per­formed by humans—the sys­tem inher­its its social requirements.

Yang, Qian, John Zim­mer­man, Aaron Ste­in­feld, and Antho­ny Toma­sic. 2016. Plan­ning Adap­tive Mobile Expe­ri­ences When Wire­fram­ing. The 2016 ACM Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/2901790.2901858.

Final­ly, a feet-in-the-mud explo­ration of what it actu­al­ly means to design for machine learn­ing with the tools most com­mon­ly used by design­ers today: draw­ings and dia­grams of var­i­ous sorts. In this case the focus is on using machine learn­ing to make an inter­face adap­tive. It includes an inter­est­ing dis­cus­sion of how to bal­ance the use of implic­it and explic­it user inputs for adap­ta­tion, and how to deal with infer­ence errors. Once again the lim­i­ta­tions of cur­rent sketch­ing and pro­to­typ­ing tools is men­tioned, and relat­ed to the need for design­ers to devel­op tac­it knowl­edge about machine learn­ing. Such tac­it knowl­edge will only be gained when design­ers can work with machine learn­ing in a hands-on manner.

Supplemental material

Floyd, Chris­tiane. 1984. “A Sys­tem­at­ic Look at Pro­to­typ­ing.” In Approach­es to Pro­to­typ­ing, 1–18. Berlin, Hei­del­berg: Springer Berlin Hei­del­berg. doi:10.1007/978–3–642–69796–8_1.

I pro­vid­ed this to stu­dents so that they get some addi­tion­al ground­ing in the var­i­ous kinds of pro­to­typ­ing that are out there. It helps to pre­vent reduc­tive notions of pro­to­typ­ing, and it makes for a nice com­ple­ment to Buxton’s work on sketch­ing.

Ble­vis, E, Y Lim, and E Stolter­man. 2006. “Regard­ing Soft­ware as a Mate­r­i­al of Design.”

Some of the papers refer to machine learn­ing as a “design mate­r­i­al” and this paper helps to under­stand what that idea means. Soft­ware is a mate­r­i­al with­out qual­i­ties (it is extreme­ly mal­leable, it can sim­u­late near­ly any­thing). Yet, it helps to con­sid­er it as a phys­i­cal mate­r­i­al in the metaphor­i­cal sense because we can then apply ways of design think­ing and doing to soft­ware programming.

Status update

This is not exact­ly a now page, but I thought I would write up what I am doing at the moment since last report­ing on my sta­tus in my end-of-year report.

The major­i­ty of my work­days are spent doing free­lance design con­sult­ing. My pri­ma­ry gig has been through Eend at the Dutch Vic­tim Sup­port Foun­da­tion, where until very recent­ly I was part of a team build­ing online ser­vices. I helped out with prod­uct strat­e­gy, set­ting up a lean UX design process, and get­ting an inte­grat­ed agile design and devel­op­ment team up and run­ning. The first ser­vices are now ship­ping so it is time for me to move on, after 10 months of very grat­i­fy­ing work. I real­ly enjoy work­ing in the pub­lic sec­tor and I hope to be doing more of it in future.

So yes, this means I am avail­able and you can hire me to do strat­e­gy and design for soft­ware prod­ucts and ser­vices. Just send me an email.

Short­ly before the Dutch nation­al elec­tions of this year, Iskan­der and I gath­ered a group of fel­low tech work­ers under the ban­ner of “Tech Sol­i­dar­i­ty NL to dis­cuss the con­cern­ing lurch to the right in nation­al pol­i­tics and what our field can do about it. This has devel­oped into a small but active com­mu­ni­ty who gath­er month­ly to edu­cate our­selves and devel­op plans for col­lec­tive action. I am get­ting a huge boost out of this. Fig­ur­ing out how to be a left­ist in this day and age is not easy. The only way to do it is to prac­tice and for that reflec­tion with peers is invalu­able. Build­ing and facil­i­tat­ing a group like this is huge­ly edu­ca­tion­al too. I have learned a lot about how a com­mu­ni­ty is boot-strapped and nurtured.

If you are in the Nether­lands, your pol­i­tics are left of cen­ter, and you work in tech­nol­o­gy, con­sid­er your­self invit­ed to join.

And final­ly, the last major thing on my plate is a con­tin­u­ing effort to secure a PhD posi­tion for myself. I am get­ting great sup­port from peo­ple at Delft Uni­ver­si­ty of Tech­nol­o­gy, in par­tic­u­lar Gerd Kortuem. I am focus­ing on inter­net of things prod­ucts that have fea­tures dri­ven by machine learn­ing. My ulti­mate aim is to devel­op pro­to­typ­ing tools for design and devel­op­ment teams that will help them cre­ate more inno­v­a­tive and more eth­i­cal solu­tions. The first step for this will be to con­duct field research inside com­pa­nies who are cre­at­ing such prod­ucts right now. So I am reach­ing out to peo­ple to see if I can secure a rea­son­able amount of poten­tial col­lab­o­ra­tors for this, which will go a long way in prov­ing the fea­si­bil­i­ty of my whole plan.

If you know of any com­pa­nies that devel­op con­sumer-fac­ing prod­ucts that have a con­nect­ed hard­ware com­po­nent and make use of machine learn­ing to dri­ve fea­tures, do let me know.

That’s about it. Free­lance UX con­sult­ing, left­ist tech-work­er organ­is­ing and design-for-machine-learn­ing research. Quite hap­py with that mix, really.

Doing UX inside of Scrum

Some notes on how I am cur­rent­ly “doing user expe­ri­ence” inside of Scrum. This approach has evolved from my projects at Hub­bub as well as more recent­ly my work with ARTO and on a project at Eden­spiek­er­mann. So I have found it works with both star­tups and agency style projects.

The start­ing point is to under­stand that Scrum is intend­ed to be a con­tain­er. It is a process frame­work. It should be able to hold any oth­er activ­i­ty you think you need as a team. So if we feel we need to add UX some­how, we should try to make it part of Scrum and not some­thing that is tacked onto Scrum. Why not tack some­thing on? Because it sig­nals design is some­how dis­tinct from devel­op­ment. And the whole point of doing agile is to have cross-func­tion­al teams. If you set up a sep­a­rate process for design you are high­ly like­ly not to ben­e­fit from the full col­lec­tive intel­li­gence of the com­bined design and devel­op­ment team. So no, design needs to be inside of the Scrum container.

Stag­gered sprints are not the answer either because you are still split­ting the team into design and devel­op­ment, ham­per­ing cross-col­lab­o­ra­tion and trans­paren­cy. You’re basi­cal­ly invit­ing Tay­lorism back into your process—the very thing you were try­ing to get­ting away from.

When you are uncom­fort­able with putting design­ers and devel­op­ers all in the same team and the same process the answer is not to make your process more elab­o­rate, par­cel things up, and decrease “messy” inter­ac­tions. The answer is increas­ing con­ver­sa­tion, not elim­i­nat­ing it.

It turns out things aren’t remote­ly as com­pli­cat­ed as they appear to be. The key is under­stand­ing Scrum’s events. The big event hold­ing all oth­er events is the sprint. The sprint out­puts a releasable incre­ment of “done” prod­uct. The devel­op­ment team does every­thing required to achieve the sprint goal col­lab­o­ra­tive­ly deter­mined dur­ing sprint plan­ning. Nat­u­ral­ly this includes any design need­ed for the prod­uct. I think of this as the ‘pro­duc­tion’ type of design. It typ­i­cal­ly con­sists most­ly of UI design. There may already be some pre­lim­i­nary UI design avail­able at the start of the sprint but it does not have to be finished. 

What about the kind of design that is required for fig­ur­ing out what to build in the first place? It might not be obvi­ous at first, but Scrum actu­al­ly has an ongo­ing process which read­i­ly accom­mo­dates it: back­log refine­ment. These are all activ­i­ties required to get a prod­uct back­log item in shape for sprint plan­ning. This is emphat­i­cal­ly not a solo show for the prod­uct man­ag­er to con­duct. It is some­thing the whole team col­lab­o­rates on. Devel­op­ers and design­ers. In my expe­ri­ence design­ers are great at facil­i­tat­ing back­log refine­ment ses­sions. At the white­board, fig­ur­ing stuff out with the whole team ‘Lean UX’ style. 

I will admit prod­uct back­log refine­ment is Scrum’s weak point. Where it offers a lot of struc­ture for the sprints, it offers hard­ly any for the back­log refine­ment (or groom­ing as some call it). But that’s okay, we can evolve our own.

I like to use Kan­ban to man­age the process of back­log refine­ment. Items come into the pipeline as some­thing we want to elab­o­rate because we have decid­ed we want to build it (in some form or oth­er, can be just an exper­i­ment) in the next sprint or two. It then goes through var­i­ous stages of elab­o­ra­tion. At the very least cap­tur­ing require­ments in the form of user sto­ries or job sto­ries, doing sketch­es, a lo-fi pro­to­type, mock­ups and a hi-fi pro­to­type and final­ly break­ing the item down into work to be done and attach­ing an esti­mate to it. At this point it is ready to be part of a sprint. Cru­cial­ly, dur­ing this life­cy­cle of an item as it is being refined, we can and should do user research if we feel we need more data, or user test­ing if we feel it is too risky to com­mit to a fea­ture outright. 

For this kind of fig­ur­ing stuff out, this ‘plan­ning’ type of design, it makes no sense to have it be part of a sprint-like struc­ture because the work required to get it to a ‘ready’ state is much more unpre­dictable. The point of hav­ing a loos­er groom­ing flow is that it exists to elim­i­nate uncer­tain­ty for when we com­mit to an item in a sprint.

So between the sprint and back­log refine­ment, Scrum read­i­ly accom­mo­dates design. ‘Pro­duc­tion’ type design hap­pens inside of the sprint and design­ers are con­sid­ered part of the devel­op­ment team. ‘Plan­ning’ type of design hap­pens as part of back­log refinement.

So no need to tack on a sep­a­rate process. It keeps the process sim­ple and under­stand­able, thus increas­ing trans­paren­cy for the whole team. It pre­vents design from becom­ing a black box to oth­ers. And when we make design part of the con­tain­er process frame­work that is Scrum, we reap the rewards of the team’s col­lec­tive intel­li­gence and we increase our agility.

Prototyping is a team sport

Late­ly I have been bing­ing on books, pre­sen­ta­tions and arti­cles relat­ed to ‘Lean UX’. I don’t like the term, but then I don’t like the tech industry’s love for invent­ing a new label for every damn thing. I do like the things empha­sis­es: shared under­stand­ing, deep col­lab­o­ra­tion, con­tin­u­ous user feed­back. These are prin­ci­ples that have always implic­it­ly guid­ed the choic­es I made when lead­ing teams at Hub­bub and now also as a mem­ber of sev­er­al teams in the role of prod­uct designer.

In all these lean UX read­ings a thing that keeps com­ing up again and again is pro­to­typ­ing. Pro­to­types are the go-to way of doing ‘exper­i­ments’, in lean-speak. Oth­er things can be done as well—surveys, inter­views, whatever—but more often than not, assump­tions are test­ed with prototypes. 

Which is great! And also unsur­pris­ing as pro­to­typ­ing has real­ly been embraced by the tech world. And tools for rapid pro­to­typ­ing are get­ting a lot of atten­tion and inter­est as a result. How­ev­er, this comes with a cou­ple of risks. For one, some­times it is fine to stick to paper. But the lure of shiny pro­to­typ­ing tools is strong. You’d rather not show a crap­py draw­ing to a user. What if they hate it? How­ev­er, high fideli­ty pro­to­typ­ing is always more cost­ly than paper. So although well-inten­tioned, pro­to­typ­ing tools can encour­age waste­ful­ness, the bane of lean. 

There is a big­ger dan­ger which runs against the lean ethos, though. Some tools afford deep col­lab­o­ra­tion more than oth­ers. Let’s be real: none afford deep­er col­lab­o­ra­tion than paper and white­boards. There is one per­son behind the con­trols when pro­to­typ­ing with a tool. So in my view, one should only ever progress to that step once a team effort has been made to hash out the rough out­lines of what is to be pro­to­typed. Basi­cal­ly: always paper pro­to­type the dig­i­tal pro­to­type. Together. 

I have had a lot of fun late­ly play­ing with brows­er pro­to­types and with pro­to­typ­ing in Framer. But as I was get­ting back into all of this I did notice this risk: All of a sud­den there is a per­son on the team who does the pro­to­types. Unless this solo pro­to­typ­ing is pre­ced­ed by shared pro­to­typ­ing, this is a prob­lem. Because the rest of the team is left out of the think­ing-through-mak­ing which makes the pro­to­typ­ing process so valu­able in addi­tion to the testable arte­facts it outputs.

It is I think a key over­sight of the ‘should design­ers code’ debaters and to an extent one made by all pro­to­typ­ing tool man­u­fac­tur­ers: Indi­vid­u­als don’t pro­to­type, teams do. Pro­to­typ­ing is a team sport. And so the suc­cess of a tool depends not only on how well it sup­ports indi­vid­ual pro­to­typ­ing activ­i­ties but also how well it embeds itself in col­lab­o­ra­tive workflows. 

In addi­tion to the tools them­selves get­ting bet­ter at sup­port­ing col­lab­o­ra­tive work­flows, I would also love to see more tuto­ri­als, both offi­cial and from the com­mu­ni­ty, about how to use a pro­to­typ­ing tool with­in the larg­er con­text of a team doing some form of agile. Most tuto­ri­als now focus on “how do I make this thing with this tool”. Use­ful, up to a point. But a large part of pro­to­typ­ing is to arrive at “the thing” together. 

One of the lean UX things I devoured was this pre­sen­ta­tion by Bill Scott in which he talks about align­ing a pro­to­typ­ing and a devel­op­ment tech stack, so that the gap between design and engi­neer­ing is bridged not just with process­es but also with tool­ing. His exam­ple applies to web devel­op­ment and app devel­op­ment using web tech­nolo­gies. I won­der what a sim­i­lar approach looks like for native mobile app devel­op­ment. But this is the sort of thing I am talk­ing about: Smart think­ing about how to actu­al­ly do this lean thing in the real world. I believe organ­is­ing our­selves so that we can pro­to­type as a team is absolute­ly key. I will pick my tools and process­es accord­ing­ly in future.

All of the above is as usu­al most­ly a reminder to self: As a design­er your role is not to go off and work solo on bril­liant pro­to­types. Your role is to facil­i­tate such efforts by the whole team. Sure, there will be solo deep design­er­ly craft­ing hap­pen­ing. But it will not add up to any­thing if it is not embed­ded in a col­lab­o­ra­tive design and devel­op­ment framework.

Nobody does thor­ough­ly argued pre­sen­ta­tions quite like Sebas­t­ian. This is good stuff on ethics and design.

I decid­ed to share some thoughts it sparked via Twit­ter and end­ed up rant­i­ng a bit:

I recent­ly talked about ethics to a bunch of “behav­ior design­ers” and found myself con­clud­ing that any designed sys­tem that does not allow for user appro­pri­a­tion is fun­da­men­tal­ly uneth­i­cal because as you right­ly point out what is the good life is a per­son­al mat­ter. Impos­ing it is an inher­ent­ly vio­lent act. A lot of design is a form of tech­no­log­i­cal­ly medi­at­ed vio­lence. Get­ting peo­ple to do your bid­ding, how­ev­er well intend­ed. Which giv­en my own voca­tion and work in the past is a kind of trou­bling thought to arrive at… Help?

Sebas­t­ian makes his best point on slides 113–114. Eth­i­cal design isn’t about doing the least harm, but about doing the most good. And, to come back to my Twit­ter rant, for me the ulti­mate good is for oth­ers to be free. Hence non-pre­scrip­tive design.

(via Design­ing the Good Life: Ethics and User Expe­ri­ence Design)

On sketching

Catch­ing up with this slight­ly neglect­ed blog (it’s been 6 weeks since the last prop­er post). I’d like to start by telling you about a small thing I helped out with last week. Peter Boers­ma1 asked me to help out with one of his UX Cock­tail Hours. He was inspired by a recent IxDA Stu­dio event where, in stead of just chat­ting and drink­ing, design­ers actu­al­ly made stuff. (Gasp!) Peter want­ed to do a work­shop where atten­dees col­lab­o­rat­ed on sketch­ing a solu­tion to a giv­en design problem.

Part of my con­tri­bu­tion to the evening was a short pre­sen­ta­tion on the the­o­ry and prac­tice of sketch­ing. On the the­o­ry side, I ref­er­enced Bill Bux­ton’s list of qual­i­ties that define what a sketch is2, and empha­sized that this means a sketch can be done in any mate­r­i­al, not nec­es­sar­i­ly pen­cil and paper. Fur­ther­more I dis­cussed why sketch­ing works, using part of an arti­cle on embod­ied inter­ac­tion3. The main point there, as far as I am con­cerned is that when sketch­ing, as design­ers we have the ben­e­fit of ‘back­talk’ from our mate­ri­als, which can pro­vide us with new insights. I wrapped up the pre­sen­ta­tion with a case study of a project I did a while back with the Ams­ter­dam-based agency Info.nl4 for a social web start-up aimed at inde­pen­dent pro­fes­sion­als. In the project I went quite far in using sketch­es to not only devel­op the design, but also col­lab­o­ra­tive­ly con­struct it with the client, tech­nol­o­gists and others.

The whole thing was record­ed; you can find a video of the talk at Vimeo (thanks to Iskan­der and Alper). I also uploaded the slides to SlideShare (sans notes).

The sec­ond, and most inter­est­ing part of the evening was the work­shop itself. This was set up as fol­lows: Peter and I had pre­pared a fic­tion­al case, con­cern­ing peer-to-peer ener­gy. We used the Dutch com­pa­ny Qur­rent as an exam­ple, and asked the par­tic­i­pants to con­cep­tu­alise a way to encour­age use of Qurrent’s prod­uct range. The aim was to have peo­ple be more ener­gy effi­cient, and share sur­plus ener­gy they had gen­er­at­ed with the Qur­rent com­mu­ni­ty. The par­tic­i­pants split up in teams of around ten peo­ple each, and went to work. We gave them around one hour to design a solu­tion, using only pen and paper. After­wards, they pre­sent­ed the out­come of their work to each oth­er. For each team, we asked one par­tic­i­pant to cri­tique the work by men­tion­ing one thing he or she liked, and one thing that could be improved. The team was then giv­en a chance to reply. We also asked each team to briefly reflect on their work­ing process. At the end of the evening every­one was giv­en a chance to vote for their favourite design. The win­ner received a prize.5

Wrap­ping up, I think what I liked most about the work­shop was see­ing the many dif­fer­ent ways the teams approached the prob­lem (many of the par­tic­i­pants did not know each oth­er before­hand). Group dynam­ics var­ied huge­ly. I think it was valu­able to have each team share their expe­ri­ences on this front with each oth­er. One thing that I think we could improve was the case itself; next time I would like to pro­vide par­tic­i­pants with a more focused, more rich­ly detailed brief­ing for them to sink their teeth in. That might result in an assign­ment that is more about struc­ture and behav­iour (or even inter­face) and less about con­cepts and val­ues. It would be good to see how sketch­ing func­tions in such a context.

  1. the Nether­lands’ tallest IA and one of sev­er­al famous Peters who work in UX []
  2. tak­en from his won­der­ful book Sketch­ing User Expe­ri­ences []
  3. titled How Bod­ies Mat­ter (PDF) by Kle­mer and Takaya­ma []
  4. who were also the hosts of this event []
  5. I think it’s inter­est­ing to note that the win­ner had a remark­able con­cept, but in my opin­ion was not the best exam­ple of the pow­er of sketch­ing. Appar­ent­ly the audi­ence val­ued prod­uct over process. []

A day of playing around with multi-touch and RoomWare

Last Sat­ur­day I attend­ed a RoomWare work­shop. The peo­ple of Can­Touch were there too, and brought one of their pro­to­type mul­ti-touch tables. The aim for the day was to come up with appli­ca­tions of RoomWare (open source soft­ware that can sense pres­ence of peo­ple in spaces) and mul­ti-touch. I attend­ed pri­mar­i­ly because it was a good oppor­tu­ni­ty to spend a day mess­ing around with a table.

Atten­dance was mul­ti­fac­eted, so while pro­gram­mers were putting togeth­er a proof-of-con­cept, design­ers (such as Alexan­der Zeh, James Burke and I) came up with con­cepts for new inter­ac­tions. The proof-of-con­cept was up and run­ning at the end of then day: The table could sense who was in the room and dis­play his or her Flickr pho­tos, which you could then move around, scale, rotate, etc. in the typ­i­cal mul­ti-touch fashion.

The con­cepts design­ers came up with main­ly focused on pulling in Last.fm data (again using RoomWare’s sens­ing capa­bil­i­ties) and dis­play­ing it for group-based explo­ration. Here’s a sto­ry­board I quick­ly whipped up of one such application:

RoomWare + CanTouch + Last.fm

The sto­ry­board shows how you can add your­self from a list of peo­ple present in the room. Your top artists flock around you. When more peo­ple are added, lines are drawn between you. The thick­ness of the line rep­re­sents how sim­i­lar your tastes are, accord­ing to Last.fm’s taste-o-meter. Also, shared top artists flock in such a way as to be clos­est to all relat­ed peo­ple. Final­ly, artists can be act­ed on to lis­ten to music.

When I was sketch­ing this, it became appar­ent that ori­en­ta­tion of ele­ments should fol­low very dif­fer­ent rules from reg­u­lar screens. I chose to sketch things so that they all point out­wards, with the mid­dle of the table as the ori­en­ta­tion point.

By spend­ing a day immersed in mul­ti-touch stuff, some inter­est­ing design chal­lenges became apparent:

  • With table­top sur­faces, stuff is clos­er or fur­ther away phys­i­cal­ly. Prox­im­i­ty of ele­ments can be unin­ten­tion­al­ly inter­pret­ed as say­ing some­thing about aspects such as impor­tance, rel­e­vance, etc. Design­ers need to be even more aware of place­ment than before, plus con­ven­tions from ver­ti­cal­ly ori­ent­ed screens no longer apply. Top-of-screen becomes fur­thest away and there­fore least promi­nent in stead of most important. 
  • With group-based inter­ac­tions, it becomes tricky to deter­mine who to address and where to address him or her. Some­times the sys­tem should address the group as a whole. When 5 peo­ple are stand­ing around a table, text-based inter­faces become prob­lem­at­ic since what is leg­i­ble from one end of the table is unin­tel­li­gi­ble from the oth­er. New con­ven­tions need to be devel­oped for this as well. Alexan­der and I phi­los­o­phized about plac­ing text along cir­cles and ani­mat­ing them so that they cir­cu­late around the table, for instance.
  • Besides these, many oth­er inter­face chal­lenges present them­selves. One cru­cial piece of infor­ma­tion for solv­ing many of these is know­ing where peo­ple are locat­ed around the table. This issue can be approached from dif­fer­ent angles. By incor­po­rat­ing sen­sors in the table, detec­tion may be auto­mat­ed and inter­faces could me made to adapt auto­mat­i­cal­ly. This is the tech­no-cen­tric angle. I am not con­vinced this is the way to go, because it dimin­ish­es people’s con­trol over the expe­ri­ence. I would pre­fer to make the inter­face itself adjustable in nat­ur­al ways, so that peo­ple can mold the rep­re­sen­ta­tion to suit their con­text. With sit­u­at­ed tech­nolo­gies like this, auto-mag­i­cal adap­ta­tion is an “AI-hard” prob­lem, and the price of fail­ure is a severe­ly degrad­ed user expe­ri­ence from which peo­ple can­not recov­er because the sys­tem won’t let them.

All in all the work­shop was a won­der­ful day of tin­ker­ing with like-mind­ed indi­vid­u­als from rad­i­cal­ly dif­fer­ent back­grounds. As a design­er, I think this is one of the best way be involved with open source projects. On a day like this, tech­nol­o­gists can be exposed to new inter­ac­tion con­cepts while they are hack­ing away. At the same time design­ers get that rare oppor­tu­ni­ty to play around with tech­nol­o­gy as it is shaped. Quick-and-dirty sketch­es like the ones Alexan­der and I came up with are def­i­nite­ly the way to com­mu­ni­cate ideas. The goal is to sug­gest, not to describe, after all. Tech­nol­o­gists should feel free to elab­o­rate and build on what design­ers come up with and vice-ver­sa. I am curi­ous to see which parts of what we came up with will find their way into future RoomWare projects.

Urban procedural rhetorics — transcript of my TWAB 2008 talk

This is a tran­script of my pre­sen­ta­tion at The Web and Beyond 2008: Mobil­i­ty in Ams­ter­dam on 22 May. Since the major­i­ty of pay­ing atten­dees were local I pre­sent­ed in Dutch. How­ev­er, Eng­lish appears to be the lin­gua fran­ca of the inter­net, so here I offer a trans­la­tion. I have uploaded the slides to SlideShare and hope to be able to share a video record­ing of the whole thing soon.

Update: I have uploaded a video of the pre­sen­ta­tion to Vimeo. Many thanks to Almar van der Krogt for record­ing this.

In 1966 a num­ber of mem­bers of Pro­vo took to the streets of Ams­ter­dam car­ry­ing blank ban­ners. Pro­vo was a non­vi­o­lent anar­chist move­ment. They pri­mar­i­ly occu­pied them­selves with pro­vok­ing the author­i­ties in a “ludic” man­ner. Noth­ing was writ­ten on their ban­ners because the may­or of Ams­ter­dam had banned the slo­gans “free­dom of speech”, “democ­ra­cy” and “right to demon­strate”. Regard­less, the mem­bers were arrest­ed by police, show­ing that the author­i­ties did not respect their right to demon­strate.1

Good after­noon every­one, my name is Kars Alfrink, I’m a free­lance inter­ac­tion design­er. Today I’d like to talk about play in pub­lic space. I believe that with the arrival of ubiq­ui­tous com­put­ing in the city new forms of play will be made pos­si­ble. The tech­nolo­gies we shape will be used for play wether we want to or not. As William Gib­son writes in Burn­ing Chrome:

…the street finds its own uses for things”

For exam­ple: Skate­board­ing as we now know it — with its empha­sis on aer­i­al acro­bat­ics — start­ed in emp­ty pools like this one. That was done with­out per­mis­sion, of course…

Only lat­er half-pipes, ramps, verts (which by the way is derived from ‘ver­ti­cal’) and skateparks arrived — areas where skate­board­ing is tol­er­at­ed. Skate­board­ing would not be what it is today with­out those first few emp­ty pools.2

Con­tin­ue read­ing Urban pro­ce­dur­al rhetorics — tran­script of my TWAB 2008 talk

  1. The web­site of Gram­schap con­tains a chronol­o­gy of the Pro­vo move­ment in Dutch. []
  2. For a vivid account of the emer­gence of the ver­ti­cal style of skate­board­ing see the doc­u­men­tary film Dog­town and Z‑Boys. []

Sketching the experience of toys

A frame from the Sketch-A-Move video

Play is the high­est form of research.”

—Albert Ein­stein1

That’s what I always say when I’m play­ing games, too. 

I real­ly liked Bill Bux­ton’s book Sketch­ing User Expe­ri­ences. I like it because Bux­ton defends design as a legit­i­mate pro­fes­sion sep­a­rate from oth­er disciplines—such as engineering—while at the same time show­ing that design­ers (no mat­ter how bril­liant) can only suc­ceed in the right ecosys­tem. I also like the fact that he iden­ti­fies sketch­ing (in its many forms) as a defin­ing activ­i­ty of the design pro­fes­sion. The many exam­ples he shows are very inspiring.

One in par­tic­u­lar stood out for me, which is the project Sketch-A-Move by Anab Jain and Louise Klink­er done in 2004 at the RCA in Lon­don. The image above is tak­en from the video they cre­at­ed to illus­trate their con­cept. It’s about cars auto-mag­i­cal­ly dri­ving along tra­jec­to­ries that you draw on their roof. You can watch the video over at the book’s com­pan­ion web­site. It’s a very good exam­ple of visu­al­iz­ing an inter­ac­tive prod­uct in a very com­pelling way with­out actu­al­ly build­ing it. This was all faked, if you want to find out how, buy the book.2

The great thing about the video is not only does it illus­trate how the con­cept works, it also gives you a sense of what the expe­ri­ence of using it would be like. As Bux­ton writes:3

You see, toys are not about toys. Toys are about play and the expe­ri­ence of fun that they help fos­ter. And that is what this video real­ly shows. That, and the pow­er of video to go beyond sim­ply doc­u­ment­ing a con­cept to com­mu­ni­cat­ing some­thing about expe­ri­ence in a very vis­cer­al way.”

Not only does it com­mu­ni­cate the fun you would have play­ing with it, I think this way of sketch­ing actu­al­ly helped the design­ers get a sense them­selves of wether what they had come up with was fun. You can tell they are actu­al­ly play­ing, being sur­prised by unex­pect­ed out­comes, etc.

The role of play in design is dis­cussed by Bux­ton as well, although he admits he need­ed to be prompt­ed by a friend of his: Alex Manu, a teacher at OCAD in Toron­to writes in an email to Bux­ton:4

With­out play imag­i­na­tion dies.”

Chal­lenges to imag­i­na­tion are the keys to cre­ativ­i­ty. The skill of retriev­ing imag­i­na­tion resides in the mas­tery of play. The ecol­o­gy of play is the ecol­o­gy of the pos­si­ble. Pos­si­bil­i­ty incu­bates creativity.”

Which Bux­ton rephras­es in one of his own per­son­al mantras:5

These things are far too impor­tant to take seriously.”

All of which has made me real­ize that if I’m not hav­ing some sort of fun while design­ing, I’m doing some­thing wrong. It might be worth con­sid­er­ing switch­ing from one sketch­ing tech­nique to anoth­er. It might help me get a dif­fer­ent per­spec­tive on the prob­lem, and yield new pos­si­ble solu­tions. Bux­ton’s book is a trea­sure trove of sketch­ing tech­niques. There is no excuse for being bored while design­ing anymore.

  1. Sketch­ing User Expe­ri­ences p.349 []
  2. No, I’m not get­ting a com­mis­sion to say that. []
  3. Ibid. 1, at 325 []
  4. Ibid., at 263 []
  5. Ibid. []

Notes on play, exploration, challenge and learning

(My read­ing notes are pil­ing up so here’s an attempt to clear out at least a few of them.) 

Part of the play expe­ri­ence of many dig­i­tal games is fig­ur­ing out how the damn thing works in the first place. In Rules of Play on page 210:

[…] as the play­er plays with FLUID, inter­ac­tion and obser­va­tion reveals the under­ly­ing prin­ci­ples of the sys­tem. In this case the hid­den infor­ma­tion grad­u­al­ly revealed through play is the rules of the sim­u­la­tion itself. Part of the play of FLUID is the dis­cov­ery of the game rules as information.”

(Sad­ly, I could not find a link to the game mentioned.)

I did not give Don­ald Nor­man all the cred­it he was due in my ear­li­er post. He does­n’t have a blind spot for games. Quite the con­trary. For instance, he explains how to make sys­tems eas­i­er to learn and points to games in the process. On page 183 of The Design of Every­day Things:

One impor­tant method of mak­ing sys­tems eas­i­er to learn and to use is to make them explorable, to encour­age the user to exper­i­ment and learn the pos­si­bil­i­ties through active exploration.”

The way to do this is through direct manip­u­la­tion, writes Nor­man. He also reminds us that it’s not nec­es­sary to make any sys­tem explorable.1 But (on page 184):

[…] if the job is crit­i­cal, nov­el, or ill-spec­i­fied, or if you do not yet know exact­ly what is to be done, then you need direct, first-per­son interaction.”

So much writ­ten after DOET seems to have added lit­tle to the con­ver­sa­tion. I’m sur­prised how use­ful this clas­sic still is.

I’m remind­ed of a sec­tion of Matt Jones’s Inter­ac­tion 08 talk—which I watched yes­ter­day. He went through a num­ber of infor­ma­tion visu­al­i­sa­tions and said he’d like to add more stuff like that into Dopplr, to allow peo­ple to play with their data. He even com­pared this act of play to Will Wright’s con­cept of pos­si­bil­i­ty space.2 He also briefly men­tioned that eas­i­ly acces­si­ble tools for cre­at­ing infor­ma­tion visu­al­i­sa­tions might become a valu­able tool for design­ers work­ing with com­plex sets of data. 

Nor­man actu­al­ly points to games for inspi­ra­tion, by the way. On page 184 just before the pre­vi­ous quote:

Some com­put­er sys­tems offer direct manip­u­la­tion, first-per­son inter­ac­tions, good exam­ples being the dri­ving, fly­ing, and sports games that are com­mon­place in arcades and on home machines. In these games, the feel­ing of direct con­trol over the actions is an essen­tial part of the task.”

And so on.

One of the most use­ful parts of Dan Saf­fer­’s book on inter­ac­tion design is where he explains the dif­fer­ences between cus­tomi­sa­tion, per­son­al­i­sa­tion, adap­ta­tion and hack­ing. He notes that an adap­tive sys­tem can be designed to induce flow—balancing chal­lenge with the skill of the user. In games, there is some­thing called dynam­ic dif­fi­cul­ty adjust­ment (DDA) which has very sim­i­lar aims. 

Salen and Zim­mer­man have their doubts about DDA though. In Rules of Play on page 223 they write:

Play­ing a game becomes less like learn­ing an expres­sive lan­guage and more like being the sole audi­ence mem­ber for a par­tic­i­pa­to­ry, impro­vi­sa­tion­al per­for­mance, where the per­form­ers adjust their actions to how you inter­act with them. Are you then play­ing the game, or is it play­ing you?”

Per­haps, but it all depends on what DDA actu­al­ly adjusts. The tech­nique might be objec­tion­able in a game (where a large part of the point is over­com­ing chal­lenge) but in oth­er sys­tems many of these objec­tions do not apply.

With a suc­cess­ful adap­tive design, the prod­uct fits the user’s life and envi­ron­ment as though it were cus­tom made.”

(Design­ing for Inter­ac­tion, page 162.)

Adap­tive sys­tems explic­it­ly antic­i­pate trans­for­ma­tive play. They allow them­selves to be changed through a per­son­’s inter­ac­tions with it.3

A char­ac­ter­is­tic of good inter­ac­tion design is play­ful­ness, writes Mr. Saf­fer in his book on page 67:

Through seri­ous play, we seek out new prod­ucts, ser­vices and fea­tures and then try them to see how they work. How many times have you pushed a but­ton just to see what it did?”

The fun­ny thing is, the con­di­tions for play accord­ing to Saf­fer are very sim­i­lar to some of the basic guide­lines Nor­man offers: Make users feel com­fort­able, reduce the chance for errors and if errors do occur, make sure the con­se­quences are small—by allow­ing users to undo, for instance. 

Mr. Nor­man writes that in games “design­ers delib­er­ate­ly flout the laws of under­stand­abil­i­ty and usabil­i­ty” (p.205). Although even in games: “[the] rules [of usabil­i­ty] must be applied intel­li­gent­ly, for ease of use or dif­fi­cul­ty of use” (p.208).

By now, it should be clear mak­ing inter­ac­tions play­ful is very dif­fer­ent from mak­ing them game-like.

  1. Appar­ent­ly, “explorable” isn’t a prop­er Eng­lish word, but if it’s good enough for Mr. Nor­man it’s good enough for me. []
  2. I blogged about pos­si­bil­i­ty space before here. []
  3. Yes, I know I blogged about adap­tive design before. Also about flow and adap­ta­tion, it seems. []