Design and machine learning – an annotated reading list

Ear­li­er this year I coached Design for Inter­ac­tion mas­ter stu­dents at Delft Uni­ver­si­ty of Tech­nol­o­gy in the course Research Method­ol­o­gy. The stu­dents organ­ised three sem­i­nars for which I pro­vid­ed the claims and assigned read­ing. In the sem­i­nars they argued about my claims using the Toul­min Mod­el of Argu­men­ta­tion. The read­ings served as sources for back­ing and evidence.

The claims and read­ings were all relat­ed to my nascent research project about machine learn­ing. We delved into both design­ing for machine learn­ing, and using machine learn­ing as a design tool.

Below are the read­ings I assigned, with some notes on each, which should help you decide if you want to dive into them yourself.

Hebron, Patrick. 2016. Machine Learn­ing for Design­ers. Sebastopol: O’Reilly.

The only non-aca­d­e­m­ic piece in this list. This served the pur­pose of get­ting all stu­dents on the same page with regards to what machine learn­ing is, its appli­ca­tions of machine learn­ing in inter­ac­tion design, and com­mon chal­lenges encoun­tered. I still can’t think of any oth­er sin­gle resource that is as good a start­ing point for the sub­ject as this one.

Fiebrink, Rebec­ca. 2016. “Machine Learn­ing as Meta-Instru­ment: Human-Machine Part­ner­ships Shap­ing Expres­sive Instru­men­tal Cre­ation.” In Musi­cal Instru­ments in the 21st Cen­tu­ry, 14:137–51. Sin­ga­pore: Springer Sin­ga­pore. doi:10.1007/978–981–10–2951–6_10.

Fiebrink’s Wek­ina­tor is ground­break­ing, fun and inspir­ing so I had to include some of her writ­ing in this list. This is most­ly of inter­est for those look­ing into the use of machine learn­ing for design and oth­er cre­ative and artis­tic endeav­ours. An impor­tant idea explored here is that tools that make use of (inter­ac­tive, super­vised) machine learn­ing can be thought of as instru­ments. Using such a tool is like play­ing or per­form­ing, explor­ing a pos­si­bil­i­ty space, engag­ing in a dia­logue with the tool. For a tool to feel like an instru­ment requires a tight action-feed­back loop.

Dove, Gra­ham, Kim Hal­skov, Jodi For­l­izzi, and John Zim­mer­man. 2017. UX Design Inno­va­tion: Chal­lenges for Work­ing with Machine Learn­ing as a Design Mate­r­i­al. The 2017 CHI Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/3025453.3025739.

A real­ly good sur­vey of how design­ers cur­rent­ly deal with machine learn­ing. Key take­aways include that in most cas­es, the appli­ca­tion of machine learn­ing is still engi­neer­ing-led as opposed to design-led, which ham­pers the cre­ation of non-obvi­ous machine learn­ing appli­ca­tions. It also makes it hard for design­ers to con­sid­er eth­i­cal impli­ca­tions of design choic­es. A key rea­son for this is that at the moment, pro­to­typ­ing with machine learn­ing is pro­hib­i­tive­ly cumbersome.

Fiebrink, Rebec­ca, Per­ry R Cook, and Dan True­man. 2011. “Human Mod­el Eval­u­a­tion in Inter­ac­tive Super­vised Learn­ing.” In, 147. New York, New York, USA: ACM Press. doi:10.1145/1978942.1978965.

The sec­ond Fiebrink piece in this list, which is more of a deep dive into how peo­ple use Wek­ina­tor. As with the chap­ter list­ed above this is required read­ing for those work­ing on design tools which make use of inter­ac­tive machine learn­ing. An impor­tant find­ing here is that users of intel­li­gent design tools might have very dif­fer­ent cri­te­ria for eval­u­at­ing the ‘cor­rect­ness’ of a trained mod­el than engi­neers do. Such cri­te­ria are like­ly sub­jec­tive and eval­u­a­tion requires first-hand use of the mod­el in real time. 

Bostrom, Nick, and Eliez­er Yud­kowsky. 2014. “The Ethics of Arti­fi­cial Intel­li­gence.” In The Cam­bridge Hand­book of Arti­fi­cial Intel­li­gence, edit­ed by Kei­th Frank­ish and William M Ram­sey, 316–34. Cam­bridge: Cam­bridge Uni­ver­si­ty Press. doi:10.1017/CBO9781139046855.020.

Bostrom is known for his some­what crazy but thought­pro­vok­ing book on super­in­tel­li­gence and although a large part of this chap­ter is about the ethics of gen­er­al arti­fi­cial intel­li­gence (which at the very least is still a way out), the first sec­tion dis­cuss­es the ethics of cur­rent “nar­row” arti­fi­cial intel­li­gence. It makes for a good check­list of things design­ers should keep in mind when they cre­ate new appli­ca­tions of machine learn­ing. Key insight: when a machine learn­ing sys­tem takes on work with social dimensions—tasks pre­vi­ous­ly per­formed by humans—the sys­tem inher­its its social requirements.

Yang, Qian, John Zim­mer­man, Aaron Ste­in­feld, and Antho­ny Toma­sic. 2016. Plan­ning Adap­tive Mobile Expe­ri­ences When Wire­fram­ing. The 2016 ACM Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/2901790.2901858.

Final­ly, a feet-in-the-mud explo­ration of what it actu­al­ly means to design for machine learn­ing with the tools most com­mon­ly used by design­ers today: draw­ings and dia­grams of var­i­ous sorts. In this case the focus is on using machine learn­ing to make an inter­face adap­tive. It includes an inter­est­ing dis­cus­sion of how to bal­ance the use of implic­it and explic­it user inputs for adap­ta­tion, and how to deal with infer­ence errors. Once again the lim­i­ta­tions of cur­rent sketch­ing and pro­to­typ­ing tools is men­tioned, and relat­ed to the need for design­ers to devel­op tac­it knowl­edge about machine learn­ing. Such tac­it knowl­edge will only be gained when design­ers can work with machine learn­ing in a hands-on manner.

Supplemental material

Floyd, Chris­tiane. 1984. “A Sys­tem­at­ic Look at Pro­to­typ­ing.” In Approach­es to Pro­to­typ­ing, 1–18. Berlin, Hei­del­berg: Springer Berlin Hei­del­berg. doi:10.1007/978–3–642–69796–8_1.

I pro­vid­ed this to stu­dents so that they get some addi­tion­al ground­ing in the var­i­ous kinds of pro­to­typ­ing that are out there. It helps to pre­vent reduc­tive notions of pro­to­typ­ing, and it makes for a nice com­ple­ment to Buxton’s work on sketch­ing.

Ble­vis, E, Y Lim, and E Stolter­man. 2006. “Regard­ing Soft­ware as a Mate­r­i­al of Design.”

Some of the papers refer to machine learn­ing as a “design mate­r­i­al” and this paper helps to under­stand what that idea means. Soft­ware is a mate­r­i­al with­out qual­i­ties (it is extreme­ly mal­leable, it can sim­u­late near­ly any­thing). Yet, it helps to con­sid­er it as a phys­i­cal mate­r­i­al in the metaphor­i­cal sense because we can then apply ways of design think­ing and doing to soft­ware programming.

Status update

This is not exact­ly a now page, but I thought I would write up what I am doing at the moment since last report­ing on my sta­tus in my end-of-year report.

The major­i­ty of my work­days are spent doing free­lance design con­sult­ing. My pri­ma­ry gig has been through Eend at the Dutch Vic­tim Sup­port Foun­da­tion, where until very recent­ly I was part of a team build­ing online ser­vices. I helped out with prod­uct strat­e­gy, set­ting up a lean UX design process, and get­ting an inte­grat­ed agile design and devel­op­ment team up and run­ning. The first ser­vices are now ship­ping so it is time for me to move on, after 10 months of very grat­i­fy­ing work. I real­ly enjoy work­ing in the pub­lic sec­tor and I hope to be doing more of it in future.

So yes, this means I am avail­able and you can hire me to do strat­e­gy and design for soft­ware prod­ucts and ser­vices. Just send me an email.

Short­ly before the Dutch nation­al elec­tions of this year, Iskan­der and I gath­ered a group of fel­low tech work­ers under the ban­ner of “Tech Sol­i­dar­i­ty NL to dis­cuss the con­cern­ing lurch to the right in nation­al pol­i­tics and what our field can do about it. This has devel­oped into a small but active com­mu­ni­ty who gath­er month­ly to edu­cate our­selves and devel­op plans for col­lec­tive action. I am get­ting a huge boost out of this. Fig­ur­ing out how to be a left­ist in this day and age is not easy. The only way to do it is to prac­tice and for that reflec­tion with peers is invalu­able. Build­ing and facil­i­tat­ing a group like this is huge­ly edu­ca­tion­al too. I have learned a lot about how a com­mu­ni­ty is boot-strapped and nurtured.

If you are in the Nether­lands, your pol­i­tics are left of cen­ter, and you work in tech­nol­o­gy, con­sid­er your­self invit­ed to join.

And final­ly, the last major thing on my plate is a con­tin­u­ing effort to secure a PhD posi­tion for myself. I am get­ting great sup­port from peo­ple at Delft Uni­ver­si­ty of Tech­nol­o­gy, in par­tic­u­lar Gerd Kortuem. I am focus­ing on inter­net of things prod­ucts that have fea­tures dri­ven by machine learn­ing. My ulti­mate aim is to devel­op pro­to­typ­ing tools for design and devel­op­ment teams that will help them cre­ate more inno­v­a­tive and more eth­i­cal solu­tions. The first step for this will be to con­duct field research inside com­pa­nies who are cre­at­ing such prod­ucts right now. So I am reach­ing out to peo­ple to see if I can secure a rea­son­able amount of poten­tial col­lab­o­ra­tors for this, which will go a long way in prov­ing the fea­si­bil­i­ty of my whole plan.

If you know of any com­pa­nies that devel­op con­sumer-fac­ing prod­ucts that have a con­nect­ed hard­ware com­po­nent and make use of machine learn­ing to dri­ve fea­tures, do let me know.

That’s about it. Free­lance UX con­sult­ing, left­ist tech-work­er organ­is­ing and design-for-machine-learn­ing research. Quite hap­py with that mix, really.

Doing UX inside of Scrum

Some notes on how I am cur­rent­ly “doing user expe­ri­ence” inside of Scrum. This approach has evolved from my projects at Hub­bub as well as more recent­ly my work with ARTO and on a project at Eden­spiek­er­mann. So I have found it works with both star­tups and agency style projects.

The start­ing point is to under­stand that Scrum is intend­ed to be a con­tain­er. It is a process frame­work. It should be able to hold any oth­er activ­i­ty you think you need as a team. So if we feel we need to add UX some­how, we should try to make it part of Scrum and not some­thing that is tacked onto Scrum. Why not tack some­thing on? Because it sig­nals design is some­how dis­tinct from devel­op­ment. And the whole point of doing agile is to have cross-func­tion­al teams. If you set up a sep­a­rate process for design you are high­ly like­ly not to ben­e­fit from the full col­lec­tive intel­li­gence of the com­bined design and devel­op­ment team. So no, design needs to be inside of the Scrum container.

Stag­gered sprints are not the answer either because you are still split­ting the team into design and devel­op­ment, ham­per­ing cross-col­lab­o­ra­tion and trans­paren­cy. You’re basi­cal­ly invit­ing Tay­lorism back into your process—the very thing you were try­ing to get­ting away from.

When you are uncom­fort­able with putting design­ers and devel­op­ers all in the same team and the same process the answer is not to make your process more elab­o­rate, par­cel things up, and decrease “messy” inter­ac­tions. The answer is increas­ing con­ver­sa­tion, not elim­i­nat­ing it.

It turns out things aren’t remote­ly as com­pli­cat­ed as they appear to be. The key is under­stand­ing Scrum’s events. The big event hold­ing all oth­er events is the sprint. The sprint out­puts a releasable incre­ment of “done” prod­uct. The devel­op­ment team does every­thing required to achieve the sprint goal col­lab­o­ra­tive­ly deter­mined dur­ing sprint plan­ning. Nat­u­ral­ly this includes any design need­ed for the prod­uct. I think of this as the ‘pro­duc­tion’ type of design. It typ­i­cal­ly con­sists most­ly of UI design. There may already be some pre­lim­i­nary UI design avail­able at the start of the sprint but it does not have to be finished. 

What about the kind of design that is required for fig­ur­ing out what to build in the first place? It might not be obvi­ous at first, but Scrum actu­al­ly has an ongo­ing process which read­i­ly accom­mo­dates it: back­log refine­ment. These are all activ­i­ties required to get a prod­uct back­log item in shape for sprint plan­ning. This is emphat­i­cal­ly not a solo show for the prod­uct man­ag­er to con­duct. It is some­thing the whole team col­lab­o­rates on. Devel­op­ers and design­ers. In my expe­ri­ence design­ers are great at facil­i­tat­ing back­log refine­ment ses­sions. At the white­board, fig­ur­ing stuff out with the whole team ‘Lean UX’ style. 

I will admit prod­uct back­log refine­ment is Scrum’s weak point. Where it offers a lot of struc­ture for the sprints, it offers hard­ly any for the back­log refine­ment (or groom­ing as some call it). But that’s okay, we can evolve our own.

I like to use Kan­ban to man­age the process of back­log refine­ment. Items come into the pipeline as some­thing we want to elab­o­rate because we have decid­ed we want to build it (in some form or oth­er, can be just an exper­i­ment) in the next sprint or two. It then goes through var­i­ous stages of elab­o­ra­tion. At the very least cap­tur­ing require­ments in the form of user sto­ries or job sto­ries, doing sketch­es, a lo-fi pro­to­type, mock­ups and a hi-fi pro­to­type and final­ly break­ing the item down into work to be done and attach­ing an esti­mate to it. At this point it is ready to be part of a sprint. Cru­cial­ly, dur­ing this life­cy­cle of an item as it is being refined, we can and should do user research if we feel we need more data, or user test­ing if we feel it is too risky to com­mit to a fea­ture outright. 

For this kind of fig­ur­ing stuff out, this ‘plan­ning’ type of design, it makes no sense to have it be part of a sprint-like struc­ture because the work required to get it to a ‘ready’ state is much more unpre­dictable. The point of hav­ing a loos­er groom­ing flow is that it exists to elim­i­nate uncer­tain­ty for when we com­mit to an item in a sprint.

So between the sprint and back­log refine­ment, Scrum read­i­ly accom­mo­dates design. ‘Pro­duc­tion’ type design hap­pens inside of the sprint and design­ers are con­sid­ered part of the devel­op­ment team. ‘Plan­ning’ type of design hap­pens as part of back­log refinement.

So no need to tack on a sep­a­rate process. It keeps the process sim­ple and under­stand­able, thus increas­ing trans­paren­cy for the whole team. It pre­vents design from becom­ing a black box to oth­ers. And when we make design part of the con­tain­er process frame­work that is Scrum, we reap the rewards of the team’s col­lec­tive intel­li­gence and we increase our agility.

Prototyping is a team sport

Late­ly I have been bing­ing on books, pre­sen­ta­tions and arti­cles relat­ed to ‘Lean UX’. I don’t like the term, but then I don’t like the tech industry’s love for invent­ing a new label for every damn thing. I do like the things empha­sis­es: shared under­stand­ing, deep col­lab­o­ra­tion, con­tin­u­ous user feed­back. These are prin­ci­ples that have always implic­it­ly guid­ed the choic­es I made when lead­ing teams at Hub­bub and now also as a mem­ber of sev­er­al teams in the role of prod­uct designer.

In all these lean UX read­ings a thing that keeps com­ing up again and again is pro­to­typ­ing. Pro­to­types are the go-to way of doing ‘exper­i­ments’, in lean-speak. Oth­er things can be done as well—surveys, inter­views, whatever—but more often than not, assump­tions are test­ed with prototypes. 

Which is great! And also unsur­pris­ing as pro­to­typ­ing has real­ly been embraced by the tech world. And tools for rapid pro­to­typ­ing are get­ting a lot of atten­tion and inter­est as a result. How­ev­er, this comes with a cou­ple of risks. For one, some­times it is fine to stick to paper. But the lure of shiny pro­to­typ­ing tools is strong. You’d rather not show a crap­py draw­ing to a user. What if they hate it? How­ev­er, high fideli­ty pro­to­typ­ing is always more cost­ly than paper. So although well-inten­tioned, pro­to­typ­ing tools can encour­age waste­ful­ness, the bane of lean. 

There is a big­ger dan­ger which runs against the lean ethos, though. Some tools afford deep col­lab­o­ra­tion more than oth­ers. Let’s be real: none afford deep­er col­lab­o­ra­tion than paper and white­boards. There is one per­son behind the con­trols when pro­to­typ­ing with a tool. So in my view, one should only ever progress to that step once a team effort has been made to hash out the rough out­lines of what is to be pro­to­typed. Basi­cal­ly: always paper pro­to­type the dig­i­tal pro­to­type. Together. 

I have had a lot of fun late­ly play­ing with brows­er pro­to­types and with pro­to­typ­ing in Framer. But as I was get­ting back into all of this I did notice this risk: All of a sud­den there is a per­son on the team who does the pro­to­types. Unless this solo pro­to­typ­ing is pre­ced­ed by shared pro­to­typ­ing, this is a prob­lem. Because the rest of the team is left out of the think­ing-through-mak­ing which makes the pro­to­typ­ing process so valu­able in addi­tion to the testable arte­facts it outputs.

It is I think a key over­sight of the ‘should design­ers code’ debaters and to an extent one made by all pro­to­typ­ing tool man­u­fac­tur­ers: Indi­vid­u­als don’t pro­to­type, teams do. Pro­to­typ­ing is a team sport. And so the suc­cess of a tool depends not only on how well it sup­ports indi­vid­ual pro­to­typ­ing activ­i­ties but also how well it embeds itself in col­lab­o­ra­tive workflows. 

In addi­tion to the tools them­selves get­ting bet­ter at sup­port­ing col­lab­o­ra­tive work­flows, I would also love to see more tuto­ri­als, both offi­cial and from the com­mu­ni­ty, about how to use a pro­to­typ­ing tool with­in the larg­er con­text of a team doing some form of agile. Most tuto­ri­als now focus on “how do I make this thing with this tool”. Use­ful, up to a point. But a large part of pro­to­typ­ing is to arrive at “the thing” together. 

One of the lean UX things I devoured was this pre­sen­ta­tion by Bill Scott in which he talks about align­ing a pro­to­typ­ing and a devel­op­ment tech stack, so that the gap between design and engi­neer­ing is bridged not just with process­es but also with tool­ing. His exam­ple applies to web devel­op­ment and app devel­op­ment using web tech­nolo­gies. I won­der what a sim­i­lar approach looks like for native mobile app devel­op­ment. But this is the sort of thing I am talk­ing about: Smart think­ing about how to actu­al­ly do this lean thing in the real world. I believe organ­is­ing our­selves so that we can pro­to­type as a team is absolute­ly key. I will pick my tools and process­es accord­ing­ly in future.

All of the above is as usu­al most­ly a reminder to self: As a design­er your role is not to go off and work solo on bril­liant pro­to­types. Your role is to facil­i­tate such efforts by the whole team. Sure, there will be solo deep design­er­ly craft­ing hap­pen­ing. But it will not add up to any­thing if it is not embed­ded in a col­lab­o­ra­tive design and devel­op­ment framework.

Prototyping in the browser

When you are design­ing a web site or web app I think you should pro­to­type in the brows­er. Why? You might as well ask why pro­to­type at all. Answer: To enable con­tin­u­ous test­ing and refine­ment of your design. Since you are design­ing for the web it makes sense to do this test­ing and refine­ment with an arte­fact com­posed of the web’s material.

There are many ways to do pro­to­typ­ing. A com­mon way is to make wire­frames and then make them ‘click­able’. But when I am design­ing a web site or a web app and I get to the point where it is time to do wire­frames I often pre­fer to go straight to the browser. 

Before this step I have sketched out all the screens on paper of course. I have done mul­ti­ple sketch­es of each page. I’ve had them cri­tiqued by team mem­bers and I have reworked them. 

Drawing pictures of web pages

But then I open my draw­ing pro­gram—Sketch, in my case—and my heart sinks. Not because Sketch sucks. Sketch is great. But it some­how feels wrong to draw pic­tures of web pages on my screen. I find it cum­ber­some. My draw­ing pro­gram does not behave like a brows­er. That is to say in stead of defin­ing a bunch of rules for ele­ments and hav­ing the brows­er fig­ure out how to ren­der them on a page togeth­er I need to fol­low those rules myself in my head as I put each ele­ment in its place.

And don’t get me start­ed on how wire­frames are sup­posed to be with­out visu­al design. That is non­sense. If you are using con­trast, rep­e­ti­tion, align­ment and prox­im­i­ty, you are doing lay­out. That is visu­al design. I can’t stand wire­frames with a bad visu­al hierarchy.

If I per­se­vere, and I have a set of wire­frames in my draw­ing pro­gram, they are sta­t­ic. I can’t use them. I then need to export them to some oth­er often clunky pro­gram to make the pic­tures click­able. Which always results in a poor resem­blance of the actu­al expe­ri­ence. (I use Mar­vel. It’s okay but it is hard­ly a joy to use. For mobile apps I still use it, for web sites I pre­fer not to.)

Prototyping in the browser

When I pro­to­type in the brows­er I don’t have to deal with these issues. I am doing lay­out in a way that is native to the medi­um. And once I have some pages set up they are imme­di­ate­ly usable. So I can hand it to some­one, a team mem­ber or a test par­tic­i­pant, and let them play with it.

That is why, for web sites and web apps, I skip wire­frames alto­geth­er and pro­to­type in the brows­er. I do not know how com­mon this is in the indus­try nowa­days. So I thought I would share my approach here. It may be of use to some. 

It used to be the case that it was quite a bit of has­sle to get up and run­ning with a brows­er pro­to­type so nat­u­ral­ly open­ing a draw­ing pack­age seemed more attrac­tive. Not so any­more. Tools have come a long way. Case in point: My set­up nowa­days involves zero screw­ing around on the com­mand line.

CodeKit

The core of it is a paid-for Mac app called CodeK­it, a so-called task man­ag­er. It allows you to install a front-end devel­op­ment frame­work I like called Zurb Foun­da­tion with a cou­ple of clicks and has a built in web serv­er so you can play with your pro­to­type on any device on your local net­work. As you make changes to the code of your pro­to­type it gets auto­mat­i­cal­ly updat­ed on all your devices. No more man­u­al refresh­ing. Saves a huge amount of time.

I know you can do most of what CodeK­it does for you with stuff like Grunt but that involves tedious con­fig­u­ra­tion and work­ing the com­mand line. This is fine when you’re a devel­op­er, but not fine when you are a design­er. I want to be up and run­ning as fast as pos­si­ble. CodeK­it allows me to do that and has some oth­er fea­tures built in that are ide­al for pro­to­typ­ing which I will talk about more below. Long sto­ry short: CodeK­it has saved me a huge amount of time and is well worth the money.

Okay so on with the show. Yes, this whole pro­to­typ­ing in the brows­er thing involves ‘cod­ing’. But hon­est­ly, if you can’t write some HTML and CSS you real­ly shouldn’t be doing design for the web in the first place. I don’t care if you con­sid­er your­self a UX design­er and some­how above all this low­ly tech­ni­cal stuff. You are not. Nobody is say­ing you should become a fron­tend devel­op­er but you need to have an acquain­tance with the mate­ri­als your prod­uct is made of. Fol­low a few cours­es on Codecadamy or some­thing. There real­ly isn’t an excuse any­more these days for not know­ing this stuff. If you want to lev­el up, learn SASS.

Zurb Foundation

I like Zurb Foun­da­tion because it offers a coher­ent and com­pre­hen­sive library of ele­ments which cov­ers almost all the com­mon pat­terns found in web sites and apps. It offers a grid and some default typog­ra­phy styles as well. All of it doesn’t look flashy at all which is how I like it when I am pro­to­typ­ing. A pro­to­type at this stage does not require per­son­al­i­ty yet. Just a clear visu­al hier­ar­chy. Work­ing with Foun­da­tion is almost like play­ing with LEGO. You just click togeth­er the stuff you need. It’s pain­less and looks and works great.

I hard­ly do any styling but the few changes I do want to make I can eas­i­ly add to Foundation’s app.scss using SASS. I usu­al­ly have a few styles in there for tweak­ing some mar­gins on par­tic­u­lar ele­ments, for exam­ple a foot­er. But I try to focus on the struc­ture and behav­iour of my pages and for that I am most­ly doing HTML

GitHub

Test­ing local­ly I already men­tioned. For that, CodeK­it has you cov­ered. Of course, you want to be able to share your pro­to­type with oth­ers. For this I like to use GitHub and their Pages fea­ture. Once again, using their desk­top client, this involves zero com­mand line work. You just add the fold­er with your CodeK­it project as a new repos­i­to­ry and sync it with GitHub. Then you need to add a branch named ‘gh-pages’ and do ‘update from mas­ter’. Presto, your pro­to­type is now on the web for any­one with the URL to see and use. Per­fect if you’re work­ing in a dis­trib­uted team. 

Don’t be intim­i­dat­ed by using GitHub. Their on-board­ing is pret­ty impres­sive nowa­days. You’ll be up and run­ning in no time. Using ver­sion con­trol, even if it is just you work­ing on the pro­to­type, adds some much need­ed struc­ture and con­trol over changes. And when you are col­lab­o­rat­ing on your pro­to­type with team mem­bers it is indispensable. 

But in most cas­es I am the only one build­ing the pro­to­type so I just work on the mas­ter branch and once every while I update the gh-pages branch from mas­ter and sync it and I am done. If you use Slack you can add a GitHub bot to a chan­nel and have your team mem­bers receive an auto­mat­ic update every time you change the prototype. 

The Kit Language

If your project is of any size beyond the very small you will like­ly have repeat­ing ele­ments in your design. Head­ers, foot­ers, recur­ring wid­gets and so on. CodeK­it has recent­ly added sup­port for some­thing called the Kit Lan­guage. This adds sup­port for imports and vari­ables to reg­u­lar HTML. It is absolute­ly great for pro­to­typ­ing. For each repeat­ing ele­ment you cre­ate a ‘par­tial’ and import it wher­ev­er you need it. Vari­ables are great for chang­ing the con­tents of such repeat­ing ele­ments. CodeK­it com­piles it all into plain sta­t­ic HTML for you so your pro­to­type runs anywhere.

The Kit Lan­guage real­ly was the miss­ing piece of the puz­zle for me. With it in place I am very com­fort­able rec­om­mend­ing this way of work­ing to anyone.

So that’s my set­up: CodeK­it, Zurb Foun­da­tion and GitHub. Togeth­er they make for a very pleas­ant and pro­duc­tive way to do pro­to­typ­ing in the brows­er. I don’t imag­ine myself going back to draw­ing pic­tures of web pages any­time soon.

Design without touching the surface

I am prepar­ing two class­es at the moment. One is an intro­duc­tion to user expe­ri­ence design, the oth­er to user inter­face design. I did not come up with this divi­sion, it was part of the assign­ment. I thought it was odd at first. I wasn’t sure where one dis­ci­pline ends and the oth­er begins. I still am not sure. But I made a prag­mat­ic deci­sion to have the UX class focus on the high lev­el process of design­ing (soft­ware) prod­ucts, and the UI class focus on the visu­al aspects of a product’s inter­face. The UI class deals with a product’s sur­face, form, and to some extent also its behav­iour, but on a micro lev­el. Where­as the UX class focus­es on behav­iour on the macro lev­el. Sim­ply speaking—the UX class is about behav­iour across screens, the UI class is about behav­iour with­in screens.

The solu­tion is work­able. But I am still not entire­ly com­fort­able with it. I am not com­fort­able with the idea of being able to prac­tice UX with­out ‘touch­ing the sur­face’, so to speak. And it seems my two class­es are advo­cat­ing this. Also, I am pret­ty sure this is every­day real­i­ty for many UX prac­ti­tion­ers. Notice I say “prac­ti­tion­er”, because I am not sure ‘design­er’ is the right term in these cas­es. To be hon­est I do not think you can prac­tice design with­out doing sketch­ing and pro­to­typ­ing of some sort. (See Bill Buxton’s ‘Sketch­ing User Expe­ri­ences’ for an expand­ed argu­ment on why this is.) And when it comes to design­ing soft­ware prod­ucts this means touch­ing the sur­face, the form.

Again, the real­i­ty is, ‘UX design­er’ and ‘UI design­er’ are com­mon terms now. Cer­tain­ly here in Sin­ga­pore peo­ple know they need both to make good prod­ucts. Some prac­ti­tion­ers say they do both, oth­ers one or the oth­er. The lat­ter appears to be the most com­mon and expect­ed case. (By the way, in Sin­ga­pore no-one I’ve met talks about inter­ac­tion design.)

My con­cern is that by encour­ag­ing the prac­tice of doing UX design with­out touch­ing the sur­face of a prod­uct, we get shit­ty designs. In a process where UX and UI are seen as sep­a­rate things the risk is one comes before the oth­er. The UX design­er draws the wire­frames, the UI design­er gets to turn them into pret­ty pic­tures, with no back-and-forth between the two. An iter­a­tive process can mit­i­gate some of the dam­age such an arti­fi­cial divi­sion of labour pro­duces, but I think we still start out on the wrong foot. I think a bet­ter prac­tice might entail includ­ing visu­al con­sid­er­a­tions from the very begin­ning of the design process (as we are sketching).

Two things I came across as I was prepar­ing these class­es are some­how in sup­port of this idea. Both result­ed from a call I did for resources on user inter­face design. I asked for books about visu­al aspects, but I got a lot more.

  1. In ‘Mag­ic Ink’ Bret Vic­tor writes about how the design of infor­ma­tion soft­ware is huge­ly indebt­ed to graph­ic design and more specif­i­cal­ly infor­ma­tion design in the tra­di­tion of Tufte. (He also men­tions indus­tri­al design as an equal­ly big prog­en­i­tor of inter­ac­tion design, but for soft­ware that is main­ly about manip­u­la­tion, not infor­ma­tion.) The arti­cle is big, but the start of it is actu­al­ly a pret­ty good if unortho­dox gen­er­al intro­duc­tion to inter­ac­tion design. For soft­ware that is about learn­ing through look­ing at infor­ma­tion Vic­tor says inter­ac­tion should be a last resort. So that leaves us with a task that is 80% if not more visu­al design. Touch­ing the sur­face. Which makes me think you might as well get to it as quick­ly as pos­si­ble and start sketch­ing and pro­to­typ­ing aimed not just at struc­ture and behav­iour but also form. (Hat tip to Pieter Diepen­maat for this one.)

  2. In ‘Jump­ing to the End’ Matt Jones ram­bles enter­tain­ing­ly about design fic­tion. He argues for pay­ing atten­tion to details and that a lot of the design he prac­tices is about ‘sig­na­ture moments’ aka micro-inter­ac­tions. So yeah, again, I can’t imag­ine design­ing these effec­tive­ly with­out doing sketch­ing and pro­to­typ­ing of the sort that includes the visu­al. And in fact Matt men­tions this more or less at one point, when he talks about the fact that his team’s deliv­er­ables at Google are almost all visu­al. They are high fideli­ty mock­ups, ani­ma­tions, videos, and so on. These then become the start­ing points for fur­ther devel­op­ment. (Hat tip to Alexan­der Zeh for this one.)

In sum­ma­ry, I think dis­tin­guish­ing UX design from UI design is non­sense. Because you can­not prac­tice design with­out sketch­ing and pro­to­typ­ing. And you can­not sketch and pro­to­type a soft­ware prod­uct with­out touch­ing its sur­face. In stead of tak­ing visu­al design for grant­ed, or talk­ing about it like it is some innate tal­ent, some kind of mag­i­cal skill some peo­ple are born with and oth­ers aren’t, user expe­ri­ence prac­ti­tion­ers should con­sid­er being less enam­oured with acquir­ing more skills from busi­ness, mar­ket­ing and engi­neer­ing and in stead prac­tice at the skills that define the fields user expe­ri­ence design is indebt­ed to the most: graph­ic design and indus­tri­al design. In oth­er words, you can’t do user expe­ri­ence design with­out touch­ing the surface.

Nobody does thor­ough­ly argued pre­sen­ta­tions quite like Sebas­t­ian. This is good stuff on ethics and design.

I decid­ed to share some thoughts it sparked via Twit­ter and end­ed up rant­i­ng a bit:

I recent­ly talked about ethics to a bunch of “behav­ior design­ers” and found myself con­clud­ing that any designed sys­tem that does not allow for user appro­pri­a­tion is fun­da­men­tal­ly uneth­i­cal because as you right­ly point out what is the good life is a per­son­al mat­ter. Impos­ing it is an inher­ent­ly vio­lent act. A lot of design is a form of tech­no­log­i­cal­ly medi­at­ed vio­lence. Get­ting peo­ple to do your bid­ding, how­ev­er well intend­ed. Which giv­en my own voca­tion and work in the past is a kind of trou­bling thought to arrive at… Help?

Sebas­t­ian makes his best point on slides 113–114. Eth­i­cal design isn’t about doing the least harm, but about doing the most good. And, to come back to my Twit­ter rant, for me the ulti­mate good is for oth­ers to be free. Hence non-pre­scrip­tive design.

(via Design­ing the Good Life: Ethics and User Expe­ri­ence Design)

On sketching

Catch­ing up with this slight­ly neglect­ed blog (it’s been 6 weeks since the last prop­er post). I’d like to start by telling you about a small thing I helped out with last week. Peter Boers­ma1 asked me to help out with one of his UX Cock­tail Hours. He was inspired by a recent IxDA Stu­dio event where, in stead of just chat­ting and drink­ing, design­ers actu­al­ly made stuff. (Gasp!) Peter want­ed to do a work­shop where atten­dees col­lab­o­rat­ed on sketch­ing a solu­tion to a giv­en design problem.

Part of my con­tri­bu­tion to the evening was a short pre­sen­ta­tion on the the­o­ry and prac­tice of sketch­ing. On the the­o­ry side, I ref­er­enced Bill Bux­ton’s list of qual­i­ties that define what a sketch is2, and empha­sized that this means a sketch can be done in any mate­r­i­al, not nec­es­sar­i­ly pen­cil and paper. Fur­ther­more I dis­cussed why sketch­ing works, using part of an arti­cle on embod­ied inter­ac­tion3. The main point there, as far as I am con­cerned is that when sketch­ing, as design­ers we have the ben­e­fit of ‘back­talk’ from our mate­ri­als, which can pro­vide us with new insights. I wrapped up the pre­sen­ta­tion with a case study of a project I did a while back with the Ams­ter­dam-based agency Info.nl4 for a social web start-up aimed at inde­pen­dent pro­fes­sion­als. In the project I went quite far in using sketch­es to not only devel­op the design, but also col­lab­o­ra­tive­ly con­struct it with the client, tech­nol­o­gists and others.

The whole thing was record­ed; you can find a video of the talk at Vimeo (thanks to Iskan­der and Alper). I also uploaded the slides to SlideShare (sans notes).

The sec­ond, and most inter­est­ing part of the evening was the work­shop itself. This was set up as fol­lows: Peter and I had pre­pared a fic­tion­al case, con­cern­ing peer-to-peer ener­gy. We used the Dutch com­pa­ny Qur­rent as an exam­ple, and asked the par­tic­i­pants to con­cep­tu­alise a way to encour­age use of Qurrent’s prod­uct range. The aim was to have peo­ple be more ener­gy effi­cient, and share sur­plus ener­gy they had gen­er­at­ed with the Qur­rent com­mu­ni­ty. The par­tic­i­pants split up in teams of around ten peo­ple each, and went to work. We gave them around one hour to design a solu­tion, using only pen and paper. After­wards, they pre­sent­ed the out­come of their work to each oth­er. For each team, we asked one par­tic­i­pant to cri­tique the work by men­tion­ing one thing he or she liked, and one thing that could be improved. The team was then giv­en a chance to reply. We also asked each team to briefly reflect on their work­ing process. At the end of the evening every­one was giv­en a chance to vote for their favourite design. The win­ner received a prize.5

Wrap­ping up, I think what I liked most about the work­shop was see­ing the many dif­fer­ent ways the teams approached the prob­lem (many of the par­tic­i­pants did not know each oth­er before­hand). Group dynam­ics var­ied huge­ly. I think it was valu­able to have each team share their expe­ri­ences on this front with each oth­er. One thing that I think we could improve was the case itself; next time I would like to pro­vide par­tic­i­pants with a more focused, more rich­ly detailed brief­ing for them to sink their teeth in. That might result in an assign­ment that is more about struc­ture and behav­iour (or even inter­face) and less about con­cepts and val­ues. It would be good to see how sketch­ing func­tions in such a context.

  1. the Nether­lands’ tallest IA and one of sev­er­al famous Peters who work in UX []
  2. tak­en from his won­der­ful book Sketch­ing User Expe­ri­ences []
  3. titled How Bod­ies Mat­ter (PDF) by Kle­mer and Takaya­ma []
  4. who were also the hosts of this event []
  5. I think it’s inter­est­ing to note that the win­ner had a remark­able con­cept, but in my opin­ion was not the best exam­ple of the pow­er of sketch­ing. Appar­ent­ly the audi­ence val­ued prod­uct over process. []

A day of playing around with multi-touch and RoomWare

Last Sat­ur­day I attend­ed a RoomWare work­shop. The peo­ple of Can­Touch were there too, and brought one of their pro­to­type mul­ti-touch tables. The aim for the day was to come up with appli­ca­tions of RoomWare (open source soft­ware that can sense pres­ence of peo­ple in spaces) and mul­ti-touch. I attend­ed pri­mar­i­ly because it was a good oppor­tu­ni­ty to spend a day mess­ing around with a table.

Atten­dance was mul­ti­fac­eted, so while pro­gram­mers were putting togeth­er a proof-of-con­cept, design­ers (such as Alexan­der Zeh, James Burke and I) came up with con­cepts for new inter­ac­tions. The proof-of-con­cept was up and run­ning at the end of then day: The table could sense who was in the room and dis­play his or her Flickr pho­tos, which you could then move around, scale, rotate, etc. in the typ­i­cal mul­ti-touch fashion.

The con­cepts design­ers came up with main­ly focused on pulling in Last.fm data (again using RoomWare’s sens­ing capa­bil­i­ties) and dis­play­ing it for group-based explo­ration. Here’s a sto­ry­board I quick­ly whipped up of one such application:

RoomWare + CanTouch + Last.fm

The sto­ry­board shows how you can add your­self from a list of peo­ple present in the room. Your top artists flock around you. When more peo­ple are added, lines are drawn between you. The thick­ness of the line rep­re­sents how sim­i­lar your tastes are, accord­ing to Last.fm’s taste-o-meter. Also, shared top artists flock in such a way as to be clos­est to all relat­ed peo­ple. Final­ly, artists can be act­ed on to lis­ten to music.

When I was sketch­ing this, it became appar­ent that ori­en­ta­tion of ele­ments should fol­low very dif­fer­ent rules from reg­u­lar screens. I chose to sketch things so that they all point out­wards, with the mid­dle of the table as the ori­en­ta­tion point.

By spend­ing a day immersed in mul­ti-touch stuff, some inter­est­ing design chal­lenges became apparent:

  • With table­top sur­faces, stuff is clos­er or fur­ther away phys­i­cal­ly. Prox­im­i­ty of ele­ments can be unin­ten­tion­al­ly inter­pret­ed as say­ing some­thing about aspects such as impor­tance, rel­e­vance, etc. Design­ers need to be even more aware of place­ment than before, plus con­ven­tions from ver­ti­cal­ly ori­ent­ed screens no longer apply. Top-of-screen becomes fur­thest away and there­fore least promi­nent in stead of most important. 
  • With group-based inter­ac­tions, it becomes tricky to deter­mine who to address and where to address him or her. Some­times the sys­tem should address the group as a whole. When 5 peo­ple are stand­ing around a table, text-based inter­faces become prob­lem­at­ic since what is leg­i­ble from one end of the table is unin­tel­li­gi­ble from the oth­er. New con­ven­tions need to be devel­oped for this as well. Alexan­der and I phi­los­o­phized about plac­ing text along cir­cles and ani­mat­ing them so that they cir­cu­late around the table, for instance.
  • Besides these, many oth­er inter­face chal­lenges present them­selves. One cru­cial piece of infor­ma­tion for solv­ing many of these is know­ing where peo­ple are locat­ed around the table. This issue can be approached from dif­fer­ent angles. By incor­po­rat­ing sen­sors in the table, detec­tion may be auto­mat­ed and inter­faces could me made to adapt auto­mat­i­cal­ly. This is the tech­no-cen­tric angle. I am not con­vinced this is the way to go, because it dimin­ish­es people’s con­trol over the expe­ri­ence. I would pre­fer to make the inter­face itself adjustable in nat­ur­al ways, so that peo­ple can mold the rep­re­sen­ta­tion to suit their con­text. With sit­u­at­ed tech­nolo­gies like this, auto-mag­i­cal adap­ta­tion is an “AI-hard” prob­lem, and the price of fail­ure is a severe­ly degrad­ed user expe­ri­ence from which peo­ple can­not recov­er because the sys­tem won’t let them.

All in all the work­shop was a won­der­ful day of tin­ker­ing with like-mind­ed indi­vid­u­als from rad­i­cal­ly dif­fer­ent back­grounds. As a design­er, I think this is one of the best way be involved with open source projects. On a day like this, tech­nol­o­gists can be exposed to new inter­ac­tion con­cepts while they are hack­ing away. At the same time design­ers get that rare oppor­tu­ni­ty to play around with tech­nol­o­gy as it is shaped. Quick-and-dirty sketch­es like the ones Alexan­der and I came up with are def­i­nite­ly the way to com­mu­ni­cate ideas. The goal is to sug­gest, not to describe, after all. Tech­nol­o­gists should feel free to elab­o­rate and build on what design­ers come up with and vice-ver­sa. I am curi­ous to see which parts of what we came up with will find their way into future RoomWare projects.

Urban procedural rhetorics — transcript of my TWAB 2008 talk

This is a tran­script of my pre­sen­ta­tion at The Web and Beyond 2008: Mobil­i­ty in Ams­ter­dam on 22 May. Since the major­i­ty of pay­ing atten­dees were local I pre­sent­ed in Dutch. How­ev­er, Eng­lish appears to be the lin­gua fran­ca of the inter­net, so here I offer a trans­la­tion. I have uploaded the slides to SlideShare and hope to be able to share a video record­ing of the whole thing soon.

Update: I have uploaded a video of the pre­sen­ta­tion to Vimeo. Many thanks to Almar van der Krogt for record­ing this.

In 1966 a num­ber of mem­bers of Pro­vo took to the streets of Ams­ter­dam car­ry­ing blank ban­ners. Pro­vo was a non­vi­o­lent anar­chist move­ment. They pri­mar­i­ly occu­pied them­selves with pro­vok­ing the author­i­ties in a “ludic” man­ner. Noth­ing was writ­ten on their ban­ners because the may­or of Ams­ter­dam had banned the slo­gans “free­dom of speech”, “democ­ra­cy” and “right to demon­strate”. Regard­less, the mem­bers were arrest­ed by police, show­ing that the author­i­ties did not respect their right to demon­strate.1

Good after­noon every­one, my name is Kars Alfrink, I’m a free­lance inter­ac­tion design­er. Today I’d like to talk about play in pub­lic space. I believe that with the arrival of ubiq­ui­tous com­put­ing in the city new forms of play will be made pos­si­ble. The tech­nolo­gies we shape will be used for play wether we want to or not. As William Gib­son writes in Burn­ing Chrome:

…the street finds its own uses for things”

For exam­ple: Skate­board­ing as we now know it — with its empha­sis on aer­i­al acro­bat­ics — start­ed in emp­ty pools like this one. That was done with­out per­mis­sion, of course…

Only lat­er half-pipes, ramps, verts (which by the way is derived from ‘ver­ti­cal’) and skateparks arrived — areas where skate­board­ing is tol­er­at­ed. Skate­board­ing would not be what it is today with­out those first few emp­ty pools.2

Con­tin­ue read­ing Urban pro­ce­dur­al rhetorics — tran­script of my TWAB 2008 talk

  1. The web­site of Gram­schap con­tains a chronol­o­gy of the Pro­vo move­ment in Dutch. []
  2. For a vivid account of the emer­gence of the ver­ti­cal style of skate­board­ing see the doc­u­men­tary film Dog­town and Z‑Boys. []