Design and machine learning – an annotated reading list

Ear­li­er this year I coached Design for Inter­ac­tion mas­ter stu­dents at Delft Uni­ver­si­ty of Tech­nol­o­gy in the course Research Method­ol­o­gy. The stu­dents organ­ised three sem­i­nars for which I pro­vid­ed the claims and assigned read­ing. In the sem­i­nars they argued about my claims using the Toul­min Mod­el of Argu­men­ta­tion. The read­ings served as sources for back­ing and evidence.

The claims and read­ings were all relat­ed to my nascent research project about machine learn­ing. We delved into both design­ing for machine learn­ing, and using machine learn­ing as a design tool.

Below are the read­ings I assigned, with some notes on each, which should help you decide if you want to dive into them yourself.

Hebron, Patrick. 2016. Machine Learn­ing for Design­ers. Sebastopol: O’Reilly.

The only non-aca­d­e­m­ic piece in this list. This served the pur­pose of get­ting all stu­dents on the same page with regards to what machine learn­ing is, its appli­ca­tions of machine learn­ing in inter­ac­tion design, and com­mon chal­lenges encoun­tered. I still can’t think of any oth­er sin­gle resource that is as good a start­ing point for the sub­ject as this one.

Fiebrink, Rebec­ca. 2016. “Machine Learn­ing as Meta-Instru­ment: Human-Machine Part­ner­ships Shap­ing Expres­sive Instru­men­tal Cre­ation.” In Musi­cal Instru­ments in the 21st Cen­tu­ry, 14:137–51. Sin­ga­pore: Springer Sin­ga­pore. doi:10.1007/978–981–10–2951–6_10.

Fiebrink’s Wek­ina­tor is ground­break­ing, fun and inspir­ing so I had to include some of her writ­ing in this list. This is most­ly of inter­est for those look­ing into the use of machine learn­ing for design and oth­er cre­ative and artis­tic endeav­ours. An impor­tant idea explored here is that tools that make use of (inter­ac­tive, super­vised) machine learn­ing can be thought of as instru­ments. Using such a tool is like play­ing or per­form­ing, explor­ing a pos­si­bil­i­ty space, engag­ing in a dia­logue with the tool. For a tool to feel like an instru­ment requires a tight action-feed­back loop.

Dove, Gra­ham, Kim Hal­skov, Jodi For­l­izzi, and John Zim­mer­man. 2017. UX Design Inno­va­tion: Chal­lenges for Work­ing with Machine Learn­ing as a Design Mate­r­i­al. The 2017 CHI Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/3025453.3025739.

A real­ly good sur­vey of how design­ers cur­rent­ly deal with machine learn­ing. Key take­aways include that in most cas­es, the appli­ca­tion of machine learn­ing is still engi­neer­ing-led as opposed to design-led, which ham­pers the cre­ation of non-obvi­ous machine learn­ing appli­ca­tions. It also makes it hard for design­ers to con­sid­er eth­i­cal impli­ca­tions of design choic­es. A key rea­son for this is that at the moment, pro­to­typ­ing with machine learn­ing is pro­hib­i­tive­ly cumbersome.

Fiebrink, Rebec­ca, Per­ry R Cook, and Dan True­man. 2011. “Human Mod­el Eval­u­a­tion in Inter­ac­tive Super­vised Learn­ing.” In, 147. New York, New York, USA: ACM Press. doi:10.1145/1978942.1978965.

The sec­ond Fiebrink piece in this list, which is more of a deep dive into how peo­ple use Wek­ina­tor. As with the chap­ter list­ed above this is required read­ing for those work­ing on design tools which make use of inter­ac­tive machine learn­ing. An impor­tant find­ing here is that users of intel­li­gent design tools might have very dif­fer­ent cri­te­ria for eval­u­at­ing the ‘cor­rect­ness’ of a trained mod­el than engi­neers do. Such cri­te­ria are like­ly sub­jec­tive and eval­u­a­tion requires first-hand use of the mod­el in real time. 

Bostrom, Nick, and Eliez­er Yud­kowsky. 2014. “The Ethics of Arti­fi­cial Intel­li­gence.” In The Cam­bridge Hand­book of Arti­fi­cial Intel­li­gence, edit­ed by Kei­th Frank­ish and William M Ram­sey, 316–34. Cam­bridge: Cam­bridge Uni­ver­si­ty Press. doi:10.1017/CBO9781139046855.020.

Bostrom is known for his some­what crazy but thought­pro­vok­ing book on super­in­tel­li­gence and although a large part of this chap­ter is about the ethics of gen­er­al arti­fi­cial intel­li­gence (which at the very least is still a way out), the first sec­tion dis­cuss­es the ethics of cur­rent “nar­row” arti­fi­cial intel­li­gence. It makes for a good check­list of things design­ers should keep in mind when they cre­ate new appli­ca­tions of machine learn­ing. Key insight: when a machine learn­ing sys­tem takes on work with social dimensions—tasks pre­vi­ous­ly per­formed by humans—the sys­tem inher­its its social requirements.

Yang, Qian, John Zim­mer­man, Aaron Ste­in­feld, and Antho­ny Toma­sic. 2016. Plan­ning Adap­tive Mobile Expe­ri­ences When Wire­fram­ing. The 2016 ACM Con­fer­ence. New York, New York, USA: ACM. doi:10.1145/2901790.2901858.

Final­ly, a feet-in-the-mud explo­ration of what it actu­al­ly means to design for machine learn­ing with the tools most com­mon­ly used by design­ers today: draw­ings and dia­grams of var­i­ous sorts. In this case the focus is on using machine learn­ing to make an inter­face adap­tive. It includes an inter­est­ing dis­cus­sion of how to bal­ance the use of implic­it and explic­it user inputs for adap­ta­tion, and how to deal with infer­ence errors. Once again the lim­i­ta­tions of cur­rent sketch­ing and pro­to­typ­ing tools is men­tioned, and relat­ed to the need for design­ers to devel­op tac­it knowl­edge about machine learn­ing. Such tac­it knowl­edge will only be gained when design­ers can work with machine learn­ing in a hands-on manner.

Supplemental material

Floyd, Chris­tiane. 1984. “A Sys­tem­at­ic Look at Pro­to­typ­ing.” In Approach­es to Pro­to­typ­ing, 1–18. Berlin, Hei­del­berg: Springer Berlin Hei­del­berg. doi:10.1007/978–3–642–69796–8_1.

I pro­vid­ed this to stu­dents so that they get some addi­tion­al ground­ing in the var­i­ous kinds of pro­to­typ­ing that are out there. It helps to pre­vent reduc­tive notions of pro­to­typ­ing, and it makes for a nice com­ple­ment to Buxton’s work on sketch­ing.

Ble­vis, E, Y Lim, and E Stolter­man. 2006. “Regard­ing Soft­ware as a Mate­r­i­al of Design.”

Some of the papers refer to machine learn­ing as a “design mate­r­i­al” and this paper helps to under­stand what that idea means. Soft­ware is a mate­r­i­al with­out qual­i­ties (it is extreme­ly mal­leable, it can sim­u­late near­ly any­thing). Yet, it helps to con­sid­er it as a phys­i­cal mate­r­i­al in the metaphor­i­cal sense because we can then apply ways of design think­ing and doing to soft­ware programming.

Status update

This is not exact­ly a now page, but I thought I would write up what I am doing at the moment since last report­ing on my sta­tus in my end-of-year report.

The major­i­ty of my work­days are spent doing free­lance design con­sult­ing. My pri­ma­ry gig has been through Eend at the Dutch Vic­tim Sup­port Foun­da­tion, where until very recent­ly I was part of a team build­ing online ser­vices. I helped out with prod­uct strat­e­gy, set­ting up a lean UX design process, and get­ting an inte­grat­ed agile design and devel­op­ment team up and run­ning. The first ser­vices are now ship­ping so it is time for me to move on, after 10 months of very grat­i­fy­ing work. I real­ly enjoy work­ing in the pub­lic sec­tor and I hope to be doing more of it in future.

So yes, this means I am avail­able and you can hire me to do strat­e­gy and design for soft­ware prod­ucts and ser­vices. Just send me an email.

Short­ly before the Dutch nation­al elec­tions of this year, Iskan­der and I gath­ered a group of fel­low tech work­ers under the ban­ner of “Tech Sol­i­dar­i­ty NL to dis­cuss the con­cern­ing lurch to the right in nation­al pol­i­tics and what our field can do about it. This has devel­oped into a small but active com­mu­ni­ty who gath­er month­ly to edu­cate our­selves and devel­op plans for col­lec­tive action. I am get­ting a huge boost out of this. Fig­ur­ing out how to be a left­ist in this day and age is not easy. The only way to do it is to prac­tice and for that reflec­tion with peers is invalu­able. Build­ing and facil­i­tat­ing a group like this is huge­ly edu­ca­tion­al too. I have learned a lot about how a com­mu­ni­ty is boot-strapped and nurtured.

If you are in the Nether­lands, your pol­i­tics are left of cen­ter, and you work in tech­nol­o­gy, con­sid­er your­self invit­ed to join.

And final­ly, the last major thing on my plate is a con­tin­u­ing effort to secure a PhD posi­tion for myself. I am get­ting great sup­port from peo­ple at Delft Uni­ver­si­ty of Tech­nol­o­gy, in par­tic­u­lar Gerd Kortuem. I am focus­ing on inter­net of things prod­ucts that have fea­tures dri­ven by machine learn­ing. My ulti­mate aim is to devel­op pro­to­typ­ing tools for design and devel­op­ment teams that will help them cre­ate more inno­v­a­tive and more eth­i­cal solu­tions. The first step for this will be to con­duct field research inside com­pa­nies who are cre­at­ing such prod­ucts right now. So I am reach­ing out to peo­ple to see if I can secure a rea­son­able amount of poten­tial col­lab­o­ra­tors for this, which will go a long way in prov­ing the fea­si­bil­i­ty of my whole plan.

If you know of any com­pa­nies that devel­op con­sumer-fac­ing prod­ucts that have a con­nect­ed hard­ware com­po­nent and make use of machine learn­ing to dri­ve fea­tures, do let me know.

That’s about it. Free­lance UX con­sult­ing, left­ist tech-work­er organ­is­ing and design-for-machine-learn­ing research. Quite hap­py with that mix, really.

Hybrid Writing for Conversational Interfaces’ workshop

On May 24 of this year, Niels ’t Hooft and myself ran a work­shop titled ‘Hybrid Writ­ing for Con­ver­sa­tion­al Inter­faces’ at TU Delft. Our aim was twofold: teach stu­dents about writ­ing char­ac­ters and dia­log, and teach them how to pro­to­type chat interfaces. 

We spent a day with rough­ly thir­ty indus­tri­al design stu­dents alter­nat­ing between bits of the­o­ry, writ­ing exer­cis­es, instruc­tions on how to use Twine (our pro­to­typ­ing tool of choice) and closed out with a small project and a show and tell. 

I was very pleased to see pro­to­types with quite a high lev­el of com­plex­i­ty and sophis­ti­ca­tion at the end of the day. And through­out, I could tell stu­dents were enjoy­ing them­selves writ­ing and build­ing inter­ac­tive conversations.

Here’s a rough out­line of how the work­shop was structured.

  1. After briefly intro­duc­ing our­selves, Niels pre­sent­ed a mini-lec­ture on inter­ac­tive fic­tion. A high­light for me was a two-by-two of the ways in which fic­tion and soft­ware can intersect.

Four types of software-fiction hybrids

  1. I then took over and did a show and tell of the absolute basics of using Twine. Things like cre­at­ing pas­sages, link­ing them, cre­at­ing branch­es and test­ing and pub­lish­ing your story.
  2. The first exer­cise after this was for stu­dents to take what they just learned about Twine and try to cre­ate a very sim­ple inter­ac­tive story.
  3. After a cof­fee break, Niels then pre­sent­ed his sec­ond mini-lec­ture on the very basics of writ­ing. With a par­tic­u­lar focus on writ­ing char­ac­ters and dia­log. This includ­ed a handy cheat­sheet for things to con­sid­er while writing.

A cheatsheet for writing dialog

  1. In our sec­ond exer­cise stu­dents worked in pairs. They first each cre­at­ed a char­ac­ter, which they then described to each oth­er. They then first planned out the struc­ture of an encounter between these two char­ac­ters. And final­ly they col­lab­o­ra­tive­ly wrote the dia­logue for this encounter. They were required to stick to Hol­ly­wood for­mat­ting. Niels and I then did a read­ing of a few (to great amuse­ment of all present) to close out the morn­ing sec­tion of the workshop.
  2. After lunch Niels pre­sent­ed his third and final mini-lec­ture of the day, on con­ver­sa­tion­al inter­faces, rely­ing heav­i­ly on the great work of our friend Alper in his book on the subject.
  3. I then took over for the sec­ond show and tell. Here we ramped up the chal­lenge and intro­duced the Twine Tex­ting Project – a frame­work for pro­to­typ­ing con­ver­sa­tion­al inter­faces in Twine. On GitHub, you can find the starter file I had pre­pared for this section.
  4. The third and final exer­cise of the day was for stu­dents to take what they learned about writ­ing dia­log, and pro­to­typ­ing chat inter­faces, and to build an inter­ac­tive pro­to­type of a con­ver­sa­tion­al inter­face or inter­ac­tive fic­tion in chat for­mat. They could either build off of the dia­log they have cre­at­ed in the pre­vi­ous exer­cise, or start from scratch.
  5. We fin­ished the day with demos, where put the Twine sto­ry on the big screen and as a group chose what options to select. After each demo the cre­ator would open up the Twine file and walk us through how they had built it. It was pret­ty cool to see how many stu­dents had put what they had learned to very cre­ative uses.

Reflect­ing on the work­shop after­wards, we felt the struc­ture was nice­ly bal­anced between the­o­ry and prac­tice. The dif­fi­cul­ty lev­el was such that stu­dents did learn some new things which they could incor­po­rate into future projects, but still built on skills they had already acquired. The choice for Twine worked out well too since it is high­ly acces­si­ble. Non-tech­ni­cal stu­dents man­aged to cre­ate some­thing inter­ac­tive, and more advanced stu­dents could apply what they knew about code to pro­duce more sophis­ti­cat­ed prototypes. 

For future work­shops we did feel we could improve on build­ing a bridge between the writ­ing for inter­ac­tive fic­tion and writ­ing for con­ver­sa­tion­al inter­faces of soft­ware prod­ucts and ser­vices. This would require some adap­ta­tion of the mini lec­tures and a slight­ly dif­fer­ent empha­sis in the exer­cis­es. The key would be to have stu­dents imag­ine exist­ing prod­ucts and ser­vices as char­ac­ters, and to then write dia­log for inter­ac­tions and pro­to­type them. For a future iter­a­tion of the work­shop, this would be worth explor­ing further.

Many thanks to Ianus Keller for invit­ing us to teach this work­shop at IDE Acad­e­my.