August 2017 – Leapfroglog

Design and machine learning – an annotated reading list

Earlier this year I coached Design for Interaction master students at Delft University of Technology in the course Research Methodology. The students organised three seminars for which I provided the claims and assigned reading. In the seminars they argued about my claims using the Toulmin Model of Argumentation. The readings served as sources for backing and evidence.

The claims and readings were all related to my nascent research project about machine learning. We delved into both designing for machine learning, and using machine learning as a design tool.

Below are the readings I assigned, with some notes on each, which should help you decide if you want to dive into them yourself.

Hebron, Patrick. 2016. Machine Learning for Designers. Sebastopol: O’Reilly.

The only non-academic piece in this list. This served the purpose of getting all students on the same page with regards to what machine learning is, its applications of machine learning in interaction design, and common challenges encountered. I still can’t think of any other single resource that is as good a starting point for the subject as this one.

Fiebrink, Rebecca. 2016. “Machine Learning as Meta-Instrument: Human-Machine Partnerships Shaping Expressive Instrumental Creation.” In Musical Instruments in the 21st Century, 14:137–51. Singapore: Springer Singapore. doi:10.1007/978–981–10–2951–6_10.

Fiebrink’s Wekinator is groundbreaking, fun and inspiring so I had to include some of her writing in this list. This is mostly of interest for those looking into the use of machine learning for design and other creative and artistic endeavours. An important idea explored here is that tools that make use of (interactive, supervised) machine learning can be thought of as instruments. Using such a tool is like playing or performing, exploring a possibility space, engaging in a dialogue with the tool. For a tool to feel like an instrument requires a tight action-feedback loop.

Dove, Graham, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. The 2017 CHI Conference. New York, New York, USA: ACM. doi:10.1145/3025453.3025739.

A really good survey of how designers currently deal with machine learning. Key takeaways include that in most cases, the application of machine learning is still engineering-led as opposed to design-led, which hampers the creation of non-obvious machine learning applications. It also makes it hard for designers to consider ethical implications of design choices. A key reason for this is that at the moment, prototyping with machine learning is prohibitively cumbersome.

Fiebrink, Rebecca, Perry R Cook, and Dan Trueman. 2011. “Human Model Evaluation in Interactive Supervised Learning.” In, 147. New York, New York, USA: ACM Press. doi:10.1145/1978942.1978965.

The second Fiebrink piece in this list, which is more of a deep dive into how people use Wekinator. As with the chapter listed above this is required reading for those working on design tools which make use of interactive machine learning. An important finding here is that users of intelligent design tools might have very different criteria for evaluating the ‘correctness’ of a trained model than engineers do. Such criteria are likely subjective and evaluation requires first-hand use of the model in real time.

Bostrom, Nick, and Eliezer Yudkowsky. 2014. “The Ethics of Artificial Intelligence.” In The Cambridge Handbook of Artificial Intelligence, edited by Keith Frankish and William M Ramsey, 316–34. Cambridge: Cambridge University Press. doi:10.1017/CBO9781139046855.020.

Bostrom is known for his somewhat crazy but thoughtprovoking book on superintelligence and although a large part of this chapter is about the ethics of general artificial intelligence (which at the very least is still a way out), the first section discusses the ethics of current “narrow” artificial intelligence. It makes for a good checklist of things designers should keep in mind when they create new applications of machine learning. Key insight: when a machine learning system takes on work with social dimensions—tasks previously performed by humans—the system inherits its social requirements.

Yang, Qian, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning Adaptive Mobile Experiences When Wireframing. The 2016 ACM Conference. New York, New York, USA: ACM. doi:10.1145/2901790.2901858.

Finally, a feet-in-the-mud exploration of what it actually means to design for machine learning with the tools most commonly used by designers today: drawings and diagrams of various sorts. In this case the focus is on using machine learning to make an interface adaptive. It includes an interesting discussion of how to balance the use of implicit and explicit user inputs for adaptation, and how to deal with inference errors. Once again the limitations of current sketching and prototyping tools is mentioned, and related to the need for designers to develop tacit knowledge about machine learning. Such tacit knowledge will only be gained when designers can work with machine learning in a hands-on manner.

Supplemental material

Floyd, Christiane. 1984. “A Systematic Look at Prototyping.” In Approaches to Prototyping, 1–18. Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978–3–642–69796–8_1.

I provided this to students so that they get some additional grounding in the various kinds of prototyping that are out there. It helps to prevent reductive notions of prototyping, and it makes for a nice complement to Buxton’s work on sketching.

Blevis, E, Y Lim, and E Stolterman. 2006. “Regarding Software as a Material of Design.”

Some of the papers refer to machine learning as a “design material” and this paper helps to understand what that idea means. Software is a material without qualities (it is extremely malleable, it can simulate nearly anything). Yet, it helps to consider it as a physical material in the metaphorical sense because we can then apply ways of design thinking and doing to software programming.

Status update

This is not exactly a now page, but I thought I would write up what I am doing at the moment since last reporting on my status in my end-of-year report.

The majority of my workdays are spent doing freelance design consulting. My primary gig has been through Eend at the Dutch Victim Support Foundation, where until very recently I was part of a team building online services. I helped out with product strategy, setting up a lean UX design process, and getting an integrated agile design and development team up and running. The first services are now shipping so it is time for me to move on, after 10 months of very gratifying work. I really enjoy working in the public sector and I hope to be doing more of it in future.

So yes, this means I am available and you can hire me to do strategy and design for software products and services. Just send me an email.

Shortly before the Dutch national elections of this year, Iskander and I gathered a group of fellow tech workers under the banner of “Tech Solidarity NL” to discuss the concerning lurch to the right in national politics and what our field can do about it. This has developed into a small but active community who gather monthly to educate ourselves and develop plans for collective action. I am getting a huge boost out of this. Figuring out how to be a leftist in this day and age is not easy. The only way to do it is to practice and for that reflection with peers is invaluable. Building and facilitating a group like this is hugely educational too. I have learned a lot about how a community is boot-strapped and nurtured.

If you are in the Netherlands, your politics are left of center, and you work in technology, consider yourself invited to join.

And finally, the last major thing on my plate is a continuing effort to secure a PhD position for myself. I am getting great support from people at Delft University of Technology, in particular Gerd Kortuem. I am focusing on internet of things products that have features driven by machine learning. My ultimate aim is to develop prototyping tools for design and development teams that will help them create more innovative and more ethical solutions. The first step for this will be to conduct field research inside companies who are creating such products right now. So I am reaching out to people to see if I can secure a reasonable amount of potential collaborators for this, which will go a long way in proving the feasibility of my whole plan.

If you know of any companies that develop consumer-facing products that have a connected hardware component and make use of machine learning to drive features, do let me know.

That’s about it. Freelance UX consulting, leftist tech-worker organising and design-for-machine-learning research. Quite happy with that mix, really.

‘Hybrid Writing for Conversational Interfaces’ workshop

On May 24 of this year, Niels ’t Hooft and myself ran a workshop titled ‘Hybrid Writing for Conversational Interfaces’ at TU Delft. Our aim was twofold: teach students about writing characters and dialog, and teach them how to prototype chat interfaces.

We spent a day with roughly thirty industrial design students alternating between bits of theory, writing exercises, instructions on how to use Twine (our prototyping tool of choice) and closed out with a small project and a show and tell.

I was very pleased to see prototypes with quite a high level of complexity and sophistication at the end of the day. And throughout, I could tell students were enjoying themselves writing and building interactive conversations.

Here’s a rough outline of how the workshop was structured.

After briefly introducing ourselves, Niels presented a mini-lecture on interactive fiction. A highlight for me was a two-by-two of the ways in which fiction and software can intersect.

I then took over and did a show and tell of the absolute basics of using Twine. Things like creating passages, linking them, creating branches and testing and publishing your story.
The first exercise after this was for students to take what they just learned about Twine and try to create a very simple interactive story.
After a coffee break, Niels then presented his second mini-lecture on the very basics of writing. With a particular focus on writing characters and dialog. This included a handy cheatsheet for things to consider while writing.

In our second exercise students worked in pairs. They first each created a character, which they then described to each other. They then first planned out the structure of an encounter between these two characters. And finally they collaboratively wrote the dialogue for this encounter. They were required to stick to Hollywood formatting. Niels and I then did a reading of a few (to great amusement of all present) to close out the morning section of the workshop.
After lunch Niels presented his third and final mini-lecture of the day, on conversational interfaces, relying heavily on the great work of our friend Alper in his book on the subject.
I then took over for the second show and tell. Here we ramped up the challenge and introduced the Twine Texting Project – a framework for prototyping conversational interfaces in Twine. On GitHub, you can find the starter file I had prepared for this section.
The third and final exercise of the day was for students to take what they learned about writing dialog, and prototyping chat interfaces, and to build an interactive prototype of a conversational interface or interactive fiction in chat format. They could either build off of the dialog they have created in the previous exercise, or start from scratch.
We finished the day with demos, where put the Twine story on the big screen and as a group chose what options to select. After each demo the creator would open up the Twine file and walk us through how they had built it. It was pretty cool to see how many students had put what they had learned to very creative uses.

Reflecting on the workshop afterwards, we felt the structure was nicely balanced between theory and practice. The difficulty level was such that students did learn some new things which they could incorporate into future projects, but still built on skills they had already acquired. The choice for Twine worked out well too since it is highly accessible. Non-technical students managed to create something interactive, and more advanced students could apply what they knew about code to produce more sophisticated prototypes.

For future workshops we did feel we could improve on building a bridge between the writing for interactive fiction and writing for conversational interfaces of software products and services. This would require some adaptation of the mini lectures and a slightly different emphasis in the exercises. The key would be to have students imagine existing products and services as characters, and to then write dialog for interactions and prototype them. For a future iteration of the workshop, this would be worth exploring further.

Many thanks to Ianus Keller for inviting us to teach this workshop at IDE Academy.