Adapting intelligent tools for creativity

I read Alper’s book on conversational user interfaces over the weekend and was struck by this paragraph:

“The holy grail of a conversational system would be one that’s aware of itself — one that knows its own model and internal structure and allows you to change all of that by talking to it. Imagine being able to tell Siri to tone it down a bit with the jokes and that it would then actually do that.”

His point stuck with me because I think this is of particular importance to creative tools. These need to be flexible so that a variety of people can use them in different circumstances. This adaptability is what lends a tool depth.

The depth I am thinking of in creative tools is similar to the one in games, which appears to be derived from a kind of semi-orderedness. In short, you’re looking for a sweet spot between too simple and too complex.

And of course, you need good defaults.

Back to adaptation. This can happen in at least two ways on the interface level: modal or modeless. A simple example of the former would be to go into a preferences window to change the behaviour of your drawing package. Similarly, modeless adaptation happens when you rearrange some panels to better suit the task at hand.

Returning to Siri, the equivalence of modeless adaptation would be to tell her to tone it down when her sense of humor irks you.

For the modal solution, imagine a humor slider in a settings screen somewhere. This would be a terrible solution because it offers a poor mapping of a control to a personality trait. Can you pinpoint on a scale of 1 to 10 your preferred amount of humor in your hypothetical personal assistant? And anyway, doesn’t it depend on a lot of situational things such as your mood, the particular task you’re trying to complete and so on? In short, this requires something more situated and adaptive.

So just being able to tell Siri to tone it down would be the equivalent of rearranging your Photoshop palets. And in a next interaction Siri might carefully try some humor again to gauge your response. And if you encourage her, she might be more humorous again.

Enough about funny Siri for now because it’s a bit of a silly example.

Funny Siri, although she’s a bit of a Silly example, does illustrate another problem I am trying to wrap my head around. How does an intelligent tool for creativity communicate its internal state? Because it is probabilistic, it can’t be easily mapped to a graphic information display. And so our old way of manipulating state, and more specifically adapting a tool to our needs becomes very different too.

It seems to be best for an intelligent system to be open to suggestions from users about how to behave. Adapting an intelligent creative tool is less like rearranging your workspace and more like coordinating with a coworker.

My ideal is for this to be done in the same mode (and so using the same controls) as when doing the work itself. I expect this to allow for more fluid interactions, going back and forth between doing the work at hand, and meta-communication about how the system supports the work. I think if we look at how people collaborate this happens a lot, communication and meta-communication going on continuously in the same channels.

We don’t need a self-aware artificial intelligence to do this. We need to apply what computer scientists call supervised learning. The basic idea is to provide a system with example inputs and desired outputs, and let it infer the necessary rules from them. If the results are unsatisfactory, you simply continue training it until it performs well enough.

A super fun example of this approach is the Wekinator, a piece of machine learning software for creating musical instruments. Below is a video in which Wekinator’s creator Rebecca Fiebrink performs several demos.

Here we have an intelligent system learning from examples. A person manipulating data in stead of code to get to a particular desired behaviour. But what Wekinator lacks and what I expect will be required for this type of thing to really catch on is for the training to happen in the same mode or medium as the performance. The technology seems to be getting there, but there are many interaction design problems remaining to be solved.

Play in social and tangible interactions

Now that the IxDA has posted a video of my presentation at Interaction 09 to Vimeo, I thought it would be a good idea to provide a little background to the talk. I had already posted the slides to SlideShare, so a full write-up doesn’t seem necessary. To provide a little context though, I will summarize the thing.


The idea of the talk was to look at a few qualities of embodied interaction, and relate them to games and play, in the hopes of illuminating some design opportunities. Without dwelling on what embodiment really means, suffice to say that there is a school of thought that states that our thinking originates in our bodily experience of the world around us, and our relationships with the people in it. I used the example of an improvised information display I once encountered in the paediatric ward of a local hospital to highlight two qualities of embodied interaction: (1) meaning is socially constructed and (2) cognition is facilitated by tangibility.1


With regards to the first aspect — the social construction of meaning — I find it interesting that in games, you find a distinction between the official rules to a game, and the rules that are arrived at through mutual consent by the players, the latter being how the game is actually played. Using the example of an improvised manège in Habbo, I pointed out that under-specified design tends to encourage the emergence of such interesting uses. What it comes down to, as a designer, is to understand that once people get together to do stuff, and it involves the thing you’ve designed, they will layer new meanings on top of what you came up with, which is largely out of your control.


For the second aspect — cognition being facilitated by tangibility — I talked about how people use the world around them to offload mental computation. For instance, when people get better at playing Tetris, they start backtracking more than when they just started playing. They are essentially using the game’s space to think with. As an aside, I pointed out that in my experience, sketching plays a similar role when designing. As with the social construction of meaning, for epistemic action to be possible, the system in use needs to be adaptable.


To wrap up, I suggested that, when it comes to the design of embodied interactive stuff, we are struggling with the same issues as game designers. We’re both positioning ourselves (in the words of Eric Zimmerman) as meta-creators of meaning; as designers of spaces in which people discover new things about themselves, the world around them and the people in it.


I had several people come up to me afterwards, asking for sources, so I’ll list them here.

  • the significance of the social construction of meaning for interaction design is explained in detail by Paul Dourish in his book Where the Action Is
  • the research by Jean Piaget I quoted is from his book The Moral Judgement of the Child (which I first encountered in Rules of Play, see below)
  • the concept of ideal versus real rules is from the wonderful book Rules of Play by Katie Salen and Eric Zimmerman (who in turn have taken it from Kenneth Goldstein’s article Strategies in Counting Out)
  • for a wonderful description of how children socially mediate the rules to a game, have a look at the article Beyond the Rules of the Game by Linda Hughes (collected in the Game Design Reader)
  • the Will Wright quote is from an interview in Tracy Fullerton’s book Game Design Workshop, second edition
  • for a discussion of pragmatic versus epistemic action and how it relates to interaction design, refer to the article How Bodies Matter (PDF) by Scott Klemmer, Björn Hartmann and Leila Takayama (which is rightfully recommended by Dan Saffer in his book, Designing Gestural Interfaces)
  • the Tetris research (which I first found in the previously mentioned article) is described in Epistemic Action Increases With Skill (PDF), an article by Paul Maglio and David Kirsh
  • the “play is free movement…” quote is from Rules of Play
  • the picture of the guy skateboarding is a still from the awesome documentary film Dogtown and Z-Boys
  • for a lot of great thinking on “loose fit” design, be sure to check out the book How Buildings Learn by Stewart Brand
  • the “meta-creators of meaning” quote is from Eric Zimmerman’s foreword to the aforementioned Game Design Workshop, 2nd ed.


And that’s it. Interaction 09 was a great event, I’m happy to have been a part of it. Most of the talks seem to be online now. So why not check them out? My favourites by far were John Thackara and Robert Fabricant. Thanks to the people of the IxDA for all the effort they put into increasing interaction design’s visibility to the world.

  1. For a detailed discussion of the information display, have a look at this blog post. []

Embodied interaction and improvised information displays

Recently a good friend of mine became a dad. It made me feel really old, but it also lead to an encounter with an improvised information display, which I’d like to tell you about, because it illustrates some of the things I have learnt from reading Paul Dourish’s Where the Action Is.

My friend’s son was born a bit too early, so we went to see him (the son) at the neonatology ward of the local hospital. It was there that I saw this whiteboard with stickers, writing and the familiar magnets on it:

Tracing of a photo of an improvised information display in a hospital neonatology ward consisting of a whiteboard, magnets, stickers and writing

(I decided to trace the photo I took of it and replace the names with fictional ones.)

Now, at first I only noticed parts of what was there. I saw the patient names on the left-hand side, and recognised the name of my friend’s son. I also noticed that on the right-hand side, the names of all the nurses on duty were there. I did not think much more of it.

Before leaving, my friend walked up to the whiteboard and said something along the lines of “yes, this is correct,” and touched one of the green magnets that was in the middle of the board as if to confirm this. It was then that my curiosity was piqued, and I asked my friend to explain what the board meant.

It turns out it was a wonderful thing, something I’ll call an improvised information display, for lack of a better word. What I had not seen the first time around, but were pointed out by my friend:

  1. There is a time axis along the top of the board. By placing a green magnet at the height of a child’s name somewhere along this axis, parents can let the staff know when they intend to visit. This is important for many reasons. One being that it helps the nurses time the moment a child will be fed so that the parents can be present. So in the example, the parents of ‘Faramond’ will be visiting around 21:00 hours.
  2. There are different colour magnets behind the children’s names, and behind the nurses’ names. This shows which nurse is responsible for which child. For instance, ‘Charlotte’ is in charge of ‘Once’s’ care.

Dourish’s book has influenced the way I look at things like this. It has made me more aware of their unique value. Whereas before I would think that something like this could be done better by a proper designer, with digital means, I now think the grasp-able aspect of such a display is vital. I also now believe that the prominent role of users in shaping the display is vital. Dourish writes:1

“What embodied interaction adds to existing representational practice is the understanding that representations are also themselves artefacts. Not only do they allow users to “reach through” and act upon the entity being represented, but they can also themselves be acted upon—picked up, examined, manipulated and rearranged.”

Parents and nurses reach through the display I saw in the neonatology ward to act upon the information about visiting times and responsibility of care. But they also act on the components of the display itself to manipulate the meaning they have.

In fact, this is how the display was constructed in the first place! The role of the designer in this display was limited to the components themselves. Designers were responsible for the affordances of the whiteboard, the magnets, the erasable markers and stickers, which enabled users to produce the information display they needed. In the words of Dourish:2

“Principle: Users, not designers, create and communicate meaning.”

“Principle: Users, not designers, manage coupling.”

It is the nurses and the parents and the social practice they together constitute that gives rise to the meaning of the display. What the board means is obvious to them, because they have ‘work’ that needs to be done together. It was not obvious to me, because I am not part of that group. It was not a designer that decided what the meaning of the different colours of the magnets were. It was a group of users who coupled meaning to the components they had available to them.

It might be a radical example, but I think this does demonstrate what people can do if the right components are made available to them, and they are allowed to make their own meaning with them. I think it is important for designers to realise this, and allow for this kind of manipulation of the products and services they shape. Clearly, Dourish’s notion of embodied interaction is a key to designing for adaptation and hacking. When it comes to this, today’s whiteboards, magnets and markers seem to do a better job than many of our current digital technologies.

  1. Page 169 []
  2. Page 170 []

Slides and summary for ‘More Than Useful’

Update: The video and slides are now available on the conference site.

The conference From Business to Buttons 2008 aimed to bring together the worlds of business and interaction design. I was there to share my thoughts on the applicability of game design concepts to interaction design. You’ll find my slides and a summary of my argument below.

I really enjoyed attending this conference. I met a bunch of new and interesting people and got to hang out with some ‘old’ friends. Many thanks to InUse for inviting me.

Diagram summarizing my FBTB 2008 talk

The topic is pretty broad so I decided to narrow things down to a class of product that is other-than-everyday — meaning both wide and deep in scope. Using Norman’s The Design of Everyday Things as a starting point, I wanted to show that these products require a high level of explorability that is remarkably similar to play. After briefly examining the phenomenon of play itself I moved on to show applications of this understanding to two types of product: customizable & personalizable ones, and adaptive ones.

For the former, I discussed how game design frameworks such as MDA can help with sculpting the parameter space, using ‘experience’ as the starting point. I also looked at how games support players in sharing stories and speculated about ways this can be translated to both digital and physical products.

For the latter — adaptive products — I focussed on the ways in which they induce flow and how they can recommend stuff to people. With adaptation, designers need to formulate rules. This can be done using techniques from game design, such as Daniel Cook’s skill chains. Successful rules-based design can only happen in an iterative environment using lots of sketching.

The presentation was framed by a slightly philosophical look at how certain games subliminally activate cognitive processes and could thus be used to allow for new insights. I used Breakout and Portal as examples of this. I am convinced there is an emerging field of playful products that interaction designers should get involved with.

Sources referenced in this presentation:1

As usual, many thanks to all the Flickr photographers who’ve shared their images under a CC license. I’ve linked to the originals from the slides. Any image not linked to is probably mine.

  1. Most of these are offline books or papers, those that aren’t have been hyperlinked to their source. []

Notes on play, exploration, challenge and learning

(My reading notes are piling up so here’s an attempt to clear out at least a few of them.)

Part of the play experience of many digital games is figuring out how the damn thing works in the first place. In Rules of Play on page 210:

“[…] as the player plays with FLUID, interaction and observation reveals the underlying principles of the system. In this case the hidden information gradually revealed through play is the rules of the simulation itself. Part of the play of FLUID is the discovery of the game rules as information.”

(Sadly, I could not find a link to the game mentioned.)

I did not give Donald Norman all the credit he was due in my earlier post. He doesn’t have a blind spot for games. Quite the contrary. For instance, he explains how to make systems easier to learn and points to games in the process. On page 183 of The Design of Everyday Things:

“One important method of making systems easier to learn and to use is to make them explorable, to encourage the user to experiment and learn the possibilities through active exploration.”

The way to do this is through direct manipulation, writes Norman. He also reminds us that it’s not necessary to make any system explorable.1 But (on page 184):

“[…] if the job is critical, novel, or ill-specified, or if you do not yet know exactly what is to be done, then you need direct, first-person interaction.”

So much written after DOET seems to have added little to the conversation. I’m surprised how useful this classic still is.

I’m reminded of a section of Matt Jones’s Interaction 08 talk—which I watched yesterday. He went through a number of information visualisations and said he’d like to add more stuff like that into Dopplr, to allow people to play with their data. He even compared this act of play to Will Wright’s concept of possibility space.2 He also briefly mentioned that easily accessible tools for creating information visualisations might become a valuable tool for designers working with complex sets of data.

Norman actually points to games for inspiration, by the way. On page 184 just before the previous quote:

“Some computer systems offer direct manipulation, first-person interactions, good examples being the driving, flying, and sports games that are commonplace in arcades and on home machines. In these games, the feeling of direct control over the actions is an essential part of the task.”

And so on.

One of the most useful parts of Dan Saffer’s book on interaction design is where he explains the differences between customisation, personalisation, adaptation and hacking. He notes that an adaptive system can be designed to induce flow—balancing challenge with the skill of the user. In games, there is something called dynamic difficulty adjustment (DDA) which has very similar aims.

Salen and Zimmerman have their doubts about DDA though. In Rules of Play on page 223 they write:

“Playing a game becomes less like learning an expressive language and more like being the sole audience member for a participatory, improvisational performance, where the performers adjust their actions to how you interact with them. Are you then playing the game, or is it playing you?”

Perhaps, but it all depends on what DDA actually adjusts. The technique might be objectionable in a game (where a large part of the point is overcoming challenge) but in other systems many of these objections do not apply.

“With a successful adaptive design, the product fits the user’s life and environment as though it were custom made.”

(Designing for Interaction, page 162.)

Adaptive systems explicitly anticipate transformative play. They allow themselves to be changed through a person’s interactions with it.3

A characteristic of good interaction design is playfulness, writes Mr. Saffer in his book on page 67:

“Through serious play, we seek out new products, services and features and then try them to see how they work. How many times have you pushed a button just to see what it did?”

The funny thing is, the conditions for play according to Saffer are very similar to some of the basic guidelines Norman offers: Make users feel comfortable, reduce the chance for errors and if errors do occur, make sure the consequences are small—by allowing users to undo, for instance.

Mr. Norman writes that in games “designers deliberately flout the laws of understandability and usability” (p.205). Although even in games: “[the] rules [of usability] must be applied intelligently, for ease of use or difficulty of use” (p.208).

By now, it should be clear making interactions playful is very different from making them game-like.

  1. Apparently, “explorable” isn’t a proper English word, but if it’s good enough for Mr. Norman it’s good enough for me. []
  2. I blogged about possibility space before here. []
  3. Yes, I know I blogged about adaptive design before. Also about flow and adaptation, it seems. []

Adaptive design and transformative play

2006APR201648 by bootload on Flickr

Allowing people to change parts of your product is playful. It has also always ‘just’ seemed like a good thing to do to me. You see this with with people who become passionate about a thing they use often: They want to take it apart, see how it works, put it back together again, maybe add some stuff, replace something else… I’ve always liked the idea of passionate people wanting to change something about a thing I designed. And it’s always been a disappointment when I’d find out that they did not, or worse—wanted to but weren’t able to.

Apparently this is what people call adaptive design. But if you Google that, you won’t find much. In fact, there’s remarkably little written about it. I was put on the term’s trail by Matt Webb and from there found my way to Dan Hill’s site. There’s a lot on the topic there, but if I can recommend one piece it’s the interview he did for Dan Saffer’s book on interaction design. Read it. It’s full of wonderful ideas articulated 100 times better than I’ll ever be able to.

So why is adaptive design conducive to the playfulness of a user experience? I’m not sure. One aspect of it might be the fact that as a designer you explicitly relinquish some control over the final experience people have with your…stuff.1 As Matt Webb noted in an end-of-the-year post, in stead of saying to people: “Here’s something I made. Go on—play with it.” You say: “Here’s something I made—let’s play with it together.”

This makes a lot of sense if you don’t think of the thing under design as something that’ll be consumed but something that will be used to create. It sounds easy but again is surprisingly hard. It’s like we have been infected with this hard-to-kill idea that makes us think we can only consume whereas we are actually all very much creative beings.2 I think that’s what Generation C is really about.

A sidetrack: In digital games, for a long time developments have been towards games as media that can be consumed. The real changes in digital games are: One—there’s a renewed interest in games as activities (particularly in the form of casual games). And two—there’s an increase in games that allow themselves to be changed in meaningful ways. These developments make the term “replay value” seem ready for extinction. How can you even call something that isn’t interesting to replay a game?3

In Rules of Play, Salen and Zimmerman describe the phenomenon of transformative play—where the “free movement within a more rigid structure” changes the mentioned structure itself (be it intended or not). They hold it as one of the most powerful forms of play. Think of a simple house rule you made up the last time you played a game with some friends. The fact that on the web the rules that make up the structures we designed are codified in software should not be an excuse to disallow people to change them.

That’s true literacy: When you can both read and write in a medium (as Alan Kay would have it). I’d like to enable people to do that. It might be hopelessly naive, but I don’t care—it’s a very interesting challenge.

  1. That’s a comfortable idea to all of the—cough—web 2.0 savvy folk out there. But it certainly still is an uncomfortable thought to many. And I think it’d surprise you to find out how many people who claim to be “hip to the game” will still refuse to let go. []
  2. Note I’m not saying we can all be designers, but I do think people can all create meaningful things for themselves and others. []
  3. Yes, I am a ludologist. So shoot me. []