Machine Learning for Designers’ workshop

On Wednes­day Péter Kun, Hol­ly Rob­bins and myself taught a one-day work­shop on machine learn­ing at Delft Uni­ver­si­ty of Tech­nol­o­gy. We had about thir­ty master’s stu­dents from the indus­tri­al design engi­neer­ing fac­ul­ty. The aim was to get them acquaint­ed with the tech­nol­o­gy through hands-on tin­ker­ing with the Wek­ina­tor as cen­tral teach­ing tool.

Photo credits: Holly Robbins
Pho­to cred­its: Hol­ly Rob­bins

Background

The rea­son­ing behind this work­shop is twofold.

On the one hand I expect design­ers will find them­selves work­ing on projects involv­ing machine learn­ing more and more often. The tech­nol­o­gy has cer­tain prop­er­ties that dif­fer from tra­di­tion­al soft­ware. Most impor­tant­ly, machine learn­ing is prob­a­bilis­tic in stead of deter­min­is­tic. It is impor­tant that design­ers under­stand this because oth­er­wise they are like­ly to make bad deci­sions about its appli­ca­tion.

The sec­ond rea­son is that I have a strong sense machine learn­ing can play a role in the aug­men­ta­tion of the design process itself. So-called intel­li­gent design tools could make design­ers more effi­cient and effec­tive. They could also enable the cre­ation of designs that would oth­er­wise be impos­si­ble or very hard to achieve.

The work­shop explored both ideas.

Photo credits: Holly Robbins
Pho­to cred­its: Hol­ly Rob­bins

Format

The struc­ture was rough­ly as fol­lows:

In the morn­ing we start­ed out pro­vid­ing a very broad intro­duc­tion to the tech­nol­o­gy. We talked about the very basic premise of (super­vised) learn­ing. Name­ly, pro­vid­ing exam­ples of inputs and desired out­puts and train­ing a mod­el based on those exam­ples. To make these con­cepts tan­gi­ble we then intro­duced the Wek­ina­tor and walked the stu­dents through get­ting it up and run­ning using basic exam­ples from the web­site. The final step was to invite them to explore alter­na­tive inputs and out­puts (such as game con­trollers and Arduino boards).

In the after­noon we pro­vid­ed a design brief, ask­ing the stu­dents to pro­to­type a data-enabled object with the set of tools they had acquired in the morn­ing. We assist­ed with tech­ni­cal hur­dles where nec­es­sary (of which there were more than a few) and closed out the day with demos and a group dis­cus­sion reflect­ing on their expe­ri­ences with the tech­nol­o­gy.

Photo credits: Holly Robbins
Pho­to cred­its: Hol­ly Rob­bins

Results

As I tweet­ed on the way home that evening, the results were… inter­est­ing.

Not all groups man­aged to put some­thing togeth­er in the admit­ted­ly short amount of time they were pro­vid­ed with. They were most often stymied by get­ting an Arduino to talk to the Wek­ina­tor. Max was often picked as a go-between because the Wek­ina­tor receives OSC mes­sages over UDP, where­as the quick­est way to get an Arduino to talk to a com­put­er is over ser­i­al. But Max in my expe­ri­ence is a fick­le beast and would more than once crap out on us.

The groups that did build some­thing main­ly assem­bled pro­to­types from the exam­ples on hand. Which is fine, but since we were main­ly work­ing with the exam­ples from the Wek­ina­tor web­site they tend­ed towards the inter­ac­tive instru­ment side of things. We were hop­ing for explo­rations of IoT prod­uct con­cepts. For that more hand-rolling was required and this was only achiev­able for the stu­dents on the high­er end of the tech­ni­cal exper­tise spec­trum (and the more tena­cious ones).

The dis­cus­sion yield­ed some inter­est­ing insights into men­tal mod­els of the tech­nol­o­gy and how they are affect­ed by hands-on expe­ri­ence. A com­ment I heard more than once was: Why is this con­sid­ered learn­ing at all? The Wek­ina­tor was not per­ceived to be learn­ing any­thing. When chal­lenged on this by reit­er­at­ing the under­ly­ing prin­ci­ples it became clear the black box nature of the Wek­ina­tor ham­pers appre­ci­a­tion of some of the very real achieve­ments of the tech­nol­o­gy. It seems (for our stu­dents at least) machine learn­ing is stuck in a grey area between too-high expec­ta­tions and too-low recog­ni­tion of its capa­bil­i­ties.

Next steps

These results, and oth­ers, point towards some obvi­ous improve­ments which can be made to the work­shop for­mat, and to teach­ing design stu­dents about machine learn­ing more broad­ly.

  1. We can improve the toolset so that some of the heavy lift­ing involved with get­ting the var­i­ous parts to talk to each oth­er is made eas­i­er and more reli­able.
  2. We can build exam­ples that are geared towards the prac­tice of design­ing IoT prod­ucts and are ready for adap­ta­tion and hack­ing.
  3. And final­ly, and prob­a­bly most chal­leng­ing­ly, we can make the work­ings of machine learn­ing more trans­par­ent so that it becomes eas­i­er to devel­op a feel for its capa­bil­i­ties and short­com­ings.

We do intend to improve and teach the work­shop again. If you’re inter­est­ed in host­ing one (either in an edu­ca­tion­al or pro­fes­sion­al con­text) let me know. And stay tuned for updates on this and oth­er efforts to get design­ers to work in a hands-on man­ner with machine learn­ing.

Spe­cial thanks to the bril­liant Ianus Keller for con­nect­ing me to Péter and for allow­ing us to pilot this crazy idea at IDE Acad­e­my.

References

Sources used dur­ing prepa­ra­tion and run­ning of the work­shop:

  • The Wek­ina­tor – the UI is infu­ri­at­ing­ly poor but when it comes to get­ting start­ed with machine learn­ing this tool is unmatched.
  • Arduino – I have become par­tic­u­lar­ly fond of the MKR1000 board. Add a lithi­um-poly­mer bat­tery and you have every­thing you need to pro­to­type IoT prod­ucts.
  • OSC for ArduinoCNMAT’s imple­men­ta­tion of the open sound con­trol (OSC) encod­ing. Key puz­zle piece for get­ting the above two tools talk­ing to each oth­er.
  • Machine Learn­ing for Design­ers – my pre­ferred intro­duc­tion to the tech­nol­o­gy from a design­er­ly per­spec­tive.
  • A Visu­al Intro­duc­tion to Machine Learn­ing – a very acces­si­ble visu­al expla­na­tion of the basic under­pin­nings of com­put­ers apply­ing sta­tis­ti­cal learn­ing.
  • Remote Con­trol Theremin – an exam­ple project I pre­pared for the work­shop demo­ing how to have the Wek­ina­tor talk to an Arduino MKR1000 with OSC over UDP.

Week 169

Fiona Raby once told me that the major­i­ty of her work with stu­dents at the RCA was about psy­chol­o­gy. After a week like this, I can see where she’s com­ing from. With­out going into too much detail, I had my work cut out for me with a new group of stu­dents who I will be work­ing with on a design research project at the HKU. After a first meet­ing with the team and a kick-off with the client the next day, it became clear I was deal­ing with a group with some seri­ous moti­va­tion­al issues. The trick was to fig­ure out where it all was com­ing from. To do this it was vital to try and see things as they real­ly are in stead of as they were pre­sent­ed to me by the group. After sev­er­al addi­tion­al ses­sions (mess­ing with my sched­ule but that comes with the ter­ri­to­ry) I had it fig­ured out more or less and have for­mu­lat­ed a plan to deal with it. Psy­chol­o­gy.

In between all that crazi­ness my week con­sist­ed of:

  • Work­ing with my two new interns at Hub­bub. We reflect­ed on their expe­ri­ences at the Nat­ur­al Net­work­ing Fes­ti­val and pre­sent­ed a post-mortem of the first game to Thieu after attend­ing one of the Learn­ing Lab mee­tups.
  • Sketch­ing out addi­tions to the PLAY Pilots web­site nec­es­sary to sup­port the Zes­baans instal­la­tion for the Nether­lands Film Fes­ti­val. These will launch next week in time for the installation’s unveil­ing on Thurs­day.
  • Pre­sent­ing my pre­lim­i­nary list of inter­ac­tive works suit­able for next year’s Twee­t­akt fes­ti­val. This is my first time curat­ing an event oth­er than This hap­pened. I am keen to mash up play­ful inter­ac­tion design with the fringes of game design and it seems Twee­t­akt are up for it too. Hap­py days.
  • Anoth­er full day of work on Maguro. Best part of which was a few qui­et hours to bang out a first playable paper pro­to­type of the game. Con­ver­gence is a bitch but always reward­ing when it hap­pens.
  • Today, I hung out at BUROPONY and took care of a few odds and ends for their web­site. In return work has start­ed on a last bit of Hub­bub cor­po­rate iden­ti­ty: a design for the box to hold our busi­ness-slash-col­lectible play­ing cards.

And with that I am sign­ing off. A train is tak­ing me from Rot­ter­dam to Utrecht, per­haps I will be in time to catch the tail end of fri­day drinks at the Dutch Game Gar­den. Nev­er a dull moment there.

Stay hungry, stay foolish”

I grad­u­at­ed from the Utrecht School of the Arts in 2002. Now, less than sev­en years lat­er, I am men­tor­ing a group of five stu­dents who will be doing the same come Sep­tem­ber this year. I took a pho­to of them today, here it is:

Bright young bunch

From left to right, here’s who they are and what they’re up to:

  • Chris­ti­aan is tech lead on Hol­lan­dia, an action adven­ture game inspired by Dutch folk­lore. His research looks at ways to close the gap between cre­atives and tech­nol­o­gists in small teams, using agile tech­niques.
  • Kjell is design­ing a series of exper­i­men­tal games using voice as their only input. He’s research­ing what game mechan­ics work best with voice con­trol.
  • Max­ine is game design­er on the afore­men­tioned Hol­lan­dia game. Her research looks at the trans­la­tion of the play expe­ri­ence of phys­i­cal toys to dig­i­tal games. (In of Hol­lan­dia, you’ll be using a Wiimote to con­trol the spin­ning top used by the hero­ine.)
  • Paul is build­ing a physics-based plat­form puz­zle game for two play­ers. His research looks at the design of mean­ing­ful col­lab­o­ra­tive play.
  • Eva is mak­ing a space sim­u­la­tion game with real­is­tic physics and com­plex con­trols. She’s research­ing what kinds of fun are elicit­ed by such games.

Prac­ti­cal­ly speak­ing, men­tor­ing these guys means that I see them once a week for a 15-minute ses­sion. In this we dis­cuss the past week’s progress and their plans for the next. They’ve set their own briefs, and are expect­ed to be high­ly self-reliant. My task con­sists of mak­ing sure they stay on track and their work is rel­e­vant, both from an edu­ca­tion­al and a pro­fes­sion­al per­spec­tive. It’s chal­leng­ing work, but a lot of fun. It forces me to make explic­it the stuff I’ve picked up pro­fes­sion­al­ly. It’s also a lot about devel­op­ing a sense for where each stu­dent indi­vid­u­al­ly can improve and encour­ag­ing them to chal­lenge them­selves in those areas.

I’m look­ing for­ward to see­ing what they’ll deliv­er come Sep­tem­ber, when it’s their turn to grad­u­ate, and go out to con­quer the world.

What I’ve been up to lately

You might be won­der­ing what’s been going on at the Leapfrog stu­dio late­ly, since I haven’t real­ly post­ed any­thing sub­stan­tial here in a while. Quite some stuff has hap­pened — and I’ll hope­ful­ly get back into post­ing longer arti­cles soon — but for now, here’s a list of more or less inter­est­ing things I have been doing:

This hap­pened – Utrecht

We had our first This hap­pened – Utrecht on Novem­ber 3. I think we suc­ceed­ed in cre­at­ing an event that real­ly looks at the craft of inter­ac­tion design. I’m hap­py to say we’re plan­ning to do three events next year — all at The­ater Kikker in Utrecht — and we’ve got lots of cool speak­ers in mind. If you want to make sure you won’t miss them, sub­scribe to our newslet­ter (in Dutch).1

Teach­ing

My stu­dents are near­ing the end of their project. They’ve been hard at work cre­at­ing con­cepts for mobile social games with a musi­cal com­po­nent; they came up with 20 in total. Now they’re pro­to­typ­ing two of them, and I must say it’s look­ing good. They’ll have to present the games to the project’s com­mis­sion­er — a major mobile phone man­u­fac­tur­er — some­where the begin­ning of Jan­u­ary 2009. I hope to be able to share some of the results here after­wards.

Office space

Since Decem­ber 1 I am a res­i­dent of the Dutch Game Gar­den’s Busi­ness Club. That means I now have a nice office smack in the cen­tre of Utrecht. The building’s home to lots of won­der­ful games com­pa­nies, some, like me, oper­at­ing on the fringes — like Fource­Labs and Monoban­da. If you’re curi­ous and would like to drop by for a tour, a cof­fee and some con­ver­sa­tion, let me know.

Brain­storm

I was invit­ed do help com­pose one of the cas­es for the ‘Grote Ams­ter­damse Water­brain­wave’. A one-day brain­storm in which 45 stu­dents from var­i­ous insti­tu­tions were asked to come up with water-relat­ed inno­va­tions that would make the Nether­lands a sig­nif­i­cant glob­al play­er once again. It was organ­ised by the Port of Ams­ter­dam, Water­net and Verleden van Ned­er­land2. I also attend­ed the day itself as an out­side expert on games and the cre­ative indus­try in gen­er­al. Read a report of the event at FD.nl (in Dutch).

Book

Dan Saf­fer’s book Design­ing Ges­tur­al Inter­faces has been pub­lished by O’Reilly and is now avail­able. Turn to page 109 and you’ll find a sto­ry­board by yours tru­ly used for illus­tra­tion pur­pos­es. That’s the first time any work of mine is fea­tured in print, so nat­u­ral­ly I’m quite proud. I have yet to receive my copy, but got a sneak peek this week­end and I must say it looks promis­ing. If you’re a design­er need­ing to get up to speed with mul­ti-touch, phys­i­cal com­put­ing and such, this should be a good place to start.

That’s about it for now. There’s a lot of excit­ing stuff in the works, the out­comes of which I will hope­ful­ly be able to share with you in 2009.

  1. The cre­ators of This hap­pened in Lon­don have been nom­i­nat­ed for a best of the year award by the Design Muse­um, by the way. Well-deserved, I would say! []
  2. A cross-media cam­paign aimed at increas­ing aware­ness of Dutch nation­al his­to­ry. []

Teaching design for mobile social play

Last week, the group project I am coach­ing at the Utrecht School of the Arts kicked off. The project is part of the school’s mas­ter of arts pro­gram. The group con­sists of ten stu­dents with very dif­fer­ent back­grounds, rang­ing from game design & devel­op­ment to audio design, as well as arts man­age­ment, media stud­ies, and more. Their assign­ment is to come up with a num­ber of con­cepts for games that incor­po­rate mobile phones, social inter­ac­tions, audio and the web. Nokia Research Cen­ter has com­mis­sioned the project, and Jus­si Holopainen, game design researcher and co-author of Pat­terns in Game Design, is the client. In the project brief there is a strong empha­sis on sketch­ing and pro­to­typ­ing, and dis­ci­plined doc­u­men­ta­tion of the design process. The stu­dents are work­ing full time on the project and it will run for around 4 months.

I am very hap­py with the oppor­tu­ni­ty to coach this group. It’s a new chal­lenge for me as a teacher — mov­ing away from teach­ing the­o­ry and into the area of facil­i­ta­tion. I am also look­ing for­ward to see­ing what the stu­dents will come up with, of course, as the domain they are work­ing in over­laps huge­ly with my inter­ests. So far, work­ing with Jus­si has proven to be very inspi­ra­tional, so I am get­ting some­thing out of it as a design­er too.

On presentations

One of the most enjoy­able things about attend­ing con­fer­ences is see­ing a lot of peo­ple pre­sent­ing in var­i­ous ways. A while ago I chal­lenged my own pre­sent­ing skills by doing a Pecha Kucha. Today, I attend­ed a class (part of a didac­tics course) on giv­ing lec­tures. Two promi­nent lec­tur­ers (Giep Hagoort and Jeroen van Mas­trigt) from with­in the Utrecht School of Arts gave us a taste of their own unique pre­sen­ta­tion for­mat and the way they pre­pared for a talk.

This trig­gered some things in my head, such as stuff I’d seen before on the web and that could be help­ful to the peo­ple attend­ing the class. A lot of them didn’t seem to be too famil­iar with it, so I’ve decid­ed to col­lect them here. Maybe they’ll come in handy to those who pass by here: