Tag: AI

Keep the tension: designing with generative AI

Slides from my keynote at the CLICKNL Design Innovation Sessions, Digital Futures theme session, Dutch Design Week, 21 October 2026, Natlab, Eindhoven.

CLICKNL invited me to give this keynote. I saw it as a great opportunity to intervene in the debate about design and generative AI. My take is informed by my past work on contestability. That work is all about approaching “AI” as an arena where various parties compete over how it is made and how it is used. My take is also informed by my commitments to practice theory and political realism. Practice theory means looking at what people and things do together, rather than at technologies in isolation. Political realism means looking at politics as it actually happens. It asks who does what to whom, for whose benefit, instead of starting from abstract moral principles.

In the talk I argue that neither wholesale rejection nor uncritical embrace of generative AI will get us very far. GenAI deskills designers and opens up new ways of designing at the same time. It is built on the past work of designers, which raises the question of whose work we build on, and who gets to call the results their own. And it automates work, but in the process work may get delegated more than replaced, transforming practices as a result. Rather than trying to resolve these tensions, I look for emerging design practices that hold them and use them as an engine for moving forward. I discuss three cases: Vera van der Burg’s Text-to-Clay, Blaze Type’s test of Monotype’s AI type tool, and The Call by Holly Herndon and Mat Dryhurst. They all go beyond consuming off-the-shelf models. They treat pipelines and workflows as an object of design. And in all three there is an awareness of the value of effort and friction. I end with an open question. If AI is not one thing, not an agent, and neither our savior nor our doom, what metaphor does your mind go to?

I hope this talk helps designers think in concrete terms about how we might “do AI” differently from the dominant ways. And I hope it helps them break free from the simplistic for-or-against debates that get us nowhere.

CLICKNL DDW 2026 DIS Digital Futures keynote slides – Kars Alfrink – Keep the tensionDownload

Works discussed

Studio Dumbar

From Text-to-Clay, Vera van der Burg

Blaze Type and Monotype’s AI type tool

The Call, Holly Herndon and Mat Dryhurst

CRTA visual identity, Bruketa&Žinić&Grey

Mud to Models and Soil to Servers workshops

  • Fieke Jansen, Sunjoo Lee, Kars Alfrink and Lena Trotereau, From Mud to Models: Regenerative Futures for Community AI (conference workshop), TH/NGS 2025, Amsterdam, 12 Dec 2025, and From Soil to Servers: Regenerative Futures for Community Computing (workshop), Creative Coding Utrecht, Utrecht, 27 Mar 2026; Kars Alfrink and Sunjoo Lee, with Advaita Jairam, From Soil to Servers: Pairing Material Practice with Speculative Design to Rethink Computing Infrastructure (conference workshop), Philosophy of Human-Technology Relations Conference (PHTR 2026), The Hague, 22 Sep 2026.
  • TU Delft, “From Mud to Models: A Workshop on Regenerative AI Futures”, 17 Dec 2025. https://www.tudelft.nl/2025/io/december/from-mud-to-models-a-workshop-on-regenerative-ai-futures
  • Kars Alfrink, Sunjoo Lee, Lena Trotereau and Fieke Jansen, “Mud and models: Does making matter for regenerative AI speculation?”, in Iskander Smit and Andrea Krajewski (eds.), The State of Responsible Technology – RESIZE<REMIX<REGEN (RIOT 2026), Stichting ThingsCon Amsterdam, June 2026, pp. 34–39. https://doi.org/10.5281/zenodo.20569440

Further reading

  • Alfrink, K. (2025). People’s compute: Design and the politics of AI infrastructures. OSF. https://doi.org/10/q8kp
  • Caplan, L. (2025). Introduction: Contesting models – art and democracy since the 1990s. In L. Caplan & K. Greaves (Eds.), Model collapse: European contemporary art in a time of democratic crisis (pp. 1–20). Manchester University Press. http://www.jstor.org/stable/jj.36233585.6
  • Geuss, R. (2008). Philosophy and real politics. Princeton University Press.
  • Kuutti, K., & Bannon, L. J. (2014). The turn to practice in HCI: Towards a research agenda. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3543–3552). ACM. https://doi.org/10.1145/2556288.2557111
  • Löwgren, J., & Stolterman, E. (2004). Thoughtful interaction design: A design perspective on information technology. MIT Press.
  • McGowan, T. (2016). Capitalism and desire: The psychic cost of free markets. Columbia University Press.
  • McGowan, T. (2019). Emancipation after Hegel: Achieving a contradictory revolution. Columbia University Press.
  • Morozov, E. (2024, August). AI and the techno-utopian path not taken. Le Monde diplomatique (English edition). https://mondediplo.com/2024/08/07ai-cold-war
  • Parsons, P. C. (2026). Myths and ironies of AI-assisted design. In Designing Interactive Systems Conference (DIS ’26) (pp. 523–535). ACM. https://doi.org/10.1145/3800645.3813086
  • Porfirio, D., McDermott, I., Chen, H.-M., Satake, S., Kanda, T., & LaToza, T. D. (2026). Designing for robot wranglers: A synthesis of literature and practice. In Designing Interactive Systems Conference (DIS ’26) (pp. 4284–4298). ACM. https://doi.org/10.1145/3800645.3813070
  • Retkowsky, J., Hafermalz, E., & Huysman, M. (2026). Harnessing a “spirited technology”: How working with generative AI collapses the creative process. Academy of Management Discoveries, 12(3), 363–387. https://doi.org/10.5465/amd.2024.0280
  • Retkowsky, J., Hafermalz, E., Huysman, M., & Odijk, D. (2026, September 16). Research: Gen AI is collapsing creative processes. Harvard Business Review. https://hbr.org/2026/09/research-gen-ai-is-collapsing-creative-processes
  • Salvaggio, E. (2026, August 16). The system from nowhere. Cybernetic Forests. https://mail.cyberneticforests.com/the-system-from-nowhere/
  • Sayers, S. (2011). Alienation as a critical concept. International Critical Thought, 1(3), 287–304. https://doi.org/10.1080/21598282.2011.609265
  • Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2024). The curse of recursion: Training on generated data makes models forget. arXiv. https://arxiv.org/abs/2305.17493
  • Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568–1580. https://www.sciencedirect.com/science/article/pii/S0048733313000930
  • Suchman, L. (2023). The uncontroversial ‘thingness’ of AI. Big Data & Society, 10(2), 1–5. https://doi.org/10.1177/20539517231206794
  • Unger, G., & Narayanan, A. (2026, October 14). Steering AI beyond automation: A research agenda and plan for action. AI as Normal Technology. https://www.normaltech.ai/p/steering-ai-beyond-automation
  • van Maanen, G. (2022). AI ethics, ethics washing, and the need to politicize data ethics. Digital Society, 1(2), Article 9. https://doi.org/10.1007/s44206-022-00013-3
  • Wright, E. O. (2010). Envisioning real utopias. Verso.

On AI discourse

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

The original ran to about 296 words, with 3 references.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

A short post on the author’s distance from online AI discourse. Drawing on Lee Vinsel’s notion of hype and criti-hype and Jodi Dean’s “communicative capitalism”, it argues that online debate is not a public sphere: platforms profit from the circulation of opinions, and that circulation substitutes for political action. For someone working on AI politics the conclusion is practical rather than principled: building power happens through organising and institutions, not in threads.

Postdoc update – December 2025

Six months since my last update. The pace hasn’t slowed. Here’s what I’ve been up to and what’s on the horizon for the next six months or so. But first, a very welcome December break. Happy holidays, dear reader.

Happenings

People’s Compute at Goldsmiths: On September 18, I presented my research agenda, “People’s Compute: Design and the Politics of AI Infrastructures,” at the Politics of AI symposium at Goldsmiths, University of London. Many thanks to Dan McQuillan, Fieke Jansen, Jo Lindsay Walton, and Pat Brody for the invitation and for putting together such a thought-provoking program. Read the transcript.

Digital Autonomy Unconference: On October 3, I attended the Digital Autonomy Unconference in Amsterdam, which was organized in collaboration with Code for NL and focused on enhancing digital autonomy within Dutch public institutions. The Digital Autonomy Competence Center was also launched at this event, for which I serve as a research associate. Read the news item.

Master’s Graduation Projects: Two more of my students have graduated. Ameya Sawant completed a project about designer autonomy and GenAI (August 29, with Fernando Secomandi as chair). David Mieras completed a project about the responsible use of AI in policy preparation (October 28, with Lianne Simonse as chair).

Personal Grant: I mentioned going for a personal grant the last time around. Unfortunately, I did not advance to the final round. However, I did receive some useful feedback and will try again next year. Onwards and upwards.

Designing Responsible AI: Sara Colombo, Francesca Mauri, and I ran the second iteration of our master’s elective course, which builds on responsible research and innovation, value-sensitive design, and design fiction. See the course description here. A more detailed write-up of how the course works is forthcoming.

International Contestable AI Workshop: On November 18, I had the pleasure of hosting a delegation from Denmark and the UK for a full-day workshop about Contestable AI at TU Delft. Read the report.

Enterprise UX: On November 21, I delivered an invited talk titled “Reclaiming Autonomy: Designing AI-Enhanced Work Tools That Empower Users” at the Enterprise UX conference in Amersfoort. Thanks to Peter Boersma for the invitation. The references to Office Space and Luddism were surprisingly well-received. Read the transcript.

Your difficult design doctor, holding forth at Enterprise UX.

NIAS Workshop: I participated in a workshop at NIAS on November 26-27, exploring permacomputing, server collectives, and networks of consent. Incredibly inspiring, it has given me many new ideas for approaching my own ongoing research. Many thanks to John Boy for the invitation. View the event page.

Stop the Cuts (continued): The fight against cuts to higher education continues. On December 9 we once again went on strike and I joined the demonstration in Amsterdam with over 7,000 participants. Our far-right government may have fallen, but the cuts remain on the table. Now is the time to maintain pressure on the parties forming a government. If you work in academia and want to act, join a union (Aob or FNV) and sign up for the WOinActie newsletter.

Advisory Today, Co-Decisive Tomorrow? A paper based on a year-long participant observation of a smart city project in Amsterdam, co-authored with Mike de Kreek, Tessa Steenkamp, and Martijn de Waal (part of the Human Values for Smarter Cities project), has been accepted for the 2026 Participatory Design Conference. Very pleased about that one. A preprint will be up once we submit the final camera-ready version, I think.

ThingsCon TH/NGS: At this year’s ThingsCon conference on December 12, Fieke Jansen, Sunjoo Lee, Lena Trotereau, and I ran a workshop titled “From Mud to Models” exploring regenerative futures for community AI. Thanks to Iskander Smit for bringing us together. A report is forthcoming. Update: read the report on the TU Delft website.

Participants building clocks powered by mud batteries at our ThingsCon workshop.

Open Letter on AI Policy: I was part of the supporting team for an open letter calling on Dutch politicians to develop a national AI policy that promotes social progress. A special thanks goes out to Cristina Zaga for taking the initiative and leading the charge on this one, but also to the core team members Roel Dobbe, Iris van Rooij, Lilian de Jong, Wouter Nieuwenhuizen, Marcela Suarez, Wiep Hamstra, and Olivia Guest, and the supporting team that myself was part of Felienne Hermans, Mark D., Eelco Herder, Emile van Bergen, Siri Beerends, Nolen Gertz, Paul Peters, Gerry McGovern, Kars Alfrink, and Jelle van der Ster. Sign and share the letter here.

On deck

Looking ahead to the new year, I have several writing projects to complete: one chapter on contestability for an edited volume on the philosophy of engineering, and another chapter for an edited volume on community AI.

I will be wrapping up my duties as associate chair for the CHI 2026 design subcommittee. And I will also serve as associate chair for the DIS 2026 artifacts & systems subcommittee.

I will do the analysis and write-up of a field evaluation of the Vision Model Macroscope prototype (also part of the aforementioned Human Values for Smarter Cities project). I am also providing support on several other papers that will hopefully find their way into venues such as FAccT, DIS, and elsewhere.

Mockup of the Vision Model Macroscope prototype.

Finally, I am part of several small grant applications exploring topics that include the potential of computational argumentation techniques to enable more interactive implementations of contestable AI, as well as contestability in digital systems used for evidence management in international criminal justice.

That’s most of it, although not all of it, but this has gotten way too long already. Thanks for reading this far, if you have, and best wishes for 2026.

Reclaiming Autonomy: Designing AI-Enhanced Work Tools That Empower Users

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

Based on an invited talk delivered at Enterprise UX, on November 21, 2025 in Amersfoort, the Netherlands. The original ran to about 1,269 words and 16 images.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

The post asks what to question before adding an LLM feature to a work tool: whether it actually works, who gains or loses power, and which practices and skills it transforms. It frames these as questions of worker autonomy (authenticity and agency, after Prunkl), illustrated with scenes from Office Space, and lists ways AI threatens autonomy: manipulation, restricted choice, diminished competence. It then sketches a toolkit for designing AI for autonomy and applies three of its elements (reflexive design, stakeholder negotiation, collective autonomy support) to LegalMike, a Dutch legal AI platform, concluding that designers act as brokers of autonomy.

Designing Learning Experiences in a Post-ChatGPT World

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

Transcript of a talk delivered at LXDCON’25 on June 12, 2025. The original ran to about 1,706 words and 32 images.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

Starting from ChatGPT’s impact on higher education, it contrasts Ethan Mollick’s adapt-and-redesign optimism with Ian Bogost’s realist diagnosis of a crisis that technology cannot fix, using Harvard’s StatGPT Slackbot as a running example. It proposes “Conscious Adaptation”, built on three strategies: institutional realism, loss-conscious innovation, and keeping solution-building and critique as distinct but mutually informing work. It ends with three imperatives for learning experience designers: protect meaningful human effort, design for real institutions rather than ideal ones, and recognise the limits of design.

On how to think about large language models

How should we think about large language models (LLMs)? People commonly think and talk about them in terms of human intelligence. To the extent this metaphor does not accurately reflect the properties of the technology, this may lead to misguided diagnoses and prescriptions. It seems to me an LLM is not like a human or a human brain in so many ways. One crucial distinction for me is that LLMs lack individuality and subjectivity.

What are organisms that similarly lack these qualities? Coral polyps and Portuguese man o’ war come to mind, or slime mold colonies. Or maybe a single bacterium, like an E. coli. Each is essentially identical to its clones, responds automatically to chemical gradients (bringing to mind how LLMs respond to prompts), and doesn’t accumulate unique experiences in any meaningful way.

Considering all these examples, the meme about LLMs being like a shoggoth (an amorphous blob-like monster originating from the speculative fiction of Howard Philips Lovecraft) is surprisingly accurate. The thing about these metaphors though is that it’s about as hard to reason about such organisms as it is to reason about LLMs. So to use them as a metaphor for thinking about LLMs won’t work. A shoggoth is even less helpful because the reference will only be familiar to those who know their H.P. Lovecraft.

So perhaps we should abandon metaphorical thinking and think historically instead. LLMs are a new language technology. As with previous technologies, such as the printing press, when they are introduced, our relationship to language changes. How does this change occur?

I think the change is dialectical. First, we have a relationship to language that we recognize as our own. Then, a new technology destabilizes this relationship, alienating us from the language practice. We no longer see our own hand in it. And we experience a lack of control over language practice. Finally, we reappropriate this language use in our practices. In this process of reappropriation, language practice as a whole is transformed. And the cycle begins again.

For an example of this dialectical transformation of language practice under the influence of new technology, we can take Eisenstein’s classic account of the history of the printing press (1980). Following its introduction many things changed about how we relate to language. Our engagement with language shifted from a primarily oral one to a visual and deliberative one. Libraries became more abundantly stocked, leading to the practice of categorization and classification of works. Preservation and analysis of stable texts became a possibility. The solitary reading experience gained prominence, producing a more private and personal relationship between readers and texts. Concerns about information overload first reared its head.

All of these things were once new and alien to humans. Now we consider them part of the natural order of things. They weren’t predetermined by the technology, they emerged through this active tug of war between groups in society about what the technology would be used for, mediated by the affordances of the technology itself.

In concrete material terms, what does an LLM consist of? An LLM is just numerical values stored in computer memory. It is a neural network architecture consisting of billions of parameters in weights and biases, organized in matrices. The storage is distributed across multiple devices. System software loads these parameters and enables the calculation of inferences. This all runs in physical data centers housing computing infrastructure, power, cooling, and networking infrastructure. Whenever people start talking about LLMs having agency or being able to reason, I remind myself of these basic facts.

A printing press, although a cleverly designed, engineered, and manufactured device, is similarly banal when you break it down to its essential components. Still, the ultimate changes to how we relate to language have been profound. From these first few years of living with LLMs, I think it is not unreasonable to think they will cause similar upheavals. What is important for me is to recognize how we become alienated from language, and to see ourselves as having agency in reappropriating LLM-mediated language practice as our own.

On mapping AI value chains

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

Report on a workshop held at CSCW 2024 (November 2024). The original ran to about 761 words and 1 image.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

The workshop, “From Stem to Stern: Contestability Along AI Value Chains”, looked at contesting AI not only at the point of individual decisions but across the whole chain, from resource extraction to deployment. The post shares the AI value chain mapping canvas (CC-BY, Yurrita & Alfrink) used in the first group activity, explains how the 50-minute mapping exercise with ChatGPT as the example case was run, and names its influences (value chain mapping, Wardley mapping). Takeaways include the vocabulary gap between legal and technical communities, the need to move from individual grievance to collective and upstream contestation, and the importance of Global South perspectives. It credits the full workshop team.

On autonomy, design, and AI

In my thesis, I use autonomy to build the normative case for contestability. It so happens that this year’s theme at the Delft Design for Values Institute is also autonomy. On October 15, 2024, I participated in a panel discussion on autonomy to kick things off. I collected some notes on autonomy that go beyond the conceptualization I used in my thesis. I thought it might be helpful and interesting to collect some of them here in adapted form.

The notes I brought included, first of all, a summary of the ecumenical conceptualization of autonomy concerning automated decision-making systems offered by Alan Rubel, Clinton Castro, and Adam Pham (2021). They conceive of autonomy as effective self-governance. To be autonomous, we need authentic beliefs about our circumstances and the agency to act on our plans. Regarding algorithmic systems, they offer this notion of a reasonable endorsement test—the degree to which a system can be said to respect autonomy depends on its reliability, the stakes of its outputs, the degree to which subjects can be held responsible for inputs, and the distribution of burdens across groups.

Second, I collected some notes from several pieces by James Muldoon, which get into notions of freedom and autonomy that were developed in socialist republican thought by the likes of Luxemburg, Kautsky, and Castoriadis (2020, 2021a, 2021b). This story of autonomy is sociopolitical rather than moral. This approach is quite appealing for someone interested in non-ideal theory in a realist mode like myself. The account of autonomy Muldoon offers is one where individual autonomy hinges on greater group autonomy and stronger bonds of association between those producing and consuming technologies. Freedom is conceived of as collective self-determination.

And then third and finally, there’s this connected idea of relational autonomy, which to a degree is part of the account offered by Rubel et al., but in the conceptions here more radical in how it seeks to create distance from liberal individualism (e.g., Christman, 2004; Mhlambi & Tiribelli, 2023; Westlund, 2009). In this, individual capacity for autonomous choice is shaped by social structures. So freedom becomes realized through networks of care, responsibility, and interdependence.

That’s what I am interested in: accounts of autonomy that are not premised on liberal individualism and that give us some alternative handle on the problem of the social control of technology in general and of AI in particular.

From my point of view, the implications of all this for design and AI include the following.

First, to make a fairly obvious but often overlooked point, the degree to which a given system impacts people’s autonomy depends on various factors. It makes little sense to make blanket statements about AI destroying our autonomy and so on.

Second, in value-sensitive design terms, you can think about autonomy as a value to be balanced against others—in the case where you take the position that all values can be considered equally important, at least in principle. Or you can consider autonomy more like a precondition for people to live with technology in concordance with their values, making autonomy take precedence over other values. The sociopolitical and relational accounts above point in this direction.

Third, suppose you buy into the radical democratic idea of technology and autonomy. In that case, it follows that it makes little sense to admonish individual designers about respecting others’ autonomy. They may be asked to privilege technologies in their designs that afford individual and group autonomy. But designers also need organization and emancipation more often than not. So it’s about building power. The power of workers inside the organizations that develop technologies and the power of communities that “consume” those same technologies. 

With AI, the fact is that, in reality, in the cases I look at, the communities that AI is brought to bear on have little say in the matter. The buyers and deployers of AI could and should be made more accountable to the people subjected to AI.

Democratizing AI Through Continuous Adaptability: The Role of DevOps

Below are the abstract and slides for my contribution to the TILTing Perspectives 2024 panel “The mutual shaping of democratic practices & AI,” moderated by Merel Noorman.

Slides

democratizing-ai-through-continuous-adaptability-the-role-of-devops-tilting-perspectives-2024Download

Abstract

Contestability

This presentation delves into democratizing artificial intelligence (AI) systems through contestability. Contestability refers to the ability of AI systems to remain open and responsive to disputes throughout their lifecycle. It approaches AI systems as arenas where groups compete for power over designs and outcomes.

Autonomy, democratic agency, legitimation

We identify contestability as a critical system quality for respecting people’s autonomy. This includes their democratic agency: their ability to legitimate policies. This includes policies enacted by AI systems.

For a decision to be legitimate, it must be democratically willed or rely on “normative authority.” The democratic pathway should be constrained by normative bounds to avoid arbitrariness. The appeal to authority should meet the “access constraint,” which ensures citizens can form beliefs about policies with a sufficient degree of agency (Peter, 2020 in Rubel et al., 2021).

Contestability is the quality that ensures mechanisms are in place for subjects to exercise their democratic agency. In the case of an appeal to normative authority, contestability mechanisms are how subjects and their representatives gain access to the information that will enable them to evaluate its justifiability. In this way, contestability satisfies the access constraint. In the case of democratic will, contestability-by-design practices are how system development is democratized. The autonomy account of legitimation adds the normative constraints that should bind this democratic pathway.

Himmelreich (2022) similarly argues that only a “thick” conception of democracy will address some of the current shortcomings of AI development. This is a pathway that not only allows for participation but also includes deliberation over justifications.

The agonistic arena

Elsewhere, we have proposed the Agonistic Arena as a metaphor for thinking about the democratization of AI systems (Alfrink et al., 2024). Contestable AI embodies the generative metaphor of the Arena. This metaphor characterizes public AI as a space where interlocutors embrace conflict as productive. Seen through the lens of the Arena, public AI problems stem from a need for opportunities for adversarial interaction between stakeholders.

This metaphorical framing suggests prescriptions to make more contentious and open to dispute the norms and procedures that shape:

  1. AI system design decisions on a global level, and
  2. human-AI system output decisions on a local level (i.e., individual decision outcomes), establishing new dialogical feedback loops between stakeholders that ensure continuous monitoring.

The Arena metaphor encourages a design ethos of revisability and reversibility so that AI systems embody the agonistic ideal of contingency.

Post-deployment malleability, feedback-ladenness

Unlike physical systems, AI technologies exhibit a unique malleability post-deployment.

For example, LLM chatbots optimize their performance based on a variety of feedback sources, including interactions with users, as well as feedback collected through crowd-sourced data work.

Because of this open-endedness, democratic control and oversight in the operations phase of the system’s lifecycle become a particular concern.

This is a concern because while AI systems are dynamic and feedback-laden (Gilbert et al., 2023), many of the existing oversight and control measures are static, one-off exercises that struggle to track systems as they evolve over time.

DevOps

The field of DevOps is pivotal in this context. DevOps focuses on system instrumentation for enhanced monitoring and control for continuous improvement. Typically, metrics for DevOps and their machine learning-specific MLOps offshoot emphasize technical performance and business objectives.

However, there is scope to expand these to include matters of public concern. The matters-of-concern perspective shifts the focus on issues such as fairness or discrimination, viewing them as challenges that cannot be resolved through universal methods with absolute certainty. Rather, it highlights how standards are locally negotiated within specific institutional contexts, emphasizing that such standards are never guaranteed (Lampland & Star, 2009, Geiger et al., 2023).

MLOps Metrics

In the context of machine learning systems, technical metrics focus on model accuracy. For example, a financial services company might use Area Under The Curve Receiver Operating Characteristics (AUC-ROC) to continuously monitor and maintain the performance of their fraud detection model in production.

Business metrics focus on cost-benefit analyses. For example, a bank might use a cost-benefit matrix to balance the potential revenue from approving a loan against the risk of default, ensuring that the overall profitability of their loan portfolio is optimized.

Drift

These metrics can be monitored over time to detect “drift” between a model and the world. Training sets are static. Reality is dynamic. It changes over time. Drift occurs when the nature of new input data diverges from the data a model was trained on. A change in performance metrics may be used to alert system operators, who can then investigate and decide on a course of action, e.g., retraining a model on updated data. This, in effect, creates a feedback loop between the system in use and its ongoing development.

An expansion of these practices in the interest of contestability would require:

  1. setting different metrics,
  2. exposing these metrics to additional audiences, and
  3. establishing feedback loops with the processes that govern models and the systems they are embedded in.

Example 1: Camera Cars

Let’s say a city government uses a camera-equipped vehicle and a computer vision model to detect potholes in public roads. In addition to accuracy and a favorable cost-benefit ratio, citizens, and road users in particular, may care about the time between a detected pothole and its fixing. Or, they may care about the distribution of potholes across the city. Furthermore, when road maintenance appears to be degrading, this should be taken up with department leadership, the responsible alderperson, and council members.

Example 2: EV Charching

Or, let’s say the same city government uses an algorithmic system to optimize public electric vehicle (EV) charging stations for green energy use by adapting charging speeds to expected sun and wind. EV drivers may want to know how much energy has been shifted to greener time windows and its trends. Without such visibility on a system’s actual goal achievement, citizens’ ability to legitimate its use suffers. As I have already mentioned, democratic agency, when enacted via the appeal to authority, depends on access to “normative facts” that underpin policies. And finally, professed system functionality must be demonstrated as well (Raji et al., 2022).

DevOps as sociotechnical leverage point for democratizing AI

These brief examples show that the DevOps approach is a potential sociotechnical leverage point. It offers pathways for democratizing AI system design, development, and operations.

DevOps can be adapted to further contestability. It creates new channels between human and machine actors. One of DevOps’s essential activities is monitoring (Smith, 2020), which presupposes fallibility, a necessary precondition for contestability. Finally, it requires and provides infrastructure for technical flexibility so that recovery from error is low-cost and continuous improvement becomes practically feasible.

The mutual shaping of democratic practices & AI

Zooming out further, let’s reflect on this panel’s overall theme, picking out three elements: legitimation, representation of marginalized groups, and dealing with conflict and contestation after implementation and during use.

Contestability is a lever for demanding justifications from operators, which is a necessary input for legitimation by subjects (Henin & Le Métayer, 2022). Contestability frames different actors’ stances as adversarial positions on a political field rather than “equally valid” perspectives (Scott, 2023). And finally, relations, monitoring, and revisability are all ways to give voice to and enable responsiveness to contestations (Genus & Stirling, 2018).

And again, all of these things can be furthered in the post-deployment phase by adapting the DevOps lens.

Bibliography

  • Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022). Contestable AI by Design: Towards a Framework. Minds and Machines, 33(4), 613–639. https://doi.org/10/gqnjcs
  • Alfrink, K., Keller, I., Yurrita Semperena, M., Bulygin, D., Kortuem, G., & Doorn, N. (2024). Envisioning Contestability Loops: Evaluating the Agonistic Arena as a Generative Metaphor for Public AI. She Ji: The Journal of Design, Economics, and Innovation, 10(1), 53–93. https://doi.org/10/gtzwft
  • Geiger, R. S., Tandon, U., Gakhokidze, A., Song, L., & Irani, L. (2023). Making Algorithms Public: Reimagining Auditing From Matters of Fact to Matters of Concern. International Journal of Communication, 18(0), Article 0.
  • Genus, A., & Stirling, A. (2018). Collingridge and the dilemma of control: Towards responsible and accountable innovation. Research Policy, 47(1), 61–69. https://doi.org/10/gcs7sn
  • Gilbert, T. K., Lambert, N., Dean, S., Zick, T., Snoswell, A., & Mehta, S. (2023). Reward Reports for Reinforcement Learning. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 84–130. https://doi.org/10/gs9cnh
  • Henin, C., & Le Métayer, D. (2022). Beyond explainability: Justifiability and contestability of algorithmic decision systems. AI & SOCIETY, 37(4), 1397–1410. https://doi.org/10/gmg8pf
  • Himmelreich, J. (2022). Against “Democratizing AI.” AI & SOCIETY. https://doi.org/10/gr95d5
  • Lampland, M., & Star, S. L. (Eds.). (2008). Standards and Their Stories: How Quantifying, Classifying, and Formalizing Practices Shape Everyday Life (1st edition). Cornell University Press.
  • Peter, F. (2020). The Grounds of Political Legitimacy. Journal of the American Philosophical Association, 6(3), 372–390. https://doi.org/10/grqfhn
  • Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022). The Fallacy of AI Functionality. 2022 ACM Conference on Fairness, Accountability, and Transparency, 959–972. https://doi.org/10/gqfvf5
  • Rubel, A., Castro, C., & Pham, A. K. (2021). Algorithms and autonomy: The ethics of automated decision systems. Cambridge University Press.
  • Scott, D. (2023). Diversifying the Deliberative Turn: Toward an Agonistic RRI. Science, Technology, & Human Values, 48(2), 295–318. https://doi.org/10/gpk2pr
  • Smith, J. D. (2020). Operations anti-patterns, DevOps solutions. Manning Publications.
  • Treveil, M. (2020). Introducing MLOps: How to scale machine learning in the enterprise (First edition). O’Reilly.

Design and machine learning – an annotated reading list

Earlier this year I coached Design for Interaction master students at Delft University of Technology in the course Research Methodology. The students organised three seminars for which I provided the claims and assigned reading. In the seminars they argued about my claims using the Toulmin Model of Argumentation. The readings served as sources for backing and evidence.

The claims and readings were all related to my nascent research project about machine learning. We delved into both designing for machine learning, and using machine learning as a design tool.

Below are the readings I assigned, with some notes on each, which should help you decide if you want to dive into them yourself.

Hebron, Patrick. 2016. Machine Learning for Designers. Sebastopol: O’Reilly.

The only non-academic piece in this list. This served the purpose of getting all students on the same page with regards to what machine learning is, its applications of machine learning in interaction design, and common challenges encountered. I still can’t think of any other single resource that is as good a starting point for the subject as this one.

Fiebrink, Rebecca. 2016. “Machine Learning as Meta-Instrument: Human-Machine Partnerships Shaping Expressive Instrumental Creation.” In Musical Instruments in the 21st Century, 14:137–51. Singapore: Springer Singapore. doi:10.1007/978–981–10–2951–6_10.

Fiebrink’s Wekinator is groundbreaking, fun and inspiring so I had to include some of her writing in this list. This is mostly of interest for those looking into the use of machine learning for design and other creative and artistic endeavours. An important idea explored here is that tools that make use of (interactive, supervised) machine learning can be thought of as instruments. Using such a tool is like playing or performing, exploring a possibility space, engaging in a dialogue with the tool. For a tool to feel like an instrument requires a tight action-feedback loop.

Dove, Graham, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. The 2017 CHI Conference. New York, New York, USA: ACM. doi:10.1145/3025453.3025739.

A really good survey of how designers currently deal with machine learning. Key takeaways include that in most cases, the application of machine learning is still engineering-led as opposed to design-led, which hampers the creation of non-obvious machine learning applications. It also makes it hard for designers to consider ethical implications of design choices. A key reason for this is that at the moment, prototyping with machine learning is prohibitively cumbersome.

Fiebrink, Rebecca, Perry R Cook, and Dan Trueman. 2011. “Human Model Evaluation in Interactive Supervised Learning.” In, 147. New York, New York, USA: ACM Press. doi:10.1145/1978942.1978965.

The second Fiebrink piece in this list, which is more of a deep dive into how people use Wekinator. As with the chapter listed above this is required reading for those working on design tools which make use of interactive machine learning. An important finding here is that users of intelligent design tools might have very different criteria for evaluating the ‘correctness’ of a trained model than engineers do. Such criteria are likely subjective and evaluation requires first-hand use of the model in real time.

Bostrom, Nick, and Eliezer Yudkowsky. 2014. “The Ethics of Artificial Intelligence.” In The Cambridge Handbook of Artificial Intelligence, edited by Keith Frankish and William M Ramsey, 316–34. Cambridge: Cambridge University Press. doi:10.1017/CBO9781139046855.020.

Bostrom is known for his somewhat crazy but thoughtprovoking book on superintelligence and although a large part of this chapter is about the ethics of general artificial intelligence (which at the very least is still a way out), the first section discusses the ethics of current “narrow” artificial intelligence. It makes for a good checklist of things designers should keep in mind when they create new applications of machine learning. Key insight: when a machine learning system takes on work with social dimensions—tasks previously performed by humans—the system inherits its social requirements.

Yang, Qian, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning Adaptive Mobile Experiences When Wireframing. The 2016 ACM Conference. New York, New York, USA: ACM. doi:10.1145/2901790.2901858.

Finally, a feet-in-the-mud exploration of what it actually means to design for machine learning with the tools most commonly used by designers today: drawings and diagrams of various sorts. In this case the focus is on using machine learning to make an interface adaptive. It includes an interesting discussion of how to balance the use of implicit and explicit user inputs for adaptation, and how to deal with inference errors. Once again the limitations of current sketching and prototyping tools is mentioned, and related to the need for designers to develop tacit knowledge about machine learning. Such tacit knowledge will only be gained when designers can work with machine learning in a hands-on manner.

Supplemental material

Floyd, Christiane. 1984. “A Systematic Look at Prototyping.” In Approaches to Prototyping, 1–18. Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978–3–642–69796–8_1.

I provided this to students so that they get some additional grounding in the various kinds of prototyping that are out there. It helps to prevent reductive notions of prototyping, and it makes for a nice complement to Buxton’s work on sketching.

Blevis, E, Y Lim, and E Stolterman. 2006. “Regarding Software as a Material of Design.”

Some of the papers refer to machine learning as a “design material” and this paper helps to understand what that idea means. Software is a material without qualities (it is extremely malleable, it can simulate nearly anything). Yet, it helps to consider it as a physical material in the metaphorical sense because we can then apply ways of design thinking and doing to software programming.