Leapfroglog – A weblog by Kars Alfrink.

Designing Learning Experiences in a Post-ChatGPT World

Transcript of a talk delivered at LXDCON’25 on June 12, 2025.

My name is Kars. I am a postdoc at TU Delft. I research contestable AI—how to use design to ensure AI systems remain subject to societal control. I teach the responsible design of AI systems. In a previous life, I was a practicing designer of digital products and services. I will talk about designing learning experiences in a post-ChatGPT world.

Let’s start at this date.

This is when OpenAI released an early demo of ChatGPT. The chatbot quickly went viral on social media. Users shared examples of what it could do. Stories and samples included everything from travel planning to writing fables to coding computer programs. Within five days, the chatbot had attracted over one million users.

Fast forward to today, 2 years, 6 months, and 14 days later, we’ve seen a massive impact across domains, including on education.

For example, the article on the left talks about how AI cheating has become pervasive in higher education. It is fundamentally undermining the educational process itself. Students are using ChatGPT for nearly every assignment while educators struggle with ineffective detection methods and question whether traditional academic work has lost all meaning.

The one on the right talks about how students are accusing professors of being hypocritical. Teachers are using AI tools for things like course materials and grading while telling students they cannot use them.

What we’re looking at is a situation where academic integrity was already in question, on top of that, both students and faculty are quickly adopting AI, and institutions aren’t really ready for it.

These transformations in higher education give me pause. What should we change about how we design learning experiences given this new reality?

So, just to clarify, when I mention “AI” in this talk, I’m specifically referring to generative AI, or GenAI, and even more specifically, to chatbots that are powered by large language models, like ChatGPT.

Throughout this talk I will use this example of a learning experience that makes use of GenAI. Sharad Goel, Professor at Harvard Kennedy School, developed an AI Slackbot named “StatGPT” that aims to enhance student learning through interactive engagement.

It was tested in a statistics course with positive feedback from students. They described it as supportive and easily accessible, available anytime for student use. There are plans to implement StatGPT in various other courses. They say it assists in active problem-solving and consider it an example of how AI can facilitate learning, rather than replace it.

The debate around GenAI and learning has become polarized. I see the challenge as trying to find a balance. On one side, there’s complete skepticism about AI, and on the other, there’s this blind acceptance of it. What I propose is that we need an approach I call Conscious Adaptation: moving forward with full awareness of what’s being transformed.

To build the case for this approach, I will be looking at two common positions in the debates around AI and education. I’ll be focusing on four pieces of writing.

Two of them are by Ethan Mollick, from his blog. He’s a professor at the University of Pennsylvania specializing in innovation and entrepreneurship, known for his work on the potential of AI to transform different fields.

The other two pieces are by Ian Bogost, published at The Atlantic. He’s a media studies scholar, author, and game designer who teaches at Washington University. He’s known for his sobering, realist critiques of the impact of technology on society.

These, to me, exemplify two strands of the debate around AI in education.

Ethan Mollick’s position, in essence, is that AI in education is an inevitable transformation that educators must embrace and redesign around, not fight.

You could say Mollick is an optimist. But he is also really clear-eyed about how much disruption is going on. He even refers to it as the “Homework Apocalypse.” He talks about some serious issues: there are failures in detection, students are not learning as well (with exam performance dropping by about 17%), and there are a lot of misunderstandings about AI on both sides—students and faculty.

But his perspective is more about adapting to a tough situation. He’s always focused on solutions, constantly asking, “What can we do about this?” He believes that with thoughtful human efforts, we can really influence the outcomes positively.

On the other hand, Ian Bogost’s view is that AI has created an unsolvable crisis that’s fundamentally breaking traditional education and leaving teachers demoralized.

Bogost, I would describe as a realist. He accepts the inevitability of AI, noting that the “arms race will continue” and that technology will often outpace official policies. He also highlights the negative impact on faculty morale, the dependency of students, and the chaos in institutions.

He’s not suggesting that we should ban AI or go back to a time before it existed. He sees AI as something that might be the final blow to a profession that’s already struggling with deeper issues. At the same time, he emphasizes the need for human agency by calling out the lack of reflection and action from institutions.

So, they both observe the same reality, but they look at it differently. Mollick sees it as an engineering challenge—one that’s complicated but can be tackled with smart design. On the other hand, Bogost views it as a social issue that uncovers deeper problems that can’t just be fixed with technology.

Mollick thinks it’s possible to rebuild after a sort of collapse, while Bogost questions if the institutions that are supposed to do that rebuilding are really fit for the job.

Mollick would likely celebrate it as an example of co-intelligence. Bogost would likely ask what the rollout of the bot would be at the expense of, or what deeper problems its deployment unveils.

Getting past the conflict between these two views isn’t just about figuring out the best technical methods or the right order of solutions. The real challenge lies in our ability as institutions to make real changes, and we need to be careful that focusing on solutions doesn’t distract us from the important discussions we need to have.

I see three strategies that work together to create an approach that addresses the conflict between these two perspectives in a way that I believe will be more effective.

First, institutional realism is about designing interventions assuming institutions will resist change, capture innovations, or abandon initiatives. Given this, we could focus on individual teacher practices, learner-level tools, and changes that don’t require systemic transformation. We could treat every implementation as a diagnostic probe revealing actual (vs. stated) institutional capacity.

Second, loss-conscious innovation is about before implementing AI-enhanced practices, explicitly identifying what human learning processes, relationships, or skills are being replaced. We could develop metrics that track preservation alongside progress. We could build “conservation” components into new approaches to protect irreplaceable educational values.

Third, and finally, we should recognize that Mollick-style solution-building and Bogost-style critical analysis serve different but essential roles. Practitioners need actionable guidance; while the broader field needs diagnostic consciousness. We should avoid a false synthesis but instead maintain both approaches as distinct intellectual work that informs each other.

In short, striking a balance may not be the main focus; it’s more about taking practical actions while considering the overall context. Progress is important, but it’s also worth reflecting on what gets left behind. Conscious adaptation.

So, applying these strategies to Harvard’s chatbot, we could ask: (1) How can we create a feedback loop between an intervention like this and the things it uncovers about institutional limits, so that those can be addressed in the appropriate place? (2) How can we measure what value this bot adds for students and for teachers? What is it replacing, what is it adding, what is it making room for? (3) What critique of learning at Harvard is implied by this intervention?

What does all of this mean, finally, for LXD? This is an LXD conference, so I don’t need to spend a lot of time explaining what it is. But let’s just use this basic definition as a starting point. It’s about experiences, it’s about centering the learner, it’s about achieving learning outcomes, etc.

Comparing my conscious adaptation approach to what typifies LXD, I can see a number of alignments.

Both LXD and Conscious Adaptation prioritize authentic human engagement over efficiency. LXD through human-centered design, conscious adaptation through protecting meaningful intellectual effort from AI displacement.

LXD’s focus on holistic learning journeys aligns with both Mollick’s “effort is the point” and Bogost’s concern that AI shortcuts undermine the educational value embedded in struggle and synthesis.

LXD’s experimental, prototype-driven approach mirrors my “diagnostic pragmatism”—both treat interventions as learning opportunities that reveal what actually works rather than pursuing idealized solutions.

So, going back one final time to Harvard’s bot, an LXD practice aligned in this way would lead us to ask: (1) Is this leveraging GenAI to protect and promote genuine intellectual effort? (2) Are teachers and learners meaningfully engaged in the ongoing development of this technology? (3) Is this prototype properly embedded, so that its potential to create learning for the organization can be realized?

So, where does this leave us as learning experience designers? I see three practical imperatives for Conscious Adaptation.

First, we need to protect meaningful human effort while leveraging AI’s strengths. Remember that “the effort is the point” in learning. Rather than asking “can AI do this?”, we should ask “should it?” Harvard’s bot works because it scaffolds thinking rather than replacing it. We should use AI for feedback and iteration while preserving human work for synthesis and struggle.

Second, we must design for real institutions, not ideal ones. Institutions resist change, capture innovations, and abandon initiatives. We need to design assuming limited budgets, overworked staff, and competing priorities. Every implementation becomes a diagnostic probe that reveals what resistance actually tells us about institutional capacity.

Third, we have to recognize the limits of design. AI exposes deeper structural problems like grade obsession, teacher burnout, and test-driven curricula. You can’t design your way out of systemic issues, and sometimes the best move is recognizing when the problem isn’t experiential at all.

This is Conscious Adaptation—moving forward with eyes wide open.

Thanks.

On the design and regulation of technology

The following is a section from a manuscript in press on the similarities and differences in approaches to explainable and contestable AI in design and law (Schmude et al., 2025). It ended up on the cutting room floor, but it is the kind of thing I find handy to refer back to, so I chose to share it here.

The responsible design of AI, including practices that seek to make AI systems more explainable and contestable, must somehow relate to legislation and regulations. Writing about responsible research and innovation (RRI) more broadly, Stilgoe et al. (2013) assert that RRI, which we would say includes design, must be embedded in regulation. But does it really make sense to think of the relationship between design and regulation in this way? Understood abstractly, there are in fact at least four ways in which we can think about the relationship between the design and regulation of technology (Figure 1).

Figure 1: We see four possible ways that the relationship between (D) the design and (R) regulation of technology can be conceptualized: (1) design and regulation are independent spheres, (2) design and regulation partially overlap, (3) design is embedded inside of regulation, or (4) regulation is embedded inside design. In all cases, we assume an interactional relation between the two spheres.

To establish the relationship between design and regulation, we first need to establish how we should think about regulation, and related concepts such as governance and policymaking more generally. One straightforward definition would be that regulation entails formal rules and enforcement mechanisms that constrain behavior. These are backed by authority—typically state authority, but increasingly also non-stake actors. Regulation and governance are interactive and mutually constitutive. Regulation is one mechanism within governance systems. Governance frameworks establish contexts for regulation. Policymaking connects politics to governance by translating political contestation into actionable frameworks. Politics, then, influences all these domains: policymaking, governance, and regulation. And they, in turn, operate within and reciprocally shape society writ large. See Table 1 for working definitions of ‘regulation’ and associated concepts.

Concept	Definition
Regulation	Formal rules and enforcement mechanisms that constrain behavior, typically state-backed but increasingly emerging from non-state actors (industry self-regulation, transnational regulatory bodies).
Governance	Broader arrangements for coordinating social action across public, private, and civil society spheres through both formal and informal mechanisms.
Policymaking	Process of formulating courses of action to address public problems.
Politics	Contestation of power, interests, and values that shapes governance arrangements.
Society	Broader context of social relations, norms, and institutions.

Table 1: Working definitions of ‘regulation’ and associated concepts.

What about design? Scholars of regulation have adopted the notion of ‘regulation by design’ (RBD) to refer to the inscribing of rules into the world through the creation and implementation of technological artifacts. Prifti (2024) identifies two prevailing approaches to RBD: The essentialist view treats RBD as policy enactments, or “rules for design.” By contrast, the functionalist view treats design as a mere instrument, or “ruling by design.” We agree with Prifti when he states that both approaches are limited. Essentialism neglects the complexity of regulatory environments, while functionalism neglects the autonomy and complexity of design as a practice.

Prifti proposes a pragmatist reconstruction that views regulation as a rule-making activity (“regulativity”) performed through social practices including design (the “rule of design”). Design is conceptualized as a contextual, situated social practice that performs changes in the environment, rather than just a tool or set of rules. Unlike the law, markets, or social norms, which rely on incentives and sanctions, design can simply disable the possibility of non-compliance, making it a uniquely powerful form of regulation. The pragmatist approach distinguishes between regulation and governance, with governance being a meta-regulative activity that steers how other practices (like design) regulate. This reconceptualization helps address legitimacy concerns by allowing for greater accountability for design practices that might bypass democratic processes.

Returning to the opening question then, out of the four basic ways in which the relationship between design and regulation can be drawn (Figure 1), if we were to adopt Prifti’s pragmatist view, Type 3 would most accurately capture the relationship, with design being one of a variety of more specific ways in which regulation (understood as regulativity) actually makes changes in the world. These other forms of regulatory practice are not depicted in the figure. This seems to align with Stilgoe et al.’s aforementioned position that responsible design must be embedded within regulation. Although there is a slight nuance to our position: Design is conceived of as a form of regulation always, regardless of active work on the part of designers to ‘embed’ their work inside regulatory practices. Stilgoe et al.’s admonition can be better understood as a normative claim: Responsible designers would do well to understand and align their design work with extant laws and regulations. Furthermore, following Prifti, design is beholden to governance and must be reflexively aware of how governance steers its practices (cf. Figure 2).

Figure 2: Conceptual model of the relationship between design, ‘classical’ regulation (i.e., law-making), and governance. Both design and law-making are forms of regulation (i.e., ‘regulativity’). Governance steers how design and law-making regulate, and design and law-making are both accountable to (and reflexively aware of) governance.

Bibliography

Prifti, K. (2024). The theory of ‘Regulation By Design’: Towards a pragmatist reconstruction. Technology and Regulation, 2024, 152–166. https://doi.org/10/g9dr24
Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568–1580. https://doi.org/10/f5gv8h

Postdoc update – July 2025

I am over one year into my postdoc at TU Delft. Where did the time go? By way of an annual report, here’s a rundown of my most notable outputs and activities since the previous update from June 2024. And also, some notes on what I am up to now.

Happenings

Participatory AI and ML Engineering: On 13 February 2024 at a Human Values for Smarter Cities meeting and on 11 June 2024 at a Cities Coalition for Digital Rights meeting, I presented a talk on participatory AI and ML engineering (blogged here). This has since evolved into a study I am currently running with the working title “Vision Model Macroscope.” We are designing, building, and evaluating an interface that allows municipal workers to understand and debate value-laden technical decisions made by machine learning engineers in the construction of camera vehicles. For the design, I am collaborating with CLEVER°FRANKE. The study is part of the Human Values for Smarter Cities projected headed up by the Civic Interaction Design group at AUAS.

Envisioning Contestability Loops: My article “Envisioning Contestability Loops: Evaluating the Agonistic Arena as a Generative Metaphor for Public AI” (with Ianus Keller, Mireia Yurrita Semperena, Denis Bulygin, Gerd Kortuem, and Neelke Doorn) was published in She Ji on 17 June 2024. (I had already published the infographic “Contestability Loops for Public AI,” which the article revolves around, on 17 April 2024.) Later in the year, on 5 September 2024, I ran the workshop that the study builds on as a ThingsCon Salon. And on 27 September 2024, I presented the article at Lawtomation Days in Madrid, Spain, as part of the panel “Methods in law and technology research: inter- and cross-disciplinary challenges and opportunities,” chaired by Kostina Prifti (slides). (Also, John Thackara said nice things about the article online.)

Contestability Loops for Public AI infographic

Envisioning Contestability Loops workshop at ThingsCon Salon in progress.

Democratizing AI Through Continuous Adaptability: I presented on “Democratizing AI Through Continuous Adaptability: The Role of DevOps” at the TILTing Perspectives 2024 panel “The mutual shaping of democratic practices & AI,” which was chaired and moderated by Merel Noorman on 14 July 2024. I later reprised this talk at NWO ICT.OPEN on 16 April 2025 as part of the track “Human-Computer Interaction and Societal Impact in the Netherlands,” chaired by Armağan Karahanoğlu and Max Birk (PDF of slides).

From Stem to Stern: I was part of the organizing team of the CSCW 2024 workshop “From Stem to Stern: Contestability Along AI Value Chains,” which took place as a hybrid one-day session on 9 November 2024. I blogged a summary and some takeaways of the workshop here. Shoutout to Agathe Balayn and Yulu Pi for leading this endeavor.

Contestable AI Talks: I was invited to speak on my PhD research at various meetings and events organized by studios, agencies, consultancies, schools, and public sector organizations. On 3 September 2024, at the data design agency CLEVER°FRANKE (slides). On 10 January 2025, at the University of Utrecht Computational Sociology group. On 19 February 2025, at digital ethics consultancy The Green Land (slides). On 6 March 2024, at Communication and Multimedia Design Amsterdam (slides). And on 17 March 2025, at the Advisory Board on Open Government and Information Management.

Designing Responsible AI: Over the course of 2024, Sara Colombo, Francesca Mauri, and I developed and taught for the first time a new Integrated Product Design master’s elective, “Designing Responsible AI” (course description). Later, on 28 March 2025, I was invited by my colleagues Alessandro Bozzon and Carlo van der Valk to give a single-morning interactive lecture on part of the same content at the course AI Products and Services (slides).

Books that represent the range of theory covered in the course “Designing Responsible AI.”

Stop the Cuts: On 2 July 2024, a far-right government was sworn in in the Netherlands (it has since fallen). They intended to cut funding to education by €2 billion. A coalition of researchers, teachers, students, and others organized to protest and strike in response. I was present at several of these actions: The alternative opening of the academic year in Utrecht on 2 September 2024. Local walkouts on 14 November 2024 (I participated in Utrecht). Mass demonstration in The Hague on 25 November 2024. Local actions on 11 December 2024 (I participated in Delft). And finally, for now at least, on 24 April 2025, at the Delft edition of the nationwide relay strike. If you read this, work in academia, and want to act, join a union (I am a member of the AOb), and sign up for the WOinActie newsletter.

End of the march during the 24 April 2025 strike in Delft.

Panels: Over the past months, I was a panelist at several events. On 22 October 2024, at the Design & AI Symposium as part of the panel “Evolving Perspectives on AI and Design,” together with Iohanna Nicenboim and Jesse Benjamin, moderated by Mathias Funk (blog post). On 13 December 2024 at TH/NGS as part of the panel “Rethink Design: Book Launch and Panel Discussion on Designing With AI” chaired by Roy Bendor (video). On 12 March 2025, at the panel “Inclusive AI: Approaches to Digital Inclusion,” chaired by Nazli Cila and Taylor Stone.

Slide I used during my panel contribution at the Design & AI symposium.

Design for Human Autonomy: I was part of several activities organized by the Delft Design for Values institute related to their annual theme of autonomy (led by Michael Klenk). I was a panelist on 15 October 2024 during the kick-off event (blog post). I wrote the section on designing AI for autonomy for the white paper edited by Udo Pesch (preprint). And during the closing symposium, master’s graduation student Ameya Sawant, whom I am coaching (with Fernando Secomandi acting as chair), was honored as a finalist in the thesis competition.

Master Graduation Students: Four master students that I coached during their thesis projects graduated, which between them explored technology’s role in society through AI-mediated civic engagement, generative AI implementation in public services, experimental approaches to AI trustworthiness, and urban environmental sensing—Nina te Groen (with Achilleas Psyilidis as chair), Romée Postma (with Roy Bendor), Eline Oei (with Giulia Calabretta), and Jim Blom (with Tomasz Jaskiewicz).

Architecting for Contestability: On 22 November 2025, I ran a single-day workshop about contestability for government-employed ICT architects participating in the Digital Design & Architecture course offered by the University of Twente, on invitation from Marijn Janssen (slides).

Qualitative Design Research: On 17 December 2024, I delivered a lecture on qualitative design research for the course Empirical Design Research, on invitation from my colleague Himanshu Verma (slides). Later, on 22 April 2025, I delivered a follow-up in the form of a lecture on reflexive thematic analysis for the course Product Futures Studio, coordinated by Holly McQuillan (slides).

Democratic Generative Things: On 6 June 2025 I joined the ThingsCon unconference to discuss my contribution to the RIOT report, “Embodied AI and collective power: Designing democratic generative things” (preprint). The report was edited by edited by Andrea Krajewski and Iskander Smit.

Me, holding forth during the ThingsCon RIOT unconference.

Learning Experience Design: I delivered the closing invited talk at LXDCON on 12 June 2025, reflecting on the impact of GenAI on the fields of education and design for learning (slides). Many thanks to Niels Floor for the invitation.

People’s Compute: I published a preprint of my position paper “People’s Compute: Design and the Politics of AI Infrastructures” over at OSF on 14 April 2025. I emailed it to peers and received over a dozen encouraging responses. It was also somehow picked up by Evgeny Morozov’s The Syllabus with some nice commentary attached.

On deck

So what am I up to at the moment? Keeping nice and busy.

I am co-authoring several articles, papers, and book chapters on topics including workplace automation, AI transparency, contestability in engineering, AI design and regulation, computational argumentation, explainable and participatory AI, and AI infrastructure politics. I do hope at least some of these will see the light of day in the coming months.
I am preparing a personal grant application that builds on the vision laid out in People’s Compute.
I will be delivering an invited talk at Enterprise UX on 21 November 2025.
I am acting as a scientific advisor to a center that is currently being established, which focuses on increasing digital autonomy within Dutch government institutions.
I will be co-teaching Designing Responsible AI again in Q1 of the next academic year.
I’ll serve as an associate chair on the CHI 2026 design subcommittee.
And I have signed up to begin our university’s teaching qualification certification.

Whew. That’s it. Thanks for reading (skimming?) if you’ve made it all the way to the end. I will try to circle back and do another update, maybe a little sooner than this one, say in six months’ time.

On how to think about large language models

How should we think about large language models (LLMs)? People commonly think and talk about them in terms of human intelligence. To the extent this metaphor does not accurately reflect the properties of the technology, this may lead to misguided diagnoses and prescriptions. It seems to me an LLM is not like a human or a human brain in so many ways. One crucial distinction for me is that LLMs lack individuality and subjectivity.

What are organisms that similarly lack these qualities? Coral polyps and Portuguese man o’ war come to mind, or slime mold colonies. Or maybe a single bacterium, like an E. coli. Each is essentially identical to its clones, responds automatically to chemical gradients (bringing to mind how LLMs respond to prompts), and doesn’t accumulate unique experiences in any meaningful way.

Considering all these examples, the meme about LLMs being like a shoggoth (an amorphous blob-like monster originating from the speculative fiction of Howard Philips Lovecraft) is surprisingly accurate. The thing about these metaphors though is that it’s about as hard to reason about such organisms as it is to reason about LLMs. So to use them as a metaphor for thinking about LLMs won’t work. A shoggoth is even less helpful because the reference will only be familiar to those who know their H.P. Lovecraft.

So perhaps we should abandon metaphorical thinking and think historically instead. LLMs are a new language technology. As with previous technologies, such as the printing press, when they are introduced, our relationship to language changes. How does this change occur?

I think the change is dialectical. First, we have a relationship to language that we recognize as our own. Then, a new technology destabilizes this relationship, alienating us from the language practice. We no longer see our own hand in it. And we experience a lack of control over language practice. Finally, we reappropriate this language use in our practices. In this process of reappropriation, language practice as a whole is transformed. And the cycle begins again.

For an example of this dialectical transformation of language practice under the influence of new technology, we can take Eisenstein’s classic account of the history of the printing press (1980). Following its introduction many things changed about how we relate to language. Our engagement with language shifted from a primarily oral one to a visual and deliberative one. Libraries became more abundantly stocked, leading to the practice of categorization and classification of works. Preservation and analysis of stable texts became a possibility. The solitary reading experience gained prominence, producing a more private and personal relationship between readers and texts. Concerns about information overload first reared its head.

All of these things were once new and alien to humans. Now we consider them part of the natural order of things. They weren’t predetermined by the technology, they emerged through this active tug of war between groups in society about what the technology would be used for, mediated by the affordances of the technology itself.

In concrete material terms, what does an LLM consist of? An LLM is just numerical values stored in computer memory. It is a neural network architecture consisting of billions of parameters in weights and biases, organized in matrices. The storage is distributed across multiple devices. System software loads these parameters and enables the calculation of inferences. This all runs in physical data centers housing computing infrastructure, power, cooling, and networking infrastructure. Whenever people start talking about LLMs having agency or being able to reason, I remind myself of these basic facts.

A printing press, although a cleverly designed, engineered, and manufactured device, is similarly banal when you break it down to its essential components. Still, the ultimate changes to how we relate to language have been profound. From these first few years of living with LLMs, I think it is not unreasonable to think they will cause similar upheavals. What is important for me is to recognize how we become alienated from language, and to see ourselves as having agency in reappropriating LLM-mediated language practice as our own.

On mapping AI value chains

At CSCW 2024, back in November of last year, we* ran a workshop titled “From Stem to Stern: Contestability Along AI Value Chains.” With it, we wanted to address a gap in contestable AI research. Current work focuses mainly on contesting specific AI decisions or outputs (for example, appealing a decision made by an automated content moderation system). But we should also look at contestability across the entire AI value chain—from raw material extraction to deployment and impact (think, for example, of data center activists opposing the construction of new hyperscales). We aimed to explore how different stakeholders can contest AI systems at various points in this chain, considering issues like labor conditions, environmental impact, and data collection practices often overlooked in contestability discussions.

The workshop mixed presentations with hands-on activities. In the morning, researchers shared their work through short talks, both in person and online. The afternoon focused on mapping out where and how people can contest AI systems, from data collection to deployment, followed by detailed discussions of the practical challenges involved. We had both in-person and online participants, requiring careful coordination between facilitators. We wrapped up by synthesizing key insights and outlining future research directions.

I was responsible for being a remote facilitator most of the day. But Mireia and I also prepared and ran the first group activity, in which we mapped a typical AI value chain. I figured I might as well share the canvas we used for that here. It’s not rocket science, but it held up pretty well, so maybe some other people will get some use out of it. The canvas was designed to offer a fair bit of scaffolding for thinking through what decision points there are along the chain that are potentially value-laden.

AI value chain mapping canvas (licensed CC-BY 4.0 Mireia Yurrita & Kars Alfrink, 2024). Download PDF.

Here’s how the activity worked: We covered about 50 minutes doing a structured mapping exercise where participants identified potential contestation points along an AI value chain, using ChatGPT as an example case. The activity used a Miro board with a preliminary map showing different stages of AI development (infrastructure setup, data management, AI development, etc.). Participants first brainstormed individually for 10 minutes, adding value-laden decisions and noting stakeholders, harms, benefits, and values at stake. They then collaborated to reorganize and discuss the map for 15 minutes. The activity concluded with participants using dot voting (3 votes each) to identify the most impactful contestation sites, which were then clustered and named to feed into the next group activity.

The activity design drew from two main influences: typical value chain mapping methodologies (e.g., Mapping Actors along Value Chains, 2017), which usually emphasize tracking actors, flows, and contextual factors, and Wardley mapping (Wardley, 2022), which is characterized by the idea of a structured progression along an x-axis with an additional dimension on the y-axis.

The canvas design aimed to make AI system development more tangible by breaking it into clear phases (from infrastructure through governance) while considering visibility and materiality through the y-axis. We ultimately chose to use a familiar system (ChatGPT). This, combined with the activity’s structured approach, helped participants identify concrete opportunities for intervention and contestation along the AI value chain, which we could build on during the rest of the workshop.

I got a lot out of this workshop. Some of the key takeaways that emerged out of the activities and discussions include:

There’s a disconnect between legal and technical communities, from basic terminology differences to varying conceptions of key concepts like explainability, highlighting the need for translation work between disciplines.
We need to move beyond individual grievance models to consider collective contestation and upstream interventions in the AI supply chain.
We also need to shift from reactive contestation to proactive design approaches that build in contestability from the start.
By virtue of being hybrid, we were lucky enough to have participants from across the globe. This helped drive home to me the importance of including Global South perspectives and considering contestability beyond Western legal frameworks. We desperately need a more inclusive and globally-minded approach to AI governance.

Many thanks to all the workshop co-organizers for having me as part of the team and to Agathe and Yulu, in particular, for leading the effort.

* The full workshop team consisted of Agathe Balayn, Yulu Pi, David Gray Widder, Mireia Yurrita, Sohini Upadhyay, Naveena Karusala, Henrietta Lyons, Cagatay Turkay, Christelle Tessono, Blair Attard-Frost, Ujwal Gadiraju, and myself.

On autonomy, design, and AI

In my thesis, I use autonomy to build the normative case for contestability. It so happens that this year’s theme at the Delft Design for Values Institute is also autonomy. On October 15, 2024, I participated in a panel discussion on autonomy to kick things off. I collected some notes on autonomy that go beyond the conceptualization I used in my thesis. I thought it might be helpful and interesting to collect some of them here in adapted form.

The notes I brought included, first of all, a summary of the ecumenical conceptualization of autonomy concerning automated decision-making systems offered by Alan Rubel, Clinton Castro, and Adam Pham (2021). They conceive of autonomy as effective self-governance. To be autonomous, we need authentic beliefs about our circumstances and the agency to act on our plans. Regarding algorithmic systems, they offer this notion of a reasonable endorsement test—the degree to which a system can be said to respect autonomy depends on its reliability, the stakes of its outputs, the degree to which subjects can be held responsible for inputs, and the distribution of burdens across groups.

Second, I collected some notes from several pieces by James Muldoon, which get into notions of freedom and autonomy that were developed in socialist republican thought by the likes of Luxemburg, Kautsky, and Castoriadis (2020, 2021a, 2021b). This story of autonomy is sociopolitical rather than moral. This approach is quite appealing for someone interested in non-ideal theory in a realist mode like myself. The account of autonomy Muldoon offers is one where individual autonomy hinges on greater group autonomy and stronger bonds of association between those producing and consuming technologies. Freedom is conceived of as collective self-determination.

And then third and finally, there’s this connected idea of relational autonomy, which to a degree is part of the account offered by Rubel et al., but in the conceptions here more radical in how it seeks to create distance from liberal individualism (e.g., Christman, 2004; Mhlambi & Tiribelli, 2023; Westlund, 2009). In this, individual capacity for autonomous choice is shaped by social structures. So freedom becomes realized through networks of care, responsibility, and interdependence.

That’s what I am interested in: accounts of autonomy that are not premised on liberal individualism and that give us some alternative handle on the problem of the social control of technology in general and of AI in particular.

From my point of view, the implications of all this for design and AI include the following.

First, to make a fairly obvious but often overlooked point, the degree to which a given system impacts people’s autonomy depends on various factors. It makes little sense to make blanket statements about AI destroying our autonomy and so on.

Second, in value-sensitive design terms, you can think about autonomy as a value to be balanced against others—in the case where you take the position that all values can be considered equally important, at least in principle. Or you can consider autonomy more like a precondition for people to live with technology in concordance with their values, making autonomy take precedence over other values. The sociopolitical and relational accounts above point in this direction.

Third, suppose you buy into the radical democratic idea of technology and autonomy. In that case, it follows that it makes little sense to admonish individual designers about respecting others’ autonomy. They may be asked to privilege technologies in their designs that afford individual and group autonomy. But designers also need organization and emancipation more often than not. So it’s about building power. The power of workers inside the organizations that develop technologies and the power of communities that “consume” those same technologies.

With AI, the fact is that, in reality, in the cases I look at, the communities that AI is brought to bear on have little say in the matter. The buyers and deployers of AI could and should be made more accountable to the people subjected to AI.

Towards a realist AI design practice?

This is a version of the opening statement I contributed to the panel “Evolving Perspectives on AI and Design” at the Design & AI symposium that was part of Dutch Design Week 2024. I had the pleasure of joining Iohanna Nicenboim and Jesse Benjamin on stage to explore what could be called the post-GenAI possibility space for design. Thanks also to Mathias Funk for moderating.

The slide I displayed:

My statement:

There’s a lot of magical thinking in the AI field today. It assumes intelligence is latent in the structure of the internet. Metaphors like AGI and superintelligence are magical in nature. AI practice is also very secretive. It relies on demonstrations. This leads to a lack of rigor and political accountability (cf. Gilbert & Lambert in VentureBeat, 2023).
Design in its idealist mode is easily fooled by such magic. For example, in a recent report, the Dutch Court of Audit states that 35% of government AI systems are not known to meet expectations (cf. Raji et al., 2022).
What is needed is design in a realist mode. Realism focuses on who does what to whom in whose interest (cf. Geuss, 2008, 23 in von Busch & Palmås, 2023). Applied to AI the question becomes who gets to do AI to whom? This isn’t to say we should consider AI technologies completely inert. It mediates our being in the world (Verbeek, 2021). But we should also not consider it an independent force that’s just dragging us along.
The challenge is to steer a path between, on the one hand, wholesale cynical rejection and naive, optimistic, unconditional embrace, on the other hand.
In my own work, what that looks like is to use design to make things that allow me to go into situations where people are building and using AI systems. And to use those things as instruments to ask questions related to human autonomy, social control, and collective freedom in the face of AI.
The example shown is an animated short depicting a design fiction scenario involving intelligent camera cars used for policy execution in urban public space. I used this video to talk to civil servants about the challenges facing governments who want to ensure citizens remain in control of the AI systems they deploy (cf. Alfrink et al., 2023).
Why is this realist? Because the work looks at how some groups of people use particular forms of actually existing AI to do things to other people. The work also foregrounds the competing interests that are at stake. And it frames AI as neither fully autonomous nor fully passive, but as a thing that mediates peoples’ perceptions and actions.
There are more examples besides this. But I will stop here. I just want to reiterate that I think we need a realist approach to the design of AI.

Participatory AI and ML engineering

In the first half of this year, I’ve presented several versions of a brief talk on participatory AI. I figured I would post an amalgam of these to the blog for future reference. (Previously, on the blog, I posted a brief lit review on the same topic; this talk builds on that.)

So, to start, the main point of this talk is that many participatory approaches to AI don’t engage deeply with the specifics of the technology. One such specific is the translation work engineers do to make a problem “learnable” by a machine (Kang, 2023). From this perspective, the main question to ask becomes, how does translation happen in our specific projects? Should citizens be involved in this translation work? If so, how to achieve this?

Before we dig into the state of participatory AI, let’s begin by clarifying why we might want to enable participation in the first place. A common motivation is a lack of democratic control over AI systems. (This is particularly concerning when AI systems are used for government policy execution. These are the systems I mostly look at in my own research.) And so the response is to bring the people into the development process, and to let them co-decide matters.

In these cases, participation can be understood as an enabler of democratic agency, i.e., a way for subjects to legitimate the use of AI systems (cf. Peter, 2020 in Rubel et al., 2021). Peter distinguishes two pathways: a normative one and a democratic one. Participation can be seen as an example of the democratic pathway to legitimation. A crucial detail Peter mentions here, which is often overlooked in participatory AI literature, is that normative constraints must limit the democratic pathway to avoid arbitrariness.

So, what is the state of participatory AI research and practice? I will look at each in turn next.

As mentioned, I previously posted on the state of participatory AI research, so I won’t repeat that in full here. (For the record, I reviewed Birhane et al. (2022), Bratteteig & Verne (2018), Delgado et al. (2023), Ehsan & Riedl (2020), Feffer et al. (2023), Gerdes (2022), Groves et al. (2023), Robertson et al. (2023), Sloane et al. (2020), and Zytko et al. (2022).) Elements that jump out include:

Superficial and unrepresentative involvement.
Piecemeal approaches that have minimal impact on decision-making.
Participants with a consultative role rather than that of active decision-makers.
A lack of bridge-builders between stakeholder perspectives.
Participation washing and exploitative community involvement.
Struggles with the dynamic nature of technology over time.
Discrepancies between the time scales for users to evaluate design ideas versus the pace at which systems are developed.
A demand for participation to enhance community knowledge and to actually empower them.

Taking a step back, if I were to evaluate the state of the scientific literature on participatory AI, it strikes me that many of these issues are not new to AI. They have been present in participatory design more broadly for some time already. Many of these issues are also not necessarily specific to AI. The ones I would call out include the issues related to AI system dynamism, time scales of participation versus development, and knowledge gaps between various actors in participatory processes (and, relatedly, the lack of bridge-builders).

So, what about practice? Let’s look at two reports that I feel are a good representation of the broader field: Framework for Meaningful Stakeholder Involvement by ECNL & SocietyInside, and Democratizing AI: Principles for Meaningful Public Participation by Data & Society.

Framework for Meaningful Stakeholder Involvement is aimed at businesses, organizations, and institutions that use AI. It focuses on human rights, ethical assessment, and compliance. It aims to be a tool for planning, delivering, and evaluating stakeholder engagement effectively, emphasizing three core elements: Shared Purpose, Trustworthy Process, and Visible Impact.

Democratizing AI frames public participation in AI development as a way to add legitimacy and accountability and to help prevent harmful impacts. It outlines risks associated with AI, including biased outcomes, opaque decision-making processes, and designers lacking real-world impact awareness. Causes for ineffective participation include unidirectional communication, socioeconomic barriers, superficial engagement, and ineffective third-party involvement. The report uses environmental law as a reference point and offers eight guidelines for meaningful public participation in AI.

Taking stock of these reports, we can say that the building blocks for the overall process are available to those seriously looking. The challenges facing participatory AI are, on the one hand, economic and political. On the other hand, they are related to the specifics of the technology at hand. For the remainder of this piece, let’s dig into the latter a bit more.

Let’s focus on translation work done by engineers during model development.

For this, I build on work by Kang (2023), which focuses on the qualitative analysis of how phenomena are translated into ML-compatible forms, paying specific attention to the ontological translations that occur in making a problem learnable. Translation in ML means transforming complex qualitative phenomena into quantifiable and computable forms. Multifaceted problems are converted into a “usable quantitative reference” or “ground truth.” This translation is not a mere representation of reality but a reformulation of a problem into mathematical terms, making it understandable and processable by ML algorithms. This transformation involves a significant amount of “ontological dissonance,” as it mediates and often simplifies the complexity of real-world phenomena into a taxonomy or set of classes for ML prediction. The process of translating is based on assumptions and standards that may alter the nature of the ML task and introduce new social and technical problems.

So what? I propose we can use the notion of translation as a frame for ML engineering. Understanding ML model engineering as translation is a potentially useful way to analyze what happens at each step of the process: What gets selected for translation, how the translation is performed, and what the resulting translation consists of.

So, if we seek to make participatory AI engage more with the technical particularities of ML, we could begin by identifying translations that have happened or might happen in our projects. We could then ask to what extent these acts of translation are value-laden. For those that are, we could think about how to communicate these translations to a lay audience. A particular challenge I expect we will be faced with is what the meaningful level of abstraction for citizen participation during AI development is. We should also ask what the appropriate ‘vehicle’ for citizen participation will be. And we should seek to move beyond small-scale, one-off, often unrepresentative forms of direct participation.

Bibliography

Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022). Power to the People? Opportunities and Challenges for Participatory AI. Equity and Access in Algorithms, Mechanisms, and Optimization, 1–8. https://doi.org/10/grnj99
Bratteteig, T., & Verne, G. (2018). Does AI make PD obsolete?: Exploring challenges from artificial intelligence to participatory design. Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial – Volume 2, 1–5. https://doi.org/10/ghsn84
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023). The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–23. https://doi.org/10/gs8kvm
Ehsan, U., & Riedl, M. O. (2020). Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. In C. Stephanidis, M. Kurosu, H. Degen, & L. Reinerman-Jones (Eds.), HCI International 2020—Late Breaking Papers: Multimodality and Intelligence (pp. 449–466). Springer International Publishing. https://doi.org/10/gskmgf
Feffer, M., Skirpan, M., Lipton, Z., & Heidari, H. (2023). From Preference Elicitation to Participatory ML: A Critical Survey & Guidelines for Future Research. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 38–48. https://doi.org/10/gs8kvx
Gerdes, A. (2022). A participatory data-centric approach to AI Ethics by Design. Applied Artificial Intelligence, 36(1), 2009222. https://doi.org/10/gs8kt4
Groves, L., Peppin, A., Strait, A., & Brennan, J. (2023). Going public: The role of public participation approaches in commercial AI labs. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1162–1173. https://doi.org/10/gs8kvs
Kang, E. B. (2023). Ground truth tracings (GTT): On the epistemic limits of machine learning. Big Data & Society, 10(1), 1–12. https://doi.org/10/gtfgvx
Peter, F. (2020). The Grounds of Political Legitimacy. Journal of the American Philosophical Association, 6(3), 372–390. https://doi.org/10/grqfhn
Robertson, S., Nguyen, T., Hu, C., Albiston, C., Nikzad, A., & Salehi, N. (2023). Expressiveness, Cost, and Collectivism: How the Design of Preference Languages Shapes Participation in Algorithmic Decision-Making. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10/gr6q2t
Rubel, A., Castro, C., & Pham, A. K. (2021). Algorithms and autonomy: The ethics of automated decision systems. Cambridge University Press.
Sloane, M., Moss, E., Awomolo, O., & Forlano, L. (2020). Participation is not a Design Fix for Machine Learning. arXiv:2007.02423 [Cs]. http://arxiv.org/abs/2007.02423
Zytko, D., J. Wisniewski, P., Guha, S., P. S. Baumer, E., & Lee, M. K. (2022). Participatory Design of AI Systems: Opportunities and Challenges Across Diverse Users, Relationships, and Application Domains. Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, 1–4. https://doi.org/10/gs8kv6

Democratizing AI Through Continuous Adaptability: The Role of DevOps

Below are the abstract and slides for my contribution to the TILTing Perspectives 2024 panel “The mutual shaping of democratic practices & AI,” moderated by Merel Noorman.

Slides

democratizing-ai-through-continuous-adaptability-the-role-of-devops-tilting-perspectives-2024 Download

Abstract

Contestability

This presentation delves into democratizing artificial intelligence (AI) systems through contestability. Contestability refers to the ability of AI systems to remain open and responsive to disputes throughout their lifecycle. It approaches AI systems as arenas where groups compete for power over designs and outcomes.

Autonomy, democratic agency, legitimation

We identify contestability as a critical system quality for respecting people’s autonomy. This includes their democratic agency: their ability to legitimate policies. This includes policies enacted by AI systems.

For a decision to be legitimate, it must be democratically willed or rely on “normative authority.” The democratic pathway should be constrained by normative bounds to avoid arbitrariness. The appeal to authority should meet the “access constraint,” which ensures citizens can form beliefs about policies with a sufficient degree of agency (Peter, 2020 in Rubel et al., 2021).

Contestability is the quality that ensures mechanisms are in place for subjects to exercise their democratic agency. In the case of an appeal to normative authority, contestability mechanisms are how subjects and their representatives gain access to the information that will enable them to evaluate its justifiability. In this way, contestability satisfies the access constraint. In the case of democratic will, contestability-by-design practices are how system development is democratized. The autonomy account of legitimation adds the normative constraints that should bind this democratic pathway.

Himmelreich (2022) similarly argues that only a “thick” conception of democracy will address some of the current shortcomings of AI development. This is a pathway that not only allows for participation but also includes deliberation over justifications.

The agonistic arena

Elsewhere, we have proposed the Agonistic Arena as a metaphor for thinking about the democratization of AI systems (Alfrink et al., 2024). Contestable AI embodies the generative metaphor of the Arena. This metaphor characterizes public AI as a space where interlocutors embrace conflict as productive. Seen through the lens of the Arena, public AI problems stem from a need for opportunities for adversarial interaction between stakeholders.

This metaphorical framing suggests prescriptions to make more contentious and open to dispute the norms and procedures that shape:

AI system design decisions on a global level, and
human-AI system output decisions on a local level (i.e., individual decision outcomes), establishing new dialogical feedback loops between stakeholders that ensure continuous monitoring.

The Arena metaphor encourages a design ethos of revisability and reversibility so that AI systems embody the agonistic ideal of contingency.

Post-deployment malleability, feedback-ladenness

Unlike physical systems, AI technologies exhibit a unique malleability post-deployment.

For example, LLM chatbots optimize their performance based on a variety of feedback sources, including interactions with users, as well as feedback collected through crowd-sourced data work.

Because of this open-endedness, democratic control and oversight in the operations phase of the system’s lifecycle become a particular concern.

This is a concern because while AI systems are dynamic and feedback-laden (Gilbert et al., 2023), many of the existing oversight and control measures are static, one-off exercises that struggle to track systems as they evolve over time.

DevOps

The field of DevOps is pivotal in this context. DevOps focuses on system instrumentation for enhanced monitoring and control for continuous improvement. Typically, metrics for DevOps and their machine learning-specific MLOps offshoot emphasize technical performance and business objectives.

However, there is scope to expand these to include matters of public concern. The matters-of-concern perspective shifts the focus on issues such as fairness or discrimination, viewing them as challenges that cannot be resolved through universal methods with absolute certainty. Rather, it highlights how standards are locally negotiated within specific institutional contexts, emphasizing that such standards are never guaranteed (Lampland & Star, 2009, Geiger et al., 2023).

MLOps Metrics

In the context of machine learning systems, technical metrics focus on model accuracy. For example, a financial services company might use Area Under The Curve Receiver Operating Characteristics (AUC-ROC) to continuously monitor and maintain the performance of their fraud detection model in production.

Business metrics focus on cost-benefit analyses. For example, a bank might use a cost-benefit matrix to balance the potential revenue from approving a loan against the risk of default, ensuring that the overall profitability of their loan portfolio is optimized.

Drift

These metrics can be monitored over time to detect “drift” between a model and the world. Training sets are static. Reality is dynamic. It changes over time. Drift occurs when the nature of new input data diverges from the data a model was trained on. A change in performance metrics may be used to alert system operators, who can then investigate and decide on a course of action, e.g., retraining a model on updated data. This, in effect, creates a feedback loop between the system in use and its ongoing development.

An expansion of these practices in the interest of contestability would require:

setting different metrics,
exposing these metrics to additional audiences, and
establishing feedback loops with the processes that govern models and the systems they are embedded in.

Example 1: Camera Cars

Let’s say a city government uses a camera-equipped vehicle and a computer vision model to detect potholes in public roads. In addition to accuracy and a favorable cost-benefit ratio, citizens, and road users in particular, may care about the time between a detected pothole and its fixing. Or, they may care about the distribution of potholes across the city. Furthermore, when road maintenance appears to be degrading, this should be taken up with department leadership, the responsible alderperson, and council members.

Example 2: EV Charching

Or, let’s say the same city government uses an algorithmic system to optimize public electric vehicle (EV) charging stations for green energy use by adapting charging speeds to expected sun and wind. EV drivers may want to know how much energy has been shifted to greener time windows and its trends. Without such visibility on a system’s actual goal achievement, citizens’ ability to legitimate its use suffers. As I have already mentioned, democratic agency, when enacted via the appeal to authority, depends on access to “normative facts” that underpin policies. And finally, professed system functionality must be demonstrated as well (Raji et al., 2022).

DevOps as sociotechnical leverage point for democratizing AI

These brief examples show that the DevOps approach is a potential sociotechnical leverage point. It offers pathways for democratizing AI system design, development, and operations.

DevOps can be adapted to further contestability. It creates new channels between human and machine actors. One of DevOps’s essential activities is monitoring (Smith, 2020), which presupposes fallibility, a necessary precondition for contestability. Finally, it requires and provides infrastructure for technical flexibility so that recovery from error is low-cost and continuous improvement becomes practically feasible.

The mutual shaping of democratic practices & AI

Zooming out further, let’s reflect on this panel’s overall theme, picking out three elements: legitimation, representation of marginalized groups, and dealing with conflict and contestation after implementation and during use.

Contestability is a lever for demanding justifications from operators, which is a necessary input for legitimation by subjects (Henin & Le Métayer, 2022). Contestability frames different actors’ stances as adversarial positions on a political field rather than “equally valid” perspectives (Scott, 2023). And finally, relations, monitoring, and revisability are all ways to give voice to and enable responsiveness to contestations (Genus & Stirling, 2018).

And again, all of these things can be furthered in the post-deployment phase by adapting the DevOps lens.

Bibliography

Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022). Contestable AI by Design: Towards a Framework. Minds and Machines, 33(4), 613–639. https://doi.org/10/gqnjcs
Alfrink, K., Keller, I., Yurrita Semperena, M., Bulygin, D., Kortuem, G., & Doorn, N. (2024). Envisioning Contestability Loops: Evaluating the Agonistic Arena as a Generative Metaphor for Public AI. She Ji: The Journal of Design, Economics, and Innovation, 10(1), 53–93. https://doi.org/10/gtzwft
Geiger, R. S., Tandon, U., Gakhokidze, A., Song, L., & Irani, L. (2023). Making Algorithms Public: Reimagining Auditing From Matters of Fact to Matters of Concern. International Journal of Communication, 18(0), Article 0.
Genus, A., & Stirling, A. (2018). Collingridge and the dilemma of control: Towards responsible and accountable innovation. Research Policy, 47(1), 61–69. https://doi.org/10/gcs7sn
Gilbert, T. K., Lambert, N., Dean, S., Zick, T., Snoswell, A., & Mehta, S. (2023). Reward Reports for Reinforcement Learning. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 84–130. https://doi.org/10/gs9cnh
Henin, C., & Le Métayer, D. (2022). Beyond explainability: Justifiability and contestability of algorithmic decision systems. AI & SOCIETY, 37(4), 1397–1410. https://doi.org/10/gmg8pf
Himmelreich, J. (2022). Against “Democratizing AI.” AI & SOCIETY. https://doi.org/10/gr95d5
Lampland, M., & Star, S. L. (Eds.). (2008). Standards and Their Stories: How Quantifying, Classifying, and Formalizing Practices Shape Everyday Life (1st edition). Cornell University Press.
Peter, F. (2020). The Grounds of Political Legitimacy. Journal of the American Philosophical Association, 6(3), 372–390. https://doi.org/10/grqfhn
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022). The Fallacy of AI Functionality. 2022 ACM Conference on Fairness, Accountability, and Transparency, 959–972. https://doi.org/10/gqfvf5
Rubel, A., Castro, C., & Pham, A. K. (2021). Algorithms and autonomy: The ethics of automated decision systems. Cambridge University Press.
Scott, D. (2023). Diversifying the Deliberative Turn: Toward an Agonistic RRI. Science, Technology, & Human Values, 48(2), 295–318. https://doi.org/10/gpk2pr
Smith, J. D. (2020). Operations anti-patterns, DevOps solutions. Manning Publications.
Treveil, M. (2020). Introducing MLOps: How to scale machine learning in the enterprise (First edition). O’Reilly.

PhD update – June 2024

I am writing this final PhD update as a freshly minted doctor. On Thursday, May 23, 2024, I successfully defended my thesis, ‘Contestable Artificial Intelligence: Constructive Design Research for Public Artificial Intelligence Systems that are Open and Responsive to Dispute.’

I started the PhD on September 1, 2018 (read the very first update posted on that day here). So, that’s five years, eight months, 23 days from start to finish. It has been quite the journey, and I feel happy and relieved to have completed it. I am proud of the work embodied in the thesis. Most of all, I am thankful for the transformative learning experience, none of which would have been possible without the support of my supervisors Gerd, Ianus, and Neelke.

On the day itself, I was honored to have as my external committee members professors Dignum, Löwgren, van Zoonen, and van de Poel, professor Voûte as the chair, and Joost and Mireia as my paranymphs.

The thesis PDF can be downloaded at the TU Delft repository, and a video of the proceedings is available on YouTube.

Me, with a copy of the thesis, shortly before starting the layperson’s talk. Photo: Roy Borghouts.

Recent events

Reviewing my notes since the last update, below are some more notable things that happened in the past eight months.

I ran a short workshop on AI Pedagogy Through A Design Lens, together with Hosana Morales, at the TU Delft spring symposium on AI education. Read the post.
A story about my research was published on the TU Delft industrial design engineering website in the run-up to my defense on May 14, 2024. Read the story.
I updated and ran the fifth and final iteration of the AI & Society industrial design engineering master elective course from February 28 through April 10, 2024. A previous version is documented here, which I plan to update sometime in the near future.
I gave a talk titled Contestable AI: Designing for Human Autonomy at the Amsterdam UX meetup on February 21, 2024. Download the slides.
The outcomes of a design sprint on tools for third-party scrutiny, organized by the Responsible Sensing Lab, which took inspiration from my research, were published on December 7, 2023. Read the report.
I was interviewed by Mireia Yurrita Semperena for a DCODE podcast episode titled Beyond Values in Algorithmic Design, published November 6, 2023. Listen to the episode.
Together with Claudio Sarra and Marco Almada, I hosted an online seminar titled Building Contestable Systems on October 26, 2023. Read the thread.
I was a panelist at the Design & AI Symposium 2023 on October 18, 2023.
A paper I co-authored titled When ‘Doing Ethics’ Meets Public Procurement of Smart City Technology – an Amsterdam Case Study, was presented by first author Mike de Kreek at IASDR 2023 on October 9-13. Read the paper.

Looking ahead

I will continue at TU Delft as a postdoctoral researcher and will stay focused on design, AI, and politics, but I will try to evolve my research into something that builds on my thesis work but adds a new angle.

The Envisioning Contestability Loops article mentioned in previous updates is now in press with She Ji, which I am very pleased about. It should be published “soon.”

Upcoming appearances include a brief talk on participatory AI at a Cities Coalition for Digital Rights event and a presentation as part of a panel on The Mutual Shaping Of Democratic Practices And AI at TILTing Perspectives 2024.

That’s it for this final PhD update. I will probably continue these posts under a new title. We’ll see.