Category: Talks

Designing socially useful computing, together

Today I gave a short talk at a teach-in on the politics of data centres. The teach-in was part of the Stop Dirty Data Centres European Days of Action on 6 and 7 November, when local groups across Europe protested and called for a moratorium on large data centres.

The teach-in took place during a support demonstration organised by Fossielvrij NL at the Plimsollweg in Amsterdam Sloterdijk. That’s the site where Pure DC is building a hyperscale data centre for Microsoft, despite the city’s ban on hyperscales. Fieke Jansen and Charis Papaevangelou organised a line-up of Dutch university teachers, each of whom had five minutes, outdoors and without slides, to explain one part of what is at stake. Thanks to them for inviting me, to Fossielvrij NL for organising the demo and to Geef Tegengas for their direct action.

A rough abstract of what I said:

I don’t need to convince anyone at a demo like this that we should resist the buildout of big corporate AI. The question I asked was: what could computing be instead? I shared three lessons from my research as a designer. First, look past the app to the infrastructure underneath it. To really be able to shape applications, people need a say in the stuff they are built on top of. Second, you can leave an app, but you can’t leave the infrastructure, so pushing back takes durable organisations with members, in addition to individual protest. I used De Digitale Stad from back in 1994 as an early example of community computing. Third, in addition to opposing big corporate AI, I am mainly interested in exploring alternative forms of computing. Like the Luddites, we can reject exploitative technological practices without rejecting all technology. One test is whether a system resists extraction or accelerates it. This brings me to design. In my view design is always political, so designers should join communities and communities should become designers, both to resist extractive computing and to build regenerative computing that meets social needs.

Update 9 November 2026: Delta interviewed me about the teach-in. Read it in English or Dutch.

Kars speaks from a red lectern on a flatbed trailer in a grassy field, gesturing with one hand. A handwritten sign on the lectern reads "nobody asked for AI." People sit on the grass in the foreground, some in keffiyehs, and one holds up an Extinction Rebellion sign. Others stand behind them, and a BIJ1 flag flies in the crowd. Behind the trailer, a white construction fence surrounds the half-built Microsoft data centre: concrete pillars with rebar sticking out of the tops, scaffolding with a Palestinian flag, and a crane. Office towers rise against a blue sky.
Speaking at the teach-in at the Plimsollweg, Amsterdam, 6 November 2026. Photo: Dorotea Pace / Fossielvrij NL.

Keep the tension: designing with generative AI

Slides from my keynote at the CLICKNL Design Innovation Sessions, Digital Futures theme session, Dutch Design Week, 21 October 2026, Natlab, Eindhoven.

CLICKNL invited me to give this keynote. I saw it as a great opportunity to intervene in the debate about design and generative AI. My take is informed by my past work on contestability. That work is all about approaching “AI” as an arena where various parties compete over how it is made and how it is used. My take is also informed by my commitments to practice theory and political realism. Practice theory means looking at what people and things do together, rather than at technologies in isolation. Political realism means looking at politics as it actually happens. It asks who does what to whom, for whose benefit, instead of starting from abstract moral principles.

In the talk I argue that neither wholesale rejection nor uncritical embrace of generative AI will get us very far. GenAI deskills designers and opens up new ways of designing at the same time. It is built on the past work of designers, which raises the question of whose work we build on, and who gets to call the results their own. And it automates work, but in the process work may get delegated more than replaced, transforming practices as a result. Rather than trying to resolve these tensions, I look for emerging design practices that hold them and use them as an engine for moving forward. I discuss three cases: Vera van der Burg’s Text-to-Clay, Blaze Type’s test of Monotype’s AI type tool, and The Call by Holly Herndon and Mat Dryhurst. They all go beyond consuming off-the-shelf models. They treat pipelines and workflows as an object of design. And in all three there is an awareness of the value of effort and friction. I end with an open question. If AI is not one thing, not an agent, and neither our savior nor our doom, what metaphor does your mind go to?

I hope this talk helps designers think in concrete terms about how we might “do AI” differently from the dominant ways. And I hope it helps them break free from the simplistic for-or-against debates that get us nowhere.

CLICKNL DDW 2026 DIS Digital Futures keynote slides – Kars Alfrink – Keep the tensionDownload

Works discussed

Studio Dumbar

From Text-to-Clay, Vera van der Burg

Blaze Type and Monotype’s AI type tool

The Call, Holly Herndon and Mat Dryhurst

CRTA visual identity, Bruketa&Žinić&Grey

Mud to Models and Soil to Servers workshops

  • Fieke Jansen, Sunjoo Lee, Kars Alfrink and Lena Trotereau, From Mud to Models: Regenerative Futures for Community AI (conference workshop), TH/NGS 2025, Amsterdam, 12 Dec 2025, and From Soil to Servers: Regenerative Futures for Community Computing (workshop), Creative Coding Utrecht, Utrecht, 27 Mar 2026; Kars Alfrink and Sunjoo Lee, with Advaita Jairam, From Soil to Servers: Pairing Material Practice with Speculative Design to Rethink Computing Infrastructure (conference workshop), Philosophy of Human-Technology Relations Conference (PHTR 2026), The Hague, 22 Sep 2026.
  • TU Delft, “From Mud to Models: A Workshop on Regenerative AI Futures”, 17 Dec 2025. https://www.tudelft.nl/2025/io/december/from-mud-to-models-a-workshop-on-regenerative-ai-futures
  • Kars Alfrink, Sunjoo Lee, Lena Trotereau and Fieke Jansen, “Mud and models: Does making matter for regenerative AI speculation?”, in Iskander Smit and Andrea Krajewski (eds.), The State of Responsible Technology – RESIZE<REMIX<REGEN (RIOT 2026), Stichting ThingsCon Amsterdam, June 2026, pp. 34–39. https://doi.org/10.5281/zenodo.20569440

Further reading

  • Alfrink, K. (2025). People’s compute: Design and the politics of AI infrastructures. OSF. https://doi.org/10/q8kp
  • Caplan, L. (2025). Introduction: Contesting models – art and democracy since the 1990s. In L. Caplan & K. Greaves (Eds.), Model collapse: European contemporary art in a time of democratic crisis (pp. 1–20). Manchester University Press. http://www.jstor.org/stable/jj.36233585.6
  • Geuss, R. (2008). Philosophy and real politics. Princeton University Press.
  • Kuutti, K., & Bannon, L. J. (2014). The turn to practice in HCI: Towards a research agenda. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3543–3552). ACM. https://doi.org/10.1145/2556288.2557111
  • Löwgren, J., & Stolterman, E. (2004). Thoughtful interaction design: A design perspective on information technology. MIT Press.
  • McGowan, T. (2016). Capitalism and desire: The psychic cost of free markets. Columbia University Press.
  • McGowan, T. (2019). Emancipation after Hegel: Achieving a contradictory revolution. Columbia University Press.
  • Morozov, E. (2024, August). AI and the techno-utopian path not taken. Le Monde diplomatique (English edition). https://mondediplo.com/2024/08/07ai-cold-war
  • Parsons, P. C. (2026). Myths and ironies of AI-assisted design. In Designing Interactive Systems Conference (DIS ’26) (pp. 523–535). ACM. https://doi.org/10.1145/3800645.3813086
  • Porfirio, D., McDermott, I., Chen, H.-M., Satake, S., Kanda, T., & LaToza, T. D. (2026). Designing for robot wranglers: A synthesis of literature and practice. In Designing Interactive Systems Conference (DIS ’26) (pp. 4284–4298). ACM. https://doi.org/10.1145/3800645.3813070
  • Retkowsky, J., Hafermalz, E., & Huysman, M. (2026). Harnessing a “spirited technology”: How working with generative AI collapses the creative process. Academy of Management Discoveries, 12(3), 363–387. https://doi.org/10.5465/amd.2024.0280
  • Retkowsky, J., Hafermalz, E., Huysman, M., & Odijk, D. (2026, September 16). Research: Gen AI is collapsing creative processes. Harvard Business Review. https://hbr.org/2026/09/research-gen-ai-is-collapsing-creative-processes
  • Salvaggio, E. (2026, August 16). The system from nowhere. Cybernetic Forests. https://mail.cyberneticforests.com/the-system-from-nowhere/
  • Sayers, S. (2011). Alienation as a critical concept. International Critical Thought, 1(3), 287–304. https://doi.org/10.1080/21598282.2011.609265
  • Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2024). The curse of recursion: Training on generated data makes models forget. arXiv. https://arxiv.org/abs/2305.17493
  • Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568–1580. https://www.sciencedirect.com/science/article/pii/S0048733313000930
  • Suchman, L. (2023). The uncontroversial ‘thingness’ of AI. Big Data & Society, 10(2), 1–5. https://doi.org/10.1177/20539517231206794
  • Unger, G., & Narayanan, A. (2026, October 14). Steering AI beyond automation: A research agenda and plan for action. AI as Normal Technology. https://www.normaltech.ai/p/steering-ai-beyond-automation
  • van Maanen, G. (2022). AI ethics, ethics washing, and the need to politicize data ethics. Digital Society, 1(2), Article 9. https://doi.org/10.1007/s44206-022-00013-3
  • Wright, E. O. (2010). Envisioning real utopias. Verso.

Designing AI for Autonomy as a Political Project

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

Based on a talk delivered at the AI & Algorithm Seminar hosted by the Dutch Data Protection Authority on 25 March 2026 at ECC Leiden. The original ran to about 1,449 words and 5 images, with 11 references.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

The post defines autonomy as authenticity plus agency, shows how AI threatens both, and argues that ethics- and rights-based approaches protect individuals from above but do not build collective autonomy from below: they offer guardrails, not arenas. As an example of an agonistic arena it presents the Vision Model Macroscope, a prototype built with CLEVER°FRANKE around Amsterdam’s scan-vehicle trash detection case, and findings from four focus groups with municipal workers. It closes with three moves for designing AI for autonomy: from individuals to collectives, from applications to infrastructure, and from abstract principles to actual power relations.

People’s Compute: Design and the Politics of AI Infrastructures

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

This post is adapted from a talk given at “The Politics of AI: Governance, Resistance, Alternatives” at Goldsmiths, University of London, on 18 September 2025. The original ran to about 1,103 words and 13 images.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

The post argues that AI infrastructure (data centres, compute, platforms, APIs) constrains what designers can do at the application level, and that every smart device reinforces Big Tech’s control of that layer. It recalls Amsterdam’s De Digitale Stad as an early example of community-owned digital infrastructure and of its capture through privatisation. Drawing on the socialist republican ideal of freedom as collective self-determination, it proposes three moves for democratic AI design: from applications to infrastructures, from individuals to collectives, and from idealism to realism. It closes by positioning this agenda between technocratic management and populist anti-politics and points to the full paper, an OSF preprint.

Reclaiming Autonomy: Designing AI-Enhanced Work Tools That Empower Users

This post was taken down on 26 October 2026. A revised version will appear at this address.

About the original

Based on an invited talk delivered at Enterprise UX, on November 21, 2025 in Amersfoort, the Netherlands. The original ran to about 1,269 words and 16 images.

Abstract

Generated by Claude, a language model, from the original text; not written by the author.

The post asks what to question before adding an LLM feature to a work tool: whether it actually works, who gains or loses power, and which practices and skills it transforms. It frames these as questions of worker autonomy (authenticity and agency, after Prunkl), illustrated with scenes from Office Space, and lists ways AI threatens autonomy: manipulation, restricted choice, diminished competence. It then sketches a toolkit for designing AI for autonomy and applies three of its elements (reflexive design, stakeholder negotiation, collective autonomy support) to LegalMike, a Dutch legal AI platform, concluding that designers act as brokers of autonomy.

Participatory AI and ML engineering

In the first half of this year, I’ve presented several versions of a brief talk on participatory AI. I figured I would post an amalgam of these to the blog for future reference. (Previously, on the blog, I posted a brief lit review on the same topic; this talk builds on that.)

So, to start, the main point of this talk is that many participatory approaches to AI don’t engage deeply with the specifics of the technology. One such specific is the translation work engineers do to make a problem “learnable” by a machine (Kang, 2023). From this perspective, the main question to ask becomes, how does translation happen in our specific projects? Should citizens be involved in this translation work? If so, how to achieve this?

Before we dig into the state of participatory AI, let’s begin by clarifying why we might want to enable participation in the first place. A common motivation is a lack of democratic control over AI systems. (This is particularly concerning when AI systems are used for government policy execution. These are the systems I mostly look at in my own research.) And so the response is to bring the people into the development process, and to let them co-decide matters.

In these cases, participation can be understood as an enabler of democratic agency, i.e., a way for subjects to legitimate the use of AI systems (cf. Peter, 2020 in Rubel et al., 2021). Peter distinguishes two pathways: a normative one and a democratic one. Participation can be seen as an example of the democratic pathway to legitimation. A crucial detail Peter mentions here, which is often overlooked in participatory AI literature, is that normative constraints must limit the democratic pathway to avoid arbitrariness.

So, what is the state of participatory AI research and practice? I will look at each in turn next.

As mentioned, I previously posted on the state of participatory AI research, so I won’t repeat that in full here. (For the record, I reviewed Birhane et al. (2022), Bratteteig & Verne (2018), Delgado et al. (2023), Ehsan & Riedl (2020), Feffer et al. (2023), Gerdes (2022), Groves et al. (2023), Robertson et al. (2023), Sloane et al. (2020), and Zytko et al. (2022).) Elements that jump out include:

  • Superficial and unrepresentative involvement.
  • Piecemeal approaches that have minimal impact on decision-making.
  • Participants with a consultative role rather than that of active decision-makers.
  • A lack of bridge-builders between stakeholder perspectives.
  • Participation washing and exploitative community involvement.
  • Struggles with the dynamic nature of technology over time.
  • Discrepancies between the time scales for users to evaluate design ideas versus the pace at which systems are developed.
  • A demand for participation to enhance community knowledge and to actually empower them.

Taking a step back, if I were to evaluate the state of the scientific literature on participatory AI, it strikes me that many of these issues are not new to AI. They have been present in participatory design more broadly for some time already. Many of these issues are also not necessarily specific to AI. The ones I would call out include the issues related to AI system dynamism, time scales of participation versus development, and knowledge gaps between various actors in participatory processes (and, relatedly, the lack of bridge-builders).

So, what about practice? Let’s look at two reports that I feel are a good representation of the broader field: Framework for Meaningful Stakeholder Involvement by ECNL & SocietyInside, and Democratizing AI: Principles for Meaningful Public Participation by Data & Society.

Framework for Meaningful Stakeholder Involvement is aimed at businesses, organizations, and institutions that use AI. It focuses on human rights, ethical assessment, and compliance. It aims to be a tool for planning, delivering, and evaluating stakeholder engagement effectively, emphasizing three core elements: Shared Purpose, Trustworthy Process, and Visible Impact.

Democratizing AI frames public participation in AI development as a way to add legitimacy and accountability and to help prevent harmful impacts. It outlines risks associated with AI, including biased outcomes, opaque decision-making processes, and designers lacking real-world impact awareness. Causes for ineffective participation include unidirectional communication, socioeconomic barriers, superficial engagement, and ineffective third-party involvement. The report uses environmental law as a reference point and offers eight guidelines for meaningful public participation in AI.

Taking stock of these reports, we can say that the building blocks for the overall process are available to those seriously looking. The challenges facing participatory AI are, on the one hand, economic and political. On the other hand, they are related to the specifics of the technology at hand. For the remainder of this piece, let’s dig into the latter a bit more.

Let’s focus on translation work done by engineers during model development.

For this, I build on work by Kang (2023), which focuses on the qualitative analysis of how phenomena are translated into ML-compatible forms, paying specific attention to the ontological translations that occur in making a problem learnable. Translation in ML means transforming complex qualitative phenomena into quantifiable and computable forms. Multifaceted problems are converted into a “usable quantitative reference” or “ground truth.” This translation is not a mere representation of reality but a reformulation of a problem into mathematical terms, making it understandable and processable by ML algorithms. This transformation involves a significant amount of “ontological dissonance,” as it mediates and often simplifies the complexity of real-world phenomena into a taxonomy or set of classes for ML prediction. The process of translating is based on assumptions and standards that may alter the nature of the ML task and introduce new social and technical problems.

So what? I propose we can use the notion of translation as a frame for ML engineering. Understanding ML model engineering as translation is a potentially useful way to analyze what happens at each step of the process: What gets selected for translation, how the translation is performed, and what the resulting translation consists of.

So, if we seek to make participatory AI engage more with the technical particularities of ML, we could begin by identifying translations that have happened or might happen in our projects. We could then ask to what extent these acts of translation are value-laden. For those that are, we could think about how to communicate these translations to a lay audience. A particular challenge I expect we will be faced with is what the meaningful level of abstraction for citizen participation during AI development is. We should also ask what the appropriate ‘vehicle’ for citizen participation will be. And we should seek to move beyond small-scale, one-off, often unrepresentative forms of direct participation.

Bibliography

  • Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022). Power to the People? Opportunities and Challenges for Participatory AI. Equity and Access in Algorithms, Mechanisms, and Optimization, 1–8. https://doi.org/10/grnj99
  • Bratteteig, T., & Verne, G. (2018). Does AI make PD obsolete?: Exploring challenges from artificial intelligence to participatory design. Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial – Volume 2, 1–5. https://doi.org/10/ghsn84
  • Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023). The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–23. https://doi.org/10/gs8kvm
  • Ehsan, U., & Riedl, M. O. (2020). Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. In C. Stephanidis, M. Kurosu, H. Degen, & L. Reinerman-Jones (Eds.), HCI International 2020—Late Breaking Papers: Multimodality and Intelligence (pp. 449–466). Springer International Publishing. https://doi.org/10/gskmgf
  • Feffer, M., Skirpan, M., Lipton, Z., & Heidari, H. (2023). From Preference Elicitation to Participatory ML: A Critical Survey & Guidelines for Future Research. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 38–48. https://doi.org/10/gs8kvx
  • Gerdes, A. (2022). A participatory data-centric approach to AI Ethics by Design. Applied Artificial Intelligence, 36(1), 2009222. https://doi.org/10/gs8kt4
  • Groves, L., Peppin, A., Strait, A., & Brennan, J. (2023). Going public: The role of public participation approaches in commercial AI labs. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1162–1173. https://doi.org/10/gs8kvs
  • Kang, E. B. (2023). Ground truth tracings (GTT): On the epistemic limits of machine learning. Big Data & Society, 10(1), 1–12. https://doi.org/10/gtfgvx
  • Peter, F. (2020). The Grounds of Political Legitimacy. Journal of the American Philosophical Association, 6(3), 372–390. https://doi.org/10/grqfhn
  • Robertson, S., Nguyen, T., Hu, C., Albiston, C., Nikzad, A., & Salehi, N. (2023). Expressiveness, Cost, and Collectivism: How the Design of Preference Languages Shapes Participation in Algorithmic Decision-Making. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10/gr6q2t
  • Rubel, A., Castro, C., & Pham, A. K. (2021). Algorithms and autonomy: The ethics of automated decision systems. Cambridge University Press.
  • Sloane, M., Moss, E., Awomolo, O., & Forlano, L. (2020). Participation is not a Design Fix for Machine Learning. arXiv:2007.02423 [Cs]. http://arxiv.org/abs/2007.02423
  • Zytko, D., J. Wisniewski, P., Guha, S., P. S. Baumer, E., & Lee, M. K. (2022). Participatory Design of AI Systems: Opportunities and Challenges Across Diverse Users, Relationships, and Application Domains. Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, 1–4. https://doi.org/10/gs8kv6

Democratizing AI Through Continuous Adaptability: The Role of DevOps

Below are the abstract and slides for my contribution to the TILTing Perspectives 2024 panel “The mutual shaping of democratic practices & AI,” moderated by Merel Noorman.

Slides

democratizing-ai-through-continuous-adaptability-the-role-of-devops-tilting-perspectives-2024Download

Abstract

Contestability

This presentation delves into democratizing artificial intelligence (AI) systems through contestability. Contestability refers to the ability of AI systems to remain open and responsive to disputes throughout their lifecycle. It approaches AI systems as arenas where groups compete for power over designs and outcomes.

Autonomy, democratic agency, legitimation

We identify contestability as a critical system quality for respecting people’s autonomy. This includes their democratic agency: their ability to legitimate policies. This includes policies enacted by AI systems.

For a decision to be legitimate, it must be democratically willed or rely on “normative authority.” The democratic pathway should be constrained by normative bounds to avoid arbitrariness. The appeal to authority should meet the “access constraint,” which ensures citizens can form beliefs about policies with a sufficient degree of agency (Peter, 2020 in Rubel et al., 2021).

Contestability is the quality that ensures mechanisms are in place for subjects to exercise their democratic agency. In the case of an appeal to normative authority, contestability mechanisms are how subjects and their representatives gain access to the information that will enable them to evaluate its justifiability. In this way, contestability satisfies the access constraint. In the case of democratic will, contestability-by-design practices are how system development is democratized. The autonomy account of legitimation adds the normative constraints that should bind this democratic pathway.

Himmelreich (2022) similarly argues that only a “thick” conception of democracy will address some of the current shortcomings of AI development. This is a pathway that not only allows for participation but also includes deliberation over justifications.

The agonistic arena

Elsewhere, we have proposed the Agonistic Arena as a metaphor for thinking about the democratization of AI systems (Alfrink et al., 2024). Contestable AI embodies the generative metaphor of the Arena. This metaphor characterizes public AI as a space where interlocutors embrace conflict as productive. Seen through the lens of the Arena, public AI problems stem from a need for opportunities for adversarial interaction between stakeholders.

This metaphorical framing suggests prescriptions to make more contentious and open to dispute the norms and procedures that shape:

  1. AI system design decisions on a global level, and
  2. human-AI system output decisions on a local level (i.e., individual decision outcomes), establishing new dialogical feedback loops between stakeholders that ensure continuous monitoring.

The Arena metaphor encourages a design ethos of revisability and reversibility so that AI systems embody the agonistic ideal of contingency.

Post-deployment malleability, feedback-ladenness

Unlike physical systems, AI technologies exhibit a unique malleability post-deployment.

For example, LLM chatbots optimize their performance based on a variety of feedback sources, including interactions with users, as well as feedback collected through crowd-sourced data work.

Because of this open-endedness, democratic control and oversight in the operations phase of the system’s lifecycle become a particular concern.

This is a concern because while AI systems are dynamic and feedback-laden (Gilbert et al., 2023), many of the existing oversight and control measures are static, one-off exercises that struggle to track systems as they evolve over time.

DevOps

The field of DevOps is pivotal in this context. DevOps focuses on system instrumentation for enhanced monitoring and control for continuous improvement. Typically, metrics for DevOps and their machine learning-specific MLOps offshoot emphasize technical performance and business objectives.

However, there is scope to expand these to include matters of public concern. The matters-of-concern perspective shifts the focus on issues such as fairness or discrimination, viewing them as challenges that cannot be resolved through universal methods with absolute certainty. Rather, it highlights how standards are locally negotiated within specific institutional contexts, emphasizing that such standards are never guaranteed (Lampland & Star, 2009, Geiger et al., 2023).

MLOps Metrics

In the context of machine learning systems, technical metrics focus on model accuracy. For example, a financial services company might use Area Under The Curve Receiver Operating Characteristics (AUC-ROC) to continuously monitor and maintain the performance of their fraud detection model in production.

Business metrics focus on cost-benefit analyses. For example, a bank might use a cost-benefit matrix to balance the potential revenue from approving a loan against the risk of default, ensuring that the overall profitability of their loan portfolio is optimized.

Drift

These metrics can be monitored over time to detect “drift” between a model and the world. Training sets are static. Reality is dynamic. It changes over time. Drift occurs when the nature of new input data diverges from the data a model was trained on. A change in performance metrics may be used to alert system operators, who can then investigate and decide on a course of action, e.g., retraining a model on updated data. This, in effect, creates a feedback loop between the system in use and its ongoing development.

An expansion of these practices in the interest of contestability would require:

  1. setting different metrics,
  2. exposing these metrics to additional audiences, and
  3. establishing feedback loops with the processes that govern models and the systems they are embedded in.

Example 1: Camera Cars

Let’s say a city government uses a camera-equipped vehicle and a computer vision model to detect potholes in public roads. In addition to accuracy and a favorable cost-benefit ratio, citizens, and road users in particular, may care about the time between a detected pothole and its fixing. Or, they may care about the distribution of potholes across the city. Furthermore, when road maintenance appears to be degrading, this should be taken up with department leadership, the responsible alderperson, and council members.

Example 2: EV Charching

Or, let’s say the same city government uses an algorithmic system to optimize public electric vehicle (EV) charging stations for green energy use by adapting charging speeds to expected sun and wind. EV drivers may want to know how much energy has been shifted to greener time windows and its trends. Without such visibility on a system’s actual goal achievement, citizens’ ability to legitimate its use suffers. As I have already mentioned, democratic agency, when enacted via the appeal to authority, depends on access to “normative facts” that underpin policies. And finally, professed system functionality must be demonstrated as well (Raji et al., 2022).

DevOps as sociotechnical leverage point for democratizing AI

These brief examples show that the DevOps approach is a potential sociotechnical leverage point. It offers pathways for democratizing AI system design, development, and operations.

DevOps can be adapted to further contestability. It creates new channels between human and machine actors. One of DevOps’s essential activities is monitoring (Smith, 2020), which presupposes fallibility, a necessary precondition for contestability. Finally, it requires and provides infrastructure for technical flexibility so that recovery from error is low-cost and continuous improvement becomes practically feasible.

The mutual shaping of democratic practices & AI

Zooming out further, let’s reflect on this panel’s overall theme, picking out three elements: legitimation, representation of marginalized groups, and dealing with conflict and contestation after implementation and during use.

Contestability is a lever for demanding justifications from operators, which is a necessary input for legitimation by subjects (Henin & Le Métayer, 2022). Contestability frames different actors’ stances as adversarial positions on a political field rather than “equally valid” perspectives (Scott, 2023). And finally, relations, monitoring, and revisability are all ways to give voice to and enable responsiveness to contestations (Genus & Stirling, 2018).

And again, all of these things can be furthered in the post-deployment phase by adapting the DevOps lens.

Bibliography

  • Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022). Contestable AI by Design: Towards a Framework. Minds and Machines, 33(4), 613–639. https://doi.org/10/gqnjcs
  • Alfrink, K., Keller, I., Yurrita Semperena, M., Bulygin, D., Kortuem, G., & Doorn, N. (2024). Envisioning Contestability Loops: Evaluating the Agonistic Arena as a Generative Metaphor for Public AI. She Ji: The Journal of Design, Economics, and Innovation, 10(1), 53–93. https://doi.org/10/gtzwft
  • Geiger, R. S., Tandon, U., Gakhokidze, A., Song, L., & Irani, L. (2023). Making Algorithms Public: Reimagining Auditing From Matters of Fact to Matters of Concern. International Journal of Communication, 18(0), Article 0.
  • Genus, A., & Stirling, A. (2018). Collingridge and the dilemma of control: Towards responsible and accountable innovation. Research Policy, 47(1), 61–69. https://doi.org/10/gcs7sn
  • Gilbert, T. K., Lambert, N., Dean, S., Zick, T., Snoswell, A., & Mehta, S. (2023). Reward Reports for Reinforcement Learning. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 84–130. https://doi.org/10/gs9cnh
  • Henin, C., & Le Métayer, D. (2022). Beyond explainability: Justifiability and contestability of algorithmic decision systems. AI & SOCIETY, 37(4), 1397–1410. https://doi.org/10/gmg8pf
  • Himmelreich, J. (2022). Against “Democratizing AI.” AI & SOCIETY. https://doi.org/10/gr95d5
  • Lampland, M., & Star, S. L. (Eds.). (2008). Standards and Their Stories: How Quantifying, Classifying, and Formalizing Practices Shape Everyday Life (1st edition). Cornell University Press.
  • Peter, F. (2020). The Grounds of Political Legitimacy. Journal of the American Philosophical Association, 6(3), 372–390. https://doi.org/10/grqfhn
  • Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022). The Fallacy of AI Functionality. 2022 ACM Conference on Fairness, Accountability, and Transparency, 959–972. https://doi.org/10/gqfvf5
  • Rubel, A., Castro, C., & Pham, A. K. (2021). Algorithms and autonomy: The ethics of automated decision systems. Cambridge University Press.
  • Scott, D. (2023). Diversifying the Deliberative Turn: Toward an Agonistic RRI. Science, Technology, & Human Values, 48(2), 295–318. https://doi.org/10/gpk2pr
  • Smith, J. D. (2020). Operations anti-patterns, DevOps solutions. Manning Publications.
  • Treveil, M. (2020). Introducing MLOps: How to scale machine learning in the enterprise (First edition). O’Reilly.

AI pedagogy through a design lens

At a TU Delft spring symposium on AI education, Hosana and I ran a short workshop titled “AI pedagogy through a design lens.” In it, we identified some of the challenges facing AI teaching, particularly outside of computer science, and explored how design pedagogy, particularly the practices of studios and making, may help to address them. The AI & Society master elective I’ve been developing and teaching over the past five years served as a case study. The session was punctuated by brief brainstorming using an adapted version of the SQUID gamestorming technique. Below are the slides we used.

AI pedagogy through a design lensDownload

“Geen transparantie zonder tegenspraak” — betoog voor première documentaire transparante laadpaal

Het onderstaande korte betoog sprak ik uit tijdens de online premiere van de documentaire over de transparante laadpaal op donderdag 18 maart 2021.

Ik had laatst contact met een internationale “thought leader” op het gebied van “tech ethics”. Hij vertelde mij dat hij heel dankbaar is voor het bestaan van de transparante laadpaal omdat het zo’n goed voorbeeld is van hoe design kan bijdragen aan eerlijke technologie.

Dat is natuurlijk ontzettend leuk om te horen. En het past in een bredere trend in de industrie gericht op het transparant en uitlegbaar maken van algoritmes. Inmiddels is het zelfs zo ver dat wetgeving uitlegbaarheid (in sommige gevallen) verplicht stelt.

In de documentaire hoor je meerdere mensen vertellen (mijzelf inbegrepen) waarom het belangrijk is dat stedelijke algoritmes transparant zijn. Thijs benoemt heel mooi twee redenen: Enerzijds het collectieve belang om democratische controle op de ontwikkeling van stedelijke algoritmes mogelijk te maken. Anderzijds is er het individuele belang om je recht te kunnen halen als een systeem een beslissing maakt waarmee je het (om wat voor reden dan ook) niet eens bent.

En inderdaad, in beide gevallen (collectieve controle en individuele remedie) is transparantie een randvoorwaarde. Ik denk dat we met dit project een hoop problemen qua design en techniek hebben opgelost die daarbij komen kijken. Tegelijkertijd doemt er een nieuwe vraag aan de horizon op: Als we begrijpen hoe een slim systeem werkt, en we zijn het er niet mee eens, wat dan? Hoe krijg je vervolgens daadwerkelijk invloed op de werking van het systeem?

Ik denk dat we onze focus zullen moeten gaan verleggen van transparantie naar wat ik tegenspraak of in goed Engels “contestability” noem.

Ontwerpen voor tegenspraak betekent dat we na moeten gaan denken over de middelen die mensen nodig hebben voor het uitoefenen van hun recht op menselijke interventie. Ja, dit betekent dat we informatie moeten aanleveren over het hoe en waarom van individuele beslissingen. Transparantie dus. Maar het betekent ook dat we nieuwe kanalen en processen moeten inrichten waarmee mensen verzoeken kunnen indienen voor het herzien van een beslissing. We zullen na moeten gaan denken over hoe we dergelijke verzoeken beoordelen, en hoe we er voor zorgen dat het slimme systeem in kwestie “leert” van de signalen die we op deze manier oppikken uit de samenleving.

Je zou kunnen zeggen dat ontwerpen van transparantie eenrichtingsverkeer is. Informatie stroomt van de ontwikkelende partij, naar de eindgebruiker. Bij het ontwerpen voor tegenspraak gaat het om het creëren van een dialoog tussen ontwikkelaars en burgers.

Ik zeg burgers want niet alleen klassieke eindgebruikers worden geraakt door slimme systemen. Allerlei andere groepen worden ook, vaak indirect beïnvloed.

Dat is ook een nieuwe ontwerp uitdaging. Hoe ontwerp je niet alleen voor de eindgebruiker (zoals bij de transparante laadpaal de EV bestuurder) maar ook voor zogenaamde indirecte belanghebbenden, bijvoorbeeld bewoners van straten waar laadpalen geplaatst worden, die geen EV rijden, of zelfs geen auto, maar evengoed een belang hebben bij hoe stoepen en straten worden ingericht.

Deze verbreding van het blikveld betekent dat we bij het ontwerpen voor tegenspraak nóg een stap verder kunnen en zelfs moeten gaan dan het mogelijk maken van remedie bij individuele beslissingen.

Want ontwerpen voor tegenspraak bij individuele beslissingen van een reeds uitgerold systeem is noodzakelijkerwijs post-hoc en reactief, en beperkt zich tot één enkele groep belanghebbenden.

Zoals Thijs ook min of meer benoemt in de documentaire beïnvloed slimme stedelijke infrastructuur de levens van ons allemaal, en je zou kunnen zeggen dat de design en technische keuzes die bij de ontwikkeling daarvan gemaakt worden intrinsiek ook politieke keuzes zijn.

Daarom denk ik dat we er niet omheen kunnen om het proces dat ten grondslag ligt aan deze systemen zelf, ook zo in te richten dat er ruimte is voor tegenspraak. In mijn ideale wereld is de ontwikkeling van een volgende generatie slimme laadpalen daarom participatief, pluriform en inclusief, net als onze democratie dat zelf ook streeft te zijn.

Hoe we dit soort “contestable” algoritmes precies vorm moeten geven, hoe ontwerpen voor tegenspraak moeten gaan werken, is een open vraag. Maar een aantal jaren geleden wist niemand nog hoe een transparante laadpaal er uit zou moeten zien, en dat hebben we ook voor elkaar gekregen.

Update (2021-03-31 16:43): Een opname van het gehele event is nu ook beschikbaar. Het bovenstaande betoog start rond 25:14.

“Contestable Infrastructures” at Beyond Smart Cities Today

I’ll be at Beyond Smart Cities Today the next couple of days (18-19 September). Below is the abstract I submitted, plus a bibliography of some of the stuff that went into my thinking for this and related matters that I won’t have the time to get into.

In the actually existing smart city, algorithmic systems are increasingly used for the purposes of automated decision-making, including as part of public infrastructure. Algorithmic systems raise a range of ethical concerns, many of which stem from their opacity. As a result, prescriptions for improving the accountability, trustworthiness and legitimacy of algorithmic systems are often based on a transparency ideal. The thinking goes that if the functioning and ownership of an algorithmic system is made perceivable, people understand them and are in turn able to supervise them. However, there are limits to this approach. Algorithmic systems are complex and ever-changing socio-technical assemblages. Rendering them visible is not a straightforward design and engineering task. Furthermore such transparency does not necessarily lead to understanding or, crucially, the ability to act on this understanding. We believe legitimate smart public infrastructure needs to include the possibility for subjects to articulate objections to procedures and outcomes. The resulting “contestable infrastructure” would create spaces that open up the possibility for expressing conflicting views on the smart city. Our project is to explore the design implications of this line of reasoning for the physical assets that citizens encounter in the city. Because after all, these are the perceivable elements of the larger infrastructural systems that recede from view.

  • Alkhatib, A., & Bernstein, M. (2019). Street-Level Algorithms. 1–13. https://doi.org/10.1145/3290605.3300760
  • Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media and Society, 20(3), 973–989. https://doi.org/10.1177/1461444816676645
  • Centivany, A., & Glushko, B. (2016). “Popcorn tastes good”: Participatory policymaking and Reddit’s “AMAgeddon.” Conference on Human Factors in Computing Systems – Proceedings, 1126–1137. https://doi.org/10.1145/2858036.2858516
  • Crawford, K. (2016). Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics. Science Technology and Human Values, 41(1), 77–92. https://doi.org/10.1177/0162243915589635
  • DiSalvo, C. (2010). Design, Democracy and Agonistic Pluralism. Proceedings of the Design Research Society Conference, 366–371.
  • Hildebrandt, M. (2017). Privacy As Protection of the Incomputable Self: Agonistic Machine Learning. SSRN Electronic Journal, 1–33. https://doi.org/10.2139/ssrn.3081776
  • Jackson, S. J., Gillespie, T., & Payette, S. (2014). The Policy Knot: Re-integrating Policy, Practice and Design. CSCW Studies of Social Computing, 588–602. https://doi.org/10.1145/2531602.2531674
  • Jewell, M. (2018). Contesting the decision: living in (and living with) the smart city. International Review of Law, Computers and Technology. https://doi.org/10.1080/13600869.2018.1457000
  • Lindblom, L. (2019). Consent, Contestability, and Unions. Business Ethics Quarterly. https://doi.org/10.1017/beq.2018.25
  • Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 205395171667967. https://doi.org/10.1177/2053951716679679
  • Van de Poel, I. (2016). An ethical framework for evaluating experimental technology. Science and Engineering Ethics, 22(3), 667–686. https://doi.org/10.1007/s11948-015-9724-3