The text below is an excerpt from a research paper in cultural studies originally written in German under the title “It’s a feature, not a glitch. Eine Diskussion über die Verbindung zwischen Klassifizierungen in KI-Modellen und kulturellem und politischem Backlash".

Classifying is a fundamental human action

Every Saturday morning I write a grocery list. This list consists of a hierarchical arrangement of categories like ‘snacks’, ‘vegetables’ and ‘fruit’, to which specific items like ‘apples’, ‘strawberries’ and ‘lemons’ are assigned.

But hang on! Are strawberries actually fruit, or do they belong in the ‘nuts’ category? What would happen if I didn’t go grocery shopping myself? Would anyone else know what to make of the fact that I’ve listed strawberries in the ‘fruit’ column? Probably! Because in the supermarket they’re almost certainly displayed next to the raspberries, in other words, in the fruit section and not on the shelf next to the cashew nuts.
[...]
Classifications structure and organise actions and are embedded in social life through tools and technologies (Abend 2023: 239). Classifying is, according to Susan Leigh Star and Geoffrey Bowker (1999), as well as Pierre Bourdieu (2018), a fundamental human activity. However, because they are embedded in society and in existing infrastructures, categories can become habitual, taken for granted, and invisible (Bowker/Star 1999: 319). Therefore, they become powerful and serve as an instrument of power (Crawford 2021: 127). This has consequences, as classifications also shape highly complex technological systems like AI. Not every categorisation, nor every categorical error, is as innocent as my grocery list.
[...]
This also raises the question of how classifications, categories, and standards themselves operate. In their book Sorting Things Out, Geoffrey C. Bowker and Susan Leigh Star (1999) demonstrated that classification systems, such as medical categories or ethnic categories, do not merely reflect reality, but also help to shape it. By defining which categories exist and what their definitions are, a classification scheme exercises power. It determines which differences are considered relevant. 
[...]
AI models, particularly those based on deep learning, rely on large datasets and classification algorithms, which, for example, categorise objects and people into predefined categories. These categories, in turn, are based on assumptions and values regarding what is relevant and how the world should be ‘divided up’. 
[...]
One topic that is repeatedly raised, debated, and described in both academic and public discourse on AI models is their distortion and bias, and the resulting (social) effects. Beyond that, the consequences of the underlying, pre-existing classification practices employed in these models are also of interest to the social sciences.
[...] 
The classification scheme always forms a pivotal point of social organisation, moral order, and technical integration (Bowker/Star 1999: 33).
[...]  

One example: ImageNet

ImageNet, developed by Fei-Fei Li et al. at Stanford in 2009, is regarded as one of the best-known and most widely used datasets in the development of deep learning. Over the past decade, it has become a cornerstone for the training of machine learning models and a benchmark for tasks such as image classification and object recognition (Luccioni/Crawford 2024: 2).
[...]
The most comprehensive version of ImageNet, known as ImageNet 21K, contains 21,841 categories (synsets) comprising just over 14 million images. ImageNet 21K (Autumn 2011 version) remains widely used in AI development to this day.
[...]
One example of one of these categories is a photograph of a smiling woman in a bikini labelled as a ‘slattern’, ‘slut’, ‘slovenly woman’ or ‘trollop’ (cf. Crawford/Paglen 2019). This example is representative of others and of the overarching phenomenon that many of the categories used are ideologically charged.
[...] 

Datasets form epistemological boundaries

A key question is therefore: What kind of political stance (ideology or values) influences the classification and labelling of images in datasets? To put it another way: What worldview has been transferred into the data, consciously or unconsciously, and what are the implications of this? The datasets, in a sense, form epistemological boundaries (Crawford/Paglen 2019). At the same time, they influence which normative patterns of behaviour are adopted, supported and reproduced by AI models. A specific slice of reality emerges, which is perpetuated and has a feedback effect on society. 
[...]
The practice of labelling human identity based on an image is an act of exercising power. According to Os Keyes et al. (2021: 165), this does not identify the truth, but merely naturalises a particular perspective. Automated identity recognition is fundamentally an essentialist project, based on the assumption of the existence of fixed, essential and visible attributes that can be used by a central authority to define a person’s identity (Luccioni/Crawford 2024: 12). Reducing complex social realities to rigid categories fails to recognise the diversity of people and robs them of their agency. This aspect is important when considering that AI systems trained on such data are used in sensitive areas such as medicine, the justice system or education.
[...] 

Another example: IBM Diversity in Faces (DiF)

ImageNet is an example of how ‘unintentionally’ problematic categories find their way into an AI dataset. Another interesting case is IBM’s Diversity in Faces (DiF), as it clearly illustrates how (past) colonial, pseudoscientific theories and anthropometric practices find their way into the datasets used to train AI models. In 2019, IBM sought to address concerns regarding bias and distortion in its system by announcing a ‘more inclusive’ dataset.
[...] 
In consequence, the developers took facial symmetry and skull shape into account in their classifications to create a complete picture of the face.
[...]
IBM is thus following the historical practices of 19th-century anthropometry. This pseudoscientific practice used physical measurements to categorise people ‘racially’ and to establish hierarchies. In this process, ethnicity became a classification of both the body and the mind (Scheuerman et al. 2021: 7). Through standards and categories, certain groups of people are constructed as ‘other’ and, as a result, are often subordinated or ‘weeded out’. The ideology behind this is known as eugenics. Facial analysis tools echo these imperialist ideologies, which underpinned the development of physiognomy and other scientific classification projects. This means that such AI technologies have the potential to reinforce racist, sexist and cisnormative beliefs and practices (Scheuerman et al. 2021: 2). 
[...]
Here, too, the practice of classification entails a centralisation of power: the power to decide which differences matter (Crawford 2021: 132). Simone Browne (2010: 135) defines this as ‘digital epidermalization’. The exercise of power through the disembodied gaze of surveillance technologies, which alienates the subject by producing a truth about the body and identity that contradicts the subject’s self-perception. And this happens invisibly and unchallenged. 
[...]

The result is a self-perpetuating cycle of discrimination

If the basis for categorising people is racism, technology becomes a threat to entire groups within the population (Schroder 2019). Whilst the logics of classification are treated as if they were natural and fixed, they are in fact fluid and contingent. This affects not only the people who are classified. The impact of this classification also shapes the classification itself. The philosopher Ian Hacking (2006: 23) refers to this as a ‘looping effect’. 
[...]
Those who do not belong to a marginalised group affected by the distortions of classification may not even notice them. This is because, due to their standardisation, they have long been part of the infrastructure and are therefore largely invisible. However, they constitute a register of power, are fundamentally political, and can be used to justify forms of violence and oppression (Crawford 2021). The result is a self-reinforcing cycle of discrimination which, under the guise of technical neutrality, increases social inequalities (ibid.: 131).
[...]

It's a planned social-engineering scheme – by design, it's a new world order

The question of whether AI will have an impact can safely be set aside. The relevant questions now are: by whom, how, where, and when will these impacts be felt (Floridi et al. 2018: 690)? And finally, we must discuss who has the power to build and use these systems?
[...]
In times of cultural backlash, it is precisely those voices that insist on supposedly ‘natural’, immutable categories that are gaining strength. Modern technology provides a new platform for old ideologies. In this way, AI can become an instrument of reactionary thinking by reproducing its premises.
[...]
In this manner, the technical infrastructure creates the foundation for a shift in values towards a reactionary society. Classifications in AI models thus institutionalise and constitute specific worldviews. Through their widespread implementation across diverse areas of society, the values associated with them become part of social life under the guise of technological objectivity. Through the (conscious or unconscious) use of AI models, more or less everyone is, in turn, involved in shaping these worldviews. This leads to a technologically mediated experience of reality that becomes an echo chamber for an ideological (for example, racist) distortion, even though social reality is in fact far more diverse.
[...]
It is becoming apparent, however, that technological possibilities do not construct a one-to-one replication of historical realities. It is not, for example, a matter of an identical repetition of colonial theories used to legitimise positions of hegemony. Rather, through the complex web of the global political arms race surrounding artificial intelligence (cf. Tortoise 2024), which is also intertwined with commercial and state interests, these technological possibilities are leading to an additional competition for interpretative authority over the construction of social reality. Technology is the catalyst here. "Our human, social, and civic dilemmas are becoming technical. And our technical dilemmas are becoming human, social, and civic" (Christian 2020: 13). 
[...]
After 21 January 2025, a photograph went viral. It was taken at Donald J. Trump’s second inauguration as President of the United States (see Getty Images 2025). The image shows Mark Zuckerberg, CEO of Meta and Facebook; Lauren Sanchez; Amazon founder Jeff Bezos; Google CEO Sundar Pichai; and Elon Musk, CEO of Tesla and SpaceX. They are all standing in the front row whilst Donald J. Trump takes the oath of office, thereby pledging his commitment to the democratic values of the American nation. Nothing conveys the interplay between technology, ideologies and political actors more vividly than this image. It demonstrates not only symbolically but explicitly that whoever possesses the dominant material power in society also possesses the dominant intellectual power (Marx/Engels 1970 [1846]: 35).

Updated note: All of this is all the more relevant when we consider the AI-assisted actions of ICE agents in the US, or the use of AI warfare to identify human targets. In other words, to select those who are to be killed.

If you’d like to find out more or discuss any specific points, please don’t hesitate to get in touch!