Checking AI? Why we need more discourse analytical enlightenment

The following text is an extended version of the presentation given at the DiscoursNet Workshop “The Digitalization of the Knowledge Economy: how AI is changing our view on the world” (8.–9.5.2026). It presents preliminary thoughts for a more extended research on generative AI in third party funding and research evaluation.

I would like to offer you a simple yet, I would say, significant perspective.

We misconceive AI because we misconceive what texts are and how they work.

Or, a bit more nuanced:

Todays understanding of AI, specifically AI Chatbots, is very much rooted in our collective representation of what texts are and how they work socially. And I would argue, that this is a source of trouble, because our collective textual practices – how we do things with texts in our everyday life – are largely marked by normative but often mistaken presuppostions about what a text is, how its truth is constituted, and what role the subject plays for this, both as an author and as a reader.

I would argue, that this misconception becomes most obvious in how we deal with the shortcomings of AI Chatbots. And there is no lack of complaints about AI Chatbots not doing, what we expected them to do.

Now, for me it is less interesting, that AI Chatbots disappoint us at times, but how we talk about this.

Some of the most common ways to describe such disappointments are complaints about “the AI” hallucinating, inventing, or even lying.

To give you just one, more or less random, but striking example, here is an article by Scott Bureau (2026) from the webpage of the Rochester Institute of Technology.

There is a clear separation of good AI and Bad AI, while the good one is telling the truth, being authentic and genuine, the bad one is not only inaccurate, but it hallucinates, fabricates and misrepresents.

While such wordings as you can see in the image are critical of AI on first sight, even normatively criticising it („lie“), they are at the same time narrowing the problem down to an exception, to something more or less easily manageable. The AI hallucinates, where we find it to be wrong, but usually tells the truth and can be expected to do so. You see this opposition in the image. (Note that my interest here is the popular representation of AI chatbots in this context, not so much the actual study described in the article.)

On a technical level this misses the point, because “inventions” are not a bug of AI Chatbots, but its feature. Nothing goes wrong, when AI hallucinates. It does, what it is supposed to do. And, in fact, it also hallucinates, where we find it to tell the “truth”. Paola Lopez has clearly described this in the context of AI generated imagery: “We attach the label ‘truth’ or ‘hallucination’ to individual outputs after the fact (and we don’t even agree on the correct label amongst ourselves)” (Lopez 2025: 65, transl. D.A.).

But, instead of questioning how LLMs work, the everyday critique of AI ist mostly limited to purported „errors“, that can be rectified in fine-tuning the models.

Against this background, I would argue, that there is inadequate trust both

• in AI as technology and
• in the subject as remedy to remaining shortcomming.

This is where the title of my presentation comes in: Checking AI.

According to the idea that AI Chatbots generally tell the „truth“, but are occasionally mistaken, or, on a bad day, so to say, might even maliciously lie, the current discourse invokes the subject as a remedy to such shortcomings – the subject both as a vigilant reader and, what is more, as a vigilant co-author.

I want to turn to one example for this, which is from the core of the scientific process. The German Research Foundation recently published guidelines on how to use AI as a reviewer for research grants. While the admission of AI Chatbots was so far limited to applicants, now reviewers are allowed to use them, too. What is interesting is, how a legitimate usage is delineated from an illegitimate one:

“AI may be used for textual and structural support, literature searches and formal text checks, while the academic analysis, evaluation and recommendation remain the exclusive responsibility of the individual reviewer.” (DFG 2026a)

This contains an implicit assumption of a clear separation of the core reviewing tasks (analysing, evaluating, recommending) and a seemingly neutral legwork. A similar separation can also be found in the recommendations for other Grants. However, what belongs on which side can be seen as contested. What is considered being part of the legwork in one context might be considered a key aspect of scientific evaluation in another.

Mostly in line with the DFG guidelines, the European Research Council (ERC) forbids “delegating” evaluation tasks to AI Chatbots and exemplifies what delegation would be as follows:

“For example, reviewers may not use an AI system to:
– summarise the proposal to avoid reading it in full,
– provide any form of assessment, explicit or implicit, of the merit of the proposal” (ERC 2026).

Now, summarising is exactly the task many researchers hope AI Chatbots are helpful for. At the same time, both, the DFG and the ERC, explicitly allow to use AI Chatbots for literature research on the proposals topic. So, implicitly, they define literature not to be part of the core evaluation practices, that cannot be delegated to AI systems.

For applicants, the separation has even more consequences: The DFG guidelines for applicants require them to explicitly disclose the use of AI Chatbots in the application, except if the usage of AI does not “affect the scientific content of the proposal (e.g. checking grammar, style, orthography, translating programs)” (DFG 2026b, transl. D.A.).

At this point, it is not so much decisive which kind of separation of the scientific core and the mere accompanying and decorative hem is the right one, but that there is no clear separation in the first place. These guidelines draw a line, so to say. But, while these guidelines are meant to provide clarity about right and wrong behaviour in the application process, they are more like drawing a line in dry sand. While there seems to be clear distinction from a bird’s eye view, from a practical perspective it is next to impossible to definitely know which practices are still on the good side, and which ones on the bad side. So it is still up to applicants and reviewers to constantly work on this distinction, to re-negotiate it while applying it.

At the end of the day, the intricate problem, who is responsible for the text in a LLM-human co-authorship, is resolved by a simple declaration: the reviewer/applicant is attributed the „exclusive responsibility“. This declaration of responsibility is quite typical. In the “Living guidelines on the responsible use of AI in research, published by the European Commission, it is even the first “recommendation” for researchers: Researchers should

“1. Remain ultimately responsible for scientific output” (European Commission 2026: 7).

Now it is remarkable that responsibility is framed as a normative duty here. And, in fact, one of the attached bullet points in the document makes clear, that the research already factually is responsible.

“AI systems are neither authors nor co-authors. Authorship implies agency and responsibility, so it lies with human researchers” (European Commission 2026: 7).

So, its less a question of becoming responsible, but of considering the consequences of a responsibility, that is declared beforehand. While the AI guild lines promise to provide certainty for the researchers, the declaration of responsibility is a bit like hanging up a warning sign “Caution. Enter at your own risk”. While the guidelines indicate some measures that can, or rather should be taken by the researchers, if something goes wrong it will allays retrospectively be the researchers fault anyway.

Consider this passage from the same DFG guidelines for reviewers, quoted above:

“AI-generated content must not be adopted uncritically. It must be checked for accuracy, bias and disciplinary appropriateness. Responsibility for the content remains with the reviewer” (DFG 2026a).

So “checking” the content the AI provides is the way to ‘take’ the responsibility, which has been declared. It is the subjects individual task to ward of any dangers for the scientific process, that could arise form the use of AI Chatbots in applications and reviews, such as loosing “objectivity” or “diversity”:

“Without careful scrutiny, such distortions could undermine the objectivity and diversity of academic assessments; this must be avoided at all cost” (DFG 2026a).

Focussing on the individual responsibility of the researcher, also prefigures, where to look for resolution for emerging problems: more or less isolated acts of verification. The idea of „checking AI“ is basically, that there is the content provided by generative Chatbots on the one hand, a set of truths on the other hand, and a subject that maps both and finds any deviation between them.

Now, I would argue that the underlying assumptions about texts and their truth are fundamentally at odds with what we know from discourse analysis, the sociology of knowledge & science and technology studies.

I will limit myself to two aspects, to shed some light on this claim.

Form/Content?

I think that there is a broad consent, that texts are not just formal containers of ideas or content (cf. Angermuller 2014, 64). Rather, how we say something affects what we are saying, or more generally: textual forms constitute a content This might even be one of the common grounds between (post-)structuralist and hermeneutic approaches to discourse (Angermuller 2014, Keller 2013).

If we follow this insight, than it subverts the careful efforts in the quoted documents, to clearly distinguish the scientific legwork, that can be unproblematically left to AI Chatbots, and the science proper, that cannot be delegated. But with this, it also subverts the declared responsibility of the subject: if stylistic corrections are admissible, and if there is no clear line between what is said and how it is said – is the researcher now responsible, or not?

In a way this leads to the vast amount of things that have been written on discourse and the subject, discussing and challenging assumptions about authorship, intentions and truth (think of Roland Barthes’ “death of the author” and of Michel Foucaults “author function”, to mention just two influential texts, Barthes 1968, Foucault 1994).

Verificationism

In the discourse on AI Chatbots in grant applications texts are mostly conceived of as collections of isolated statements, that all can be and must be separately check for their truth value. This is a form of a naive verificationism, which is very much at odds with how we empirically deal with texts – even in academia with it’s particular textual practices – and how our knowledge is structured.

Alfred Schutz (1944) used the nice image of a hypsography of knowledge. So our knowledge is like a alpine landscape. Our explicit knowledge is quite limited and rather a bit scattered, like summits, breaking through the clouds.

It is surrounded and connected by areas of implicit knowledge, hearsay knowledge, vague assumptions in a misty valley.

Quite some aspects are even submerged. And this points to the fact, that, other than the verificationism assumes, there are not only things that we don’t know and that we need to verify, but more often than not, we don’t know, what we don’t know, how deep the submerged areas of alpine lakes are, so to say.

In fact, explicit knowledge, that is up for verification, is a quite limited part of our knowledge. Even in science –where we value questioning assumptions, taking a closer look and create specific constellations where experiences are empowered to make our purportedly self-evident truths shatter – most of our knowledge is highly mediated; and even talking about books, we have not read, can be considered part of a legitimate part of the scientific process (cf. Bayard 2007).

What is more, the idea that language would be nothing but a collection of propositions, that are either true or false, has long been rejected – be it by pointing to the holistic nature of speech (Quine 2004) or to the empractical and deictic dimension of speech (Bühler 1999).

In general, it seems that the relevance of truth for our evaluation of texts is widely overestimated. Our knowledge is more often than not dependent rather on plausibility than on truth. And this is the perfect margin for manoeuvre for AI Chatbots, because they were in the end not created as truth machines, but as plausibility machines. As inventor of texts, truth is a byproduct of LLMs, while coherence and plausibility is a key requirement.

Overall, I think the discourse on AI in third party funding and evaluation puts unwarranted trust in both: AI Chatbots and in the subject. In contrast, I would argue that we need to deflate our trust in the technology and our trust in the subject.

And I think discourse analysis can play an important part in this regard. While Chatbots creating “sensible” texts felt like magic for many users, for a discourse analysis this does mostly confirm, that subjective intentions and meaning do play a subordinate role in discourses and that the (empirical) distribution of forms covers large parts of language in society. Think of Zelig Harris with his distributionalist approach, having in fact coined the term “analyse du discourse” (Harris 1969) or of the spectrum of corpus linguistics and lexicometry (Scholz 2019). This is deflating AI by a more prosaic idea of what „texts“ are and how they „make sense“.

At the same time, discourse analysis rejects the idea of an autonomous subject as the source of the discourse, a subject „behind the text“, so to say. It points out how the subject is in fact woven into the textual practices, and not so much a self-conscious judge of truths (cf. Foucault 1994). Such a perspective deflates hopes, that the subject can serve as a remedy to technological problems, that it will ‘set everything right’.

In this sense our common discourse on AI Chatbots and their shortcomings would need – to echo Niklas Luhmann’s concept of a sociological enlightenment – some discourse analytical enlightenment. Not so much as to establish an autonomous subject, but to keep us aware of the fact, that the idealized separations of AI legwork and the responsible subject wont resolve the practical problems that AI Chatbots bring with them for our knowledge – not because of their deficiencies, but because of their genuine mode or operation.

LITERATUR:

Angermuller, Johannes 2014: Poststructuralist Discourse Analysis, Basingstoke/New York, Palgrave Macmillan.

Barthes, Roland 1968: La mort de l’auteur, in: Le bruissement de la langue, Paris: Seuil, pp. 61–67.

Bayard, Pierre 2007: Comment parler des livres que l’on n’a pas lus? Paris: Les Éditions de Minuit.

Bühler, Karl 1999: Sprachtheorie. Die Darstellungsfunktion der Sprache, Stuttgart: Lucius & Lucius.

Bureau, Scott, 2026: Research reveals which popular generative AI chatbots lie, Rochester Institute of Technology, https://www.rit.edu/news/research-reveals-which-popular-generative-ai-chatbots-lie (20.5.2026).

DFG 2026a: Artificial Intelligence in the Review Process. Deutsche Forschungsgemeinschaft (DFG). https://doi.org/10.5281/zenodo.18886256 (18.5.2026).

DFG 2026b: KI in der Antragsstellung, https://www.dfg.de/de/grundlagen-themen/digitale-themen/ki/antragstellung (18.5.2026).

ERC 2026: The use of AI in grant proposal evaluation. Guidelines for ERC panel members and remote reviewers, https://erc.europa.eu/system/files/2026-03/Use-AI-grant-proposal-evaluation.pdf (18.5.2026).

European Comission 2026: Living guidelines on the responsible use of generative AI in research. ERA Forum Stakeholders’ document, 3. version, Brussels, https://research-and-innovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en?filename=ec_rtd_ai-guidelines.pdf (18.5.2026).

Foucault, Michel 1994: Qu’est-ce qu’un auteur? in: Dits et écrits, Paris: Gallimard, pp. 789–821.

Harris, Zelig 1969: Analyse du discourse, Languages, 13, 8–45.

Lopez, Paola 2025: Über die Wahrheitseigenschaft, Merkur, 79, 903, pp. 49–56.

Keller, Reiner 2013: Doing Discourse Research. An Introduction for Social Scientists, London et al.: Sage.

Quine, Willard V. 2004: Fünf Marksteine des Empirismus, in: Stekeler-Weithofer, Pirmin (ed.): Geschichte der Philosophie in Text und Darstellung. Gegenwart, Stuttgart: Reclam, pp. 181–190.

Scholz, Ronny 2019: Lexicometry: A Quantifying Heuristic for Social Scientists in Dsicourse Studies, in. Scholz, Ronny (ed.): Quantifying Approaches to Discourse for Social Scientists, Cham: Palgrave Macmillan, pp. 123–153.

Schutz, Alfred 1944: The Stranger: An Essay in Social Psychology, American Journal of Sociology, 4, 6, pp. 499–507.