The Problem of Artificial Intelligence Participation in Scientific Research Process: The Case of ChatGPT-4o and the Khabar and Insh? Distinction in Arabic Rhetoric

Abstract

Artificial intelligence (AI) tools, characterized by capabilities such as natural language processing, memory retention, intelligent response generation, and reasoning, have increasingly been used in social science research. This raises concerns about the accuracy and reliability of the data produced by these tools, and as well as their potential to cause ethical violations in academic research. Moreover, when the sources cited do not align with the information provided, the validity of resulting interpretations becomes questionable. For these reasons, while the ethical implications of incorporating AI tools into research in the language, social and human sciences are being debated, empirical studies are also being conducted to assess how AI manages this process. To our knowledge, this study is the first in the field to examine the use of an AI tool as a source in Arabic language studies and to identify resulting issues related to informational accuracy and reliability. In addition, since the topic presented to ChatGPT appears not only in Arabic rhetorical sources but also in disciplines such as theology, philosophy, and logic, it allows the AI tool to perform broader information retrieval. Moreover, the controversial and interpretative nature of the topic is crucial for assessing the AI tool's capacity to process, analyze, and interpret the subject from multiple perspectives. This is particularly important because, in similar studies, controversial issues are often avoided, and the focus remains solely on the factual accuracy AI-generated content. ChatGPT is available in both free and paid versions, with the latest model, ChatGPT-4o, currently accessible to users. In this study, ChatGPT-4o was evaluated using a qualitative data analysis approach. The assessment focused on several criteria; the accuracy of the cited sources with respect to the referenced work and authors; the diversity of disciplinary fields reflected in the literature review; the tool's ability to identify and cite primary and secondary sources. Additionally, the accuracy of the factual information provided and the consistency of responses to identical questions posed at different times were examined. In order to test the use of the AI tool in the scientific research process, the khabar-insh & amacr; distinction, an issue widely debated in Arabic rhetoric, was chosen. Rhetoricians, who classify all sentences in the language as either khabar or insh & amacr; within a holistic framework, base this distinction on whether a sentence can be judged as true or false. Accordingly, sentences that are capable of being either true or false are classified as khabar, while those that are not considered as insh & amacr;. Although few in number, various objections to this classification have been raised, some from the classical tradition, but mostly from the modern scholarship. Our study is structured around several key questions: What the khabar-insh & amacr; distinction? How did it emerge? What are the major theories or perspectives on this distinction? What are the sources of these perspectives? And how does the AI tool respond when required to choose between them? The study begins by briefly introducing AI, followed by an overview of the khabar-insh & amacr; distinction is generally defined in rhetoric and other Islamic sciences. It then summarizes the main theories and debates surrounding the khabar-insh & amacr; distinction, before presenting the study's methodology, limitations, data analysis, and findings. Finally, the responses provided by ChatGPT-4o were categorized under thematic headings, analyzed and paraphrased for evaluation. The study found that some of the information quoted by ChatGPT-4o could not be verified in the cited sources. In addition, it was revealed that some of the cited sources did not exist, and others were misattributed to incorrect authors. In terms of the subject familiarity, ChatGPT-4o failed to mention ongoing debates and provided only general information. Even when specific theories were introduced and references to scholarly disagreement were made, its responses were limited to literal definitions. The definitions provided did not reflect the content of sources that contain detailed discussions. For these reasons, using ChatGPT as a source in scientific, at its current stage of development, is not advisable for obtaining reliable and accurate information.