STYLOMETRIC CLASSIFICATION OF AI-GENERATED TEXTS: COMPARATIVE EVALUATION OF MACHINE LEARNING MODELS

Abstract

The rapid proliferation of large language models (LLMs), such as ChatGPT and Deepseek, has made it increasingly difficult to distinguish between AI-generated and human-written text.This study evaluates the effectiveness of stylometric analysis as a transparent and interpretable method for detecting synthetic content. A balanced dataset of 30,000 short-form responses (10,000 per class: Human, ChatGPT, Deepseek) was constructed. While the Human and ChatGPT responses were sourced from an existing dataset, the Deepseek responses were generated using standardized prompts to ensure consistency. Each response was transformed into a vector of 12 anually engineered features capturing lexical richness, syntactic structure, and readability.The study involved five classifiers: Logistic Regression, Support Vector Machine, Random Forest, Gradient Boosting, and Decision Tree. Each was trained and evaluated on multiclass and binary classification tasks. Randomized hyperparameter tuning was applied to enhance performance.The tuned Random Forest achieved the highest results, with macro-averaged F1-scores of 0.84 (multiclass) and 0.86 (binary), and accuracy over 87 %. Gradient Boosting and SVM showed comparably strong performance, confirming the robustness of ensemble and margin- based methods in this context. Key features such as Simpson’s Index, type-token ratio, and sentence length proved most informative. The results confirm that stylometric features, despite their simplicity, can reliably distinguish between human and AI-generated text. The results indicate that this approach demonstrates clear potential and, when used in combination with other methods, can contribute effectively to the identification of AI-generated content. Additionally, generating datasets using open-source models with the Ollama framework enables affordable and scalable experimentation without relying on commercial APIs. This is particularly beneficial for early stage research and academic environments with limited resources.