RESEARCH OF METHODS FOR TEXT VECTORIZATION IN THE TASKS OF VALIDATION THE ANSWERS PRESENTED IN NATURAL LANGUAGE

Abstract

Intellectualization of the process of processing natural language texts in the tasks of automated testing determines the relevance of the research. Since open-type answers in testing systems are natural language texts, the problem of their processing refers to the applied problem of word processing. All applied problems of word processing, the solution of which takes place with the use of machine learning, neural networks, require vectorization – the conversion of text into digital values. The aim of the article is to research the models, methods of vectorization of texts in the problems of processing answers given in natural language. At the first stage, the basic applied problems of word processing are investigated, as a result of which their classification is given. The assignment of the problem of checking natural language answers within the framework of this research to the problem of text classification and semantic analysis is substantiated. In the second stage, the basic models of text representation in digital form are analyzed: bagof- words and distributive semantics. The application of the bag-of-words model for the problem of processing open-ended answers is substantiated, as the vocabulary used to encode the collection of correct answers and the frequency of words with which they are used in the answers of “training” and “test” sets are enough to determine the answer class. It is noted that the vector of features in this problem is the frequency of tokens (symbolic or verbal uni-, bi-, n-grams) of the dictionary, formed by the training sample, in the answers of the “training” and “test” data sets. In the third stage, the approaches of calculating the vector of characteristics are investigated: absolute frequency (TF), relative frequency (TF-IDF), compatible information (PWI), the advantages and disadvantages of each of them are determined. At the last stage for vectorization of texts in problems of processing of the answers given in natural language, the following combinations of sets of signs are offered: model bag-ofwords and TF; bag-of-words and TF-IDF model; verbal n-grams and TF-IDF; symbol n-grams and TF-IDF; model bag-of-words and TF-PWI. The proposed sets of features and their combinations are a means of improving the machine learning model for the task of checking the answers given in natural language. Further research will be aimed at developing a model of machine learning of this problem and its experimental testing with the proposed sets of features in order to obtain an effective mathematical model.