DATA PROCESSING AND ANALYSIS ON THE EXAMPLE OF THE SPAMBASE DATASET USING MACHINE LEARNING LIBRARIES

Abstract

The article analyzes the Spambase dataset with data on e-mails classified as spam and nonspam. A detailed analysis of this data frame is provided with information about the data in the columns (properties) and records. The dataset was uploaded to the Google CoLab software development environment for programming and further analysis. NumPy, Pandas, Matplotlib, Sklearn, and Imblearn libraries are used for scientific calculations and data analysis in Python. Their combinations allow developers and researchers to effectively work with structured data, perform various operations, visualize results and solve complex data analysis and processing tasks. To better understand the material, reviewed basic theoretical information about data forecasting. Definitions of machine learning, artificial intelligence, and data science are provided. Machine learning categories such as supervised, unsupervised, and reinforcement learning are also described. The main types of features used in machine learning models are considered: qualitative, ordinal, and quantitative. The Heart Disease dataset was also presented, describing and labelling important definitions such as features matrix X, feature vector, properties, targets vector Y. The need to break up the whole dataset into training, validation and testing datasets for correct evaluation and model verification is described. The use of L1 and L2 loss functions to evaluate model performance is explained, and the advantages and disadvantages of each approach are indicated. The analysis of the Spambase dataset in the Google Colab environment continued. Histograms were constructed to represent the distribution of data by different properties for two classes: spam and non-spam. Analyzed histograms for properties word_freq_credit, char_freq_! and capital_run_length_total. The split() function from the NumPy library splits the data into training, validation, and testing sets. For the training dataset, classes were rebalanced using the random oversampling method (RandomOverSampler). As a result, new instances were created for the less-represented class of e-mails containing spam.