A METHODOLOGY FOR OPTIMIZING DATA PROCESSING AND TRANSFER IN INFORMATION SYSTEMS
Keywords:
data warehouse, ETL process, analytical system, data transfer efficiency, data processingAbstract
Given the rapid development of digital technologies, companies increasingly face the need to process large volumes of data, which complicates the efficient execution of business analytics based on operational databases. Performing analytical operations directly on databases intended for daily business activities is not always an optimal solution. A normalized relational database used in such cases may consist of dozens or even thousands of tables, each with its own unique names and levels of detail. As a result, the process of locating and retrieving the necessary data for specific analyses can be complex and time-consuming.Moreover, modern companies often use multiple business applications that interact with different databases. The integration of such architecturally incompatible systems for analytical purposes can lead to a decline in data quality. Therefore, this study proposes a methodology for optimizing data processing and transfer on a previously modeled data warehouse.The main focus of the research was on designing a relational database and a data warehouse using a hybrid approach, configuring the ETL process between the relational database and the data warehouse using SSIS packages, and comparing the efficiency of data transfer across tables of varying sizes. To assess the effectiveness of the proposed methodology, the following criteria were considered: the speed of transferring a specified volume of data from a simulated relational system to the analytical environment, as well as the minimization of the use of additional tools and human involvement during the process.The study included the development of a relational data model, its population with suboptimal data, the creation of an analytical system, and the configuration of data transfer and processing processes using various tools and parameters. The results demonstrated the effectiveness of using the SSIS Fast Load type for transferring and processing data of different volumes, showing an 11 % performance improvement over the use of native T-SQL tools and a 98 % improvement compared to the SSIS Plain Load type.
References
Saraswat P., Raj S. Educational data mining and data warehouse design using business intelligence. International Journal of Innovative Research in Computer Science & Technology. 2022. P. 97–101. URL: https://doi.org/10.55524/ijircst.2022.10.1.17
Data Optimization for Industrial IoT-Based Recommendation Systems / M. Beshley et al. Electronics. 2022. Vol. 12, no. 1. P. 33. URL: https://doi.org/10.3390/electronics12010033
Novoselova O., Ruzheynikov A., Gavrilov A. Analytical Processing of Applied Tasks Conceptual Models at Design of Information-Active Systems. EPJ Web of Conferences. 2019. Vol. 224. P. 06008. URL: https://doi.org/10.1051/epjconf/201922406008
Jaroli P., Masson P. Data Warehousing and OLAP Technology (Data warehousing). International Journal of Engineering Trends and Technology. 2017. Vol. 51, no. 1. P. 45–50. URL: https://doi.org/10.14445/22315381/ijett-v51p208
A Data Warehouse Approach for Business Intelligence / G. Garani et al. 2019 IEEE 28th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), Napoli, Italy, 12–14 June 2019. 2019. URL: https://doi.org/10.1109/wetice.2019.00022
Rachapudi S. Holistic Data Management: Integrating OLTP and OLAP in Financial Systems. International Journal of Science and Research (IJSR). 2024. Vol. 13, no. 10. P. 1115–1118. URL: https://doi.org/10.21275/sr241015110206
