MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning
- PDF / 1,240,640 Bytes
- 13 Pages / 595.32 x 841.92 pts (A4) Page_size
- 66 Downloads / 175 Views
Wu et al. / Front Inform Technol Electron Eng 2020 21(7):1034-1046
Frontiers of Information Technology & Electronic Engineering www.jzus.zju.edu.cn; engineering.cae.cn; www.springerlink.com ISSN 2095-9184 (print); ISSN 2095-9230 (online) E-mail: [email protected]
MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning* Zhao-qi WU1, Jin WEI2,3, Fan ZHANG†‡1, Wei GUO1, Guang-wei XIE2,3 1
National Digital Switching System Engineering & Technological R&D Center, Zhengzhou 450002, China 2
School of Computer Science, Fudan University, Shanghai 200433, China 3
Data Arena Institute, Fudan University, Shanghai 200433, China †
E-mail: [email protected]
Received Mar. 1, 2019; Revision accepted Nov. 14, 2019; Crosschecked June 2, 2020
Abstract: With the growing amount of information and data, object-oriented storage systems have been widely used in many applications, including the Google File System, Amazon S3, Hadoop Distributed File System, and Ceph, in which load balancing of metadata plays an important role in improving the input/output performance of the entire system. Unbalanced load on the metadata server leads to a serious bottleneck problem for system performance. However, most existing metadata load balancing strategies, which are based on subtree segmentation or hashing, lack good dynamics and adaptability. In this study, we propose a metadata dynamic load balancing (MDLB) mechanism based on reinforcement learning (RL). We learn that the Q_learning algorithm and our RL-based strategy consist of three modules, i.e., the policy selection network, load balancing network, and parameter update network. Experimental results show that the proposed MDLB algorithm can adjust the load dynamically according to the performance of the metadata servers, and that it has good adaptability in the case of sudden change of data volume. Key words: Object-oriented storage system; Metadata; Dynamic load balancing; Reinforcement learning; Q_learning https://doi.org/10.1631/FITEE.1900121 CLC number: TP338.8
1 Introduction The increasing amount of data provides huge challenge to data storage systems, especially with the advent of big data era in the fields of engineering and information service. With the introduction of a series of storage systems such as the Google File System (GFS) (Ghemawat et al., 2003), Amazon S3 (Palankar et al., 2008), Hadoop Distributed File System (HDFS) (Shvachko et al., 2010), and Ceph (Wang FY et al., 2013), distributed storage technology has developed rapidly and become a supporting technology of stor‡
Corresponding author Project supported by the National Natural Science Foundation of China (Nos. 61572520 and 61521003) ORCID: Zhao-qi WU, https://orcid.org/0000-0001-7857-2875; Fan ZHANG, https://orcid.org/0000-0001-7456-8377 © Zhejiang University and Springer-Verlag GmbH Germany, part of Springer Nature 2020 *
age at the stage of cloud computing and big data. In the current distributed file system, metadata of files is separated from data access. Metadata is stored on the metada
Data Loading...
