Communication-efficient distributed multi-task learning with matrix sparsity regularization
- PDF / 1,483,609 Bytes
- 33 Pages / 439.37 x 666.142 pts Page_size
- 11 Downloads / 234 Views
Communication-efficient distributed multi-task learning with matrix sparsity regularization Qiang Zhou1 · Yu Chen1 · Sinno Jialin Pan1 Received: 3 May 2019 / Revised: 20 July 2019 / Accepted: 16 September 2019 © The Author(s), under exclusive licence to Springer Science+Business Media LLC, part of Springer Nature 2019
Abstract This work focuses on distributed optimization for multi-task learning with matrix sparsity regularization. We propose a fast communication-efficient distributed optimization method for solving the problem. With the proposed method, training data of different tasks can be geo-distributed over different local machines, and the tasks can be learned jointly through the matrix sparsity regularization without a need to centralize the data. We theoretically prove that our proposed method enjoys a fast convergence rate for different types of loss functions in the distributed environment. To further reduce the communication cost during the distributed optimization procedure, we propose a data screening approach to safely filter inactive features or variables. Finally, we conduct extensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our proposed method. Keywords Distributed learning · Multi-task learning · Acceleration
1 Introduction Multi-task learning (MTL) (Caruana 1997) aims to jointly learn multiple machine learning tasks by exploiting their commonality to boost the generalization performance of each task. Similar to many standard machine learning techniques, in MTL, a single machine is assumed to be able to access all training data over different tasks. However, in practice, especially in the context of smart city, training data for different tasks is owned by different organizations and geo-distributed over different local machines, and centralizing the data may result in expensive cost of data transmission and cause privacy and security issues. Take personal-
Editors: Kee-Eung Kim and Jun Zhu.
B
Sinno Jialin Pan [email protected] Qiang Zhou [email protected] Yu Chen [email protected]
1
Nanyang Technological University, Singapore, Singapore
123
Machine Learning
ized healthcare as a motivating example. In this context, learning a personalized healthcare prediction model from each user’s personal data including his/her profile and various sensor readings from his/her mobile device is considered as a different task. On one hand, the personal data may be too sparse to learn a precise prediction model for each task, and thus MTL is desired. On the other hand, some of the users may not be willing to share their personal data, which results in a failure of applying standard MTL methods. Thus, a distributed MTL algorithm is more preferred. However, if frequent communication is required for the distributed MTL algorithm to obtain an optimal prediction model for each task, users have to pay for expensive cost on data transmission, which is not practical. Therefore, designing a communication-efficient MTL algorithm in the distributed computing en
Data Loading...
