TY - GEN
T1 - GradMFL
T2 - 21st International Conference on Algorithms and Architectures for Parallel Processing, ICA3PP 2021
AU - Tong, Guanghui
AU - Li, Gaolei
AU - Wu, Jun
AU - Li, Jianhua
N1 - Publisher Copyright:
© 2022, Springer Nature Switzerland AG.
PY - 2022
Y1 - 2022
N2 - The massive datasets are often collected under non-IID distribution scenarios, which enforces existing federated learning (FL) frameworks to be still struggling on the model accuracy and convergence. To achieve heterogeneity-aware collaborative training, the FL server aggregates gradients from different clients to ingest and transfer common knowledge behind non-IID data, while leading to information loss and bias due to statistical weighting. To address the above issues, we propose a Gradient Memory-based Federated Learning (GradMFL) framework, which enables Hierarchical Knowledge Transferring over Non-IID Data. In GradMFL, a data clustering method is proposed to categorize Non-IID data to IID data according to the similarity. And then, in order to enable beneficial knowledge transferring between hierarchical clusters, we also present a multi-stage model training mechanism using gradient memory, constraining the updating directions. Experiments on solving a set of classification tasks based on benchmark datasets have shown the strong performance of good accuracy and high efficiency.
AB - The massive datasets are often collected under non-IID distribution scenarios, which enforces existing federated learning (FL) frameworks to be still struggling on the model accuracy and convergence. To achieve heterogeneity-aware collaborative training, the FL server aggregates gradients from different clients to ingest and transfer common knowledge behind non-IID data, while leading to information loss and bias due to statistical weighting. To address the above issues, we propose a Gradient Memory-based Federated Learning (GradMFL) framework, which enables Hierarchical Knowledge Transferring over Non-IID Data. In GradMFL, a data clustering method is proposed to categorize Non-IID data to IID data according to the similarity. And then, in order to enable beneficial knowledge transferring between hierarchical clusters, we also present a multi-stage model training mechanism using gradient memory, constraining the updating directions. Experiments on solving a set of classification tasks based on benchmark datasets have shown the strong performance of good accuracy and high efficiency.
KW - Federated learning
KW - Gradient memory
KW - Hierarchical clustering
KW - Knowledge transferring
KW - Non-IID data
UR - http://www.scopus.com/inward/record.url?scp=85126218806&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85126218806&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-95384-3_38
DO - 10.1007/978-3-030-95384-3_38
M3 - Conference contribution
AN - SCOPUS:85126218806
SN - 9783030953836
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 612
EP - 626
BT - Algorithms and Architectures for Parallel Processing - 21st International Conference, ICA3PP 2021, Proceedings
A2 - Lai, Yongxuan
A2 - Wang, Tian
A2 - Jiang, Min
A2 - Xu, Guangquan
A2 - Liang, Wei
A2 - Castiglione, Aniello
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 3 December 2021 through 5 December 2021
ER -