FLGQM: Robust Federated Learning Based on Geometric and Qualitative Metrics
Abstract
:1. Introduction
2. Materials and Methods
2.1. Related Work
2.1.1. Federated Learning
2.1.2. Poisoning Attacks in Federated Learning
2.1.3. Existing Robust Federated Learning
2.2. FLGQM
2.2.1. Overview
2.2.2. Aggregation Rule
Algorithm 1 FLGQM. |
|
3. Results and Discussion
3.1. Experimental Setup
3.1.1. Datasets
- MINST [35]: A widely used dataset in machine learning, it serves as the standard benchmark for digit recognition tasks. It comprises 60,000 training images and 10,000 test images, with each grayscale image representing a handwritten digit (ranging from 0 to 9) in a 28 × 28 pixel format.
- CIFAR-10 [36]: Curated by the Canadian Institute for Advanced Research (CIFAR), it is a widely used dataset in machine learning. It consists of 60,000 32 × 32 pixel color images categorized into 10 distinct classes. These classes represent objects such as airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.
3.1.2. Poisoning Attack
- Label flip** attack [37]: For each sample of each malicious client, we cyclically shift the corresponding label l to , where Y represents the total number of labels (e.g., in the MNIST dataset) .
- LIE attack [9]: The Little Is Enough (LIE) attack adds a little noise on the model that is the average of all uploaded local models in the non-adversarial setting. The perturbation used in the attack is carefully chosen to be large enough to significantly impact the global model but small enough to avoid detection by the Byzantine Robust Aggregation algorithm.
- Fang attack [8]: The malicious client crafts poisoned local model updates that are intentionally opposite to the benign updates, allowing them to bypass the defense mechanism of Krum.
- AGRT attack [13]: The goal of the aggregation algorithm tailored (AGRT) attack is to maximize the effect on the global model in an optimization problem during each round of FL, i.e., to find the best malicious update that is not easily detected by the defense mechanism, while also achieving the maximum attack effect. Formally, the attacker builds a malicious update using the following optimization problem.
3.1.3. Baseline Aggregation Rules
- FedAvg [1]: The new global model is obtained by calculating the average of all local models.
- Median [18]: The median directly takes the median of the coordinates of each dimension of all local models to obtain the new global model.
- Krum [15]: In Krum, the Euclidean distance is employed to score the local models, and subsequently, the highest-rated model is chosen as the global model. For the ith client, assuming the existence of f malicious clients, the local model’s score can be determined as follows.
- Trimmed Mean [18]: The trimmed mean is an aggregation rule that takes into account each model parameter and operates based on coordinates. For every model parameter, the server gathers all values from local models and arranges them in ascending order. Subsequently, it excludes the largest and smallest values, computes their average, and utilizes this average as the corresponding parameter value in the global model.
3.1.4. Performance Metrics and Federated Learning System Setup
3.1.5. Global Model
3.1.6. Federated Learning Method Parameter Setting
3.2. Experimental Results
3.2.1. FLGQM Can Achieve Two Defensive Goals
- Fidelity. As evident from Table 3, FLGQM achieves fidelity as the accuracy in the non-adversarial setting is comparable to the baseline (FedAvg) on both datasets.
- Robustness. As can be seen from Table 3, FLGQM achieves robustness as its accuracy under all attacks has very little change compared to the baseline (FedAvg) on both datasets.
3.2.2. The Variant of FLGQM
3.2.3. Impact of Malicious Clients’ Number
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1273–1282. [Google Scholar]
- Biggio, B.; Nelson, B.; Laskov, P. Poisoning attacks against support vector machines. ar**. ar**&author=Cao,+X.&author=Fang,+M.&author=Liu,+J.&author=Gong,+N.Z.&publication_year=2020&journal=ar**v" class='google-scholar' target='_blank' rel='noopener noreferrer'>Google Scholar]
- **e, C.; Koyejo, S.; Gupta, I. Zeno: Distributed stochastic gradient descent with suspicion-based fault-tolerance. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 10–15 June 2019; pp. 6893–6901. [Google Scholar]
- Zhao, B.; Sun, P.; Wang, T.; Jiang, K. Fedinv: Byzantine-robust federated learning by inversing local model updates. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 20–27 February 2022; pp. 9171–9179. [Google Scholar]
- Mahloujifar, S.; Mahmoody, M.; Mohammed, A. Universal multi-party poisoning attacks. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 10–15 June 2019; pp. 4274–4283. [Google Scholar]
- **a, Q.; Tao, Z.; Hao, Z.; Li, Q. FABA: An algorithm for fast aggregation against byzantine attacks in distributed neural networks. In Proceedings of the International Joint Conference on Artificial Intelligence, Macao, China, 10–16 August 2019; pp. 4824–4830. [Google Scholar]
- Fung, C.; Yoon, C.J.; Beschastnikh, I. Mitigating sybils in federated learning poisoning. ar**v 2018, ar**v:1808.04866. [Google Scholar]
- Wan, W.; Hu, S.; Lu, J.; Zhang, L.Y.; **, H.; He, Y. Shielding Federated Learning: Robust Aggregation with Adaptive Client Selection. ar**v 2022, ar**v:2204.13256. [Google Scholar]
- Pillutla, K.; Kakade, S.M.; Harchaoui, Z. Robust Aggregation for Federated Learning. IEEE Trans. Signal Process. 2022, 70, 1142–1154. [Google Scholar] [CrossRef]
- Muoz-Gonz¢lez, L.K.; Co, T.; Lupu, E.C. Byzantine-robust federated machine learning through adaptive model averaging. ar**v 2019, ar**v:1909.05125. [Google Scholar]
- Xu, J.; Huang, S.L.; Song, L.; Lan, T. Byzantine-robust federated learning through collaborative malicious gradient filtering. In Proceedings of the 42nd International Conference on Distributed Computing Systems (ICDCS 2022), Bologna, Italy, 10–13 July 2022; pp. 1223–1235. [Google Scholar]
- Geng, G.; Cai, T.; Yang, Z. Better safe than sorry: Constructing byzantine-robust federated learning with synthesized trust. Electronics 2023, 12, 2926. [Google Scholar] [CrossRef]
- Wang, T.; Zhao, B.; Fang, L. FLForest: Byzantine-robust Federated Learning through Isolated Forest. In Proceedings of the 28th International Conference on Parallel and Distributed Systems (ICPADS), Nan**g, China, 10–12 January 2023; pp. 296–303. [Google Scholar]
- Cao, X.; Lai, L. Distributed gradient descent algorithm robust to an arbitrary number of byzantine attackers. IEEE Trans. Signal Process. 2019, 67, 5850–5864. [Google Scholar] [CrossRef]
- Rodríguez-Barroso, N.; Martínez-Cámara, E.; Luzón, M.V.; Herrera, F. Dynamic defense against byzantine poisoning attacks in federated learning. Future Gener. Comput. Syst. 2022, 133, 1–9. [Google Scholar] [CrossRef]
- Cao, X.; Zhang, Z.; Jia, J.; Gong, N.Z. Flcert: Provably secure federated learning against poisoning attacks. IEEE Trans. Inf. Forensics Secur. 2022, 17, 3691–3705. [Google Scholar] [CrossRef]
- Kang, J.; **ong, Z.; Niyato, D.; **e, S.; Zhang, J. Incentive Mechanism for Reliable Federated Learning: A Joint Optimization Approach to Combining Reputation and Contract Theory. IEEE Internet Things J. 2019, 6, 10700–10714. [Google Scholar] [CrossRef]
- Deng, L. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Process. Mag. 2012, 29, 141–142. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Hinton, G.; Chen, C.F.R.; Fan, Q.; Panda, R. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 357–366. [Google Scholar]
- Tolpegin, V.; Truex Gursoy, S.M.E.; Liu, L. Data poisoning attacks against federated learning systems. In Proceedings of the Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, Guildford, UK, 14–18 September 2020; pp. 480–501. [Google Scholar]
Defense | Technique | Geometric | Qualitative | Central Dataset | Non-IID Data |
---|---|---|---|---|---|
Krum/Multi-Krum | Euclidean distance | ✓ | ✗ | Not needed | ✗ |
Bulyan | Krum + trimmed median | ✓ | ✗ | Not needed | ✗ |
RFA | Geometric median | ✓ | ✗ | Not needed | ✗ |
FoolsGold | Contribution similarity | ✓ | ✗ | Not needed | ✓ |
Zeno | Calculates score by clean small dataset | ✗ | ✓ | Needed | ✓ |
Fltrust | Cosine similarity + Clean small dataset | ✓ | ✓ | Needed | ✓ |
MAB-RFL | Similarity + Graph theory | ✓ | ✗ | Not needed | ✓ |
FLGQM (ours) | Cosine similarity + Euclidean distance (+ Distributed score calculation) | ✓ | ✓ | Not needed | ✓ |
Explanation | MNIST | CIFAR-10 | |
---|---|---|---|
K | Number of clients | 100 | 40 |
Number of clients selected per global round | K | ||
Proportion of clients selected as training clients each global round | |||
Proportion of clients selected as union clients each global round | |||
Local epoch | 5 | ||
Global epoch | 300 | ||
b | Batch size | 16 | 64 |
Learning rate | 0.01 | 0.001 | |
Number of malicious clients | 20 | 8 | |
f | Parameter of Krum [11] | ||
Parameter of trimmed mean [15] |
Dataset | Attacks | FedAvg | Median | Trim_mean | Krum | FLTrust | MAB-RFL | FLGQM |
---|---|---|---|---|---|---|---|---|
MNIST | No attack | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 98.39 |
LF attack | - | 97.79 | 97.41 | 93.51 | 97.40 | 96.60 | 98.03 | |
LIE attack | - | 96.25 | 88.62 | 94.49 | 97.14 | 97.44 | 98.05 | |
Fang attack | - | 96.17 | 95.64 | 94.75 | 97.10 | 97.45 | 97.97 | |
AGRT attack | - | 95.18 | 95.84 | 74.19 | 97.03 | 97.14 | 97.63 | |
CIFAR-10 | No attack | 73.68 | 72.83 | 72.58 | 66.36 | 73.29 | 73.70 | 73.55 |
LF attack | - | 69.59 | 69.35 | 57.63 | 73.21 | 73.66 | 73.53 | |
LIE attack | - | 48.38 | 35.77 | 41.68 | 72.95 | 72.62 | 73.30 | |
Fang attack | - | 58.73 | 66.57 | 42.43 | 73.19 | 72.18 | 73.39 | |
AGRT attack | - | 23.06 | 53.77 | 10.00 | 72.52 | 70.55 | 72.63 |
Dataset | Attacks | FedAvg | Median | Trim_mean | Krum | FLTrust | MAB-RFL | FLGQM-NoQMS | FLGQM |
---|---|---|---|---|---|---|---|---|---|
MNIST | No attack | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 97.98 | 98.39 |
LF attack | - | 97.79 | 97.41 | 93.51 | 97.40 | 96.60 | 95.94 | 98.03 | |
LIE attack | - | 96.25 | 88.62 | 94.49 | 97.14 | 97.44 | 96.83 | 98.05 | |
Fang attack | - | 96.17 | 95.64 | 94.75 | 97.10 | 97.45 | 95.53 | 97.97 | |
AGRT attack | - | 95.18 | 95.84 | 74.19 | 97.03 | 97.14 | 87.64 | 97.63 | |
CIFAR-10 | No attack | 73.68 | 72.83 | 72.58 | 66.36 | 73.29 | 73.70 | 73.15 | 73.55 |
LF attack | - | 69.59 | 69.35 | 57.63 | 73.21 | 73.66 | 65.59 | 73.53 | |
LIE attack | - | 48.38 | 35.77 | 41.68 | 72.95 | 72.62 | 69.28 | 73.30 | |
Fang attack | - | 58.73 | 66.57 | 42.43 | 73.19 | 72.18 | 67.30 | 73.39 | |
AGRT attack | - | 23.06 | 53.77 | 10.00 | 72.52 | 70.55 | 56.34 | 72.63 |
Attacks | Percentage of Malicious Clients | FedAvg | Median | Trim_mean | Krum | FLTrust | MAB-RFL | FLGQM |
---|---|---|---|---|---|---|---|---|
LF attack | 0% | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 98.39 |
10% | - | 98.04 | 98.01 | 94.81 | 97.69 | 97.42 | 98.22 | |
20% | - | 97.79 | 97.41 | 93.51 | 97.40 | 96.60 | 98.03 | |
30% | - | 96.96 | 96.54 | 93.27 | 97.42 | 97.28 | 98.01 | |
40% | - | 94.14 | 93.71 | 93.14 | 95.78 | 97.12 | 97.87 | |
50% | - | 17.44 | - | 23.55 | 95.23 | 95.32 | 97.86 | |
LIE attack | 0% | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 98.39 |
10% | - | 98.01 | 97.95 | 94.63 | 97.57 | 97.42 | 98.26 | |
20% | - | 96.25 | 88.62 | 94.49 | 97.14 | 97.44 | 98.05 | |
30% | - | 96.06 | 84.77 | 94.46 | 97.25 | 97.22 | 98.02 | |
40% | - | 95.96 | 79.67 | 30.43 | 97.04 | 96.94 | 97.89 | |
50% | - | 16.80 | - | 11.22 | 97.10 | 96.86 | 97.64 | |
Fang attack | 0% | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 98.39 |
10% | - | 96.35 | 96.69 | 94.79 | 97.38 | 97.86 | 98.33 | |
20% | - | 96.25 | 96.54 | 94.75 | 97.10 | 97.45 | 97.97 | |
30% | - | 96.17 | 95.36 | 94.19 | 96.27 | 97.57 | 97.77 | |
40% | - | 95.31 | 9.80 | 10.00 | 94.58 | 97.42 | 97.68 | |
50% | - | 10.30 | - | 9.90 | 94.20 | 97.37 | 97.59 | |
AGRT attack | 0% | 98.51 | 98.25 | 98.22 | 95.01 | 97.84 | 97.95 | 98.39 |
10% | - | 96.98 | 96.77 | 94.88 | 97.21 | 97.23 | 98.09 | |
20% | - | 95.18 | 95.84 | 74.19 | 97.03 | 97.14 | 97.63 | |
30% | - | 95.09 | 90.03 | 66.84 | 96.99 | 96.12 | 97.45 | |
40% | - | 94.98 | 9.80 | 10.00 | 96.37 | 96.10 | 97.41 | |
50% | - | 9.80 | - | 9.80 | 95.49 | 96.00 | 97.20 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, S.; Xu, X.; Wang, M.; Wu, F.; Ji, Y.; Zhu, C.; Zhang, Q. FLGQM: Robust Federated Learning Based on Geometric and Qualitative Metrics. Appl. Sci. 2024, 14, 351. https://doi.org/10.3390/app14010351
Liu S, Xu X, Wang M, Wu F, Ji Y, Zhu C, Zhang Q. FLGQM: Robust Federated Learning Based on Geometric and Qualitative Metrics. Applied Sciences. 2024; 14(1):351. https://doi.org/10.3390/app14010351
Chicago/Turabian StyleLiu, Shangdong, ** Xu, Musen Wang, Fei Wu, Yimu Ji, Chenxi Zhu, and Qurui Zhang. 2024. "FLGQM: Robust Federated Learning Based on Geometric and Qualitative Metrics" Applied Sciences 14, no. 1: 351. https://doi.org/10.3390/app14010351