Decision Tree-Based Federated Learning: A Survey
Abstract
:1. Introduction
- This survey categorizes and summarizes federated decision tree models, explains the innovative points of each scheme, and compares the differences and connections between different schemes in multiple aspects.
- We elaborate on the main issues currently facing privacy in federated learning and summarize the performance differences brought about by using different cryptographic techniques and privacy protection schemes in training federated decision tree models.
- We consider using decision trees as the underlying model for federated learning, which involves a large amount of computation and communication between multiple parties during the training process. Therefore, we discuss iterative and aggregation strategies in federated learning to improve the convergence and communication efficiency of the model.
- Finally, we provide prospects for future research directions in this field.
2. Federated Decision Tree
3. Security Scheme
- Model Inference Attack: Attackers can infer sensitive information about training data by observing the output of the federated decision tree model [70,71,72]. Attackers can establish training data or infer specific eigenvalues by utilizing the probability distribution or decision path of the output, thereby infringing on the privacy of participants.
- Model/Data Poisoning Attack: Attackers may attempt to manipulate model updates or gradients of participants during federated learning, or use malicious data for model training, causing malicious interference to the aggregated decision tree model. This may lead to a decrease in the performance of the final model or produce misleading results during the inference stage [73,74,75].
- Aggregation Information Leakage Attack: Attackers can infer the data information of participants by observing the aggregation process of the model parameters or gradients [76,77]. By analyzing the changes in aggregation results, attackers may obtain sensitive information about data distribution or features.
- Malicious Behavior: Participants in federated learning may engage in malicious behavior, such as providing false model updates [78,79], tampering with data labels [80], or manipulating the aggregation process [81,82,83]. This malicious behavior may undermine the accuracy and reliability of the federated decision tree model and may also threaten data privacy.
3.1. Cryptographic Technology
3.2. Different Privacy
3.3. Data Security Aggregation
- Current research on decision trees in federated learning is mostly focused on VFL. In a horizontal setting, it is difficult to aggregate directly through model parameters like neural networks, as different participants use different features to split the intermediate nodes. In VFL, it is more feasible to establish synchronization layer-by-layer among the parties.
- We summarized the existing security schemes applied in the federal decision tree model, including, but not limited to, HE, MPC, DP and secure aggregation. We explained the application of various technologies in federated decision trees, as well as the privacy protection capabilities and performance advantages and disadvantages they provide.
- The introduction of security technology has had an impact on the cost of computation and communication, as well as on the accuracy of models. Following review, we believe that balancing privacy protection and model accuracy in designing security technology solutions is a direction for further research.
4. Efficiency Scheme
- Our efficiency evaluation of the federated decision tree is divided into the training stage and the prediction stage. We summarized the existing improvement plans and provided an overview of the experimental results.
- When training a decision tree model in a federated environment, many factors, such as safety, accuracy, and efficiency need to be considered. Currently, most solutions focus on one or two of them, and there is room for improvement.
5. Discussion
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Zhuang, J.; Yu, J.; Ding, Y.; Qu, X.; Hu, Y. Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment. IEEE Trans. Multimed. 2023, 26, 1361–1372. [Google Scholar] [CrossRef]
- Peng, W.; Hu, Y.; Yu, J.; ** attacks on malware detection systems. Neural Comput. Appl. 2020, 32, 14781–14800. [Google Scholar] [CrossRef]
- **a, Q.; Tao, Z.; Hao, Z.; Li, Q. FABA: An algorithm for fast aggregation against byzantine attacks in distributed neural networks. In Proceedings of the IJCAI, Macao, China, 10–16 August 2019. [Google Scholar]
- **e, C.; Koyejo, S.; Gupta, I. Zeno: Distributed stochastic gradient descent with suspicion-based fault-tolerance. In Proceedings of the ICML, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 6893–6901. [Google Scholar]
- Li, L.; Xu, W.; Chen, T.; Giannakis, G.B.; Ling, Q. RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. In Proceedings of the AAAI, Honolulu, HI, USA, 29–31 January 2019; Volume 33, pp. 1544–1551. [Google Scholar]
- Yang, S.; Ren, B.; Zhou, X.; Liu, L. Parallel distributed logistic regression for vertical federated learning without third-party coordinator. ar**v 2019, ar**v:1911.09824. [Google Scholar]
- Zhang, Y.; Zhu, H. Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning. ar**v 2020, ar**v:2007.06849. [Google Scholar]
- Paillier, P. Public-key cryptosystems based on composite degree residuosity classes. In Proceedings of the EUROCRYPT, Prague, Czech Republic, 2–6 May 1999; Springer: Berlin/Heidelberg, Germany, 1999; pp. 223–238. [Google Scholar]
- Goldreich, O. Secure multi-party computation. In Manuscript Preliminary Version; Citeseer: University Park, PA, USA, 1998; Volume 78. [Google Scholar]
- Bonawitz, K.; Ivanov, V.; Kreuter, B.; Marcedone, A.; McMahan, H.B.; Patel, S.; Ramage, D.; Segal, A.; Seth, K. Practical secure aggregation for federated learning on user-held data. ar**v 2016, ar**v:1611.04482. [Google Scholar]
- Fredrikson, M.; Jha, S.; Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the CCS, Denver Colorado, USA, 12–16 October 2015; pp. 1322–1333. [Google Scholar]
- Shokri, R.; Stronati, M.; Song, C.; Shmatikov, V. Membership inference attacks against machine learning models. In Proceedings of the SP, San Jose, CA, USA, 22–24 May 2017; pp. 3–18. [Google Scholar]
- Mohassel, P.; Zhang, Y. Secureml: A system for scalable privacy-preserving machine learning. In Proceedings of the SP, San Jose, CA, USA, 22–24 May 2017; pp. 19–38. [Google Scholar]
- Papernot, N.; Abadi, M.; Erlingsson, U.; Goodfellow, I.; Talwar, K. Semi-supervised knowledge transfer for deep learning from private training data. ar**v 2016, ar**v:1610.05755. [Google Scholar]
- Dwork, C.; McSherry, F.; Nissim, K.; Smith, A. Calibrating noise to sensitivity in private data analysis. In Proceedings of the TCC, New York, NY, USA, 4–7 March 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 265–284. [Google Scholar]
- Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H.B.; Mironov, I.; Talwar, K.; Zhang, L. Deep learning with differential privacy. In Proceedings of the CCS, Vienna, Austria, 25–27 October 2016; pp. 308–318. [Google Scholar]
- Dey, A.K.; Martin, C.F.; Ruymgaart, F.H. Input recovery from noisy output data, using regularized inversion of the Laplace transform. IEEE Trans. Inf. Theory 1998, 44, 1125–1130. [Google Scholar] [CrossRef]
- McHutchon, A.; Rasmussen, C. Gaussian process training with input noise. NeurIPS 2011, 24, 1341–1349. [Google Scholar]
- Awan, J.; Kenney, A.; Reimherr, M.; Slavković, A. Benefits and pitfalls of the exponential mechanism with applications to hilbert spaces and functional pca. In Proceedings of the ICML, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 374–384. [Google Scholar]
- Liu, X.; Li, Q.; Li, T.; Chen, D. Differentially private classification with decision tree ensemble. Appl. Soft Comput. 2018, 62, 807–816. [Google Scholar] [CrossRef]
- **ang, T.; Li, Y.; Li, X.; Zhong, S.; Yu, S. Collaborative ensemble learning under differential privacy. In Proceedings of the WI, Santiago, Chile, 7 November 2018; pp. 73–87. [Google Scholar]
- Fletcher, S.; Islam, M.Z. A Differentially Private Decision Forest. AusDM 2015, 15, 99–108. [Google Scholar]
- Yang, S.; Li, N.; Sun, D.; Du, Q.; Liu, W. A differential privacy preserving algorithm for greedy decision tree. In Proceedings of the ICBASE, IEEE, Zhuhai, China, 24–26 September 2021; pp. 229–237. [Google Scholar]
- Mironov, I. Rényi differential privacy. In Proceedings of the CSF, IEEE, Santa Barbara, CA, USA, 21–25 August 2017; pp. 263–275. [Google Scholar]
- Shi, L.; Shu, J.; Zhang, W.; Liu, Y. HFL-DP: Hierarchical federated learning with differential privacy. In Proceedings of the GLOBECOM, IEEE, Madrid, Spain, 7–11 December 2021; pp. 1–7. [Google Scholar]
- Wu, Z.; Li, Q.; He, B. Practical vertical federated learning with unsupervised representation learning. TBD ar**v 2022, ar**v:2208.10278. [Google Scholar] [CrossRef]
- Bonawitz, K.; Ivanov, V.; Kreuter, B.; Marcedone, A.; McMahan, H.B.; Patel, S.; Ramage, D.; Segal, A.; Seth, K. Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the CCS, Dallas, TX, USA, 31 October 2017; pp. 1175–1191. [Google Scholar]
- Bittau, A.; Erlingsson, U.; Maniatis, P.; Mironov, I.; Raghunathan, A.; Lie, D.; Rudominer, M.; Kode, U.; Tinnes, J.; Seefeld, B. Prochlo: Strong privacy for analytics in the crowd. In Proceedings of the SOSP, Shanghai, China, 29–31 October 2017; pp. 441–459. [Google Scholar]
- Erlingsson, U.; Feldman, V.; Mironov, I.; Raghunathan, A.; Song, S.; Talwar, K.; Thakurta, A. Encode, shuffle, analyze privacy revisited: Formalizations and empirical evaluation. ar**v 2020, ar**v:2001.03618. [Google Scholar]
- Sun, L.; Qian, J.; Chen, X.; Yu, P.S. Ldp-fl: Practical private aggregation in federated learning with local differential privacy. ar**v 2020, ar**v:2007.15789. [Google Scholar]
- Erlingsson, U.; Feldman, V.; Mironov, I.; Raghunathan, A.; Talwar, K.; Thakurta, A. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the SODA, SIAM, San Diego, CA, USA, 6–9 January 2019; pp. 2468–2479. [Google Scholar]
- Liu, R.; Cao, Y.; Chen, H.; Guo, R.; Yoshikawa, M. Flame: Differentially private federated learning in the shuffle model. In Proceedings of the AAAI, Virtual, 2–9 February 2021; Volume 35, pp. 8688–8696. [Google Scholar]
- McMahan, H.B.; Moore, E.; Ramage, D.; Hampson, S.; y Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the AISTATS, PMLR, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1273–1282. [Google Scholar]
- Weinberg, A.I.; Last, M. Selecting a representative decision tree from an ensemble of decision-tree models for fast big data classification. J. Big Data 2019, 6, 1–17. [Google Scholar] [CrossRef]
- Kwatra, S.; Torra, V. A Survey on Tree Aggregation. In Proceedings of the FUZZ-IEEE, IEEE, Luxembourg, Luxembourg, 11–14 July 2021; pp. 1–6. [Google Scholar]
- Kargupta, H.; Park, B. A fourier spectrum-based approach to represent decision trees for mining data streams in mobile environments. TKDE 2004, 16, 216–229. [Google Scholar] [CrossRef]
- Miglio, R.; Soffritti, G. The comparison between classification trees through proximity measures. Comput. Stat. Data. An. 2004, 45, 577–593. [Google Scholar] [CrossRef]
- Caruana, R.; Niculescu-Mizil, A.; Crew, G.; Ksikes, A. Ensemble selection from libraries of models. In Proceedings of the ICML, Banff, AL, Canada, 4–8 July 2004; p. 18. [Google Scholar]
- Tian, Y.; Feng, Y. Rase: Random subspace ensemble classification. J. Mach. Learn. Res. 2021, 22, 2019–2111. [Google Scholar]
- Chen, M.; Shlezinger, N.; Poor, H.V.; Eldar, Y.C.; Cui, S. Communication-efficient federated learning. Proc. Natl. Acad. Sci. USA 2021, 118, e2024789118. [Google Scholar] [CrossRef] [PubMed]
- Chen, H.Y.; Chao, W.L. Fedbe: Making bayesian model ensemble applicable to federated learning. ar**v 2020, ar**v:2009.01974. [Google Scholar]
- Antunes, R.S.; da Costa, C.A.; Küderle, A. Federated learning for healthcare: Systematic review and architecture proposal. ACM Trans. Intell. Syst. Technol. 2022, 13, 1–23. [Google Scholar] [CrossRef]
- Kasturi, A.; Ellore, A.R.; Hota, C. Fusion learning: A one shot federated learning. In Proceedings of the ICCS, Amsterdam, The Netherlands, 3–5 June 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 424–436. [Google Scholar]
- Li, M.; Chen, Y.; Wang, Y.; Pan, Y. Efficient asynchronous vertical federated learning via gradient prediction and double-end sparse compression. In Proceedings of the ICARCV, Shenzhen, China, 13–15 December 2020; pp. 291–296. [Google Scholar]
- Chiti, F.; Fantacci, R.; Picano, B. A matching theory framework for tasks offloading in fog computing for IoT systems. IEEE Internet Things J. 2018, 5, 5089–5096. [Google Scholar] [CrossRef]
- Arisdakessian, S.; Wahab, O.A.; Mourad, A.; Otrok, H.; Guizani, M. A survey on iot intrusion detection: Federated learning, game theory, social psychology and explainable ai as future directions. IEEE Internet Things J. 2022, 10, 4059–4092. [Google Scholar] [CrossRef]
- Wehbi, O.; Arisdakessian, S.; Wahab, O.A.; Otrok, H.; Otoum, S.; Mourad, A.; Guizani, M. FedMint: Intelligent Bilateral Client Selection in Federated Learning with Newcomer IoT Devices. IEEE Internet Things J. 2023, 10, 20884–20898. [Google Scholar] [CrossRef]
- Li, Y.; Feng, Y.; Qian, Q. FDPBoost: Federated differential privacy gradient boosting decision trees. J. Inf. Secur. Appl. 2023, 74, 103468. [Google Scholar] [CrossRef]
- Hu, Y.; Zhang, Y.; Gong, D.; Sun, X. Multi-participant federated feature selection algorithm with particle swarm optimizaiton for imbalanced data under privacy protection. IEEE Trans. Artif. Intell. 2022, 4, 1002–1016. [Google Scholar] [CrossRef]
- Courbariaux, M.; Bengio, Y.; David, J. Binaryconnect: Training deep neural networks with binary weights during propagations. NeurIPS 2015, 28, 3123–3131. [Google Scholar]
- Devos, L.; Meert, W.; Davis, J. Fast gradient boosting decision trees with bit-level data structures. In Proceedings of the ECML-PKDD, 19–23 September 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 590–606. [Google Scholar]
- Shi, Y.; Ke, G.; Chen, Z.; Zheng, S.; Liu, T. Quantized Training of Gradient Boosting Decision Trees. ar**v 2022, ar**v:2207.09682. [Google Scholar]
- Fu, M.; Zhang, C.; Hu, C.; Wu, T.; Dong, J.; Zhu, L. Achieving Verifiable Decision Tree Prediction on Hybrid Blockchains. Entropy 2023, 25, 1058. [Google Scholar] [CrossRef] [PubMed]
- Zhang, J.; Fang, Z.; Zhang, Y.; Song, D. Zero knowledge proofs for decision tree predictions and accuracy. In Proceedings of the CCS, Virtual Event, USA, 9–13 November 2020; pp. 2039–2053. [Google Scholar]
- Wang, H.; Deng, Y.; **e, X. Public Verifiable Private Decision Tree Prediction. In Proceedings of the Inscrypt, Guangzhou, China, 11–14 December 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 247–256. [Google Scholar]
- Wen, H.; Fang, J.; Wu, J.; Zheng, Z. Transaction-based hidden strategies against general phishing detection framework on ethereum. In Proceedings of the ISCAS, Daegu, Republic of Korea, 22–28 May 2021; pp. 1–5. [Google Scholar]
- Joshi, K.; Bhatt, C.; Shah, K.; Parmar, D.; Corchado, J.M.; Bruno, A.; Mazzeo, P.L. Machine-learning techniques for predicting phishing attacks in blockchain networks: A comparative study. Algorithms 2023, 16, 366. [Google Scholar] [CrossRef]
- Ali, M.N.; Imran, M.; din, M.S.U.; Kim, B.S. Low rate DDoS detection using weighted federated learning in SDN control plane in IoT network. Appl. Sci. 2023, 13, 1431. [Google Scholar] [CrossRef]
- Kazmi, S.H.A.; Qamar, F.; Hassan, B.; Nisar, K.; Chowdhry, B.S. Survey on joint paradigm of 5G and SDN emerging mobile technologies: Architecture, security, challenges and research directions. Wirel. Pers Commun 2023, 130, 2753–2800. [Google Scholar] [CrossRef]
Proposed Model | HFL/VFL | Tree Algorithm | Security Measure | Performance Improvement | ||
---|---|---|---|---|---|---|
Security | Accuracy | Efficiency | ||||
Tree-based FL [35] | HFL | GBDT | DP + SecAgg | ✓ | ✓ | |
FEDXGB [33] | HFL | XGBoost | HE + SS | ✓ | ✓ | |
F-XGBoost [45] | HFL | XGBoost | K-Anon | ✓ | ✓ | |
Federated Forest [37] | HFL | Extra trees | LDP | ✓ | ✓ | |
DFedForest [46] | HFL | RF | Blockchain | ✓ | ✓ | |
FL-XGBoost [47] | HFL | XGBoost | Encryption | ✓ | ||
FedXGB [48] | HFL | XGBoost | SS | ✓ | ✓ | |
DPBoost [49] | HFL | GBDT | DP | ✓ | ||
SimFL [50] | HFL | XGBoost | LSH | ✓ | ||
eFL-Boost [36] | HFL | GBDT | SecAgg | ✓ | ||
Pri Fed GBDT [38] | HFL | GBDT | RDP | ✓ | ✓ | ✓ |
SecureBoost [23] | VFL | XGBoost | HE | ✓ | ||
SecureBoost+ [51] | VFL | XGBoost | HE | ✓ | ✓ | |
Pivot [34] | VFL | RF & GBDT | HE + MPC | ✓ | ✓ | |
Secure XGBoost [52] | VFL | XGBoost | SecEnclave | ✓ | ||
FLSectree [53] | VFL | XGBoost | Encryption | ✓ | ✓ | |
FedXGBoost [54] | VFL | XGBoost | DP | ✓ | ||
VF2Boost [39] | VFL | GBDT | SecAgg | ✓ | ||
FEVERLESS [55] | VFL | XGBoost | SecAgg + CDP | ✓ | ✓ | |
Fed-EINI [32] | VFL | RF & GBDT | HE | ✓ | ✓ | |
FedGBF [56] | VFL | RF & GBDT | Encryption | ✓ | ||
FedRF [57] | VFL | RF | Encryption | ✓ | ||
OpBoost [58] | VFL | XGBoost | LDP | ✓ | ✓ | |
VPRF [21] | VFL | RF | HE | ✓ | ✓ | |
SGBoost [59] | VFL | XGBoost | SS + FE + SHE | ✓ | ✓ | ✓ |
VF-CART [40] | VFL | CART | HE | ✓ | ||
PriVDT [60] | VFL | GBDT | FSS | ✓ | ||
FederBoost [24] | HFL/VFL | GBDT | SecAgg + DP | ✓ | ✓ |
Mechanisms | Technology | Principle | Advantages | Disadvantages |
---|---|---|---|---|
Data ambiguity | DP | Central/local noise provided to perturb data or gradient values. | High computational efficiency; Low communication overhead; Post-processing, protecting published data. | Decreased accuracy and availability of training models. |
Process encryption | HE | Gradient encryption, operating on ciphertext. | Strict privacy protection. | Unable to handle complex operations; Low computational efficiency; High storage overhead. |
SMC | Intermediate data are private and cannot be learned by other parties. | Prevent man-in-the-middle attack and data leakage. | Low computational efficiency; High communication overhead. |
Aggregating decision trees | Structure-based: according to the hierarchical structure of the tree, then aggregating different layers. Classifying the samples in the sub-nodes in the hierarchy. |
Weight-based: considering the division of the tree as a set, and aggregating the weight values of the samples in the set. | |
Logic-based: considering the setting up of the decision tree as a set of logical rules, and then aggregating the logical expressions. | |
Dataset-based: fitting the results of multiple decision trees onto a complete dataset. | |
Selecting decision trees | In one iteration, selecting the single tree that best represents the information of all datasets as the global model. |
Training Stage | Incremental learning. |
Model compression and pruning. | |
Parallel and asynchronous computing. | |
Sampling and subsampling. | |
Prediction Stage | Client inference. |
Compression model and quantization. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, Z.; Gai, K. Decision Tree-Based Federated Learning: A Survey. Blockchains 2024, 2, 40-60. https://doi.org/10.3390/blockchains2010003
Wang Z, Gai K. Decision Tree-Based Federated Learning: A Survey. Blockchains. 2024; 2(1):40-60. https://doi.org/10.3390/blockchains2010003
Chicago/Turabian StyleWang, Zijun, and Keke Gai. 2024. "Decision Tree-Based Federated Learning: A Survey" Blockchains 2, no. 1: 40-60. https://doi.org/10.3390/blockchains2010003