ILoRA: A Unified Framework for Federated Learning with Heterogeneous Client Adaptation

Federated learning has revolutionized how AI models are trained across decentralized devices, but integrating it with parameter-efficient fine-tuning s like Low-Rank Adaptation (LoRA) has exposed critical s under client heterogeneity. A new study introduces ILoRA, a unified framework that tackles initialization instability, rank incompatibility, and client drift in federated LoRA fine-tuning, promising more stable and accurate model training without compromising privacy. This innovation could reshape how large foundation models are adapted in real-world scenarios, from mobile applications to edge computing, by ensuring that diverse client capabilities and data distributions no longer hinder collaborative learning.

ILoRA addresses three core s identified in federated LoRA systems. First, initialization-induced instability arises from random initialization of LoRA parameters across clients, which misaligns adaptation subspaces and slows convergence. Second, rank incompatibility and aggregation error occur when averaging LoRA parameters of different ranks, leading to biased global models due to dimension mismatches. Third, client drift under non-IID data exacerbates local-global update divergence, impairing generalization. To overcome these, ILoRA integrates QR-based orthonormal initialization to ensure all clients start in a coherent subspace, concatenated QR aggregation to fuse heterogeneous-rank updates while preserving information, and an AdamW optimizer with rank-aware control variates to correct local updates and mitigate drift.

Extensive experiments on computer vision and natural language processing benchmarks validate ILoRA's superiority. On datasets like CIFAR-10, CIFAR-100, and Tiny-ImageNet using models such as ViT-Base and Swin-Base, ILoRA consistently outperforms existing s like FedIT, FLoRA, LoRA-FAIR, and FFA-LoRA. For instance, in heterogeneous settings with ViT-Base and AdamW optimizer, ILoRA achieved up to 87.51% accuracy on CIFAR-100, improving over LoRA-FAIR by 1.10% and FedIT by 2.32%. In NLP tasks with RoBERTa under non-IID data distributions, ILoRA reached an average accuracy of 85.02% across seven benchmarks, with its enhanced variant ILoRA-S hitting 86.11%, setting state-of-the-art performance on six out of seven datasets.

Of ILoRA extend across multiple domains, enabling more efficient and private AI adaptations in fields like autonomous systems, personalized assistants, and secure data analytics. By reducing communication costs through QR-based compression—achieving O(rs · max(d, k)) overhead compared to FLoRA's O(rtotal · max(d, k))—ILoRA makes federated learning scalable for large-scale deployments. Its theoretical convergence guarantees, supported by analyses showing O(1/√SKT) rates with bounded heterogeneity effects, ensure robust performance even under extreme data skewness and rank variations, paving the way for broader adoption in resource-constrained environments.

Despite its strengths, ILoRA has limitations, such as assumptions of subspace consistency and potential sensitivity to very large rank disparities beyond tested ranges. The framework's reliance on QR decomposition may introduce computational overhead in low-resource settings, and empirical validation primarily covers vision and NLP tasks, leaving other domains like audio or multimodal data less explored. Future work could extend ILoRA to dynamic rank adjustments and broader parameter-efficient s, enhancing its applicability in evolving federated ecosystems.