Federated Learning in Big Model Era: Domain-Specific Multimodal Large Models

by   Zengxiang Li, et al.

Multimodal data, which can comprehensively perceive and recognize the physical world, has become an essential path towards general artificial intelligence. However, multimodal large models trained on public datasets often underperform in specific industrial domains. This paper proposes a multimodal federated learning framework that enables multiple enterprises to utilize private domain data to collaboratively train large models for vertical domains, achieving intelligent services across scenarios. The authors discuss in-depth the strategic transformation of federated learning in terms of intelligence foundation and objectives in the era of big model, as well as the new challenges faced in heterogeneous data, model aggregation, performance and cost trade-off, data privacy, and incentive mechanism. The paper elaborates a case study of leading enterprises contributing multimodal data and expert knowledge to city safety operation management , including distributed deployment and efficient coordination of the federated learning platform, technical innovations on data quality improvement based on large model capabilities and efficient joint fine-tuning approaches. Preliminary experiments show that enterprises can enhance and accumulate intelligent capabilities through multimodal model federated learning, thereby jointly creating an smart city model that provides high-quality intelligent services covering energy infrastructure safety, residential community security, and urban operation management. The established federated learning cooperation ecosystem is expected to further aggregate industry, academia, and research resources, realize large models in multiple vertical domains, and promote the large-scale industrial application of artificial intelligence and cutting-edge research on multimodal federated learning.


page 1

page 2

page 3

page 5

page 8


Federated Learning: Organizational Opportunities, Challenges, and Adoption Strategies

Restrictive rules for data sharing in many industries have led to the de...

Fed-DART and FACT: A solution for Federated Learning in a production environment

Federated Learning as a decentralized artificial intelligence (AI) solut...

Federated Learning: Balancing the Thin Line Between Data Intelligence and Privacy

Federated learning holds great promise in learning from fragmented sensi...

An Efficient Industrial Federated Learning Framework for AIoT: A Face Recognition Application

Recently, the artificial intelligence of things (AIoT) has been gaining ...

A Vertical Federated Learning Method for Interpretable Scorecard and Its Application in Credit Scoring

With the success of big data and artificial intelligence in many fields,...

Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evo...

A Vertical Federated Learning Method For Multi-Institutional Credit Scoring: MICS

As more and more companies store their customers' data; various informat...

Please sign up or login with your details

Forgot password? Click here to reset