Fusion of Satellite Images and Weather Data with Transformer Networks for Downy Mildew Disease Detection

09/06/2022
by   William Maillet, et al.
0

Crop diseases significantly affect the quantity and quality of agricultural production. In a context where the goal of precision agriculture is to minimize or even avoid the use of pesticides, weather and remote sensing data with deep learning can play a pivotal role in detecting crop diseases, allowing localized treatment of crops. However, combining heterogeneous data such as weather and images remains a hot topic and challenging task. Recent developments in transformer architectures have shown the possibility of fusion of data from different domains, for instance text-image. The current trend is to custom only one transformer to create a multimodal fusion model. Conversely, we propose a new approach to realize data fusion using three transformers. In this paper, we first solved the missing satellite images problem, by interpolating them with a ConvLSTM model. Then, proposed a multimodal fusion architecture that jointly learns to process visual and weather information. The architecture is built from three main components, a Vision Transformer and two transformer-encoders, allowing to fuse both image and weather modalities. The results of the proposed method are promising achieving 97% overall accuracy.

READ FULL TEXT

page 1

page 6

research
04/12/2022

Are Multimodal Transformers Robust to Missing Modality?

Multimodal data collected from the real world are often imperfect due to...
research
08/01/2022

SiamixFormer: A Siamese Transformer Network For Building Detection And Change Detection From Bi-Temporal Remote Sensing Images

Building detection and change detection using remote sensing images can ...
research
10/04/2022

ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

3D human reconstruction from RGB images achieves decent results in good ...
research
12/05/2022

Solving the Weather4cast Challenge via Visual Transformers for 3D Images

Accurately forecasting the weather is an important task, as many real-wo...
research
08/26/2022

TFusion: Transformer based N-to-One Multimodal Fusion Block

People perceive the world with different senses, such as sight, hearing,...
research
06/13/2022

Multimodal Learning with Transformers: A Survey

Transformer is a promising neural network learner, and has achieved grea...
research
09/27/2022

Dense-TNT: Efficient Vehicle Type Classification Neural Network Using Satellite Imagery

Accurate vehicle type classification serves a significant role in the in...

Please sign up or login with your details

Forgot password? Click here to reset