DeepAI AI Chat
Log In Sign Up

PatchGame: Learning to Signal Mid-level Patches in Referential Games

by   Kamal Gupta, et al.
University of Maryland

We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of "important" image patches, while the task for the listener is to match the speaker's message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit or implicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification). Code available at


page 7

page 8

page 15

page 16

page 17


Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

The past year has witnessed a rapid development of masked image modeling...

Learning Multi-Object Positional Relationships via Emergent Communication

The study of emergent communication has been dedicated to interactive ar...

SPEAKER VGG CCT: Cross-corpus Speech Emotion Recognition with Speaker Embedding and Vision Transformers

In recent years, Speech Emotion Recognition (SER) has been investigated ...

Learning to Communicate with Strangers via Channel Randomisation Methods

We introduce two methods for improving the performance of agents meeting...

SdAE: Self-distillated Masked Autoencoder

With the development of generative-based self-supervised learning (SSL) ...

Exploration of Lightweight Single Image Denoising with Transformers and Truly Fair Training

As multimedia content often contains noise from intrinsic defects of dig...

Emergent Communication with Attention

To develop computational agents that better communicate using their own ...