b'Chuang Gan'

research

∙ 08/15/2023

A^2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models

We study the task of zero-shot vision-and-language navigation (ZS-VLN), ...

0 Peihao Chen, et al. ∙

research

∙ 07/24/2023

3D-LLM: Injecting the 3D World into Large Language Models

Large language models (LLMs) and Vision-Language Models (VLMs) have been...

0 Yining Hong, et al. ∙

research

∙ 07/22/2023

Learning Vision-and-Language Navigation from YouTube Videos

Vision-and-language navigation (VLN) requires an embodied agent to navig...

0 Kunyang Lin, et al. ∙

research

∙ 07/20/2023

Reparameterized Policy Learning for Multimodal Trajectory Optimization

We investigate the challenge of parametrizing policies for reinforcement...

0 Zhiao Huang, et al. ∙

research

∙ 07/05/2023

Building Cooperative Embodied Agents Modularly with Large Language Models

Large Language Models (LLMs) have demonstrated impressive planning abili...

0 Hongxin Zhang, et al. ∙

research

∙ 06/29/2023

An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training

We present a model that can perform multiple vision tasks and can be ada...

0 Zitian Chen, et al. ∙

research

∙ 06/27/2023

Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties

General physical scene understanding requires more than simply localizin...

0 Hsiao-Yu Tung, et al. ∙

research

∙ 06/07/2023

ModuleFormer: Learning Modular Large Language Models From Uncurated Data

Large Language Models (LLMs) have achieved remarkable results. But exist...

0 Yikang Shen, et al. ∙

research

∙ 05/04/2023

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Recent AI-assistant agents, such as ChatGPT, predominantly rely on super...

0 Zhiqing Sun, et al. ∙

research

∙ 04/27/2023

Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics

We propose a hybrid neural network (NN) and PDE approach for learning ge...

0 Pingchuan Ma, et al. ∙

research

∙ 04/19/2023

EC^2: Emergent Communication for Embodied Control

Embodied control requires agents to leverage multi-modal pre-training to...

0 Yao Mu, et al. ∙

research

∙ 04/17/2023

Hyper-Decision Transformer for Efficient Online Policy Adaptation

Decision Transformers (DT) have demonstrated strong performances in offl...

1 Mengdi Xu, et al. ∙

research

∙ 04/07/2023

Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following

Humans, even at a very early age, can learn visual concepts and understa...

0 Mingyu Ding, et al. ∙

research

∙ 04/06/2023

Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention

Humans possess a versatile mechanism for extracting structured represent...

0 Mingyu Ding, et al. ∙

research

∙ 03/29/2023

Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

Modeling sounds emitted from physical object interactions is critical fo...

0 Kun Su, et al. ∙

research

∙ 03/27/2023

DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

In this work, we aim to learn dexterous manipulation of deformable objec...

0 Sizhe Li, et al. ∙

research

∙ 03/20/2023

3D Concept Learning and Reasoning from Multi-View Images

Humans are able to accurately reason in 3D by gathering multi-view obser...

0 Yining Hong, et al. ∙

research

∙ 03/09/2023

PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

Existing approaches to system identification (estimating the physical pa...

0 Xuan Li, et al. ∙

research

∙ 03/09/2023

Planning with Large Language Models for Code Generation

Existing large language model-based code generation pipelines typically ...

0 Shun Zhang, et al. ∙

research

∙ 12/15/2022

Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners

Optimization in multi-task learning (MTL) is more challenging than singl...

1 Zitian Chen, et al. ∙

research

∙ 11/21/2022

CLAWSAT: Towards Both Robust and Accurate Code Models

We integrate contrastive learning (CL) with adversarial learning to co-o...

5 Jinghan Jia, et al. ∙

research

∙ 10/27/2022

Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation

Effective planning of long-horizon deformable object manipulation requir...

0 Xingyu Lin, et al. ∙

research

∙ 10/15/2022

Revisiting the Roles of "Text" in Text Games

Text games present opportunities for natural language understanding (NLU...

9 Yi Gu, et al. ∙

research

∙ 10/14/2022

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

We address a practical yet challenging problem of training robot agents ...

0 Peihao Chen, et al. ∙

research

∙ 10/14/2022

Learning Active Camera for Multi-Object Navigation

Getting robots to navigate to multiple objects autonomously is essential...

0 Peihao Chen, et al. ∙

research

∙ 10/12/2022

M^3Video: Masked Motion Modeling for Self-Supervised Video Representation Learning

We study self-supervised video representation learning that seeks to lea...

0 Xinyu Sun, et al. ∙

research

∙ 09/17/2022

MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

Decentralized learning has shown great promise for cooperative multi-age...

0 Kefan Su, et al. ∙

research

∙ 09/01/2022

Gait Recognition in the Wild with Multi-hop Temporal Switch

Existing studies for gait recognition are dominated by in-the-lab scenar...

0 Jinkai Zheng, et al. ∙

research

∙ 07/22/2022

Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation

This paper studies a new, practical but challenging problem, called Clas...

5 Hongbin Lin, et al. ∙

research

∙ 07/13/2022

3D Concept Grounding on Neural Fields

In this paper, we address the challenging problem of 3D concept groundin...

5 Yining Hong, et al. ∙

research

∙ 07/07/2022

Finding Fallen Objects Via Asynchronous Audio-Visual Integration

The way an object looks and sounds provide complementary reflections of ...

6 Chuang Gan, et al. ∙

research

∙ 07/05/2022

Weakly Supervised Grounding for VQA in Vision-Language Transformers

Transformers for visual-language representation learning have been getti...

0 Aisha Urooj Khan, et al. ∙

research

∙ 06/30/2022

On-Device Training Under 256KB Memory

On-device training enables the model to adapt to new data collected from...

14 Ji Lin, et al. ∙

research

∙ 06/27/2022

Prompting Decision Transformer for Few-Shot Policy Generalization

Humans can leverage prior experience and learn novel tasks from a handfu...

1 Mengdi Xu, et al. ∙

research

∙ 06/03/2022

SNAKE: Shape-aware Neural 3D Keypoint Field

Detecting 3D keypoints from point clouds is important for shape reconstr...

0 Chengliang Zhong, et al. ∙

research

∙ 05/29/2022

EfficientViT: Enhanced Linear Attention for High-Resolution Low-Computation Visual Recognition

Vision Transformer (ViT) has achieved remarkable performance in many vis...

0 Han Cai, et al. ∙

research

∙ 05/11/2022

RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation

This work considers identifying parameters characterizing a physical sys...

2 Pingchuan Ma, et al. ∙

research

∙ 05/05/2022

Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics

Differentiable physics has recently been shown as a powerful tool for so...

4 Sizhe Li, et al. ∙

research

∙ 05/05/2022

Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

This paper studies the problem of fixing malfunctional 3D objects. While...

15 Yining Hong, et al. ∙

research

∙ 05/02/2022

ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

Objects' motions in nature are governed by complex interactions and thei...

0 Zhenfang Chen, et al. ∙

research

∙ 04/04/2022

Learning Neural Acoustic Fields

Our environment is filled with rich and dynamic acoustic information. Wh...

3 Andrew Luo, et al. ∙

research

∙ 03/31/2022

DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools

We consider the problem of sequential robotic manipulation of deformable...

2 Xingyu Lin, et al. ∙

research

∙ 03/30/2022

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

We present a meta-learning framework for learning new visual concepts qu...

12 Lingjie Mei, et al. ∙

research

∙ 03/13/2022

AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

Training a generalizable 3D part segmentation network is quite challengi...

0 Xueyi Liu, et al. ∙

research

∙ 12/09/2021

PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

A critical aspect of human visual perception is the ability to parse vis...

28 Yining Hong, et al. ∙

research

∙ 12/01/2021

Graph Convolutional Module for Temporal Action Localization in Videos

Temporal action localization has long been researched in computer vision...

0 Runhao Zeng, et al. ∙

research

∙ 11/01/2021

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Contrastive learning (CL) can learn generalizable feature representation...

4 Lijie Fan, et al. ∙

research

∙ 10/28/2021

Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language

In this work, we propose a unified framework, called Visual Reasoning wi...

8 Mingyu Ding, et al. ∙

research

∙ 10/28/2021

MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

Tiny deep learning on microcontroller units (MCUs) is challenging due to...

19 Ji Lin, et al. ∙

research

∙ 10/17/2021

Network Augmentation for Tiny Deep Learning

We introduce Network Augmentation (NetAug), a new training method for im...

0 Han Cai, et al. ∙

Chuang Gan

Featured Co-authors