DeepAI AI Chat
Log In Sign Up

Efficient Meta-Learning via Error-based Context Pruning for Implicit Neural Representations

by   Jihoon Tack, et al.

We introduce an efficient optimization-based meta-learning technique for learning large-scale implicit neural representations (INRs). Our main idea is designing an online selection of context points, which can significantly reduce memory requirements for meta-learning in any established setting. By doing so, we expect additional memory savings which allows longer per-signal adaptation horizons (at a given memory budget), leading to better meta-initializations by reducing myopia and, more crucially, enabling learning on high-dimensional signals. To implement such context pruning, our technical novelty is three-fold. First, we propose a selection scheme that adaptively chooses a subset at each adaptation step based on the predictive error, leading to the modeling of the global structure of the signal in early steps and enabling the later steps to capture its high-frequency details. Second, we counteract any possible information loss from context pruning by minimizing the parameter distance to a bootstrapped target model trained on a full context set. Finally, we suggest using the full context set with a gradient scaling scheme at test-time. Our technique is model-agnostic, intuitive, and straightforward to implement, showing significant reconstruction improvements for a wide range of signals. Code is available at


page 2

page 5

page 14

page 16

page 17

page 18

page 19


Meta-Learning Sparse Implicit Neural Representations

Implicit neural representations are a promising new avenue of representi...

Torchmeta: A Meta-Learning library for PyTorch

The constant introduction of standardized benchmarks in the literature h...

CAML: Fast Context Adaptation via Meta-Learning

We propose CAML, a meta-learning method for fast adaptation that partiti...

Faster Optimization-Based Meta-Learning Adaptation Phase

Neural networks require a large amount of annotated data to learn. Meta-...

MetaHDR: Model-Agnostic Meta-Learning for HDR Image Reconstruction

Capturing scenes with a high dynamic range is crucial to reproducing ima...

Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning

Model-agnostic meta learning (MAML) is currently one of the dominating a...

Memory-Based Meta-Learning on Non-Stationary Distributions

Memory-based meta-learning is a technique for approximating Bayes-optima...