-
Procedural Content Generation through Quality Diversity
Quality-diversity (QD) algorithms search for a set of good solutions whi...
read it
-
Reinforced Data Sampling for Model Diversification
With the rising number of machine learning competitions, the world has w...
read it
-
Reinforced Data Sampling for Model Diversificatio
With the rising number of machine learning competitions, the world has w...
read it
-
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
Open-domain code generation aims to generate code in a general-purpose p...
read it
-
EvidentialMix: Learning with Combined Open-set and Closed-set Noisy Labels
The efficacy of deep learning depends on large-scale data sets that have...
read it
-
TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
Score function-based natural language generation (NLG) approaches such a...
read it
-
Generalized Sampling in Julia
Generalized sampling is a numerically stable framework for obtaining rec...
read it
A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation
This work studies the widely adopted ancestral sampling algorithms for auto-regressive language models, which is not widely studied in the literature. We use the quality-diversity (Q-D) trade-off to investigate three popular sampling algorithms (top-k, nucleus and tempered sampling). We focus on the task of open-ended language generation. We first show that the existing sampling algorithms have similar performance. After carefully inspecting the transformations defined by different sampling algorithms, we identify three key properties that are shared among them: entropy reduction, order preservation, and slope preservation. To validate the importance of the identified properties, we design two sets of new sampling algorithms: one set in which each algorithm satisfies all three properties, and one set in which each algorithm violates at least one of the properties. We compare their performance with existing sampling algorithms, and find that violating the identified properties could lead to drastic performance degradation, as measured by the Q-D trade-off. On the other hand, we find that the set of sampling algorithms that satisfies these properties performs on par with the existing sampling algorithms. Our data and code are available at https://github.com/moinnadeem/characterizing-sampling-algorithms
READ FULL TEXT
Comments
There are no comments yet.