DeepAI AI Chat
Log In Sign Up

LSH Microbatches for Stochastic Gradients: Value in Rearrangement

by   Eliav Buchnik, et al.

Metric embeddings are immensely useful representation of interacting entities such as videos, users, search queries, online resources, words, and more. Embeddings are computed by optimizing a loss function of the form of a sum over provided associations so that relation of embedding vectors reflects strength of association. Moreover, the resulting embeddings allow us to predict the strength of unobserved associations. Typically, the optimization performs stochastic gradient updates on minibatches of associations that are arranged independently at random. We propose and study here the antithesis of coordinated arrangements, which we obtain efficiently through LSH microbatching, where similar associations are grouped together. Coordinated arrangements leverage the similarity of entities evident from their association vectors. We experimentally study the benefit of tunable minibatch arrangements, demonstrating consistent reductions of 3-15% in training. Arrangement emerges as a powerful performance knob for SGD that is orthogonal and compatible with other tuning methods, and thus is a candidate for wide deployment.


page 1

page 2

page 3

page 4


A probabilistic framework for multi-view feature learning with many-to-many associations via neural networks

A simple framework Probabilistic Multi-view Graph Embedding (PMvGE) is p...

Understanding Undesirable Word Embedding Associations

Word embeddings are often criticized for capturing undesirable word asso...

Retrieving Multi-Entity Associations: An Evaluation of Combination Modes for Word Embeddings

Word embeddings have gained significant attention as learnable represent...

Embedding Knowledge Graph of Patent Metadata to Measure Knowledge Proximity

Knowledge proximity refers to the strength of association between any tw...

Are Emojis Emotional? A Study to Understand the Association between Emojis and Emotions

Given the growing ubiquity of emojis in language, there is a need for me...