Search papers, labs, and topics across Lattice.
This paper investigates the enhancement of Random Indexing (RI) embeddings through the application of weighted averaging on a sparse Positive Pointwise Mutual Information (PPMI) graph. By refining RI vectors on a fairytales corpus, the authors achieve a significant accuracy improvement from 19.4% to 30.7% on a semantic analogy set of 272 Google family-category questions. The findings suggest that while PPMI graph averaging can effectively repair weak RI initializations, it does not outperform neural network-based methods on broader datasets like text8 or achieve meaningful similarity correlations on SimLex-999.
PPMI graph averaging can boost Random Indexing accuracy by over 50%, but struggles against neural embeddings in broader contexts.
Sparse word embedding pipelines can avoid dense co-occurrence matrix materialization, dense factorization, and gradient training while still relying on sparse global corpus statistics. This paper studies Random Indexing (RI) vectors refined by weighted averaging on a sparse Positive Pointwise Mutual Information (PPMI) graph. On a fairytales corpus, the covered semantic analogy set consists of 272 Google family- category questions. On this family subset, PPMI top-K graph averaging repairs a weak RI initialization, improving accuracy from 19.4+-0.7% to 30.7+-2.9% across five seeds. Under the single tested runs, the same neighborhood averaging reduces family- subset analogy accuracy for PPMI+SVD (singular value decom- position), Binary+SVD, CBOW, and Skip-gram. Thus the method is not competitive with neural baselines on text8 and gives near- zero strict similarity correlation on SimLex-999. While Bloom filter sketches underperform RI in the tested configuration, we find that PPMI graph averaging with top-K pruning is a useful non-gradient repair for weak RI embeddings. On the fairytales dataset, PPMI top-K=50 graph averaging improves RI with accuracy going from 19.4+-0.7% to 30.7+-2.9%, and performing best with a seed42 of 34.6%.
