← Research

AMHN@NeurIPS'23 · Workshop paper

Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories

Denoising is memory retrieval in disguise

Figure from Memory in Plain Sight

Abstract

The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. Though the generative process is traditionally understood as an "iterative denoiser", there is no universally accepted language to describe it. We introduce a novel perspective to describe DMs using the mathematical language of memory retrieval from the field of energy-based Associative Memories (AMs), making efforts to keep our presentation approachable to newcomers to both of these fields. Unifying these two fields provides insight that DMs can be seen as a particular kind of AM where Lyapunov stability guarantees are bypassed by intelligently engineering the dynamics (i.e., the noise and step size schedules) of the denoising process. Finally, we present a growing body of evidence that records DMs exhibiting empirical behavior we would expect from AMs, and conclude by discussing research opportunities that are revealed by understanding DMs as a form of energy-based memory.

In the press

More

Cite

@misc{hoover2023memory,
  title={Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories}, 
  author={Benjamin Hoover and Hendrik Strobelt and Dmitry Krotov and Judy Hoffman and Zsolt Kira and Duen Horng Chau},
  year={2023},
  eprint={2309.16750},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}