AMHN@NeurIPS'23 · Workshop paper
Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories
Denoising is memory retrieval in disguise
Abstract
The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. Though the generative process is traditionally understood as an "iterative denoiser", there is no universally accepted language to describe it. We introduce a novel perspective to describe DMs using the mathematical language of memory retrieval from the field of energy-based Associative Memories (AMs), making efforts to keep our presentation approachable to newcomers to both of these fields. Unifying these two fields provides insight that DMs can be seen as a particular kind of AM where Lyapunov stability guarantees are bypassed by intelligently engineering the dynamics (i.e., the noise and step size schedules) of the denoising process. Finally, we present a growing body of evidence that records DMs exhibiting empirical behavior we would expect from AMs, and conclude by discussing research opportunities that are revealed by understanding DMs as a form of energy-based memory.
In the press
- Quanta MagazineThe Strange Physics That Gave Birth to AI
More
Cite
@misc{hoover2023memory,
title={Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories},
author={Benjamin Hoover and Hendrik Strobelt and Dmitry Krotov and Judy Hoffman and Zsolt Kira and Duen Horng Chau},
year={2023},
eprint={2309.16750},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Benjamin Hoover
Hendrik Strobelt
Dmitry Krotov
Judy Hoffman
Zsolt Kira
Polo Chau