Rethinking positional encoding
Jianqiao Zheng, Sameera Ramasinghe, Simon Lucey
7/6/2021
Keywords: Fundamentals
Venue: ARXIV 2021
Bibtex:
@article{zheng2021rethinking,
journal = {arXiv preprint arXiv:2107.02561},
booktitle = {ArXiv Pre-print},
author = {Jianqiao Zheng and Sameera Ramasinghe and Simon Lucey},
title = {Rethinking Positional Encoding},
year = {2021},
url = {http://arxiv.org/abs/2107.02561v2},
entrytype = {article},
id = {zheng2021rethinking}
}
Abstract
It is well noted that coordinate based MLPs benefit greatly -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice. Codes available at https://github.com/osiriszjq/Rethinking-positional-encoding.
Citation Graph
(Double click on nodes to open corresponding papers' pages)
(Double click on nodes to open corresponding papers' pages)
* Showing citation graph for papers within our database. Data retrieved from Semantic Scholar. For full citation graphs, visit ConnectedPapers.