Diff with StreamingLLM

[pfeatherstone]

Meta AI have just released https://arxiv.org/pdf/2309.17453.pdf
This looks related to, or at least trying to solve the same problems as, RMT.
Any thoughts?

[lucidrains]

I dont have experiments to back it up, but my intuition is that RMT is closer to a solution whereas attention sinks is a hack to make standard decoders work.

[lucidrains]

the right way to approach attention sinks would be to append learned memory key / values https://github.com/lucidrains/x-transformers#augmenting-self-attention-with-persistent-memory at each layer, but i also think relative positions will need to be redesigned

[lucidrains]

we'll just have to see what follow up papers do. sometimes you can have a new paper basically obviate previous papers, so its best not to jump on this hacky solution yet