Interesting stuff! We actually used the standard self-attention mechanism to do something similar for multi-agent spatiotemporal modeling --> https://github.com/airalcorn2/baller2vec. We then extended that work to account for the fact that agent trajectories are not statistically independent at a given time step --> https://github.com/airalcorn2/baller2vecplusplus. I think it would be pretty straightforward and interesting to drop the Performer's Fast Attention mechanism into our models!