RWKV-5 (Eagle) and RWKV-6 (Finch): Efficient and Expressive Sequence Models with Matrix-Valued States and Dynamic Recurrence

The authors present two new sequence model architectures, Eagle (RWKV-5) and Finch (RWKV-6), that improve upon the RWKV-4 architecture by incorporating multi-headed matrix-valued states and dynamic recurrence mechanisms. These advancements enhance the models' expressivity while maintaining the efficient inference and training characteristics of RNNs.