RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval

RNNs struggle with in-context retrieval, hindering their performance compared to Transformers.