Manipulating the Alignment Ability of Large Language Models through In-Context Demonstrations

In-Context Learning (ICL) can be leveraged to effectively jailbreak or guard aligned language models by crafting malicious or safe demonstrations.