"I believe I have the power to blight crops using magical prayers." "She used magical prayers to blight our crops! Kill her! She admits it!"

That actually happening or not aside, they're actually practicing Voodoo: They do use poisons to alter the mind-states of people in Haiti, to make more pliable slaves, among other more recreational uses such as "spirit journies". To a savage who would kill a wizard because they're a wizard, someone claiming to be a wizard and then using "magic potions" is going to seem plenty wizardy to them.