The Labour leader was sporting the Care Family Tartan when a local minister said he thought he would wear an “Islamic jihad scarf”