Sunan Abu Dawud #3251
Oaths and Vows (Kitab Al-Aiman Wa Al-Nudhur) 22:10
Sa'id ibn Ubaydah said:
Ibn Umar heard a man swearing: No, I swear by the Ka'bah. Ibn Umar said to him: I heard the Messenger of Allah (ﷺ) say: He who swears by anyone but Allah is polytheist.