Hunters feel they have earned more than the warlocks and are entitled to more prestige. While warlocks are not necessarily smarter but more educated and have a sense of pride in "being more powerful". Plus I heard the hunters used their invisibility to steal the warlocks cookies.