They weren't behind the walls they were outside infront of it patrolling keeping the people safe the hunters only went out there's getting treasures not looking for fights they probably avoided any encounter they could unless it directly interfered with their loot cowards