His enemies also get stronger. Remember Broly? And Cell? Goku had to push himself to his limits in order to fight them. And that was after one full year of continuous training in the Hyperbolic Time Chamber.