But he never killed anyone with the spirit bomb, plus it takes time to form, he's vulnerable while charging it, he borrows energy to help make it (the earth is pretty desolate btw) and to top it off the one kill (sorta) was almost entirely made up of borrowed energy and a combined effort of him and vegeta against kid buu. Goku couldn't use the spirit bomb.