The only real problem with this theory is in the unveiling lore book when they talk about the flower game, it’s all metaphorical. All the events happened, as far as we know but we are told that the flowers aren’t actually flowers. They are a placeholder for life. They could be a species, a person, a planet, a galaxy, etc. Even if the black garden was the garden where the flower game was played, the winnower tells us that he entered the game as well, and that now that they were in the game there were no other games, no second chances, meaning there was no going back to the garden (which is said they left in ruins during there original fight). I know the winnower may not be a reliable source but it is the only source we have.