Heya, I have finished my bachelors in physics, and since I'm also from the line of astrophysics I can tell you how to go about what you want.
You don't really need all that chemistry and biology sub courses, but you would definitely need to do your masters in physics and have cosmology or astrophysics as your special paper. By the time you finish your masters, you would definitely know what area you want to research on and hence get your PhD.
And uh, good luck. <3