Why do men lose hair with low testosterone?

Indeed, but this decrease in testosterone over the years does not seem to have an affect on the amount of hair loss, which is somewhat counterintuitive.

Not at all. Genetic hair loss has ingrained in each follicular unit of the scalp, a biological clock that limits how many growth cycles it has. The fewer the growth cycles, the more the balding. The hair on the back and sides of the head have no limit on hair cycling. Even with this explanation, the hair loss process is very complex and many factors determine the number of cycles each follicular unit has besides the genetics.