I don't doubt that there are conditions in which men and women have the same math scores. Are these conditions "neutral"? "Unbiased"? I don't think it's possible to raise a population to the age where they are able to take a math test and still call them unbiased. I don't think it makes sense to talk about the origin of the intelligence of a person as being distinct from their history.

I guess the answer that you're looking for is that I don't think there is any specific gene that evolution selected for encouraging or inhibiting math intelligence. And if there was, I don't see why it would be turned on for one sex and not the other. But we are complicated organisms, and our mathematical intelligence is the product of our complicated genes interacting with a complicated environment.