Children are conditioned to be obedient to adults and higher-authority figures. Due to their inherent lack of experience and therefore lack of knowledge on the subject, they are easily manipulated into engaging in sex.
Another important point is that children are not recognized as real human beings but rather as property of their respective parents (or legal guardians). This can sometimes be inconsistent but this is the general attitude.

The moral argument is very important because morality is inherently a societal construct. If society didn't exist and it was just you and a 12 year old girl there are no extra reasons against intercourse when compared to a 20-year old with a medical-condition you need to be mindful of.

But because society exists, just don't fuck kiddies. It's that simple.