I'm in a similar boat, I'd say most men are since it's biologically hardwired into us. I won't bring up older studies that put the age lower as they've been deprecated by more recent research, but the latest research indicates the age men of all ages find the most attractive is 20.
Personally I do think I find 20-year-olds the most attractive in general, and actually just a few days ago I was feeling down about that because my biological clock is ticking in spite of my decision to never pursue relationships. Nothing I can do about it, nothing you can do about it. As we get older, we get more and more "irrational" in our attraction to young girls and feel like it's wrong, even though it's just biology: even if we make a decision to never have children, our bodies don't care and will increasingly turn against us as the impregnation train speeds farther away. We can't undo evolution, which is what has made us want to impregnate someone at peak fertility, no matter how solid our reasons to not want that and how hard we try to not be attracted to girls considered by modern society to be too young for us.
Someone at 20 is much more likely to get pregnant than someone at 30, and someone at 30 is much more likely to get pregnant than someone at 40. That's what determines the biological aspect of the attraction. There are also societal factors leading us to find young girls more attractive: they're more likely to be open about their sexuality, dress in revealing clothes, act in a sexualised manner publicly, and so on, and the media is constantly using sexualised teens as a carrot in advertising and such. For us, biologically it makes no difference if she's 15 or 20, and often models in ads are somewhere in between those ages.
The legal ambiguity, not knowing whether she's legal or not, is a kind of forbidden fruit set up by society. It's also part of why the age of consent is often lower than the legal age for porn. There's a capitalistic motivation behind it, in addition to judicial opportunism. If people were allowed to freely form relationships of any kind with whom they want without risking shame or arrest, the status quo of modern civilisation would change and the power structures would shift to be more equal. Obviously those in power don't want that, and controlling the youth is one of their ways to ensure their own sustenance in the future. An artificial separation between age groups is being created, growing more and more as time passes. In less "civilised" societies, a 20-year-old girl marrying a 40-year-old man is often not considered weird, but in a couple of decades it will be as they become "modernised" too.
Many societies used to practise marriage at menarche or even earlier, often those barely pubescent girls being married to men old enough to be their fathers. It's still common in certain countries, even ones that are technologically modernised like India and Iran... I personally think it's fucked up for a 10-year-old or such to marry anyone, but the UN definition of "child" as anyone under 18 with the intention of enforcing a clean artificial separation between "children" and "adults" is bullshit. You don't magically become an adult when you turn 18, and it used to be universally acknowledged that childhood and adulthood are on a spectrum. Not anymore. Even some old people are starting to think of 17-year-olds as the same as toddlers, and society is intentionally infantilising teenagers while also sexualising them, a kind of carrot and stick where both the infantilisation and sexualisation are carrots for teens but sticks for anyone older than them. Teens are the most profitable demographic, so by separating them from "adults" for as long as possible, profits are maximised.
Society needs to accept "naturalism" in all of its range, or things will only get worse. It frustrates me that so many leftists are starting to get just as much in denial about "naturalism" as far-right morons, letting the UN or their country's government or whomever dictate how they should feel and suppressing their own selves. Of course, "naturalism" is not to be upheld as the sole standard upon which society is built, but for it to be increasingly actively rejected by essentially the entire political spectrum.
Getting back to what you actually said, at least I find solace in the fact that I wouldn't have a chance with them anyway and I'd only make them miserable in the long term. If you actually have some hope and can't remind yourself of a relationship-ruining flaw in yourself, it's probably even worse. I'm sorry.