Rule of thumb with poor soil is you want to build better soil.
>I've tried planting stuff in it and nothing grows but weeds
A weed is just a plant that you don't want. The plants you're seeing are what's called "pioneer" plants - they can grow in harsh, shitty conditions. In growing and dying they will help create new soil. That is the natural process. Over time more complex and massive plants grow as the soil accumulates. See pic. You can speed up the process (called succession) in various ways.
>unless you surround it all with ready-dirt and that kind of defeats the purpose doesn't it
No, this is one of the ways to speed up succession. You can bring in some soil, some compost, some mulch, i.e. the stuff to make soil. The point isn't to pull yourself up by your bootstraps but to use the resources available to you to build a sustainable system - one that eventually can keep going without you having to intervene much if at all. There's no ban on using technology or advanced resources. You just need to be aware of any technical drawbacks a particular method might have, like some fertilizers being low quality or something like that.