I don't know for sure, but I think that is true of all kinds of sailboats, and I think it has always been that way. Even the people who are sailing as children and teenagers usually drop out for a number of years. College plus maybe grad school, then starting a career, paying back student loans, etc. If no college, then usually marriage, children, trying to buy a house, etc. As you said, it usually comes down to money during that period in the 20's.
A lot of young, single adults who come out of sailing programs or racing teams opt to crew on big monohulls -- free ride, free food, free beer, and sometimes even a bunk for the night.
I think it's pretty common for people to not get back into sailing (or start sailing) with their own boat until they are somewhere in their 30's. So when we talk about getting more young people into the sport, actually people in their 30's ARE the young people.
And, on the other hand, we also lose a bunch of people in their 30's for sometimes as much as a decade while they are going through child-rearing years, and some re-surface when they are in their 40's.
If we are talking about sailboat racing, it is much more demanding timewise than most other sports when you figure that to attend a regatta, you often have to leave right after work on Friday and don't get home until late Sunday. So time can be as big an issue as money. So you need to be at a point in your life where you have disposable income and available time.
Okay, those are just age-related factors. There are lots more factors that determine when and if people get into sailing or leave sailing, but that's for another thread.
Whoops, just thought of another age-related thing -- people in their teens and 20's sometimes are involved in more physically demanding, and body-damaging, sports (e.g. motocross). When they get to the point where they cannot do those sports any more, they turn to something like sailing that they are able to do for the rest of their lives and still have the challenge of competition.