I was invited recently to appear on the highly respected PBS series WealthTrack along with Burton Malkiel, author of the investing classic A Random Walk Down Wall Street. Malkiel is a member of the investment committee of my own firm, Rebalance.

It was the premiere of host Consuelo Mack’s 10th year on the program she started and a tremendous honor for us. I admire her serious, results-based approach to financial journalism.

The list of her past interviewees on the award-winning show is a who’s who of wealth management all-stars, including folks such as the Yale Endowment’s David Swensen and widely admired fund manager Jeremy Grantham.

Mack asked us on specifically to address the retirement crisis, a topic we follow closely as retirement advisors.

I hope you get to see the entire program, which you can watch now on the Rebalance website. But I believe it worthwhile to share a major point Burt and I made to Mack during the program: Too many retirement savers aren’t making it not because they don’t save enough or because they live longer than expected, but because of our conflict-ridden retirement system.

Yes, we should save more and, yes, longevity is a real problem. But a major additional issue is the massive “returns gap” many retirement investors experience because of high fees and conflicts of interest among their own advisors.

It’s real and it’s huge. Try this on for size: Thirty-five million baby boomers are due to retire in the coming decades. In fact, it is already well under way, to the tune of 10,000 per day.

During years of saving and investing, those workers reasonably expected to at least match the returns of the stock market. From 1991 to 2010, that was essentially the experience of pension plan participants. Department of Labor data shows that workers covered by a pension plan got 8.3% per year from their managers.

Retirement savers using individual retirement accounts (IRAs), meanwhile, experienced an annualized return of 4.5%. People in 401(k) plans saw only a bit better of a return, at 5.4% per year.

In any retirement plan, a major component of success is compounding, that is, how fast money doubles. Think about it, if your money doubles in less than 10 years, you might still have another 10 or 20 to save money. With more time, your money doubles and doubles again. A dollar saved becomes $2, then that $2 turns into $4.

A person who saves $5,000 a year for 30 years thus has $664,706 for retirement. But what if your money is growing more slowly?

Sure, it doubles, but instead of nine years it might take 16 years. At 4.5% a year, you’re talking about an end point of $317,597 — the same three decades of saving but well under half the money at the end of the run!

Who would put you in such a terrible investment? Well, maybe they weren’t terrible investments after all but instead perfectly “suitable,” which is to say any advisor brought up before a regulator could easily say that the investment he chose fit your situation.

Fiduciary role

What the advisor fails to explain — and this is rampant in the retirement advising business — is just how much money he collected in commissions for selling you that fund in the first place.

As we told Mack, it’s no accident that millions of Americans are falling short. The fact that stock broker fees can cost you half your retirement is not a “bug” in the retirement system we have.

Rather, from the financial advisor perspective, it’s a feature. They get a nice fat commission without having to be clear to the client how much the advisor is paid or by whom for selling that particular fund.

There is a move on in Washington, hotly contested by the stock brokers, to change how this works, to force retirement advisors to spell out their fees and side-deal commissions. In effect, retirement advisors — stock brokers, insurance agents and others — would have to admit to their clients the facts in full and be forced act as true fiduciaries.

However that works out, the really important thing for retirement savers and investors now is to save more, plan better and lower the fees they pay.

If they consciously chose to pay more, well, it’s a free market. But the fees — and the real impact of those costs on retirement outcomes — should not be obscured from view.

Send this to a friend