Why are My IRA Returns Disappointing?

A friend of mine was trying to figure out why his financial advisor was underperforming the market so badly. 

Over the past 5 years (November 15th, 2019, to November 15th, 2024), his portfolio was up just 4.8% annually vs. 13.8% for the S&P 500.

He’s not the kind of person who checks his portfolio every week or month. He trusted his advisor to do the job for him.

But when he started scrutinizing the returns, he wanted to know why his IRA was performing the way it was so he could be informed in an upcoming annual review.

It would be convenient to say his advisor is expensive and doesn’t know what he’s doing.

But the answer is not that simple.

So we looked over his portfolio to figure out what was going on because that’s five years of solid returns — gone.

I came up with six reasons for the underperformance and tried to quantify the impact of each. 

The variability of stock market returns by asset class over the past five years played a major role.

Other issues stemmed from using a financial advisor and how he managed the portfolio. 

Hindsight is 20/20, and I’m not writing this to criticize advisors. This is for illustrative purposes, to demonstrate ways to understand your portfolio returns, especially for those disappointed while the stock markets have done so well.

Watch the Video

Six Causes of Portfolio Underperformance

You should not expect to achieve the same returns if you’re not 100% allocated to the S&P 500. 

On the surface, 4.8% compared to the S&P 500 performance of 13.8% is very disappointing for a 48-year-old.

However, a big part of the underperformance was due to small and mid-cap underperformance compared to large-cap. International stocks dramatically underperformed as well.

So, anyone with a conventionally balanced portfolio over the past five years likely underperformed the S&P 500 for the stock portion of their allocation.

In my friend’s case, other factors hurt his returns, and I tried to quantify the 9% performance difference (between 13.8% and 4.8%).

We didn’t look back at all five years of portfolio management; we only looked at his portfolio allocation today to find clues for the underperformance.

Inappropriate Asset Allocation

Impact Estimate: 1.5%

My friend is 48 years old, has moderate risk tolerance, is married, has no kids, and expects to work for at least another decade.

With a moderate risk tolerance, I expected my friend’s portfolio to be allocated at about 80/20 stocks to bonds. 

Instead, his portfolio was 70/30. This was too conservative for his risk tolerance.

I used the asset allocation calculator (try it!) on my website to give us a ballpark estimate or starting point of where his asset allocation should be. 

ideal asset allocation

This is what I expected his portfolio to look like.

Actual Asset Allocation

But this is where it was.

I compared an 80/20 stock-to-bond portfolio to a 70/30 using Portfolio Visualizer and VOO (S&P 500 ETF) and BND (total bond market ETF) as proxies to estimate the impact on his overall returns (his portfolio was more complicated, we’ll get to that).

I found that an 80/20 portfolio outperformed a 70/30 portfolio by about 1.5% over the five years.

With a more age and risk-appropriate portfolio, my friend could have had improved returns. 

Large Caps Outperformed Small/Mid Caps and International

Impact Estimate: 5.0%

Large-cap stocks outperformed small, mid-cap, and international stocks over the five years ending in November 2024.

This was due, in part, to incredible gains by stocks like Tesla, Nvidia, and the other “Mag 7”.

Most of us are exposed to small and mid-cap stocks through total market funds or small and mid-cap focused funds.

Returns were good, but not as good as large-cap. 

Using the S&P 500 and Russell 2000 (via the IWM ETF) as proxies, we can see large-cap stocks outperformed small and mid-cap by 5%.

The gap between U.S. large caps and international stocks is even wider. I used Vanguard’s VXUS Total International stock ETF as a proxy in the comparison below:

These results have had DIY investors questioning whether they should own international stocks.

Most writers I follow agree that we should not abandon international stocks. The chart below (via Charlie Bilello) shows that outperformance by U.S. or international stocks is cyclical, and the U.S. has dominated the past decade.

When this inevitably reverses, U.S. investors will want to have exposure to international stocks.

Even though his portfolio allocation to diversified stocks was a significant factor in underperformance, it’s not a reason to change to be all-in the U.S. only.

The longer one side outperforms, the more likely we see a reversal.

Unnecessarily Complicated Portfolio

Estimated Impact: Unknown

A typical retirement portfolio can be sufficiently allocated and diversified with a handful of mutual funds or ETFs.

Many investors deploy three-fund portfolios consisting of a U.S. total market stock fund, a total market ex-U.S. international stock fund, and a total market bond fund.

Fewer holdings make a portfolio simpler to monitor, manage, and modify. 

My friend’s portfolio had about a dozen holdings. I felt this was unnecessarily complicated for the portfolio size and investment objectives. 

The portfolio had some low-cost EFTs as the top holdings but also quite a few underperforming managed funds and asset overlap.

Though I can’t put a number on it, complication likely led to some underperformance. I suspect this advisor lost sight of overlap within the portfolio and was perhaps influenced by his organization’s fund recommendations/preferences.

Underperforming Managed Funds

Estimated Impact: 0.50% to 1.0%

When you work with a financial advisor, especially one associated with a larger brokerage house, you are subject to being invested in underperforming managed mutual funds with high fees. 

Why do financial advisors put clients in lousy funds?

The industry is abundant with conflicted interests and firm-approved funds and quotas.

In other words, advisors may have incentives to put clients into certain funds that may not be in their best interests. The fiduciary standard does not apply to many advisors most of the time.

Conflicts can include beneficial payments (e.g., 12b-1 fees), or the umbrella brokerage organization may have deals with fund providers and need to reach minimum investment amounts, requiring advisors to put a portion of their clients’ money into certain funds (whether these funds have a good history or not).

In my friend’s case, his advisor had him in several managed funds, most of which did not outperform their benchmarks.

This is usual. Nearly every year, more than 60% of fund managers do not beat their benchmarks. Over 5-year periods, the odds of underperformance is 90%. 

Every managed mutual fund pick has a roughly 1 in 10 chance of beating its benchmark over five years. If the advisor chooses multiple managed funds, the odds of beating the benchmarks become nearly impossible. 
The solution: buy the benchmark funds. 

The solution: buy the benchmark funds. 

For over a decade, my employer’s 401(k) had high-fee funds that underperformed because of 12b-1 fees and bad fund providers. But I didn’t have a choice; we only had so many funds to choose from with no influence on the pool of funds.

Ask your employer for better funds, they may not know the selection sucks.

With an advisor, you can scrutinize selected funds and ask questions. Perform research by comparing funds in your portfolio to their benchmarks. 

In the video for this article, I demonstrate how to compare fund performance using Morningstar Investor. 

Frequent Rebalancing

Estimated Impact: Unknown

According to my friend, his advisor made portfolio adjustments every quarter. Sometimes, advisors may feel they must “mix the bowl” to show they are doing something. 

But a more laissez-faire approach leads to better outcomes. Rebalancing should be done just once a year or less frequently. 

When the stock market is in a bull run, you want to let your winners keep winning. From what I could tell, the frequent rebalancing and portfolio juggling likely led to some underperformance, but how much? It’s hard to say. 

Advisor Fees

Estimated Impact: 1%

My friend didn’t know how much his advisor was charging him.

I’ve heard from several readers in emails and survey results who say they are unsure what their advisor charges them. 

For whatever reason, not everyone is comfortable asking, or maybe they are not all that concerned. 

One thing is for certain: financial advisors get paid, and what clients pay them lowers returns.

How they extract money from clients’ accounts may vary. As an intern with Merrill Lynch in 1997, I asked the advisor I was working with how he got paid. 

His response: 

The client never sees it.

Some firms and advisors are more transparent than others. But usually, you’ll see a fee withdrawal in the transaction list.

AUM advisors typically charge about 1% for clients under $1 million. Over a million, the client has some negotiating power. 

My Dad worked with an advisor who charged 1.2% and invested his money in mutual funds with fees near 1%.

I asked for confirmation: “So, my Dad’s returns are handicapped by 2% before the year starts?”

He replied with a hesitant and uncomfortable “Yes”. 

Hiring a financial advisor is like hiring someone to cut the grass. You are outsourcing the job.

Unfortunately, financial advisors provide a very expensive service that frequently doesn’t deliver to expectations. Monitor portfolio changes, ask questions, and accurately communicate your investment objectives. 

Remedies

As DIY investors, we already save 1% to 2% or more by not using an AUM financial advisor. So you already have an advantage.

Here are some other ways to help optimize long-term returns with appropriate risk. 

Align Asset Allocation with Risk Tolerance

Determine your stock-to-bond allocation by using my calculator or simply subtracting your age from the number 120 (conservative), 130 (moderate), or 140 (aggressive), depending on your risk tolerance — aka, the “minus your age” rule of thumb. 

That’s a place to start. But also consider the reliability of other income sources, risk tolerance, investment objectives, and when you’ll need the money.

Streamline Holdings

DIY investors can build a diversified, risk-appropriate portfolio with as little as three ETFs or mutual funds.

Having more than six holdings or owning individual stocks within a portfolio will complicate things and certainly not guarantee you’ll outperform a simple portfolio.

A streamlined and diversified portfolio will lower risk and improve returns over long-term investment horizons. Furthermore, a simpler portfolio is easier to track and modify and takes up much less of your time to manage. 

Buy the Benchmark Funds

If you want to construct a portfolio with varying stock assets besides the primary indexes, you can allocate funds to small or mid-cap growth or value funds, REIT funds, varying bond funds, or other asset classes.

But instead of trying to find the 5% to 10% of fund managers who beat their benchmarks, just buy the benchmarks. ETFs that track benchmarks (e.g., the S&P 500 or Russell 1000 Value Index) are widely available and easy to find.

Choose them to simplify your decision and meet the benchmark returns. 

Most of my retirement money is in broad total stock market index mutual funds. However, I’ve owned various managed mutual funds through various employers because they were the only options available at times. 

Over the years, I’ve moved nearly all of my money out of managed funds into total market index funds (mostly FSKAX and FSGGX).

Now, I can most forget about the money. I have no plans to spend it for at least a decade and no plans to time the market by selling.

Benchmark and index investing for DIY investors frees time for other activities and improves returns.

Annual Rebalance

I spelled out how to approach the annual rebalance process in a recent article and video.

Rebalancing at regular intervals is generally considered a sound habit, but only to maintain an appropriate asset allocation, not to try and time the market.

Avoid rebalancing based on emotional triggers like market volatility, and do not rebalance more frequently than annually. 

Choose a time of the year (January works for me), and adjust in that timeframe. 

Understand Advisor fees

A recent survey suggested that less than half of my readers have ever worked with an advisor. But a surprisingly large percent have or currently do. I’m glad you’re hear to learn for yourself.

My friend had an upcoming annual meeting with his advisor, and I suggested he ask what the fees are, how they are withdrawn from his account, and if there are any potential conflicts of interest. 

Fees are negotiable, especially as account balances rise. 

This advisor is a long-time friend of my friend, so we hope he has his best interests in mind. But as an intern, I saw that friends and family service isn’t much different than any other client.

Through his service, he sold him an insurance-investing product (e.g., whole life, universal life). I recommend that people never mix insurance and investing products, as these are primarily designed to earn commissions for their sellers and hide fees from clients.

The same objectives can be achieved with term life and normal investing accounts. 

Consider Self-Directed

If you must use an advisor, fee-only advisors charge flat rates for advice instead of an AUM fee for managing the portfolio. 

So that’s one remedy — do not work with AUM advisors.

Another is to not work with friends or family to manage your money. It will be harder to leave if you don’t like their investment and fee philosophies. 

As an intern in 1997, I shadowed a relatively new advisor who spent 95% of his time growing his client base.

He started with friends and family. Many of those probably became clients for life, partly because they were afraid of hurting their relationship while he was trying to build a career. 

That left 5% of his time for managing portfolios. Is it wise to hire someone who spends just 5% of their time doing what you’ve hired them for?

Halfway through my internship, I asked if I could see him buy a stock or mutual fund on the computer (this was before online investing became widespread). He didn’t do it often because he was always cold calling and mailing sales material to potential clients. 

Working with an advisor means outsourcing the job. You cannot expect the advisor to beat the market. So, your assets will grow slower than if you self-direct your investment portfolio using market index funds and minimal tweaking or stock picking. 

Self-directed investing gives you a 1% to 2% advantage over advisors. That’s a substantial advantage that equates to tens of thousands (even hundreds) over several years. 

Conclusion

If your retirement portfolio returns are disappointing, it’s worth conducting a similar exercise to see what’s dragging your portfolio returns. 

Figure out your current asset allocation compared to your ideal target allocation, considering your age and risk tolerance.

Use an asset allocation calculator if you’re unsure of your target asset allocation.

Research what you own. Then, look at expensive or chronically underperforming funds you own and unload them. Talk to your advisor about your holding to exchange managed funds in favor of benchmark index funds.

Remember, you can’t expect to match its returns if you’re not 100% in the S&P 500.

If you’re very aggressive, investing in all stocks, tech stocks, or other speculative investments, you may have a chance to beat the S&P 500.

Beware, when the bull market ends, more aggressive portfolios are going to underperform, so watch out.

As for hiring an advisor, I’ve heard from so many readers who will never trust their money to an advisor. They are too expensive for most to consider, and no one cares about your money more than you. 

Fee-only fiduciary advisors (the ones that are 100% fiduciary all the time) can help guide those who need help. But most of you can fend for yourselves and beat the advisors with your built-in advantage. 

However, you must consider the duel challenges of growing and preserving wealth, which becomes more complicated as we age. Managing money is not only about returns. 

Featured image via DepositPhotos used under license.