Index investing and Dr. Frankenstein's creation

Introduced in 1957, the S&P 500 is the most widely cited US equity index, with more money managed to it, by far, than any other benchmark. As such, it has become a proxy for the health of US investors, corporate America and the overall economy.

Initially a tool for measurement and understanding, the S&P has over time become a central actor in shaping investor behavior. The widespread use of S&P-based products now actively influences the market that the index was intended merely to observe.

The consequences have been many, but perhaps the most significant has been the widening gap between investment in the stock market and actual ownership of businesses through the stock market. The S&P has become a sort of Frankenstein’s science project run amok, a mix of Mary Shelley's novel and the less nuanced films that have followed. Created to do a certain thing, it is now an active participant beyond the reach of its creator. I don’t mean this in a pejorative sense. After all, it can be considered an excellent index; I do think, however, that too much weight is loaded upon it.

When the S&P was launched, it was a vast improvement over the numerous industry and early market averages that had appeared in the late nineteenth and early twentieth centuries. Those primitive measures struggled with stock splits, stock dividends and dividend payments. Over time, they tended to become distorted. But the S&P is no longer just a yardstick. It has taken on a life of its own and changed the whole nature of stock investing in ways most investors have not paused to consider.

The creature’s first sign of life independent of its components came in the form of estimates for what the index itself would “earn.” These “top-down” estimates took advantage of the growing availability of greater and more detailed macroeconomic data in the postwar decades. These forecasts were then used to calculate the market’s P/E ratio. Assigning the market a P/E number was akin to the 1931 film version of Dr. Frankenstein throwing the electric switch: If it has a P/E, it breathes. It can be cheap, it can be expensive, it can be fairly priced. It has a character.

Initially just an idea, the S&P first took on corporeal form in 1976 when the first index fund based on it appeared. By 2023, up to 40% of the index’s value was in products or derivatives indexed or benchmarked to the S&P 500.  While the index still measures the market, models based on it have become the largest single investment in the market. Market concentration has compounded this problem in recent years.

At its current size, the S&P is so large that inclusions and exclusions lead to material share price swings in the affected securities—and a voluminous academic literature. Similarly, the daily reweighting of the index is no longer a simple exercise in observation, not when trillions of dollars are in funds tracking or competing with it. When there are net inflows or outflows to these products, S&P providers buy and sell the constituent stocks (or market futures) without regard to valuation, to dividends or, frankly, to anything. They have to.

Market cap weighting, together with the immense size of products tied to it, cause the S&P to exhibit characteristics of a momentum product, buying what’s going up and selling what’s going down. But whether Main Street investors fully appreciate the consequences of the S&P 500 being both a measuring device and a momentum-oriented financial product at the same time remains unclear.

Price discovery is at risk too. With so much money flowing into index funds over the past several decades, the burden of determining the fair, negotiated price of a security has been falling on a shrinking pool of active participants.

Concentration within the index doesn’t help matters. As of the end of 2024, the top ten names in the index represented 37% of its total value. That is materially greater than a previous peak in concentration, around 25%, prior to the bursting of the internet bubble in 2000. I’m not passing judgment on today’s dominant companies or their valuation. But as a practical matter, these leading names are no longer sensitive to market movements so much as the market is sensitive to theirs.

One could say that any pooled investment product attenuates the connection between a company’s owners and the company itself. But the unprecedented scale of broad-market index investing, and the pointed narrative around passive investing, distinguishes those products from actively managed, narrower pools of invested capital with specific mandates (i.e., growth, value, small caps, dividends, specific sectors, stated geographies, and new technologies). The latter allocate capital in an intentional manner, engage in intentional price discovery, and (are supposed to) vote as proxies—a critical measure of business ownership—with equal intention.

What’s to be done?

At a minimum, investment analysts should be more familiar with equal-weighted or fundamentally weighted indices. Retail investors in particular should have a better understanding of what it means to “own the market” through an index fund.

Importantly, retirement plan sponsors and trustees need to appreciate that an attitude of “all things passive” creates as much risk as it removes. Relentlessly pushing retirees into broad index products may be the low-cost option, but, like the grave robbing the good doctor did to create his creature, not a low-risk solution.

A longer version of this article originally appeared in the journal American Affairs.