“As long as I’m not the richest man in the world, I won’t really be happy,” Bernard Arnault once said.

The day has come. Arnault, the founder and CEO of luxury goods conglomerate LVMH, went from being the richest person in the world to the fifth-most after a 20% drop in his company’s stock price, resulting in a $54 billion cut of his net worth.

Arnault, who oversees dozens of prestigious brands including Moët Hennessy, Louis Vuitton, Dior, Givenchy, and Fendi, was worth an estimated $231 billion in late March, according to the Bloomberg Billionaires Index. That placed him ahead of Tesla CEO Elon Musk, Amazon founder Jeff Bezos, and Meta CEO Mark Zuckerberg.

As of Monday, Arnault was worth an estimated $177 billion. Now, Arnault ranks lower than each of those executives, plus Oracle co-founder Larry Ellison. In the past year, LVMH stock has dropped over 16%, and is currently trading at about $132 per share; Arnault owns about 48% of the luxury conglomerate. Since the start of 2024, Arnault’s net worth is down $30 billion, making him the biggest loser on the list year-to-date.

LVMH did not respond to Fortune’s request for comment.

What’s happening at LVMH?

During the first half of 2024, LVMH reported a modest dip in revenues, but its wine and spirits divisions fared even worse.

“Maybe the current global situation, be it geopolitical or macroeconomic, doesn’t lead people to cheer up and open bottles of champagne,” Jean-Jacques Guiony, LVMH’s chief financial officer, said during the company’s earnings call in late July. “I don’t really know. The matter of fact is, is that our volumes are down double digits.”

But it’s not just LVMH that’s had a tough time in today’s market. Other major luxury brands and holding houses have reported losses this year. In fact, luxury revenues were flat during Q2 2024, the slowest growth in 15 quarters, according to a Bank of America note about luxury goods released Monday. Demand deteriorated during July, and August and September have also slowed.

Michael Kors, housed under Capri Holdings, saw its first quarter revenue drop more than 14% year-over-year, according to Capri’s first quarter fiscal 2025 results posted Aug. 8.

“Sometimes you’ll be the hottest thing on the block,” Michael Kors said in his testimony during an $8.5 billion antitrust trial involving Capri Holdings and Tapestry. “Sometimes you’ll be lukewarm. Sometimes you’ll be cold.”

Bernard Arnault’s net worth

While Arnault has lost a tremendous amount of wealth this year as the result of the struggling luxury market, with a net worth of $177 billion, he’s still wealthier than former Microsoft CEOs Bill Gates and Steve Ballmer, as well as Warren Buffett, Michael Dell, and Nvidia CEO Jensen Huang.

Story continues

Plus, Arnault has stayed active in investing this year. His family office has invested in five AI-focused startups this year amounting to hundreds of millions of dollars. Plus, in July, he joined the chase to acquire Venice’s iconic Hotel Bauer for $305 million—but got outbid.

Arnault, 75, isn’t ready to leave his post: He forced his board to extend the retirement age for its chairman and CEO from 75 to 80 so he could stay longer. But he received a letter from 93-year-old Warren Buffett saying he should’ve made the age limit higher, according to Bloomberg. Still, Arnault says he’s working 12-hour days.

“Every morning I have fun when I arrive,” Arnault told Bloomberg.

This story was originally featured on Fortune.com