Two o’clock in the morning, on a warm spring night, Bryan “Baby” Williams pulls his blood-red Bugatti Veyron into the parking lot of the Hit Factory studio in Miami. Stepping out of the driver’s-side door in blue jeans, white-on-white Uptowns, a white YMCMB hoodie, a Gucci belt, dark sunglasses and a whole lot of diamonds and tattoos, the Cash Money Records founder looks like real hip-hop—a back-in-the-day era when rappers still dressed like they came from the hood.

Baby, 42, is from the streets of New Orleans, where he and his brother Ronald “Slim” Williams founded their label 22 years ago. They broke nationally in 1998, entering into a landmark distribution deal with Universal Records and selling millions of records by the Hot Boys—Juvenile, B.G., Turk and Lil Wayne. Juve, B.G. and Turk would jump ship a few years later, complaining of financial impropriety. But Wayne remained and was rewarded with a president position at the label and his own imprint, Young Money Entertainment. He would soon become, of course, the biggest star in rap music, and would sign a pair that would become the two biggest new stars in rap music, Drake and Nicki Minaj.

Baby, meanwhile, oversees it all. As hip-hop goes through its transitions and industry heavyweights fizzle out, lose power or disappear, he and his company grow stronger and richer every year. In April, Forbes magazine named him one of The Forbes Five: Hip-Hop’s Wealthiest Artists, tallying his wealth at $100 million (a figure he disputes as being far too low). XXL sits down for a conversation with one of hip-hop’s last true moguls.

XXL: When you look around at the industry and see the lack of moguls doing it like they used to, what does that make you think?

Bryan "Baby" Williams: When I look around, through my years and my time, you know, the game, it was a lot in the field, a lot of people doing it. And I look around, and it’s like, Where the fuck did everybody go? I ain’t never understood that, but the game like that sometimes. That muthafucka’s gon’ keep ya or it’s gon’ leave ya. It’s all on you.

So you look around and feel like everyone’s gone… What does that feel like? How does that work?

Just how heavy it was, how many different brands you had, how many different artists. Now it’s a new era, really. A lot of the old acts ain’t around no more, of the era when we come in the game. The labels, the corporations, they not supporting the brands. I think one of my biggest things was I been an independent brand. I always owned my own brand. I never been funded by them. It’s P&D deal. I was always able to use my money to do everything we wanted to do, too. So when you bagged up by them and they money, and you ain’t working out… You might have one bad year, and they’ll give up. Things might not balance out. It could be any little thing. Times change, like 9/11, depression. People didn’t wanna fuck with money like they used to. In my situation, I never had them do that for me. I always floated my own money. We made our own moves.

You and your label have been through a lot of ups and downs. You had major success coming in the game in 1998, but by 2001, the Hot Boys had broken up; Juve, B.G. and Turk left the label with beef; and Lil Wayne wasn’t nearly as big as he is now. A lot of people had written you off. What were you thinking?

You know, the way I grew up was, I lost my mama at two. My daddy died when I was, like, seven. My brother died when I was, like, nine. So I been losing lives all my life. So for the acts to leave me, it really made me stronger, ’cause personal life made me this way, and growing up taught us that we have to take a loss and try to make it into a gain. And it’s all I really had in life to do was music. So they losses, really, when they left, it opened me up to have to go harder, you understand me? At that time, Wayne was young, so I had to get it in. We had to go hard as a team. So then when he came to the plate like, “Let’s do it,” that was a wrap.