The Norwegian Ruud, who took a medical timeout after the second set, succumbed 6-3, 7-5, 6-7 (4/7), 6-2 to the 39th-ranked Brooksby on Rod Laver Arena.
His early exit left the tournament without its top two seeds in week one after Nadal's shock loss on the same court Wednesday to another American in Mackenzie McDonald.
It opens the door to Novak Djokovic winning a 10th Melbourne title and temporarily derails Ruud's dreams of becoming world number one.
"I gave it my all, but in the fourth set especially wasn't enough. The first two sets were sort of closer, but I ended up getting broken a bit too many times," said the world number three.
"I will try to learn from it. I think that he played great today."
The 24-year-old Ruud's defeat ended any hope he had of assuming the world number-one ranking from Carlos Alcaraz, who is absent from Melbourne with an injury.
Only two players can now achieve the feat if they lift the trophy - Serbian superstar Djokovic or Greek third seed Stefanos Tsitsipas.
Ruud enjoyed a breakthrough 2022, reaching the final of both the French and US Opens.
He felt "tension and some slight pain in my abdominal" but said it was not responsible for the surprisingly one-sided defeat.
"It's something that has actually been with me for around two years. It just sort of tightens up my left side from serving," he said.
"It didn't affect me a lot, but I felt it a little bit in the second set. I hope it's nothing serious."
Melbourne debut
Brooksby, who is on his Australian Open debut, will face Tommy Paul next after his fellow American beat Spain's Alejandro Davidovich Fokina in five sets.
"First and foremost, Casper is a warrior, I knew it would be a great battle out there," said the 22-year-old.
"I'm just really proud of my mental resolve after the third-set battle didn't go my way, and to turn it around."
Brooksby, whose best Grand Slam result was reaching the round of 16 at the 2021 US Open, took charge in the first set, breaking twice, with the Norwegian hitting 15 unforced errors to slump behind.
Ruud, who was scheduled to meet Djokovic in the semi-finals, regrouped at the changeover and earned two break points on Brooksby's opening serve in set two, but he failed to convert.
They traded breaks through a gruelling second set before the American earned a set point at 5-6 and after a 26-shot rally moved two sets clear on a lucky net cord.
Ruud left the court for a medical timeout at the changeover, apparently for an abdominal problem.
His game didn't improve when he returned, broken two more times before staging a remarkable comeback.
He saved three match points at 3-5 as Brooksby became agitated, slamming his racquet on the ground, before taking it to a tiebreak where he broke early to take it to a fourth set.
But, undeterred, the gutsy American reset to break Ruud immediately and race to a 3-0 lead.
