Rangnick said his team had played four entertaining, intense matches and were worthy of a quarter-final berth but paid the price for poor defending that saw them go behind after 57 seconds to a buoyant Turkish side.
Merih Demiral pounced on a bungled Austrian effort to clear a corner and he doubled Turkey's lead just before the hour mark before Michael Gregoritsch pulled one back seven minutes later.
Austria flicked the switch but struggled to create substantive chances until a potent stoppage-time header from Christoph Baumgartner bounced off the turf and was saved spectacularly by goalkeeper Mert Gunok.
"If you're behind with two goals, then things aren't any easier. The team tried everything. We then scored one goal and we had enough time to achieve a draw," Rangnick told a press conference.
"It's difficult if you have Gordon Banks in goal," he said, likening Gunok's save to that famously pulled off by the late England goalkeeper from a header from Brazil's Pele in the 1970 World Cup.
Austria's elimination extends to seven decades their unfathomable failure to win a match in the knockout rounds of a major tournament.
"You also need a bit of luck. If Baumgartner's header at the end would have gone in, we could have won this game," Rangnick said.
"This was a historic chance to win, to go to the quarter-final and play against the Netherlands. I cannot believe that we're going home today. We thought that we would continue our journey here," he said.
Rangnick said his team now had to capitalise on their momentum and ensure qualification for the World Cup, which would be for the first time since 1998.
"We are in part one at the moment, and that hasn't happened for years, that we are in part one and we want to stay there. And we have a good chance to qualify for a World Cup after many years," he said.
One player who may no longer figure is striker Marko Arnautovic, who hinted Tuesday's defeat could be his last outing for the national side.
"It's very bitter, it's madness that we left the game like this," Arnautovic said. "For us, it's over. The coach said 'chin up'."
"It could be that it's my last game," he added.
