Raheem Sterling put Chelsea ahead in the first half at the City Ground on Sunday.
But Ivory Coast defender Serge Aurier levelled after the break and Chelsea were fortunate to escape without an embarrassing defeat after Forest dominated for long spells in the second half.
Chelsea are languishing in eighth place after failing to win for the sixth time in their last seven league games.
The Blues are seven points adrift of fourth placed Manchester United in the race to qualify for the Champions League via a top four finish.
"We need to be more consistent. It would help to get players back, the key ones and then you can build consistency in key places," Potter said.
"It's always tough because you have a lot of big teams with lots of top players and coaches.
"We have a long way to go so it would be a mistake to think about what could happen in five months time."
Just months after replacing the sacked Thomas Tuchel, Chelsea boss Potter is starting to come under fire as the early optimism after his move from Brighton begins to fade.
'Not good enough'
Chelsea's 2-0 win against Bournemouth on 27 December was supposed to offer hope of a brighter period in 2023.
But while it is surely too early for Potter to be worried about losing the first high-pressure job of his career, the 47-year-old needs an uptick in results in the new year to avoid putting Chelsea chairman Todd Boehly in an awkward position.
"Our performance level wasn't good enough to take the three points. We didn't move the ball fast enough," Potter said.
"Forest played a good game but we're disappointed with our performance and concerned we didn't deserve the win. It was overall below par for us."
Chelsea captain Cesar Azpilicueta added to the gloom when he said: "Since the beginning of the season we have lost too many points and we are really far from where we want to be.
"We need to have a very good second half of the season if we want to play in the Champions League next season."
Forest are third bottom having won just three games all season, but boss Steve Cooper will take heart from the way his team rattled Chelsea in the second half.
The only dark cloud for Forest was the alleged homophobic chanting from home supporters that earned condemnation from LGBTQ groups.
