Conor Gallagher's shot was stopped by the arm of West Ham midfielder Tomas Soucek late on but the referee waved play on. The match ended 1-1.
Former Chelsea player Joe Cole, working as a pundit for BT Sport, said Potter should have caused "a bit of uproar" while Rio Ferdinand compared him to Jose Mourinho.
Potter, whose side play the first leg of their Champions League last 16 tie at Borussia Dortmund later on Wednesday, said he had a responsibility not to contribute to the abuse of referees.
"The same media are talking about me being angrier but then running stories about problems with referees at grassroots level," Potter said. "They don't see the connection."
A BBC questionnaire responded to by more than 900 amateur referees in England revealed worrying levels of abuse and intimidation with multiple cases of death threats being made.
"That's not to say we don't all lose our temper; we do because it's an emotional thing," added Potter.
"I have a responsibility to myself, to Chelsea, to the game, and to act in a way that is the right thing for me."
