Edwards lashed home a rebound in the third minute of added time to inflict the Black Cats' first defeat of the season.
Sunderland looked set to continue their 100 per cent start under Regis Le Bris when Patrick Roberts opened the scoring from the penalty spot.
Daniel Ballard's own goal and Ryan Hardie's penalty saw Plymouth come from behind after the break, but they appeared to have blown all three points when Romaine Mundle levelled in the 86th minute.
However, Edwards' eased the pressure on former England and Manchester United captain Rooney, who was sacked last season after just 83 days by Birmingham.
"When they equalised you think 'oh no not again' but then Joe gets the late winner," said Rooney.
"It always feels nice to win in any game and to get off the mark in the league. It will now be when is the second win coming?"
Leeds' unbeaten start came to an end in a 1-0 home defeat to promotion rivals Burnley.
Hollywood actor Will Ferrell, who is one of a number of celebrity investors in Leeds, was in attendance at Elland Road.
But there was no fairytale ending for the home side as Luca Koleosho's stunning first half strike earned all three points for the Clarets, who climb into third.
"It's a massive three points for us, especially at such a tough place to come and where we are in the stages of our development as well," said Burnley manager Scott Parker.
Blackburn climbed up to second in the table as Yuki Ohashi scored a second-half double in a 3-0 home win against Bristol City.
Watford missed the chance to move level on points with Sunderland after being held 1-1 at home by Coventry.
Cardiff remain rooted to the foot of the table after Kenzo Goudmijn's strike earned Derby a 1-0 win at Pride Park.
Two goals in stoppage time ensured a thrilling end to the game as Sheffield Wednesday drew 1-1 with QPR at Hillsborough.
Owls captain Barry Bannan thought he had given his side victory with a superb finish but substitute Alfie Lloyd scrambled in an equaliser with the last kick of the match.
Teden Mengi's superb finish clinched Luton's first win since April as they edged past Millwall 1-0 at The Den.
