Seven-time world champion Hamilton cut a gloomy figure in the Mercedes garage after posting the 18th fastest lap in the session, more than 1.5 seconds slower than Ferrari's pace-setting Charles Leclerc at Albert Park.
Hamilton was ninth quickest in the earlier session, while teammate George Russell was third. Russell was sixth fastest in the second free-practice session.
Wolff said engineers had experimented with two different set-ups in the cars as the team tries to unlock performance amid a difficult start to the season.
"In the second session (we’ve) gone through a really quite dramatic set-up change on Lewis' (car) and that has massively backfired," he told Sky. "But this is why we're having those sessions. On the other side it was a bit better, but we were lacking performance. ... Overall it wasn’t a good day."
Both Hamilton and Russell struggled with the balance of their cars and each skidded off track in practice.
They were hardly alone, though, with most drivers battling for grip on the street circuit, including Alex Albon, who wrecked his Williams car with a crash at turn six in the first practice.
Mercedes are fourth in the constructors' championship behind leading Red Bull, Ferrari and McLaren after the opening races in Bahrain and Saudi Arabia.
"If I would say that I’m not frustrated, it would be not the truth," Wolff said of the W15 car's progress. "Certainly we are because we’re trying so much in all directions, but don’t seem to have found that silver bullet yet which helps us to get us in the right direction."
