Number 8 Faletau and scrum half Evans scored the key tries for the Dragons, who took advantage of the Ospreys being without 16 players. Full-back Gareth Owen scored the the only try for the hosts, who had the consolation of a losing bonus point.
The Dragons held a deserved 8-3 lead at the interval having shaded possession and territory in the first half.
The highlight for the men of Gwent was the 33rd minute try from Faletau, the Welsh-qualified back rower battling his way past three would-be tacklers to score in the left corner.
It was the best moment of a largely defence-dominated first 40 minutes which was played out in heavy rain.
The Dragons had much the better of the territorial advantage with two lineouts in the Ospreys' 22 before Tovey, who controlled things from fly-half, kicked his side into a 3-0 lead with a 12th minute penalty goal after the Ospreys had been caught offside.
The Ospreys had the best chance of scoring the opening try when former Dragons winger Richard Fussell went close down the left.
It was a lead the Dragons held on to until the 32nd minute when fly-half Dai Flanagan made it 3-3 from a penalty kick after referee Dave Pearson had blown up for crossing.
But a minute later Faletau was over after Gavin Thomas had made the initial break into the Ospreys' 22. To make matters worse for the home side, Flanagan missed a relatively easy penalty in the closing minutes of the half.
However, the Ospreys more than made up for it at the start of the second half with Owen going over for a try after only two minutes' play. Fussell had made a break along the ight and passed inside for the full-back to carry on and make it over.
Flanagan converted but the Ospreys only held the lead - 10-8 - for two minutes, before Tovey converted his second penalty.
Flanagan missed a penalty on 52 minutes but was successful three minutes later with another attempt as the Ospreys went back into the lead at 13-11.
But again the Ospreys' advantage was shortlived as the Dragons soon went in for their second try. A good build-up involving Thomas, Tom Willis and James Leadbeater ended with number 9 Evans nipping in for the score down the blindside.
Owen and Tovey traded penalties before the Dragons dug their heels in to claim a famous victory, only their second on the road in the Magners League since January 2009.
|Ospreys Score Card|
|Newport Gwent Dragons Score Card|