Tovey's 79th-minute strike gave the Dragons a 23-22 lead and put them within touching distance of only their second victory in Ireland since 2004.
Tovey's unconverted try was the Dragons' only first half score, but they had second half touchdowns from Lewis Evans and Aled Brew and there were successful kicks from Adam Hughes, Matthew Jones and Tovey.
Pienaar had topped off Nevin Spence's early try and added a penalty to give Ulster a 10-5 advantage at half-time.
But they fell 17-10 behind by the hour mark, and had to fight all the way with tries from replacements Tommy Seymour and Darren Cave before Pienaar's match-winning contribution.
Ireland Under-20 internationals Craig Gilroy, Luke Marshall and Paddy Jackson were part of a youthful Ulster back-line led by the steadying hand of Pienaar.
The Dragons, meanwhile, fielded largely the same team from last weekend's LV= Cup semi-final defeat to Gloucester, Hughes and Joe Bearman were the only additions in the centre and at number 8 respectively.
Ulster's debutant winger Conor Gaston dealt well with some high balls and got a couple of chances to run at his opposite number Brew.
The visitors made the first error, their full-back Martyn Thomas kicking the ball straight into touch from outside the 22.
Tim Barker won the lineout for Ulster and a pass from Marshall put his centre partner Spence through a hole in the Dragons defence and over for the try.
Pienaar knocked over the simple conversion from in front of the posts and Tovey, presented with his first place-kick, badly missed the target after Gilroy was penalised for a tackle infringement.
Tovey redeemed himself in the 13th minute when he gathered the ball and crossed unchallenged for his try after Ulster full-back Adam D'Arcy had failed to hold a chip and Brew kicked through.
The unsuccessful conversion was followed by a second penalty miss from the usually reliable Tovey and Ulster stretched their lead to 10-5 thanks to a 26th minute penalty from Pienaar, with young Jackson doing well in the build-up.
Ulster had a strong spell coming up to half-time but could not trouble the scoreboard - Gaston went close near the touchline - and Tovey was again wayward with a penalty early in the second half.
The next score was crucial and the Dragons got it to nose ahead at 12-10. The visitors picked up the pace with some powerful running and precise passing and flanker Evans spotted a gap to score from close range.
Tovey added the extras before some lacklustre tackling from Ulster invited winger Will Harries through and a final pass sent Brew over for his sixth try of the league campaign. The Dragons now seemed to have the game within their grasp.
Ulster needed to hit back quickly and they did. Four minutes later, Pedrie Wannenburg fended off a couple of tackles to lay the groundwork for fellow replacement Seymour to sprint through and grab a five-pointer.
The Dragons, who were unbeaten in their last four clashes with Ulster, kept on course with a well-struck penalty from replacement Jones.
But Ulster bounced back with a try near the posts from the fit-again Cave, who was set up by Gaston. Pienaar's conversion took the province into a two-point lead.
The Dragons won possession from the restart and rumbled forward into range for Tovey to drop his goal.
However, that good work was undone when Ulster gathered the ball from the kick-off. Their forwards patiently gained ground before Pienaar's sweet strike decided a game which could have gone either way.
Commenting afterwards on his last-gasp kick, the Springbok star said: "I just looked up and thankfully it went through the posts - it's a great result for us.
"The lads showed a lot of character to win the ball back at the end. We will celebrate this win but it was not a good performance from us and we know there are a lot of tough matches ahead of us."
|Ulster Rugby Score Card|
|Newport Gwent Dragons Score Card|