But replacement scrum-half Marshall gave Ulster the lead for the first time in the game with a 75th-minute try which, combined with eight points from another late arrival Paddy Jackson, saw them to victory.
The Ospreys had looked to be sitting pretty following a try from Hanno Dirksen and eight points from the boot of Matthew Morgan but Ulster had other ideas.
The early exchanges were cagey with both sides not wishing to give anything anyway and it took until midway through the first half for the first points to go on the board.
Ulster fly-half Niall O'Connor had the opportunity to draw first blood for the visitors but his penalty attempt was wide.
So it was the home side's nerves who were calmed first with a fine individual try from Dirksen on 24 minutes - the 21-year-old winger profiting from some suspect Ulster defending to cross.
Morgan kicked the extras to make it 7-0 but the Ospreys were pegged back when O'Connor found his range with a 32nd-minute penalty.
The Ulster No. 10 could have reduced the margin to just one point on the stroke of half-time with another penalty but he missed to leave his record at one successful effort from three in the first half.
Ulster were made to pay for that poor record almost immediately after the break when the Ospreys stretched their lead to 10-3 in the 47th minute.
Morgan, who was replacing Dan Biggar at fly-half for the clash, kicked a relatively easy penalty.
Declan Fitzpatrick - a fresh face, one of four in the starting XV for Ulster - folded under pressure in a scrum and Morgan made no mistake with the resulting kick.
Morgan was at it again minutes later with another place kick to stretch the lead to 13-3.
With Ruan Pienaar on duty for South Africa earlier in the day Ulster were lacking his presence among the half-backs and O'Connor was withdrawn on 55 minutes.
His replacement Jackson's first action was to succeed where his predecessor had failed, slotting a penalty to put the visitors within striking distance, trailing by seven points.
On the hour mark it was Morgan who was wayward with a penalty attempt with the last 20 minutes seemingly set to be decided by the next score.
Ulster had brought on the big guns with Ireland stars Stephen Ferris and Andrew Trimble joining the fray minutes earlier and the visitors got within four points after Jackson made it two successful penalty kicks from two.
Morgan's replacement Biggar missed a drop-goal attempt as they looked to ease the pressure on themselves in the final nine minutes.
And it proved to be a costly miss when Marshall's quick thinking saw the replacement touch down after John Afoa had been stopped just short of the line.
Jackson added the extras and the Ospreys were forced to taste a second successive defeat.
|Ospreys Score Card|
|Ulster Rugby Score Card|