Winger James glided through for the game's only try in the 16th minute, showing a clean pair of heels to the Glasgow cover after gobbling up a loose pass from Colin Gregor.
The second half was dominated by penalty kicks. Sweeney slotted four of his five attempts after the break, including the insurance score in the 76th minute.
The defeat ended a good run of form for the Warriors, which was sparked by recent European wins over London Wasps and the Newport Gwent Dragons.
Speaking afterwards, their head coach Sean Lineen said: "It was such a see-saw match and it eventually got to the stage that it came down to the team that had the ball least who won.
"Unfortunately it was down to mistakes and discipline that we lost the game. We need to be more intelligent and precise with the key decisions that we make.
"Cardiff had a couple of good runs. Ultimately their try came from our own error, but they deserved the win.
"I thought Peter Murchie played well, Colin Gregor was outstanding - leading by example - and Calum Forrester put in a great performance, being as it was his first game back."
Weir (shoulder) and Forrester (knee) were back in harness for Glasgow as Lineen made a raft of changes from the team that face the Dragons.
Cardiff were also missing players through injury, international duty and suspension. Second row Paul Tito captained a much-changed Blues side, and Deiniol Jones (back) and Xavier Rush (neck) bolstered the visitors' replacements bench.
Glasgow made a promising start when a penalty was awarded to them at the first scrum with Weir making the most of the opportunity.
But Johnnie Beattie then failed to gather the restart kick, allowing Cardiff to attack off the scrum and win a penalty of their own which Sweeney converted to level the sides after six minutes.
And Cardiff took the lead after with a try out of nothing. As Glasgow mounted a prolonged attack, a strong Cardiff tackle put pressure on Gregor who handed possession to James, who hared over 70 metres for the unconverted try that put the visitors into an 8-3 lead.
Glasgow, who lost their last two games to Cardiff, dug in and regained the lead with a couple of penalties from Weir, the second from all of halfway.
Sweeney split the posts again after 35 minutes to put Cardiff ahead once more but Weir's third penalty, after the Cardiff scrum were again penalised, ensured Glasgow went in at half-time with a one-point advantage.
The third quarter saw no fewer than five penalties successfully converted, with Sweeney kicking first and Weir cancelling that out at the other end of the pitch.
The match began to swing in Cardiff's favour, however. Glasgow struggled to maintain their discipline at ruck time, allowing Sweeney to add three more penalties and hand the Blues a 23-15 lead while Rob Harley was sent to the sin-bin.
Sweeney then rounded off a pleasing result for Dai Young's men with another accurate kick, topping off his 21-point salvo and guiding the Blues back up to third in the table.
|Glasgow Warriors Score Card|
|Cardiff Blues Score Card|