One hundred and seventy lawmakers voted for Rajoy, 111 against, and 68 abstained - all Socialist MPs, in line with the party's reluctant decision to let its arch-rival govern rather than trigger a third round of elections in the poll-weary country.
Rajoy pledged to plough on with economic policies deeply unpopular with the opposition which blames austerity measures taken in his first term for rising inequality.
"Do no expect me to... damage economic recovery and job creation," the 61-year-old told lawmakers in a tense pre-vote session, referring to Spain's return to growth under his watch following an economic downturn.
"There is no sense in getting rid of all reforms."
'They don't represent us'
The Socialists' decision to abstain drew stinging criticism from its rivals including far-left Podemos, and divided the party so seriously that Socialist chief Pedro Sanchez was ousted earlier this month.
Hours before the vote, Sanchez himself gave a tearful statement to the media, announcing he was quitting as a lawmaker so he would not have to abstain and allow his staunch rival Rajoy to govern.
Hundreds of protesters gathered near parliament amid a heavy police presence, unhappy about corruption and sweeping spending cuts during Rajoy's first term, shouting: "They don't represent us."
"It's going to be the same government, or similar, (as in) the past four years, which was disastrous for Spain," said Carmen Lopez, a 65-year-old retired computer technician.
In the pre-vote session, party leaders strongly criticised Rajoy and one another - just as they have done for the past 10 months as the country went through two inconclusive elections.
This unstable period saw Spain go from jubilation after polls last December that ended the two-party hold on power as millions voted for two upstart parties - to disillusion following polls in June that returned inconclusive results once again.
Rajoy's Popular Party (PP) won both elections but without enough parliamentary seats to govern alone. As no political grouping was able to agree on a viable coalition, Spain looked set for more elections.
That changed last weekend when the Socialists opted to abstain in Saturday's confidence vote after weeks of in-fighting that saw Sanchez ousted.
Rajoy's party will only have 137 out of 350 seats in parliament and will face huge opposition, forcing him to negotiate every bill.
He originally came to power in 2011 with an absolute majority.
"You are in the clear minority and under tight surveillance of this lower house. The Socialist party will devote itself to monitoring your every step," Antonio Hernando, the Socialists' parliamentary spokesman, told Rajoy.
Among Rajoy's priorities will be the 2017 budget, which may need at least five billion euros ($5.5 billion) in spending cuts to reduce the deficit in the face of EU pressure.
But further cuts are likely to face stiff opposition both in parliament and on the street.
He will also face rising separatist sentiment in the northeastern Catalonia region.
After the vote, Rajoy sought to strike a conciliatory tone.
"If we all make an effort, we can reach agreements and we have to try and turn this difficult and complex situation into an opportunity," he told reporters.
Political analyst Pablo Simon said his term in office would be the most "turbulent" ever in Spain and could prompt Rajoy to call early elections if he faces gridlock in parliament.
But he predicted it would not be as difficult for Rajoy as some have anticipated.
The Socialists will need time to regroup and will not want early elections, knowing they would fare badly after their very public breakdown, he said.
The PP also has a majority in the senate, and may be able to form pacts with smaller parties in the lower house to see laws through, Simon added.
By Marianne Barriaux / AFP