Rajoy, leader of the conservative Popular Party (PP), lost the vote of confidence with 180 votes against him and 170 in favour - the same result he obtained during the first vote on Wednesday.
He was backed only by members of his own group, by centrist upstarts Ciudadanos and a lone member from a regional party from the Canary Islands.
Political leaders now have two months from Wednesday's parliamentary vote to explore alternative alliances but if their efforts come to nothing election-weary Spanish voters will be asked to return to the polls in December.
The PP, in power since 2011, won the most seats in elections held in December and June but fell short of a majority both times as voters angry over corruption and austerity flocked to new parties.
Spain has never had a coalition government since it returned to democracy following the death of longtime dictator Francisco Franco in 1975 and the country's four main parties have been unable to cobble together a governing alliance.
In his speech before the second vote, Rajoy made a final appeal to Socialist MPs to support his bid to form a government, and so avoid "running the risk of having to return to the polls."
"We all have the obligation to ensure that Spain does not foot the bill for parliamentary rifts and stubbornness," he added.
'Betrayed Spaniards' trust'
Socialist leader Pedro Sanchez hit back at his PP rival, arguing that allegations of illegal financing and graft that have dogged the conservative party under Rajoy's watch had undermined his credibility.
"You Mr Rajoy, have betrayed Spaniards' trust," he told his rival.
During the second parliamentary vote Rajoy only needed to get more votes in favour than against, meaning he only needed the Socialists who finished second in both elections to abstain.
But Sanchez fears he would lose his own credibility with his supporters if he were to abandon his long-time opposition to Rajoy, according to members of his entourage.
The political impasse could take its toll on Spain's economic recovery and is fuelling voters' frustration.
"Enough is enough. We are the laughing stock of the entire world," said Luis Garcia Montero, a 53-year-old bank worker, as he smoked a cigarette on a street bench in Madrid before going to work.
Although Rajoy remains in office as acting prime minister, he has no power to propose legislation or spend on new infrastructure projects such as roads and railways.
Spain's public works ministry and the state companies it oversees spent 20 percent less on contracts in the first six months of this year than they did in 2015.
Lack of budget
The lack of a fully-functioning government will make it hard for Spain to submit a 2017 budget draft, outlining cuts to bring its budget deficit down to 3.1 percent next year, by an October 15 deadline.
A caretaker government can only roll over its spending plan for 2016. Rajoy will continue to negotiate with other parties to try to form a government and avoid a third election, the prime minister's office said on its website.
His supporters hope the result of regional elections in the tiny Basque Country on September 25 could change the political calculus.
The Basque Nationalist Party (PNV), which has five seats, could support Rajoy nationally if it needs the backing of the PP to govern in the Basque Country following these polls.
The PNV has backed minority PP governments in the past but Rajoy has angered the party with policies it feels weaken its efforts to protect the Basque Country's distinctive language and culture. But PNV leaders warned Friday they would not back the PP even after the Basque regional elections.
For his part Sanchez plans to speak to the leaders of Ciudadanos and new anti-austerity party Podemos to try to unblock the political impasse, a Socialist spokeswoman said.
By Daniel Silva/ AFP