Rajoy, the leader of the conservative Popular Party (PP), lost the vote of confidence with 189 votes against and 170 in favour after an often acrimonious debate in parliament.
He needed an absolute majority of 176 seats in the 350-seat assembly but was backed only by his own party, the centrist Ciudadanos and a lone MP from the Canary Islands.
The PP, in power since 2011, won the most seats in elections held in December and June but fell short of an absolute majority both times as voters angry over corruption and austerity flocked to new formations.
Another confidence vote will be held on Friday in which Rajoy only needs more votes in favour than against but there is little to indicate he will win that one either.
The Socialists, who came second in the polls, have steadfastly refused to back Rajoy. Socialist leader Pedro Sanchez told the assembly Rajoy "has no credibility".
If there is no breakthrough two months after Wednesday's parliamentary ballot, vote-weary Spaniards will be asked to return to the polls on December 25th - the date determined by timings laid out in Spanish election law.
But the Socialists and other parties have already proposed shortening the campaign so a repeat election could be held on December 18, in a sign that fresh polls are looming.
"Given the parties' entrenched positions, a third round of elections" is now the most likely scenario, said Antonio Barroso, a Spanish political analyst at Teneo Intelligence, a think tank in London.
'Damage to democracy'
Rajoy accused the Socialists of "stubbornly wanting new elections" and warned the political deadlock risks jeopardising Spain's economy and even raises questions about democracy.
"It is hard to think of anything that could cause more damage to Spanish democracy than telling citizens that their vote has been useless on two occasions and that a general election needs to be held for a third time," he said.
Spain suffered a painful economic slump when a housing bubble burst in 2008 but has bounced back to become of the eurozone's most dynamic economies, growing for 12 straight quarters.
But the political stalemate has begun to show signs of taking its toll on the economy.
Although Rajoy remains in office as acting prime minister, he has no power to propose legislation or spend on new infrastructure projects such as roads and railways.
Spain's public works ministry and the state companies it oversees spent 20 percent less on contracts in the first six months of this year than they did in 2015.
The lack of a fully-functioning government has also delayed the drafting of a 2017 budget, which needs to be approved by mid-October to comply with European Union rules.
And weary voters are growing increasingly frustrated with politicians and their inability to compromise to form a government.
"They are have no shame, they only look after their own good, not the good of the country," said Jesus Ruiz, a 48-year-old builder as he repaired a fountain near the 17th century palace in Madrid that houses the foreign ministry.
One factor that could change the political calculus is the result of regional elections in the Basque Country on September 25th.
The Basque Nationalist Party (PNV), which has five seats, could support Rajoy nationally if it needs the backing of the PP to govern in the Basque Country following the regional polls.
The PNV has backed minority PP governments in the past but Rajoy angered the party during a debate in parliament by focusing on the defence of Spanish unity.
"You did not make a minimal effort to win our support," Aitor Esteban, a party MP told Rajoy in parliament.