Sanchez, whose Socialists came second in an inconclusive general election on December 20th, was seeking parliament's approval to become prime minister.
He needed an absolute majority in the vote in the 350-seat lower house of parliament, but lost, with 219 votes against him, 130 in favour and one abstention.
Both acting Prime Minister's conservative Popular Party (PP), in power since 2011, and new far-left party Podemos, a close ally of Greece's ruling Syriza party, voted against him as expected.
Sanchez only secured the support of his own party, which won 90 seats in the election, and new market-friendly party Ciudadanos which has 40 seats.
The Socialist leader, a 44-year-old career politician dubbed "El Guapo" ("Mr. Handsome") for his good looks, will have another chance on Friday in a vote in which he would only need a simple majority.
The December 20th elections resulted in a hung parliament divided among the four main parties - none of which won enough seats to govern alone.
The PP came first in the polls - which put an end to Spain's long-running two party system - but lost its absolute majority, taking just 123 seats.
Rajoy gave up attempts to form a government after he failed to get support from other parties fed up with years of crisis and corruption scandals plaguing his grouping.
So King Felipe VI asked runner-up Sanchez, whose Socialists scored their worst result in history, to form a government.
After weeks of negotiations and ping-pong televised statements, he sealed a deal last week with Ciudadanos.
Podemos, which with its 65 seats would be a valuable partner for Sanchez, immediately suspended its talks with the Socialists over their deal with Ciudadanos, which it considers too liberal.
Rajoy dismissed Sanchez's bid to form a government as "theatre" since he did not have enough support and said it was aimed at blaming others if Spain needs fresh elections.
"This is a fictitious, unreal candidature," he said during a debate in parliament ahead of the confidence vote.
Rajoy also warned that a Socialist-led government would threaten the country's economic recovery since Sanchez had vowed to undo many of his government's key reforms.
Spain's economy grew 3.2 percent in 2015, one of the fastest rates in Europe, although the unemployment rate remains high at 20.9 percent.
Podemos leader Pablo Iglesias said his lawmakers would be voting against Sanchez because the Socialists did not appear set to lead a genuine leftist government.
"Negotiate with us, stop obeying the oligarchs," the pony-tailed political science professor said.
Sanchez appealed to leftist lawmakers on Tuesday to join him in alliance to oust the PP, arguing in the assembly that Spain voted decisively for change in last year's election.
He said a Socialist-led government would enact progressive measures such as a minimum wage increase and a gender wage-gap law, and stressed that all sides would need to compromise as no single ideological group has enough seats for a majority.
"Why don't we get together and pass as many reforms as possible to solve the most urgent problems of Spaniards?" he asked the assembly.
If Sanchez fails the second vote, a two-month countdown will then start from Wednesday, during which the wildly diverging parties will try once again to come to an agreement to govern Spain.
Failing that, new elections will be called, most likely on June 26th.
That would leave Spain in political limbo for several more months just as the country emerges from a severe economic crisis, and also faces an independence threat in the northeastern region of Catalonia.
Recent polls suggest that if Spain goes back to the polls, the four main parties would obtain roughly the same result as they did in the December election.
No candidate for prime minister has failed in both confidence votes since Spain returned to democracy following the death of longtime dictator General Francisco Franco in 1975.