Here are five reasons behind their resistance:
To avoid being Rajoy's accomplice
Allowing Rajoy's conservative Popular Party (PP) to stay in power could alienate leftwing voters who oppose austerity measures introduced by the party and are angry over corruption.
Narciso Michavila, who heads GAD3 polling firm, said the Socialists "would suffer the same fate" as Greece's once-powerful centre-left Pasok party if it backs Rajoy.
Support for Pasok plunged after it in 2012 joined a coalition led by the centre-right New Democracy party.
"Whoever appears to be Rajoy's accomplice will be dead, politically," an advisor to Socialist leader Pedro Sanchez told AFP.
"Essentially what Rajoy is asking is that we approve his investiture, his budget, everything" so he can govern as he did when the party had an absolute majority in parliament between 2011 and 2015, he added.
The Socialists lost power to Rajoy in 2011 at the height of Spain's financial crisis as voters punished its embrace of austerity, and since then, they have also seen their support dwindling following the rise of new far-left party Podemos.
"Many people do not vote for us, not because they don't like what we say, but because they do not believe that we will do it," the Sanchez advisor said.
Internal party polls show Socialist voters and sympathisers appreciate the fact that when the party says 'no' to Rajoy and 'no' to the PP, it sticks to its guns and doesn't back down, he said.
"If we give up this position (and agree to support Rajoy), we leave the field open to Podemos."
Not to be the only defeated candidate
"In Pedro Sanchez's mind, it is very important to be able to say: 'I am not the only one to have lost a parliamentary vote of confidence'," explained a member of the Socialist leader's entourage.
After the December elections, Sanchez tried to form a government, but he only won the support of the centrist Ciudadanos, becoming the first candidate not to win the backing of parliament.
His defeat triggered a fresh election in June.
Return to bipartisanship
The Socialists believe support for Podemos and Ciudadanos, two new parties which have shaken up Spanish politics in recent years, will soften if there is a third election and voters will return to the country's two traditional parties - the PP and the Socialists.
"Sanchez has lost his fear of new elections," the advisor said.
"People will tell themselves: the new politics is nice but it led to an impasse."
If there is a third election, although the PP may win more seats, the Socialists expect to win "at least 110 seats" - up from the 85 they took in June in what was their worst-ever showing.
To consolidate grip on party
Two years ago, Sanchez became the first leader of the Socialists to be directly elected by members, but he has since faced opposition from powerful regional party leaders.
If the Socialists boost their showing in a third election, Sanchez will arrive at the next party congress with his position at the helm reinforced.
"He has never been as relaxed as now. No one dares to ask publicly that he back Rajoy," Sanchez's advisor said.
By Adrien Vicente