"The 'no' to Mr Rajoy... has never been more justified," Pedro Sanchez told reporters after the Socialist party (PSOE) lost ground in polls in Galicia and the Basque country on Sunday.
Sanchez's refusal to endorse his longtime rival heightens the chances of an unprecedented third general election in a year, analysts said.
Spain, the eurozone's fourth largest economy, is being run by a caretaker government after elections in December and June saw Rajoy's conservative Popular Party (PP) win without an absolute majority and other parties fail to forge a rival coalition.
Rajoy's PP is six seats short of the absolute majority of 176 seats it needs in a parliamentary confidence vote, even with the support of centrist party Ciudadanos, and one extra seat from a minor Canary Islands party.
A Socialist abstention would be enough to enable a PP-led minority government under Rajoy, who won most votes in the two general elections but each time fell short of an overall majority.
Parliament must usher in a new government by October 31st otherwise new elections will take place, around Christmas.
Sanchez repeated his desire to form a "government of change" that ousts the PP from power and said he wants the PSOE to hold primaries on October 23rd in which he will stand again as leader.
"There is enough time for the debate to be held... and for us to form an alternative government," he said.
In Sunday's elections, the PSOE lost ground to new anti-austerity party Podemos, which is seeking to replace it as Spain's main party on the left. The PSOE finished fourth in the Basque region, behind Podemos, and was tied in Galicia with the En Marea coalition which includes Podemos with 14 seats each.