“I think the time has come to put a final full stop to this period,” said the 63-year old, removed from office last week in a no-confidence vote after former officials from his party were convicted in a bribery scandal.
“The PP must keep advancing and building its history at the service of Spaniards under the leadership of another person,” he told party members.
Rajoy said a national executive meeting would take place “soon” to call an emergency congress which would “open a new phase for our party” and elect his successor.
He added he would stay on in his post as PP head until the successor is chosen, and gave no date or deadline.
- Rajoy forced out and Sanchez becomes Spain's new PM
- PROFILE Mariano Rajoy: Spain's great survivor… until now
- Rajoy poised to fall in no-confidence vote
Rajoy's ousting was a spectacular turn of events just two weeks after his minority government managed to have its 2018 budget approved despite strong resistance.
His government's pleasure at that achievement was short-lived as a day later on May 24, a court announced it had sentenced former PP officials, businessmen and their spouses to a total of 351 years in jail for their role in a vast bribery scheme known as the Gurtel case.
It was one corruption scandal too many for the PP, which has been hit by a series of graft accusations.
The opposition Socialist party filed a motion of no-confidence, which passed on Friday. Rajoy was removed and replaced as prime minister by Socialist party chief Pedro Sanchez.
Political analysts say the rapid turn of events caught the PP off guard.
“It is pretty clear that they did not really expect this,” Roger Senserrich, a contributor to political analysis website Politikon, told AFP.
“The Spanish right has a very strong tradition, based on previous experiences with leadership replacement, that no one really openly challenges leadership until the leader says he wants to leave.”
Among those tipped to replace Rajoy is former defence minister Maria Dolores de Cospedal, currently deputy head of the PP.
Could Soraya Saenz de Santamaria be the first female leader of the PP?
Others include former deputy prime Minister Soraya Saenz de Santamaria and the president of the northwestern region of Galicia, Alberto Nunez Feijoo, who is close to the ousted prime minister.
In his speech, Rajoy defended his record, saying Spain's economy had largely recovered from the economic crisis that was racking Spain when he came to power in 2011.
Where corruption is concerned, he said many graft cases had “affected” and “shocked” him.
But he pointed out that none of his government members or the PP itself were found penally responsible in the Gurtel case, even if his party was ordered to pay back money that had been illegally obtained.
He criticised Sanchez sharply, saying he had become prime minister without having won any general elections.
“Someone who was systematically rejected by Spaniards when they were asked their opinion in the polls is governing the country,” he said of Sanchez, who lost both 2015 and 2016 elections.
He also criticised the way Sanchez, whose Socialist party only has 84 seats in the 350-strong parliament, cosied up to Catalan separatist and Basque nationalist lawmakers in order to get support for the no-confidence vote.