In a test of the national mood ahead of general elections expected in November, the conservative Popular Party lost 2.5 million voters in the late May polls from the last local elections four years ago.
Rajoy blamed his party's poor showing on the fact that most Spaniards have still not begun to feel the effects of the nation's timid economic recovery.
"Spaniards have had enough of struggling," he told a party gathering, also blaming a series of corruptions scandals involving top party members, saying "we paid dearly" for them.
While the Popular Party won the most votes overall, its portion of the vote dropped from 37 percent in 2011 to 27 percent this year.
It was also ousted from power in historical strongholds such as Madrid and Valencia.
The Socialists came in second place with 25 percent of the vote, while the new anti-austerity party Podemos, which is close to Greece's ruling Syriza, came in third place.
Rajoy said the rise in support for left-wing parties began during the European Parliament elections last year and continued during the regional and local elections.
He said this was unsurprising as it was the typical pattern after a "deep and long economic crisis".
"The results... were worse in terms of the loss of institutional power," Rajoy said.
The Spanish economy, the eurozone's fourth largest, grew by 1.4 percent last year but the jobless rate stands at 23.8 percent.
Rajoy announced six new party appointments, naming his chief of staff, Jorge Moragas, as the party's campaign chief and replacing the party's spokesman.
Also on Thursday, Catalonia's 37-year-old governing alliance split following disagreements over how to push for the region's independence, leaders said.