"I ask for forgiveness in the name of the Popular Party (PP) from all Spaniards for having put people in positions of responsibility who weren't fit for them," said Rajoy in the national parliament.
"I hear and understand the frustration of the people," he added.
"This behaviour is particularly hurtful when the Spanish people have had to put up with so many sacrifices to get our country out of the economic crisis."
Rajoy's comments came after Spanish police on Monday made 37 arrests as part of an investigation into a kickbacks scheme in which politicians and public officials allegedly received illegal commissions from companies in return for political favours including the awarding of contracts for construction and infrastructure projects.
Among those targeted in the raids were Francisco Granados, the former number two in Madrid's PP as well as the party's head of the provincial authority of León, Marcos Martínez Barazón. Several Madrid mayors from the PP are also being investigated.
All have now had their PP membership suspended.
The arrests had made people mistrustful of politicians, said Rajoy on Tuesday, but he stressed that it was unfair to tar the vast majority of people in Spain's political parties with the same brush.
The prime minister was also keen to highlight that the prime motivation of those implicated in the kickbacks scandal was greed, and had nothing to do with illegal party funding.
Those assertion comes on the same day a former secretary general for the PP, Ángel Acebes, appeared in court as part of a long-running investigation into whether the party operated an illegal slush fund.
In 2013, former party treasurer Luis Bárcenas — currently in prison on charges related to another corruption case — alleged he used such a fund to make under-the-table payments to top party figures including Rajoy. The prime minister has repeatedly denied those claims.
Rajoy's apology comes at a very sensitive time for the government with other party members wrapped up in a credit card scandal involving Spain's Caja Madrid and Bankia financial institutions.
Around 80 top executives at the banks, including ex-IMF boss and PP member Rodrigo Rato, allegedly used "phantom" credit cards to make personal purchases which were never declared according to Spain's anti-corruption prosecutor.
Prime Minister Rajoy said on Monday that Rato and all other PP members implicated in the scandal would be expelled from the party.