Rajoy was grilled in parliament after documents published in Spanish media showed he and other members of his conservative Popular Party had received kickbacks from construction companies.
He survived mounting calls for his resignation by denying he had ever received any cash but admitting he had been wrong to trust his party's former treasurer Luis Barcenas.
According to the survey published in El Mundo newspaper, 72.1 percent of Spaniards think he "is not telling the truth."
Among supporters of the Popular Party, which he has led since 2004, 43 percent do not believe he was truthful when he defended himself in parliament on Thursday.
More than 90 percent of opposition supporters think he lied and close to 60 percent of all people surveyed say he should step down.
The opinion poll was carried out by the Sigma Dos institute on a sample of 1,000 people and has a margin of error of 3.13 percent.