Rajoy described the vote as "a total failure" during a press conference he had called late on Tuesday, after mounting pressure for him to break the silence he has maintained since Sunday's vote.
During the appearance, the Spanish Prime Minister also stressed a definitive referendum on the issue of independence for Catalonia was impossible, and that reform of the Spanish constitution was the only option available.
The Spanish prime minister fiercely opposed the unofficial poll which saw 1.86 million people vote for independence out of a total eligible voting population estimated in the range of 5.4 million and 6 million.
Catalan president Artur Mas reached out to Rajoy on Tuesday, saying it was time Madrid entered into a "permanent dialogue" with Catalonia. He said he had written to Rajoy inviting him "to set the conditions for a dialogue that is permanent and as constructive as possible".
The ultimate aim of the dialogue is to hold "a definitive and politically binding consultation" vote, Mas added.
"The issue we are faced with can only be resolved at the top political level."
"He (Mas) has a legal method, but doesn't appear to have the backing to use it."
Rajoy said he wasn't prepared to accept challenges that came from an illegal starting position, and that dialogue on Catalonia was only possible — and only made sense — "when there is a minimum of institutional trust" — a reference to Mas's decision to go ahead with Sunday' poll despite a legal block by Madrid.
Addressing the issue of when Catalonia's public prosecutor would launch legal action against the Catalan president, Rajoy said the office had its powers and that he did not know what the prosecutor was going to do.
"Who am I to say (to the prosecutor) what should be done?"
He added the office had to be respected and he would respect any decisions made. "You have to respect the rules of the game."
"I've completed my obligations. I appealed (the Catalan vote) twice in the Constitutional Court. I was sensible. Many people would have liked me to take more drastic action," said the Spanish leader.
"The organizers (of the vote) might have preferred a different picture with different international ramifications. Taking other decision would have created more difficulties. I think the (Spanish) government did what it had to do," he added.
Asked why Rajoy had not meet with the Catalan president since July, the Spanish prime minister said: "Whenever Mas has wanted to see me I have seen him, whenever he has called me, I have spoken to him. But you can't ask the prime minister of the (Spanish) government to question national sovereignty.
"Most Spaniards would understand that."
There's one undeniable fact: in Catalonia, there are more Catalans than pro-independence supporters.
Hay un hecho innegable: en Cataluña hay muchísimos más catalanes que independentistas— Mariano Rajoy Brey (@marianorajoy) November 12, 2014