Rubalcaba announced he was calling a special congress on July 19th and 20th which would see a new secretary general chosen.
The opposition leader said that he would remain in the role of secretary general of the party "until it is decided who will replace me".
"I assume my responsibility for electoral results that were bad," Rubalcaba told a news conference after his party lost nine European Parliament seats in
The PSOE — one of the two largest parties in Spain along with the ruling conservative Popular Party — won only 23 percent of the vote in the European elections, sparking speculation Rubalcaba would bow out.
The Socialists now hold just 14 seats in Strasbourg out of Spain's total of 54.
Rubalcaba described the results as "very worrying" for his party.
"Later I will hand over power to another secretary general to be chosen by the congress," he said.
Rubalcaba, a veteran of Spain's political system and former deputy prime minister under José Luis Rodríguez Zapatero took over the reins of the socialist party in 2011 and presided over the worst electoral defeat the party had ever suffered.
The party only won 28.8 percent of the vote as Spaniards punished the party that had attempted to steer the country through a deep economic crisis.
Rubalcaba has failed to persuade voters that he is different from Zapatero, who stepped down in 2011 and who many accuse of not acting swiftly enough to fend off an economic crisis sparked by the collapse of a property boom that has left one in four people out of work.
In the early general election of November 2011 Rubalcaba led the Socialist Party to its worst result since Spain returned to democracy following the death of longtime dictator General Francisco Franco.
The election was won by Rajoy's Popular Party in a landslide.
Under his watch the Socialists have consistently trailed the Popular Party in opinion polls despite noisy street protests against government austerity measures and painful economic reforms such as a reform of the labour code that makes it easier to fire workers.
In Sunday's European Parliament elections backing for the Socialist Party was down to 23 percent. The Socialists; top candidate in the election, Elena Valenciano, said the results were "hard, difficult".
"What is clear is that we have not regained citizens' trust, in part because lots of people are having a hard time, and they have been having a hard time for a while. In fact, they started having hard times when we were in government," said Rubalcaba.
"We have not regained their trust because people are having a hard time and there are people that remember that this started when we were in government," he added
When the Socialists took power under Zapatero in 2004 Spain was riding a construction boom fueled by low interest rates and foreign demand for vacation homes along the country's extensive costline.
But when the international credit crisis hastened a correction that was already under way in the property sector in 2007, millions of people were thrown out of work and banks and homeowners became engulfed in debt.