"His Majesty King Juan Carlos has just told me of his wish to renounce the throne and begin the succession process of succession," Rajoy said during the press conference.
Spain's Crown Prince Felipe is to take over the reins, he added.
The 76-year-old king who took the throne in 1975 was "convinced" of his decision, and would give his own reasons for this abdication, the Spanish Prime Minister said during the brief press conference.
Rajoy described the king as the "best symbol of unity and peace" for the country, and said the abdication process would be carried "in the context of instutional stability".
The Spanish leader called an emergency press conference shortly before 10am on Monday, with immediate speculation in the Spanish press being that the annoucement concerned the position of King Juan Carlos.
Spain's monarch took power after the country had experienced nearly 40 years of dictatorship under Francisco Franco.
Tweet from Spain's Royal Palace announcing the abdication of King Juan Carlos.
La abdiciación de Juan Carlos es una oportunidad extraordinaria para dar legitimidad al trono en un referéndum #democracia— Ramón Lobo (@ramonlobo) June 2, 2014
The abdication of Juan Carlos is an extraordianary opportunity to give legitimacy to the throne through a democratic referendum, Ramón Lobo, veteran Spanish journalist with El País tweeted.
La dimisión del rey ayudará a que el verbo DIMITIR empiece a conjugarse en España #ElReyAbdica— Ramón Lobo (@ramonlobo) June 2, 2014
The resignation of the king will help the verb RESIGN become conjugated in Spain, Lobo also tweeted, a reference to the fact that very few Spanish public figures step down.
Spain's online newspaper El Confidential said the king had resigned to save the monarchy from an institutional crisis:
ÚLTIMA HORA: El Rey abdica para salvar la Monarquía de la crisis institucional.— El Confidencial (@elconfidencial) June 2, 2014
More to follow