“I will try to form a government, a stable government,” he told supporters gathered outside his party's headquarters in Madrid.
“We will need to talk a lot, negotiate and reach agreement,” he added as hundreds of supporters chanted: “Spain! Spain!”
The Popular Party won 28.71 percent of the vote, giving it 123 seats in the 350-seat parliament, well short of the 176 seats needed for an absolute majority and down from 186 seats in the outgoing assembly.
The Socialists came in second with 90 seats while two newcomers, anti-austerity party Podemos and centre-right Ciudadanos, came third and fourth respectively.
Podemos captured 69 seats while Ciudadanos got 40 seats, putting an end to over 30 years of dominance by the Popular Party and the Socialists.
The results mean the only parties able to form a majority with Rajoy would be the Socialists or Podemos, both of which are unlikely scenarios.
Going forward, King Felipe VI, the head of state, will nominate a prime minister after holding talks with the leaders of each party that has won seats in parliament.
The nominated leader must then win a vote of confidence in the assembly to take office.
On a first vote the candidate must get more than half of the 350 lawmakers in parliament in order to form a government.
If he fails he must get more votes for him than against him in a second ballot 48 hours later, meaning a candidate can be succesful if parties abstain.
If there is still a deadlock within two months of the first vote in parliament, the king must call a new election.