Obama and Rajoy crossed paths on their way in to the current G20 summit in Saint Petersburg.
The US president approached Rajoy and praised the Spanish PM on Spain's economic recovery over the last 12 months , according to the Office of the Spanish Prime Minister.
Obama said he believed that this recovery was the result of reforms set in motion by the Spanish government, the same Spanish sources reported.
Rajoy, for his part, underlined there were was work to be done.
The Spanish leader added it was necessary to keep applying measures to bring the country out of its crisis and create economic and jobs growth.
The conversation — carried out in the presence of Rajoy's translator and Spanish cabinet chief Jorge Moragas — ended in the two politicians shaking hands.
A formal meeting between the two leaders has been on the cards for some time, but is yet to eventuate.
During Thursday's short conversation, Obama apparently referred to that possibility, saying they should meet in Washington in the coming months.
The Office of the Spanish Prime Minister has previously said that such a meeting would take place in 2014.