Rato, a former Spanish economy minister, was found guilty in 2017 of paying for personal expenses with credit cards put at his disposal when he was the boss of Caja Madrid and Bankia, at a time when both banks were in difficulty.
The case shocked Spain, where it was uncovered at the height of the crisis that left many people struggling financially. Bankia later had to be nationalised.
The 69-year-old had been free on bail pending an appeal but Spain's Supreme Court last month confirmed his sentence of four years and six months.
“I ask for forgiveness to society and those people who may feel disappointed or affected,” Rato told reporters before entering the Soto del Real prison north of Madrid.
A spokeswoman for Spain's prison system confirmed Rato entered the jail just after one pm (1100 GMT).
He was economy minister and deputy prime minister in the conservative government of Jose Maria Aznar from 1996 to 2004, before going on to head the International Monetary Fund until 2007.
His subsequent career as a banker in Spain was short-lived — from 2010 to 2012. But apart from the undeclared credit cards, it also led to another banking scandal considered the country's biggest.
Thousands of small-scale investors lost their money after they were persuaded to buy shares ahead of the flotation of Bankia in 2011, with Rato at the reins.
Less than a year later, he resigned as it became known that Bankia was in dire straits.
The state injected billions of euros but faced with the scale of losses and trouble in other banks, it asked the European Union for a bailout for the entire banking sector and eventually received €41 billion.
Rato is due to stand trial in November, accused of falsifying information about Bankia's finances.
He is the third former IMF chief to fall foul of the law.
His successor Dominique Strauss-Kahn was tried in 2015 on pimping charges in a lurid sex scandal, and was acquitted.
And Christine Lagarde, who took over from Strauss-Kahn and is the current IMF chief, was found guilty of negligence over a state payout to a tycoon when she was French finance minister, though she received no penalty.