An absolute majority of 180 lawmakers voted for the motion to loud applause and shouts of "Yes we can," even if Sanchez faces a tough road ahead as he will govern with an even smaller minority than Rajoy.
The veteran bespectacled 63-year-old leader got up and shook hands with Sanchez before leaving the lower house without a word.
Rajoy had already admitted defeat minutes before the vote, knowing that an absolute majority of lawmakers as diverse as Catalan separatists and Basque nationalists had pledged their support for the no-confidence motion.
"It's been an honour -- there is none bigger -- to have been Spain's prime minister," he told parliament, with lawmakers from his conservative Popular Party (PP) giving him a standing ovation.
Sanchez, Spain's 46-year-old opposition leader, instigated the no-confidence motion last week after a court confirmed that a vast system of bribes given to former PP officials in exchange for lucrative public contracts was implemented between 1999-2005.
After years of anger over the scandals tainting the PP, corruption finally got the better of the party and sealed Rajoy's downfall.
"Today we are signing a new page in the history of democracy in our country," Sanchez told parliament prior to the vote.
But PP lawmaker Rafael Hernando told him he would be entering the prime minister's office "through the back door" after failing to win 2015 and 2016 general elections.
"For the first time we may get a prime minister who didn't win elections," he retorted.
Tough ride for Sanchez
In order to push through the no-confidence motion, the Socialists, who hold just 84 of parliament's 350 seats, had to cosy up to parties they had previously clashed with, like Catalan separatists and the anti-establishment Podemos.
Podemos has already asked to be part of his new government. As such, even if he has pledged to govern long enough to restore "institutional stability" before calling early elections, Sanchez's new government will likely be highly unstable.
Aitor Esteban of the Basque PNV nationalist party, whose support proved decisive for the motion's success, on Thursday warned that such a minority government would be "weak and difficult, complicated."
"This is going to be a constant bing, bang, boom."
A scandal too far
Although Rajoy survived a similar no-confidence vote last year, Friday's ballot draws a line under his rollercoaster time in office which began in 2011 and saw him implementing drastic spending cuts before winning re-election in 2015 and 2016.
Despite winning the last two polls, he lacked the absolute majority of his first term.
He put Spain back onto the path of growth after a devastating economic crisis although unemployment remains high, jobs precarious and many complain inequalities have risen.
His term in office was also marred by a series of corruption scandals involving former PP members.
Last week the National Court, which deals with major criminal cases, sentenced 29 people with links to the PP, including a former treasurer, to a total of 351 years in jail.
It also ordered the party to pay back 245,000 euros ($290,000) received from the scheme to help finance election campaigns.
Rajoy became Spain's first sitting prime minister to give evidence at trial when he was called as a witness in the case last year.
In its ruling, the court said the credibility of Rajoy's testimony "should be questioned".
Mari Angeles Suarez, a 65-year-old pensioner, said she had come to parliament with her husband to witness Rajoy's demise.
"We needed Rajoy to leave his party now, with all the corruption it's accumulated, and we needed change now," she told AFP.
'What moral authority'?
During Thursday's pre-vote debate, Rajoy said the corruption case "does not concern members of the government" and repeated the party's argument that only a tiny number of its politicians have been tainted by corruption.
He also hit back by listing the many graft cases involving the Socialists over the years.
"Are you Mother Teresa of Calcutta? With what moral authority do you speak?" he told Sanchez.