Both Madrid and Istanbul were tied after the first round — meaning Tokyo topped that round — and it had to be put to a second decisive vote to see who went out.
Istanbul — who have never got so near to winning the right to host the Games in their four previous bids — then garnered 49 of the 94 votes cast to progress.
However, they were to be easily beaten in the second and final round of voting with Tokyo winning the right to host their second Olympics — winning by 60 votes to 36.
It was the third successive time that Madrid failed to win the right to host the Games, finishing third for 2012 and second for 2016.
Madrid's exit recalled the shock of Chicago's first round demise in the vote for the 2016 Games won by Rio de Janeiro four years ago.
Basketball superstar Pau Gasol, a two-time Olympic silver medalist, echoed many people's suspicions that tactical voting by Tokyo supporters had seen them lose out so that they faced a weaker opponent in the second round.
"The vote was beyond our control, unfortunately it did not go the way we hoped.
"Everything seemed to indicate that once the Japanese voters went into vote in the tie-break they voted for the weakest rival so they could win in the final, which makes sense and anyone would do.
"So once we were in a tie things looked bad."
Not even the presence of the heir to the Spanish throne Crown Prince Felipe, who gave a rousing speech, nor assurances by Spanish Prime Minister Mariano Rajoy that the recession-hit economy could get the Spanish capital enough votes to at least reach the second round.
Madrid had made much of their budget being relatively small for the Games as most of the venues and the infrastructure was in place because of promises honoured in the two previous bids.
Rajoy, who had flown from St Petersburg, Russia, after attending the G20 Summit, spoke optimistically about the Spanish economy.
"Madrid offers a safe solid and reliable bid as the IOC Evaluation Commission said," commented the 58-year-old, who has been Prime Minister since 2011.
"Our organisation and finance are unquestionable, Madrid has and always will keep its promises.
"It is probably the most reasonable financial Olympic bid in history. most of the required investment has been made and what little remains is guaranteed by the Government."
Prince Felipe - the flagbearer at the 1992 Olympics in Barrcelona — said he wanted to sense what it had felt like in Spain after the Barcelona Games.
"After the 1992 Games Spain was bursting with pride, the future looked golden," he said.
"We felt that Spain was something good for the world and that Spain was moving forward.
"We need the Games for now as well as for the future."