Ancelotti, 55, had lead Real to their 10th elite European title last season but was sacked after the club finished runners-up to Barcelona in La Liga and lost to Juventus in the semi-finals of this year's Champions League.
Club president Florentino Perez told a news conference at the club's Santiago Bernabeu stadium that the services of the former AC Milan, Chelsea and Paris St Germain coach were no longer required.
"The board of directors has taken the decision to relieve Carlo Ancelotti of his duties as coach of Real Madrid," Perez said.
"It has been a very hard decision... but Real Madrid is extremely demanding and we believe it is time to give a new impulse" to the club, Perez continued.
"The time has come for a change of direction that will allow us to win more titles and reach our peak competitive level ahead of a new era," said the straight-talking Perez, who explained that Real Madrid was a worldwide reference and no-one who worked there could expect an easy time.
'Two fantastic years'
The mild-mannered Ancelotti took to Twitter immediately after the announcement to thank the players and fans for the memories.
"What remains for me is the memory of two fantastic years at Real Madrid," he said. "Thanks to the club, to the fans and to my players."
Ancelotti, who succeed Jose Mourinho as Real boss in June 2013, leaves Madrid having failed to convert a record run of victories last year into silverware at the end of this season.
Madrid had won the Club World Cup on a streak of 22 consecutive wins between September and December to add to the Copa del Rey and UEFA Super Cup in a record four-trophy haul in 2014.
But they were beaten to the Spanish league title by a treble-seeking Barcelona and eliminated by city rivals Atletico Madrid in the Copa del Rey.
Ancelotti's last chance of keeping his job was ended when one of his former clubs, Juventus, knocked Real out of the Champions League earlier this month to progress to the final.
That was too much for the club's management, despite Ancelotti's popularity with fans and players.
World Player of the Year Cristiano Ronaldo had even taken to Twitter to back the coach after Saturday's season-ending 7-3 demolition of Getafe: "Great coach and amazing person. Hope we work together next season."
Madrid however are now preparing to welcome a new coach with media reporting that Napoli's Spanish coach Rafael Benitez is among the favourites.
"It would be very good if he speaks Spanish," said Perez, indicating that an announcement would be made next week.
Ancelotti, meanwhile, has said that his future remains in coaching although he suggested he would take a year out and have back surgery if he was dismissed.
"My future is clear in my head," Ancelotti said at the weekend. "I will continue here at Real Madrid or I will stop for a year."
"I must have a back operation because of spinal stenosis that for some time creates a tingle in my hands," he told Italian newspaper Il Giornale.
"I don't know how long the period of recovery will be," he added. "I can assure you that I've had plenty of offers from Italy, England, Germany."
"This is football, especially at a club like Real Madrid it is normal to be questioned after part of the season didn't go well.
"I continue to have confidence in my ability because I like to coach."