Trailing 1-0 after the first leg in Portugal, Málaga levelled the tie on aggregate when Isco scored with a stunning strike just before half-time.
If that gave the Spanish club confidence, they were left in an even stronger position when Porto had Steven Defour sent off for a second booking shortly after the restart.
And substitute Roque Santa Cruz scored the crucial goal, heading home with 13 minutes left, as Málaga, making their first appearance in the competition, become the third club from La Liga in the quarter-final draw, along with Real Madrid and Barcelona.
"It really has been a strange year for us," said Málaga coach Manuel Pellegrini, with the Spanish club having made just as many headlines for their off-field financial problems as for their European success.
"What characterizes the players is their desire to stay focused on their sporting ambitions.
"I always said that Málaga were not in the Champions League just to make up the numbers," added the Chilean, who took Villarreal to the semi-finals in 2006.
"It would be good to avoid another Spanish team in the quarters, but we will take whatever the draw gives us."
Still unbeaten in their domestic league this season, and leading thanks to Joao Moutinho's solitary strike in the first leg last month, two-time European champions Porto would have been confident of seeing off the challenge of Málaga at La Rosaleda.
Vitor Pereira's side started strongly, too, dominating possession despite the hostile atmosphere. Malaga had not won a domestic game since losing the first leg in Portugal, and they took their time to come into the contest, but a strong finish to the first period saw them come right back into the tie.
Ten minutes of the opening half remained when Málaga produced their first chance of note, Portuguese left-back Antunes letting fly from 25 yards and forcing a fine save from Helton, who tipped the ball over for a corner.
Little Argentine striker Javier Saviola then thought he had given Málaga the lead on the night when he stabbed home from close range, but his effort was disallowed for a supposed foul by Julio Baptista on Helton.
Replays showed that Baptista was unlucky to be penalized by the Italian referee Nicola Rizzoli, but it was a maybe a sense of injustice that drove the home side to level the tie in the 43rd minute.
Manuel Iturra found Isco, who turned and spotted Helton off his line before whipping a brilliant finish into the top corner from 20 yards.
La Rosaleda erupted as the young playmaker gave another display of the talent that has him earmarked for a glorious career.
But the moment that really tipped the balance the way of Manuel Pellegrini's side came four minutes into the second half, when Porto's Belgian midfielder Defour was dismissed.
Defour had already been booked when he went rashly into a challenge on Joaquin, and Rizzoli produced a second yellow card.
The tension was now palpable, with Porto knowing that an away goal would leave Málaga staring at elimination.
But the hosts sensed blood against the 10 men, and Santa Cruz was sent off for Baptista to freshen up the attack for the closing stages.
The big Paraguayan, on loan from Manchester City, needed just three minutes to make the difference, heading low into the net from Isco's outswinging corner.
Málaga were almost there, but there were still some nervy moments to come for the home fans.
Porto thought they had that all-important away goal when Maicon headed in with 10 minutes to go, but his effort was disallowed for offside.
It was a marginal decision, but the correct one, and Pellegrini's side held on to take their place in the last eight.