Casillas broke a bone in his left hand while playing for Real against Valencia on January 23 and despite returning to training in recent weeks, has yet to win his place back in Jose Mourinho's side.
Spain coach Vicente del Bosque on Wednesday indicated that he was likely to name Casillas in his squad for the Confederations Cup in Brazil in June.
But speaking to Catalan broadcaster Barcelona Television, Xavi said of his national side colleague's predicament: "I think it's unfair."
Nevertheless, he rejected the possibility of Casillas swapping the Santiago Bernabeú for the Camp Nou.
"Iker will never come to Barcelona. He's very pro-Madrid. I can't see him leaving the club. If he came, obviously I'd happily welcome him," said Xavi.
"But honestly, I can't see him here," said the 33-year-old veteran of Spain's victorious World Cup and double European Championship-winning squad.
On Barcelona goalkeeper Victor Valdés, who in January announced that he will not renew his contract when it expires next year, Xavi said it was unlikely he would have a change of heart.
"I don't think that Valdés will go back on his decision. It's a friend who's leaving, a fixture of the club. He's a player who's fought to win his place but has had perhaps the worse press that I've seen in my career."
Asked about the upcoming Champions League semi-final against Bayern Munich, Xavi echoed the view of the Bavarian giants' skipper Philip Lahm, saying the tie could go either way.
"I think we've got a 50-50 chance in the semis. Bayern are on sparkling form. They've already won the Bundesliga title but we're two big clubs and at this level, everything's about the detail," he added.