The family from Ecuador were evicted after they were unable to keep up rental payments on their flat in Madrid's Villa de Vallecas district, Spain's El País newspaper reported.
Local authorities said they had offered assistance to the family but that they had turned down that help as they planned to move to Madrid district of Parla.
It's a claim denied by Spanish anti-evictions lobby, the PAH, which has campaigned long and hard for the family to be able to stay in the apartment.
PAH members said the family of five moved into the apartment three years ago but had seen their rents treble to €700 ($780) after a "vulture fund" bought up the property.
Madrid has sold a total of 1,860 apartments originally set aside as social housing to Fidere, the Spanish subsidiary of US hedge fund Blackstone.
"They didn't refuse to pay but asked the rent be in line with their financial situation (their only income was a subsidy of €341 a month) so that they could pay their debit when their economic situation improved," Feli Velázquez from PAH told El País of the family's situation.
The family were spared twice but on Friday morning they were evicted.
"Fidere told us they weren't [the Spanish charity] Cáritas and they had bought the apartments to make money," the PAH activists said.
Police allegedly used "plenty of violence" evicting the family, while a lawyer acting on their behalf said the eviction was a form of "social terrorism".
Only one percent of Madrid's roughly 2.6 million homes are public or social housing, compared with over 30 percent in cities such as Amsterdam, Berlin and Vienna, Antonio Gonzalez Gordillo, a Socialist lawmaker in the Madrid regional government and a specialist in urban planning told news agency AFP recently.