"He sold hashish sometimes, like many do, to buy something to eat, pay the rent, but he was not a trafficker," said Mohamed, as he drank mint tea at the Gibraltar bar in Lavapies, a multi-ethnic neighbourhood where El Khazzani once lived.
"He was a good guy, normal, who played football with us right here. It's strange that he could have done this," added the 34-year-old Moroccan builder.
El Khazzani, 25, is being questioned by French anti-terror investigators after he opened fire on a high-speed train on Friday evening before being overpowered by passengers.
The alleged attacker maintains he was only trying to rob passengers because he was hungry.
"He rarely went to mosque," said Hicham, another 34-year-old Moroccan builder as he stood at the counter of the bar, before adding that El Khazzani "sometimes" sold hashish.
El Khazzani, who was born in Tetouan in northern Morocco, was living at the time with his father, a scrap dealer, in a three storey building with a pink facade on Cabastreros street that is now occupied by Spanish and Senegalese tenants.
A Spanish court issued an arrest warrant for him in July 2014 to serve a six-month jail sentence issued by a court two years earlier for having sold seven grammes (0.25 ounces) of hashish to a 17-year-old when he was just 19 years old.
When Spain entered into recession in 2008 jobs became scarce in the neighbourhood.
"Of 50 young Moroccans, I know just ten that have a permanent job," said Allal Taouriarat, 58, who sells used items in Madrid's weekly Rastro flea market.
"Many ended up leaving for France or Belgium," he added.
El Khazzani lived in Spain between 2007 and 2014, leaving Madrid for the southern port of Algeciras which is just a 90 minute ferry ride from northern Morocco, before going to France.
Algeciras has a huge population of Moroccans, many who have moved there from other parts of Spain and across Europe "because of its proximity to their country and their families," said Jose Angel Ponce Lara, a sociologist who works to help integrate immigrants in Spain.
Mohamed said he last saw El Khazzani in Madrid a couple of years ago.
"He had come from Algecerias for a trial in Madrid, he had a small beard but I did not think he was radical," Mohamed said.
The Lavapies neighbourhood where El Khazzani lived is slowly becoming gentrified, with trendy cafes located beside shabby travel agencies advertising cheap trips to Bangladesh.
But it remains linked in Spaniards' minds to the 2004 Madrid train bombings in which 191 people were killed and more than 1,800 injured when bombs hidden in duffle bags ripped through four crowded commuter trains.
One of the 21 people who were found guilty in 2007 of involvement in the attack, Jamal Zougam, ran the shop in the neighbourhood where most of the mobile phones used to set off the bombs came from.
"Before it was a troubled neighbourhood, neglected by the authorities, but recently, since the zone draws tourists, we are very much controlled by the police," said Taouriarat.
There are five "small mosques" in the neighbourhood set discretely in the basements of buildings, he added.
"I have gone into all of them and I have never heard a speech inciting terrorism," said Taouriart.
"What we need to stop are those who brainwash our youths in the name of God. To then use them as free soldiers," added Said, a 39-year-old barber, at the terrace of the Gibraltar bar.
By Laurence Boutreux