The foreign ministry will file an official complaint to the British embassy over the incident, a spokesman said.
The incident, which occurred on Sunday, disrupted the work of the ship belonging to Spain's state Oceanographic Institute, the spokesman added.
"We consider that the waters adjacent to Gibraltar are Spanish. The ship was in Spanish waters while carrying out its tasks," he told AFP on condition of anonymity.
A Royal Navy patrol boat, the HMS Sabre, fired flares at the research ship Angeles Alvarino, after attempts to contact its crew by radio failed, according to Britain's defence ministry.
The British navy also prevented the vessel from deploying research buoys into the sea.
Britain believes the vessel strayed into its territorial waters off Gibraltar.
A Foreign Office spokesman said Britain would protest "against the incursion directly to the Spanish authorities at a senior level, making clear that such violations of the UK's sovereignty are unacceptable."
The research vessel, which is investigating geological risks to the seabed, left the disputed area after the flares were fired. It arrived at the Spanish port of Fuengirola on Monday.
"Our work in the Bay of Algeciras has finished. Thank you Royal Navy for accompanying us these days," the crew wrote in a Twitter message.
Gibraltar's environment minister, John Cortes, accused the Spanish scientists of acting as pawns to further Spain's claims over the waters around Gibraltar.
"It hurts to see scientists being forced/tricked into playing dangerous games, falsely in the name of science," he said in a Twitter message.
Madrid ceded Gibraltar in 1713 under the Treaty of Utrecht.
Britain has exercised de facto control of the waters around the rocky peninsula attached to Spain at the mouth of the Mediterranean for most of the time since then.
But Spain still claims sovereignty over the territory and the issue has for years been the source of tension between the two countries.
Britain claims a strip measuring three nautical miles (5.5 kilometres) surrounding Gibraltar as their territorial waters.
But Spain does not recognise any waters off Gibraltar as belonging to the British territory, apart from its ports.