The Angeles Alvarino, a survey ship operated by Spain's state Oceanographic Institute, entered the disputed waters on several occasions over the past 48 hours and was repeatedly told to leave by Royal Navy vessels, a Gibraltar government spokesman said.
The navy also prevented the vessel, whose crew say they are conducting a survey of an underwater canyon in the area of the Strait of Gibraltar, from deploying research buoys into the sea.
But the ship remained in the area and a British patrol boat on Sunday afternoon fired at least three warning flares as a result of its movements, according to the Gibraltar government spokesman, Britain's Foreign Office and eyewitnesses.
Gibraltar's Chief Minister Fabian Picardo said he had been in regular contact with the Commander of British Forces in the British territory, Commodore Mike Walliker, throughout the weekend in respect of the ship's
"I have discussed all the tactical decisions taken and am satisfied that they have been designed to demonstrate untrammelled sovereignty jurisdiction and control over British Gibraltar territorial waters," he added.
Gibraltar says Spanish vessels regularly stray deep into its territorial waters and the Foreign Office has attacked such incidents as provocative, but it is rare for the navy to fire flares.
A tiny enclave located on Spain's southern tip, Gibraltar has been a British territory since 1713 and is known to its 30,000 residents as "the Rock".
Gibraltar remains a source of British-Spanish tensions, with Madrid long claiming the territory should be returned to Spanish sovereignty.