Proactiva Open Arms said its vessel, the Golfo Azzurro, was some 24 nautical miles (44.5 kilometres) away from the Libyan coast at the time of the incident, well out of Libyan territorial waters which end 12 miles from the shore, according to websites which monitor maritime traffic.
But they were within a 100-mile zone within which Libya has since last week demanded that it be able to carry out rescue operations. Tripoli wants to deter foreign rescue vessels, which it accuses of collusion with people trafficking networks.
Oscar Camps, Proactiva Open Arms' director of operations, wrote on Twitter on Tuesday: "They are threatening to open fire if we don't head in the direction of their territorial waters. It's detention, pure and simple.
Camps eventually sent another message explaining that the Libyans gave up after "1 hours and 44 minutes of sequestration and tension".
The Golfo Azzurro, a 37 metre-long boat which had arrived in the zone Tuesday morning after refuelling in Malta, then headed north again.
Defend Europe, a group which protests against the migrant rescue boats, also had a vessel in the area and claimed to have heard the Libyan coastguard tell the Golfo Azzurro to "leave Libyan waters immediately or follow them to Tripoli".
"Leave and don't come back," they say they heard the Libyans tell the rescue vessel.
The Libyan coastguard, contacted by AFP, could not immediately confirm the incident.
Proactiva Open Arms already announced last week threats from the Libyan coastguard which said it had fired shots in the air while telling its other vessel, Open Arms, to leave the area.
The Libyan navy announced last Thursday the creation of a new search and rescue zone under its direct responsibility, with access banned for foreign vessels.
But such a decision should in principle be authorised by the International Maritime Organization, according to the Italian Interior Minister Marco Minniti, suggesting the coastguard of chaos-torn Libya do not have the
resources to patrol such a zone.
While waiting for further information, three other aid groups have suspended their operations in recent days.