The study in the American Journal of Human Genetics is based on a male Neanderthal whose 49,000-year-old remains were found in El Sidrón, Spain.
Until now, researchers have only sequenced the DNA of female Neanderthal fossils, and have found that one to four percent of European and Asian people's DNA can be traced to Neanderthals.
But researchers at Stanford University found that the Neanderthal's Y chromosome is completely lacking in males today.
The Y chromosome is one of two human sex chromosomes (X and Y) and is passed on exclusively from father to son.
The findings suggest that Neanderthal Y chromosomes may never have been passed along when Neanderthals and humans mingled and mated some 50,000 years ago.
That could be because women may have miscarried male fetuses sired by Neanderthals, or produced very few healthy male babies that could pass on this Y-chromosome lineage.
Researchers are probing the hypothesis that modern women's immune systems might have attacked male fetuses carrying certain Neanderthal mutations.
The scientists say they found mutations in certain immune system genes from the El Sidrón Neanderthal that have been blamed for transplant rejection when modern males donate organs to women.
"The functional nature of the mutations we found suggests to us that Neanderthal Y chromosome sequences may have played a role in barriers to gene flow, but we need to do experiments to demonstrate this and are working to plan these now," said senior author Carlos Bustamante, professor of biomedical data science and genetics at the Stanford University School of Medicine.
"We've never observed the Neanderthal Y chromosome DNA in any human sample ever tested," Bustamante added.
"That doesn't prove it's totally extinct, but it likely is."
Previous studies have shown that modern human and Neanderthal lineages diverged between 400,000 and 800,000 years ago.
The Neanderthals died out some 30,000 years ago.