Sounds at or below 70 dBA are generally considered safe. Any sound at or above 85 dBA is more likely to damage your hearing over time. Researchers have found that people who are exposed over long periods of time to noise levels at 85 dBA** or higher are at a much greater risk for hearing loss.
So if you listen music with normal converstation volumes, no issues. The risk factor goes up after that, if the levels are “people yelling” loud. You listen too loud already and hearing damage might happen.