Just because you get symptoms slowly doesn't mean it isn't neurodegeneration. Many neurodegenerative diseases like Alzheimer's and Parkinson's come on slowly over the course of years and even decades.
Last week a member of this forum posted a link to the first published study I'm aware of that proved hallucinogens can be neurotoxic: https://www.researchgate.net/publication/14821234_Degeneration_of_Purkinje_cells_in_parasagittal_zones_of_the_cerebellar_vermin_after_treatment_with_Ibogaine_or_harmaline
So we now know hallucinogens have the ability to cause necrosis, aka cell death. In addition, we also know many people who get HPPD suffer from anxiety. Science has confirmed that with increased stress follows increased cortisol which creates a surplus of the transmitter glutamate: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3181836/
In addition, we know glutamate produces free radicals which attack and kill brain cells: https://link.springer.com/article/10.1134%2FS000635090905011X
But glutamate is also excitotoxic, activating and killing NMDA receptors, which are the same receptors targeted in certain hallucinogenic drugs: https://www.sciencedaily.com/terms/excitotoxicity.htm
Finally, it's thought glutamate is the missing link to signal necrosis after ischemia (loss of oxygen) wherein during even a very brief episode of blood loss glutamate is activated and thus triggers cell death: http://www.sciencedirect.com/science/article/pii/S0301008213001305
On the topic of blood flow, it's also been noted how LSD and other hallucinogens can decrease blood circulation in certain parts of the brain: http://www.nature.com/news/brain-scans-reveal-how-lsd-affects-consciousness-1.19727
I'm just saying, between anxiety and chemical drugs there's plenty of potential for excess glutamate, loss of blood flow and a possible triggering of necrosis. When I saw a neurologist this past winter he said he didn't know what happened to me but that I had symptoms of a stroke. I also know that in the weeks leading up to taking LSD I had a period of extreme anxiety where I didn't hardly sleep for about 10 days. It was without question the worst acute stretch of anxiety I've ever had. So for me there almost had to be some sort of a glutamate overload, and I assume for those who suffered from anxiety already prior to taking drugs or had a really bad trip that triggered an incredible release of anxiety and glutamate there's a good chance they were in the same boat as I was.