Even though nothing is really shown, that transporter accident scene is one of the most disturbing things I've ever seen in a movie. It kind of stopped the movie cold for me at that point.
I recall seeing it upon release (I was about ten). I had clear memories of Star Wars in mind despite Sci-fi fantasy of that type being rare at that stage, so to see the reality of Transporter accidents (was it even eluded to in TOS?) was shocking. The line back from Starfleet, that the crew didn't want to see what emerged at their end, set my imagination going in overdrive. Classic stuff!
Yes, there was an episode where Kirk splits in two from a transporter accident. One Kirk is logical and calm, the other resorts to man's primitive barbaric nature. They used an alien dog looking thing as a test subject in the transporter when trying to fix the problem. The alien creature dies in the experiment.
I also agree the transporter accident in ST:TMP was one of the most chilling and disturbing things I've ever seen on screen. Yet we never saw in detail the bodies, deformity or guts and blood. Nowadays it would be all in your face Game Of Thrones/Walking Dead gorefest.
Thanks. I'd forgotten about the tests in The Enemy within.