First time happened to me ! Looks like Cain liked me for saving him from the cage and shared his portal with me I really liked it !
Hah! I love it when I can get this to happen, it's a well-known little secret. If you position yourself exactly underneath his gibbet where his body would land and then click on the cage, it will descend and he'll vanish but leave a free portal back to town. Guess he likes his personal space more than you