I think that a kitchen-table game of D&D is not the battleground upon which the nature of the soul will be decided. Unfortunately for us all, D&D makes testable assumptions about the nature of free will, sentience, and so forth, which end up having effects on game mechanics.
Souls are the province of necromancy. I can think of two possession-type effects: the necromantic magic jar spell and the undead ghost‘s ability of the same name. They work more or less the same way once the effect gets going: charisma to remain in control of yourself, then use your stats with the possessor’s int, wis, and cha; they can use class features, but not yours. Another spell that mentions the soul is the planar-travel-in-safety astral projection, also necromancy. And of course the raising of the dead — though reincarnate does what it says on the tin, it’s transmutation (because of the new flesh); the others are all necromancy.
Souls are a discrete object, though intangible. They have a battlemap position, are the seat of the character’s conciousness (when magic jaring, the spirit is referred to as “you” while the discarded shell is “your body”). The effects that permit soul transfer are all-or-nothing: humanoids only, and you’re either in-and-in-control or failed, no “present but observer only”, no “as-though-it-were-a-grapple-struggling-egos”.
Souls that are generated via magic jar or resurrection are natural and permanent members of the plane (your type isn’t humanoid[resurrected]; your posession isn’t affected by the turn undead ability, you don’t ping detect good and evil or get hedged by magic circles). None of this is true for ghosts. So the special sauce that makes undead the baddies that they are isn’t inherent to being unfleshed spirit — and the effects that let you briefly wander as unfleshed spirit don’t change your type! So it would be a completely acceptable thing (ontologically speaking) to have an incorporeal humanoid spirit that wasn’t undead. Dreamwalkers, say. That we don’t implies some weakness in the natural types: their souls don’t know how to self-sustain without the shield of flesh.
In my campaign’s most recent session, a character got possessed by Lady Bethroine, an ancient Illyrian (I believe in a low power curve: while I’ve been too lazy to do full stats, “DC 15 or so” and “Challenge 16 or so” are in the back of my mind) who, during The Event, turned into a living song. While she no doubt could rage out and take over an opera house (something something conductor something something single-entendre), her altered view of time and desire to see what’s what means some joy-riding first.
I’m not doing it now, but I’m also thinking voodoo practicioners, or the 5e binder. The loa are definitely not ghosts, and are definitely called upon to possess. While riding, there’s a cooperative melded state going on: The being has qualities of both. Actually, I have the middlefingerofvecna.com binder open in front of me now, and it’s a pretty good representation of that if you assume the are selfish: this deific spirit does these things while riding; it makes itself known in these ways, which since you are an adveturer we assume you do your best to suppress and minimize. Especially since the features are only available to binders, who are probably pretty good at the suppression and minimization, that makes sense.
The best match 5e has for this sort of give-and-take I need for Bethroine are sentient magic items. The precise mechanism by which they dominate their wielders (see Sentient Magic Item > Conflict) isn’t clear — is it an enchantment effect, a necromantic effect? Do they have souls to go along with their free will? The control is scalable: free rein, won a conflict, or after winning a conflict, full possession.
That final model is my ideal for Bethroine, because it lets me put a power level on her: being ridden by her is like having attunement to an intelligent item which you can’t put down.
Actually, thinking about it, so is the madstone, that little statuette Bippi picked up. Huh. Remove curse lets you sever the attunement, but since Bethroine would also require removing the unwelcome lodger, that’s going to get sticky.
Bethroine, the living song
Int +3 Wis +1 Cha +6 (Ego 17)
Communication: Verbal (1 language) but prefers emotive
Senses: Hearing out to 120 feet
Alignment: Chaotic Neutral
Special Purpose: Glory Seeker
During a conflict at low level she forces the ridden to sing about the current state of things so she can understand what’s going on. At high level, she assumes direct control.
Features (all save DCs 17, cast at the 16th level of ability):
Cast message (no boredom chance, see below) and vicious mockery at will.
Cast hideous laughter once per short rest.
Use an action to gain blindsense 30 (auditory) for 1 round.
Cast confusion once per long rest.
In all cases, the effect is song-based and somewhat infectious; the message is a snatch of music (no words, just emotion); the hideous laughter and confusion effects force their victims to sing their hearts out while performing a dance number, etc. There is a 5% chance on each ability use that Bethroine will select a new mount or become corporeal to participate herself — this is not a good thing.
Int -1 Wis -1 Cha -1
Senses: vision 30 feet normal
Alignment: Chaotic Neutral
Special Purpose: Increase Excitement
The madstone is a wand of wonder which seeks to create chaos. Each time it is used, the wielder must make a DC 12 charisma save to avoid acquiring a short term insanity. Once per long rest, it may extract this cost itself by winning an ego contest (same DC).