I think D&D made a big mistake making all Tanar’ri demons back when the Satanic Panic loosened sufficiently to permit printing the word “demon”. I make no bones about being unimpressed by the Yugoloth; the Devils as a mass noun have always been kinda weird sounding, and the Demons are a grab bag that just shouldn’t be made coherent. So here’s how I’d structure it, if I didn’t have 30 years of D&D canon to support.
Lots of things are currently fiends that shouldn’t be. Consider the Yochlol: she’s a fiend (a demon!) because Lolth is the Queen of the Demonweb Pits; as an Abyssal ruler, she needs Abyssal vassals. But everything about them is more aberration than fiend: they’re tentacled shapeshifters with mind reading powers and a poisonous cloud form. She wouldn’t even be the only demon prince(ss) served by non-demons: Graz’zt is the lord of lamia, for Goodness’ sake.
Another example: the palette-swap quasit and imp. If Imp, like Succubus (or Mephit), were not pinned to the Devil or Demon faction, we’d have more room for little irritating flappy guys. The way I’d split it is to have the monster category called Imp, and then focus on why I want different types. The serves-as-familiar-shapeshifts-poison-bite-invisible Quasit can eat the current Imp as far as I’m concerned, in this merger. The red-colored flappy guy should get a little pitchfork and just be a flame-tossing winged kobold (“Least Imp” or “Courier Imp” maybe). I’d probably put the spined devil in this type too (“Spike Imp” or “Spine Imp”). So maybe there isn’t even conceptual room for both types of mini-devil.
A third: the completely pointless dretch-and-manes-and-lemures. We not only don’t need all 3, but if the original monster manual had just said to go use a zombie for the gribblies of the abyss, we’d all have been fine with that. A zombie with fire resistance? okay, done. These monsters don’t unlock plot; by being “living beings” they at best bootstrap a bottom rung of the satanic evolutionary hierarchy, but that would be true no matter where you put the starting point: “One out of a million evil souls is so sustained by this wickedness that, rather than manifest as an undead corpse, they become a Hezrou with their former memories intact” works just as well.
This is 5e. If we sweep away the rest of the alignment-based one-plane-one-monster, what do we get?
Succubus/Incubus: These medium-sized infiltration-y fiends are as presented.
Night Hags: Ditto, these plotters and schemers stand alone. It occurs to me that we have fey hags and fiend hags; we need aberration hags to complete the warlock-patron set.
Kyton/Chain Devil: these get extracted to stand alone, too. Pain fiends, hunters and punishers. You could actually stick the Erinyes as avengers in this faction perfectly well too, if one wanted to; their association with ropes of entanglement is quite like the Kyton’s animate chains, and while they lack the Unnerving Mask, I’d give them a frightful aura (the effect of the ‘mask) without a second thought. They’re probably a manufactured, young race: Asmodeus’ created servitors or something. They’re also probably made out of humanoids via some horrific process.
Now the hard work. Devils and Demons are bound up with each other, defined in each others’ absences. The Devils on the whole have the better story beats, because the Abyss is where you (traditionally) stick weird monsters; see previous discussion on the Yochlol.
I like the Blood War, and I like its two factions well enough. I don’t like them as Devil and Demon, that feels wrong because it positions the entire races against each other and elevates the Blood War to a higher importance than worldly affairs. What I prefer, I think, is an all out war of Hell trying to conquer and colonize the Abyss, and the Abyss reacting violently on that front. I want the Abyss itself to be the battleground most of the time in this war, because it has such exciting beachheads. That is, every Devil is associated with the Blood War, but most Demons are not. The World is another conquest, and it’s the battlefront between Angels and Devils.
Team Baatezu: Make it out of two parts: Fallen Celestials and Native Baatorians.
The Fallen are the pit fiends, horned devils, and barbed devils (corresponding to solars, planetars, and deva before the fall). I suppose technically there should be a path linking deva to rakshasa (see below) also, but that’s D&D’s fault for misappropriating names. I like the reincarnating celestial being which becomes fiendish if it should falter from the path of righeousness, but the D&D deva isn’t that. These beings are probably frequently summoned: back before they fell they were friendly angels, after all, so their phone numbers were on file; the act of falling probably makes responding to the call much more optional, however.
I want my Fallen to have blasphemy powers. Maybe the mummy lord’s blasphemous word: each mortal creature within 10 feet of the speaker must make a Constitution saving throw or be stunned until the end of the speaker’s next turn, success granting 24hr immunity. Maybe just frequent access to spellcasting.
The Native Baatorians are, in D&D lore, the Ancient Baatorians, Scary Shit from Beyond the Dawn of Time, theoretically represented in modern products by the “Nupperibo” (another entry in the dretch-manes-lemure chain, pfeh). I will keep that idea around, but there’s no reason they had to die off: the Ice Devil and Bone Devil are their inheritors. There might be some others that fit well here, like Locust Devils (… Chasme, repurposed) or all of the Yugoloth, but I dunno, that might be a bridge too far.
Team Tanar’ri. These are young (the only ancient things in the Abyss are individuals and their domains. Demogorgon can be old, but Demogorgon’s servitors don’t last long enough), and exist in contrast and in order to oppose the Baatezu. There’s only a few demons which mention armies in their entry: Balor (generals), Goristro (seige engines), Hezrou (footsoldiers), Marilith (captain). So these and only these (barring anything really appropriate coming along) are the anti-Baatezu faction. Every other kind of demon is just that, a unique demon.
Rakshasa: I’d like more rakshasa at different power levels, but these lords of seeming are fine. Add more please. Per the MM, these are actually a type of devil: they’re linked to the Nine Hells. I could do without that myself, but at least it gives me a path (deva -> barbed devil -> rakshasa) that matches the cycle of reincarnation. A Rakshasa-couatl (sort of like a naga, I guess?) would be great. I’d move them out of the Nine Hells and into the
Team Yugoloth: As I mentioned, as far as I’m concerned these are Baatorians gone off the reservation. Doesn’t really contradict existing lore; Asmodeus and the Night Hags created them, but that could as easily refer to an organization as a species. I’m inclined to give them mastery over the Styx (at least aspirationally), explaining their mercenary nature somewhat, since they’re everywhere you want to be.