My mod fixes this by having a plugin remove the problematic dialogue entries. Because of this, Serana, unlike most NPCs in the game, will basically tell you to go fuck yourself if you ask her to come with you when you already have a follower. Skyrim was coded with the assumption that the player would only have one follower at a time. Unfortunately, this AI also clashes with mods like Extensible Follower Framework, which enable you to have more than one follower at a time.
She’s a rather complex entity, using her own AI to help quests in Dawnguard manipulate her follower status and “lock” her to the player when needed. I recently made a Skyrim mod to fix an annoying bug with Serana, one of the Dawnguard DLC’s characters.