The owner of a familiar changes if you either remove the familiar owner from the party or a character in a party slot lower than that of the familiar owner.
Example: Using default party of Gorris, Baern, Cadriel, Kirika, Ilauna and Felicia.
1. Let Ilauna summon a familiar. (Current familiar owner is Ilauna.)
2. Kick out Baern. Save and reload the game. (Current familiar owner is Felicia.)
3. Kick out Felicia. Save and reload the game. (Current familiar owner is still Felicia, even though she isn't member of the party. The familiar will still follow the party however.)
4. Let Baern rejoin the party. Save and reload the game. (Current familiar owner is now Baern.)
Since any party member can leave and rejoin the party at any time, you can control who will be the familiar owner in the end.Expected:
Familiar owner should always stay the same even when removing characters, including the familiar owner, from the party.