Well you’re my friend
and can you see
many times we’ve been out drinking
and many times we’ve shared our thoughts
but did you ever, ever notice
the kind of thoughts I got?
Bonnie ‘Prince’ Billy, I See A Darkness
It’s an idea specific to the Vampire games, establishing firm rules for how vampires act in stories with vampire protagonists. Lestat prefers the blood of evildoers but is happy to kill several victims every night regardless of their morality, while Louis agonises over every death. Mae compartmentalises while Caleb just can’t.
The threat of becoming a mindless beast if you can’t find a balance is, I believe, new to the genre. (And doesn’t really come into force fully until Masquerade second edition, as Chicago By Night has some functional Humanity 0 vampire NPCs.)
The closest precursor is outside the genre in the Cyberpunk RPG, where Humanity is a control over how much cybernetics you can pack in and also influences how well you deal with others. It also has roots back to the Dark Side in WEG’s Star Wars RPG which is almost entirely based on behaviour, and the Sanity system in Call Of Cthulhu - which is harsher, as you roll Sanity to resist losing it, so the rolls get harder as you progress. Humanity is easier to keep at low levels unless you drop far enough to lose Virtues as well.
It serves to channel vampire PCs away from using their powers to run roughshod over humans and leaving a trail of bloodless bodies, and can lead to players being very careful about actions that would threaten their Humanity. This can lead to highly moral PCs, and sometimes to frustrated players. Shifting what is allowable can have substantial effects on play style. Make it okay to fight other vampires, for example, and the game will probably feature more Kindred violence...