I need to speak to someone in a castle dungeon. But they won't let me in. How do I go about this? [Bannerlord]

I am trying to enter this castle for the main quest (Talk to 10 lords) but its lord has a real lance up his ass or something.

He has a prisoner in his castle which I need to speak to, except its policy not to let anyone in.

So I try to talk to the lord himself, to perhaps earn favor. But he's inside of this castle too!!

POV: