#and we see AFO smirking at nana whos the new wielder and has a son by this point in time

[mishy-mashy]

Just looking at how All For One failed to steal One For All from Banjo

So Banjo says to Midoriya the below, right

And we see in the flashbacks of the previous users that All For One actually managed to try stealing One For All from Banjo. But his eyes go to his hand, which he then grabs

This is probably where he realized he couldn't steal One For All so easily. Why didn't it work? He touched Banjo, used his Quirk; why didn't his brother come back into his possession?

He grabbed his own hand because he realized something was wrong here.

All For One couldn't steal OFA from Banjo because he didn't have enough emotion to override their wills. This also happened with En, even though we don't see it.

The above is a volume extra. Banjo and En were the current wielders of One For All when they encountered him. One For All escaped him twice when he faced the current wielder (Banjo and En) directly. The others, except Midoriya/Ninth, had been the leftover power after already passing it on.

All For One failed with Banjo because he couldn't override One For All's will. Though it's not shown as much, he also failed with En, but the volume extra hints that All For One actually got ahold of him too.

Trying to steal One For All from Banjo is probably where All For One realized it takes more than the usual touch-and-go to steal it.