Go with your feels mate. I'll be honest, I have a Perfecto-styled jacket that doesn't get that much wear because I feel like you do. I love the look but I often feel self-conscious that people will think I'm going for that "tough guy" look.
That being said, if you won't feel self-conscious, that jacket is super versatile and much more classic looking - which may be more your style.
The James is more like a modernized cafe racer w/ a spread collar. Less classic but much more subtle. I will say that a leather jacket is a loud statement in itself so I find that more subtle may be the way to go if you feel similarly.