I'm 5'10 155lbs and have the same issue. Usually find myself in between S and M for most brands.
I prefer how a size S fits when I'm wearing just one layer underneath, but with a sweater it's slightly constricting. Size M fits me just right in the shoulders, but the length and 'boxiness' of an M is an issue for me, unless I wear thick layers. However, at your height and weight I'd say go with the M. If I was a couple inches taller and slightly heavier that's what I would do.
On the other hand, I feel I'm firmly an S when it comes to ACR pants.