I'd let him know that if you are not going to be taking any responsibility for his kids then he really should be keeping them to himself....by himself!
Raising kids is a big responsibility and it could be so much easier if the burden is shared equally. Your kids really do not deserve that inequality!
You obviously do not want to ditch the relationship over kids. You have tolerated this for three years so by his reckoning you will continue to tolerate it, right?
Whatever changes you want to make will go at a pace you are comfortable with. I'd start right off the bat with ignoring his 'stare'. I would also put it into your kids' minds that you know what they see and ask them what they think of it and agree with them when it's undeniable and assure them you are working on changing the situations to positives. Reiterate strongly that it's not the kids to blame, encourage all the kids to blend and give them the opportunity to come to know each other. Once the kids are blended and you will see more peace within them, more so if you react more positively towards them, then you can work on the man himself. Ignore his stares, like I said, but be careful of drastic changes that could bring about drastic reactions so softly softly catchee ...
Chastizing kids is not always about negatives, often times kids know they did wrong and asking them what they'd do differently on a 'first offence' is a good way to go. If they repeatedly do the same things wrong then consequences are firmer, right? Also, encourage them to seek advice if they are not sure about doing something, or the outcomes of a plan.
Relationships are often hard work at some point, it's worth investing in a good marraige. Bear in mind the kids grow up! Think about the man you will be with once they are gone and if that's what you want then put in the effort NOW to blend. Sometimes, and I apologize in advance for this, guys are so caught up in stuff that they 'go with the flow' and once he sees the kids getting along he'll be beguiled and go with your program before he knows what's hit him...guys...you know you do it for love right?
More importantly, do NOT take stuff to heart if someone does, or says, something you consider to be negative. It's on them, not you. Wipe the slate, what's done is done, start over with good intentions.

When I started dating my now DH, he had 2 kids who were 9-10, I had 2 kids who were 6-8.

Early on my DH wanted to have a "talk" with my children about how HIS CHILDREN would be treated differently when they came to visit. He wanted my children to understand how difficult it is to be children of divorce, and that sometimes they would be treated "special". Note: my kids' biofather passed away when they were 3 and 5 but apparently since it wasn't "divorce" they didn't have any issues to deal with...

I nipped that in the bud IMMEDIATELY. I am an equal opportunity parent... everyone is treated the same, has the same expectations, accountabilities, rewards and punishments (age appropriate of course).

Although my DH didn't get to have that "talk" with my kids his overprotectiveness and favoritism came thru loud and clear. My children resented not only him, but his children (because of course, as much as the kids tried to get along, his kids always held themselves with a sense of entitlement). It was hard, because my kids could do something and get in trouble. His kids would wait 3 seconds and do the very same thing and nothing was said.

Please, do NOT let your situation get out of hand. Firstly, do you want your kids growing up feeling like they are second-best? And do you want them growing up seeing how you (and in turn, they) are secondary, are held to higher standards, etc?

I don't know the relationship all the kids have with each other, but if his two smell favoritism they will most likely run with it. I gave my DH the ultimatum because I refused to have my children feel like crap in their own home.