You will have to allow him to do what is allowed in your custody order.  So, if he gets set time with them.. he can take it.. or he can let his kids be with the grandparents.. unless you have a first right of refusal type of clause where you have to be offered time if he can't be present.. it's his choice.  Just like you would be able to get a sitter for them on your time.

If he is allotted time and has no restrictions on travel with the kids, you can't stop him from taking them on a trip.  He may not have been alone with them when you were together, but I'm assuming at some point since, he has been caring for them at least some on his own time.

Unless you believe he is in an active adiction phase or other danger, you may have to grit your teeth and let him be their parent on his time.  Of course, you can prepare your kids with information to deal with a situation where one of them might get lost.. even if it were on a shopping trip with you for example.  Surely you have had conversations about what to do in those cases to some extent?  

But, I understand, it can be hard to let go and give someone else that control.  But.. if the courts have ordered that he gets unsupervised time with his kids... you need to understand that as their father, he does have rights to them... just like you do.

But, you could certainly arm your kids with information cards to keep that might have phone numbers to call etc..if they get lost?

Do you think your kids are higher risk for wandering off? developmental issues that might make caring for them extra hard?

I was worried about similar. I coped by trying my best to teach my kids to be as "independently safe" as possible.

One thing i was worried about was water. My ex has a pool and often goes other places with pools or near bodies of water. I made sure my kids could swim.

I also did "getting lost" scenarios. When we would go places, i told them that if they were ever lost in public, go up to a woman with kids or a female worker and ask for help.  I made sure they both knew my phone number. Talked about safety around cars, strangers, what to do if there's a fire. If they are sitting in a car outside a store waiting for an adult to run in, lock the doors and don't open the door for anyone for any reason, same with being home alone. I didn't do it in a way to make them paranoid, but just made safety and awareness part of daily life. 

The BM in our life used to use this anxiety to try to keep my SS away from us - only granting us a few hours at a time. Trying to get a single overnight was like pulling teeth and we don't have any of the background your ex has - BM just couldn't handle not being in control. Once we got steady overnights, BM kind of freaked out because she had to trust us to take care of SS. SS is always perfectly cared for at our house, but it wasn't exactly how BM would do it and it was a control thing. We got frequent texts from BM, checking in, making sure SS went to bed on time, making sure SS had a bath, making sure SS didn't eat too much sugar, making sure SS didn't watch too much TV. She was relentless. It took DH setting firm boundaries and several years before BM got the hint that SS was perfectly fine in our care. 

At the end of the day, you at some point trusted this person enough to have children with them, regardless of the circumstances, that has to count for something. He is their Dad and he has rights to see his kids, just like you do. Like others said above, if there are actual, concrete concerns, then that's one thing, but it sounds like you're worrying about the "what ifs" and you can't do that. You may not like how he parents or what he does on his parenting time, but that is strictly a difference of opinions and parenting styles, not an actual threat of harm or danger. Something bad can happen on either parents' parenting time - illness, accident, natural disaster, etc, but it isnt enough to prevent the contact; you wouldn't like it if the shoe was on the other foot and he withheld the kids from you because of these "what ifs."