I think it's perfectly understandable that you would be nervous, given that his daughter has clearly been alienated by a toxic BM, and her few appearances in your life have been accompanied by stress and drama.  I think it's important to have an open talk with your DH about the situation and your concerns and expectations.

I have to ask - how did BM know about all these important events in your lives if SD wasn't ever having contact with your husband?

Your anxiety over this situation is completely normal and your feelings about her, understandable.

I agree that you need to have a discussion with your husband about boundaries and what ifs.

I have experienced this in the past with my DH's oldest (estranged) daughter, and anytime she wanted to be a part of his life (when she wanted something), it turned out to be very short lived. My DH, himself, put an end to the charade several years ago (she's now 32) when she cussed him out and tried to put a guilt trip on him. Haven't heard from her since.

Each time contact was made, I got anxious, but I let him know what I would and would not tolerate and for the most part, he agreed with me. His daughter, along with her mother and her mother's family, are all very toxic and manipulating, and he knows that.

Like you, I didn't want to be blamed for the relationship not working out, so, I just sat back and let it self destruct on it's own, as I knew it would.

My DH loved her (still does) and always wanted to be a part of her life and he has been deeply hurt by her over the years. But, she had been brainwashed her entire life by BM's family, to the point that she does not even believe that DH is her dad.

He has finally accepted that it just wasn't meant to be.

I have been lurking on here for a couple of years now, but your post is similar to my situation and so here's my first post.  My husband and I have been together just over 20 years, and I knew about his daughter since day one. She hadn't been in his life since she was 18 months old because of his ex gf.   When they broke up she got a restraining order against him saying he was abusive and moved across the country.  My husband at the time was young and didn't have the money to fight it.  The BM kept his daughter in the dark and wouldn't tell her anything about him.  It wasn't until she was 13 and came across her birth certificate that she had a name.  She asked her aunt (BM's sister) to contact my husband and see if she could have contact with him. He was overjoyed and said yes please give her my details.  Instead she went back to her and said he wanted nothing to do with her.  My husband waited 5 years to hear from her. When she was 18 and a young mother herself she contacted him.  She was no longer in contact with her mother or the rest of her family.  They talked a lot on Facebook and the phone but she always kept him at a distance.  So I didn't feel very threatened.  It took them a few years to finally meet in person, he drove to Wales for a long weekend to meet with her.  I stayed home (my choice) I knew she'd feel awkward enough meeting him.   The first meeting went well, and we arranged for another longer visit when I would go too.  I am not going to lie, it was hard at first.  I felt left out, he was spending a lot of time talking to her both on facebook and face time and I started worrying about how it was going to change our family dynamics.  We don't have kids together and it's always just been the two of us.  Then there was this virtual stranger who was taking up all my husband's time. So I was jealous, worried, angry a whole gambet of emotions. 

I had a long chat with DH about how I felt and my concerns and fortunately, he listened and he included me in everything. Made it clear we were a package deal.  Fast forward a few years and she and I are very close. She calls me mum. She said I've treated her more as a mum than her own. Her two boys call me nana and they've never met the BM.  It all took time, it wasn't easy going by any stretch of the imagination. But we some how made it through and things are good. I know all situations are different but I do understand how you are feeling at the moment and it's all completely natural.  I would say the most important thing is to talk to your husband as tog redux suggested.  

You are not a bad person at all. I had no idea I would have so many painful, confusing emotions before getting seriously involved with a man with a child. I had never in my life thought of myself as a bad person. I've always tried my hardest to do what I thought was right and live by a moral standard. All of that blew up after becoming a stepmother. I felt bad, evil, unworthy, bitter, hateful, and like a horrible person. Who would not want anything to do with a stepchild? Who would be angry at the person they love for having a child? I've had so many painful emotions the last several years all because the man I love has a child with another woman. I've just recently began believing I am not a horrible person, I deserve to be a mother (to my own children) just as much as anyone else. I deserve to have a family with my husband just as much as anyone else. Knowing my husband has a child he loves with another woman while struggling to have my/our own child has literally been the hardest thing in my life. So, you are not a bad person. You are a normal person going through an ongoing and incredibly difficult situation.