When there are minor children, a Will should always be used to name a guardian(s) of their persons and property. This guardian is who will be taking care of them in your absence and will also have control over their finances, both from you and for their well being. This guardian that you appoint, needless to say, is someone that you must be able to trust completely with your children and someone who will make sure that they are cared for in the way that you have planned. This person "can" of course be someone other than your X.
Alternate guardians should also be named in the event that the original guardian is for whatever reason unable to assume responsibility. Naming of guardians and alternates should not be done any other way but in a Will. This will relieve any hint of confusion after you are not able to take care of your kids yourself. Of course, if there is a surviving parent that person will be automatically named guardian if living in the same household; but, if your will specifies a different person to control the money, then this can fit your goals quite nicely.
This situation can and often gets tricky in divorce cases. Since you are divorced, the parent with legal custody of the child(ren) should designate a guardian. If you are the legal guardian, then you have the authority to designate who will care for your children after you die. Understand, however, that if somebody besides the other biological parent is named, this decision might not be binding.
When a custodial parent dies, the non-custodial parent always has priority in seeking guardianship and custody, unless that person is deemed unfit to perform the duties necessary or is unsafe to leave with children. If you are set against your "X" getting custody of your children if you were to die, you need to make sure that you or your appointed guardian will be able to prove that your "X" is unfit or unable to perform the job.
However, be aware that the court will probably have to approve who you have proposed to be the legal guardian eventually even if named in your Will. The purpose of your Will in this regard, though, is to guide the court in its judgment. It will also help avoid family arguments over who is better qualified to raise your children and will give the person you choose the authority over all others.