Genesis 24:5

And the servant said unto him, Peradventure the woman will not be willing to follow me unto this land: must I needs bring thy son again unto the land from whence thou camest?

Cross-reference

Abraham directly answers this question in 24:8, releasing the servant from the oath if the woman won't come.

In 24:39, the servant retells this same question to Rebekah's family, recounting his exchange with Abraham.