Genesis 24:5 is the same concern the servant now voices again to Rebekah's family — the woman might not follow. He's recounting the objection he originally raised to Abraham.