People have been washing their faces with soap for nearly 2000 years. I think if it caused a significant amount of damage, we would have noticed by now.
But I imagine if a person did a lifetime half-face test where they washed one side of their face exclusively with soap and the other with more gentle, non-ionic surfactants, they might see a slight difference. Although, the impact of UV exposure, gravity, aging and the environment are much more significant contributors that even if they ran this experiment, they wouldn’t see any significant difference.