Why do people doubt those who wish to join units like DELTA and DEVGRU?

No, they're not "just regular guys." They're superior to "regular guys" by several standard deviations. Genetic freaks endowed by nature with low levels of stress hormones, excellent respiration and metabolism, gifted in spatial reasoning, etc. No amount of desire will turn an average person into a gifted person.