Correct answer is (a) \(\frac{1}{\sqrt{1-2acosω+a^2}}\)
To elaborate: For the given X(ω)=1/(1-ae^-jω), |a|<1 we obtain
XI(ω)=(-asinω)/(1-2acosω+a^2) and XR(ω)=(1-acosω)/(1-2acosω+a^2)
We know that |X(ω)|=\(\sqrt{X_R (ω)^2+X_I (ω)^2}\)
Thus on calculating, we obtain
|X(ω)| = \(\frac{1}{\sqrt{1-2acosω+a^2}}\).