The correct choice is (b) \(\frac{1}{1-2acosω+a^2}\)
The explanation: Given x(n)= a^nu(n), |a|<1
The auto correlation of the above signal is
rxx(l)=\(\frac{1}{1-a^2}\) a^|l|, -∞< l <∞
According to Wiener-Khintchine Theorem,
Sxx(ω)=F{rxx(l)}=\([\frac{1}{1-a^2}]\).F{a^|l|} = \(\frac{1}{1-2acosω+a^2}\)