Correct answer is (b) \(\frac{1}{(1-\frac{1}{2} e^{-jω})(1-\frac{1}{4} e^{-jω})}\)
The best I can explain: The frequency response function of the system is
H(ω) = \(\sum_{n=0}^∞ (\frac{1}{2})^n e^{-jωn}\)
=\(\frac{1}{1-\frac{1}{2} e^{-jω}}\)
Similarly, the input sequence x(n) has a Fourier transform
X(ω)=\(\sum_{n=0}^∞ (\frac{1}{4})^n e^{-jωn}\)
=\(\frac{1}{1-\frac{1}{4} e^{-jω}}\)
Hence the spectrum of the signal at the output of the system is
Y(ω)=X(ω)H(ω)
=\(\frac{1}{(1-\frac{1}{2} e^{-jω})(1-\frac{1}{4} e^{-jω})}\).