The correct answer is (a) 20-\(\frac{10}{\sqrt{5}} sin(π/2n-26.60)+ \frac{40}{3}cosπn\)
The best I can explain: The frequency response of the system is
H(ω)=\(\sum_{n=-∞}^∞ h(n) e^{-jωn} = \frac{1}{1-\frac{1}{2} e^{-jω}}\)
For first term, ω=0=>H(0)=2
For second term, ω=π/2=>H(π/2)=\(\frac{1}{1+j\frac{1}{2}} = \frac{2}{\sqrt{5}} e^{-j26.6°}\)
For third term, ω=π=> H(π)=\(\frac{1}{1+\frac{1}{2}}\) = 2/3
Hence the response of the system to x(n) is
y(n)=20-\(\frac{10}{\sqrt{5}} sin(π/2n-26.6^0)+ \frac{40}{3}cosπn\)