Right answer is (d) 4x^3-3x
The best I can explain: We know that a chebyshev polynomial of degree N is defined as
TN(x) = cos(Ncos^-1x), |x|≤1; TN(x) = cosh(Ncosh^-1x), |x|>1
And the recursive formula for the chebyshev polynomial of order N is given as
TN(x)=2xTN-1(x)-TN-2(x)
Thus for a chebyshev filter of order 3, we obtain
T3(x)=2xT2(x)-T1(x)=2x(2x^2-1)-x=4x^3-3x.