Correct answer is (c) 16x^5-20x^3+5x
The explanation is: We know that a chebyshev polynomial of degree N is defined as
TN(x) = cos(Ncos^-1x), |x|≤1
= cosh(Ncosh^-1x), |x|>1
And the recursive formula for the chebyshev polynomial of order N is given as
TN(x)= 2xTN-1(x)-TN-2(x)
Thus for a chebyshev filter of order 5, we obtain
T5(x)=2xT4(x)-T3(x)=2x(8x^4-8x^2+1)-(4x^3-3x)=16x^5-20x^3+5x.