Right answer is (c) On unit circle
To explain: The transfer function of normalized low pass Butterworth filter is given as
HN(s).HN(-s)=\(\frac{1}{1+(\frac{s}{j})^{2N}}\)
The poles of the above equation is obtained by equating the denominator to zero.
=> \(1+(\frac{s}{j})^{2N}\)=0
=> s=(-1)^1/2N.j
=> sk=\(e^{jπ(\frac{2k+1}{2N})} e^{jπ/2}\), k=0,1,2…2N-1
The poles are therefore on a circle with radius unity.