Right option is (d) 4.5787 rad/sec
The explanation: We know that the equation for the cutoff frequency of a Butterworth filter is given as
ΩC = \(\frac{\Omega_P}{(10^{-K_P/10}-1)^{1/2N}}\)
We know that KP=-1 dB, ΩP=4 rad/sec and N=5
Upon substituting the values in the above equation, we get
ΩC=4.5787 rad/sec.