Right answer is (a) \(cos^{-1}\frac{12}{\sqrt{1302}}\)
To explain: Given that, the normal to the planes are \(\vec{n_1}=4\hat{i}+\hat{j}-2\hat{k} \,and \,\vec{n_2}=5\hat{i}-6\hat{j}+\hat{k}\)
The angle between two planes of the form \(\vec{r}.\vec{n_1}=d_1 \,and \,\vec{r}.\vec{n_2}=d_2\) is given by
cosθ=\(\left |\frac{\vec{n_1}.\vec{n_2}}{|\vec{n_1}||\vec{n_2}|}\right |\)
\(|\vec{n_1}|=\sqrt{4^2+1^2+(-2)^2}=\sqrt{21}\)
\(|\vec{n_2}|=\sqrt{5^2+(-6)^2+1^2}=\sqrt{62}\)
\(\vec{n_1}.\vec{n_2}\)=4(5)+1(-6)-2(1)=20-6-2=12
cosθ=\(\frac{12}{\sqrt{21}.\sqrt{62}}=\frac{12}{\sqrt{1302}}\)
∴θ=\(cos^{-1}\frac{12}{\sqrt{1302}}\).