Right option is (b) cosθ=\(\left |\frac{A_1 A_2+B_1 B_2+C_1 C_2}{\sqrt{A_1^2+B_1^2+C_1^2}\sqrt{A_2^2+B_2^2+C_2^2}}\right |\)
For explanation I would say: If the planes are in the Cartesian form i.e. \(A_1 x+B_1 y+C_1 z+D_1\)=0 and \(A_2 x+B_2 y+C_2 z+D_2\)=0, where \(A_1,B_1,C_1 \,and \,A_2,B_2,C_2\) are the direction ratios of the planes, then the angle between them is given by
cosθ=\(\left |\frac{A_1 A_2+B_1 B_2+C_1 C_2}{\sqrt{A_1^2+B_1^2+C_1^2} \sqrt{A_2^2+B_2^2+C_2^2}}\right |\)