The correct answer is (b) \(\vec{r}.(\frac{2\hat{i}}{\sqrt{38}}+\frac{3\hat{j}}{\sqrt{38}}-\frac{5\hat{k}}{\sqrt{38}})=\frac{7}{\sqrt{38}}\)
The best I can explain: Let \(\vec{n}=2\hat{i}+3\hat{j}-5\hat{k}\)
\(\hat{n}=\frac{\vec{n}}{|\vec{n}|}=\frac{(2\hat{i}+3\hat{j}-5\hat{k})}{\sqrt{(2^2+3^2+(-5)^2)}}=\frac{(2\hat{i}+3\hat{j}-5\hat{k})}{\sqrt{38}}\)
Hence, the required equation of the plane is \(\vec{r}.(\frac{2\hat{i}}{\sqrt{38}}+\frac{3\hat{j}}{\sqrt{38}}-\frac{5\hat{k}}{\sqrt{38}})=\frac{7}{\sqrt{38}}\)