Right choice is (b) \(\begin{bmatrix}\frac{1}{3}&-\frac{1}{6}&0\\-\frac{11}{30}&\frac{1}{12}&\frac{3}{10}\\-\frac{1}{10}&\frac{1}{4}&-\frac{1}{10}\end{bmatrix}\)
The explanation: Consider the matrix A=\(\begin{bmatrix}5&1&3\\4&2&6\\5&4&2\end{bmatrix}\)
Using the elementary row operations, we write A=IA
\(\begin{bmatrix}5&1&3\\4&2&6\\5&4&2\end{bmatrix}\)=\(\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)A
Applying R2→5R2-4R1 and R3→R3-R1
\(\begin{bmatrix}5&1&3\\0&6&18\\0&3&-1\end{bmatrix}\)=\(\begin{bmatrix}1&0&0\\-4&5&0\\-1&0&1\end{bmatrix}\)A
Applying R1→R2-6R1 and R3→R2-2R3
\(\begin{bmatrix}-30&0&0\\0&6&18\\0&0&20\end{bmatrix}\)=\(\begin{bmatrix}-10&5&0\\-4&5&0\\-2&5&-2\end{bmatrix}\)A
Applying R2→20R2-18R3
\(\begin{bmatrix}-30&0&0\\0&120&0\\0&0&20\end{bmatrix}\)=\(\begin{bmatrix}-10&5&0\\-44&10&36\\-2&5&-2\end{bmatrix}\)A
Applying R1→R1/(-30), R2→R2/120, R3→R3/20
\(\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)=\(\begin{bmatrix}\frac{1}{3}&-\frac{1}{6}&0\\-\frac{11}{30}&\frac{1}{12}&\frac{3}{10}\\-\frac{1}{10}&\frac{1}{4}&-\frac{1}{10}\end{bmatrix}\)A
A^-1=\(\begin{bmatrix}\frac{1}{3}&-\frac{1}{6}&0\\-\frac{11}{30}&\frac{1}{12}&\frac{3}{10}\\-\frac{1}{10}&\frac{1}{4}&-\frac{1}{10}\end{bmatrix}\).