Right option is (c) \(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\ \frac{1}{40}&\frac{-1}{8}&\frac{1}{5}\\ \frac{3}{40}&0&\frac{-1}{10}\end{bmatrix}\)
For explanation: Consider the matrix A=\(\begin{bmatrix}1&2&4\\5&2&4\\3&6&2\end{bmatrix}\)
Using the elementary row operation, we write A=IA
\(\begin{bmatrix}1&2&4\\5&2&4\\3&6&2\end{bmatrix}\)=\(\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)A
Applying R1→R1-R2
\(R_1 \rightarrow \frac{R_1}{-4}\)
\(\begin{bmatrix}1&0&0\\5&2&4\\3&6&2\end{bmatrix}\)=\(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\0&1&0\\0&0&1\end{bmatrix}\)A
Applying R2→R2-5R1 and R3→R3-3R1
\(\begin{bmatrix}1&0&0\\0&2&4\\0&6&2\end{bmatrix}\)=\(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\\frac{5}{4}&\frac{-1}{4}&0\\\frac{3}{4}&\frac{-3}{4}&1\end{bmatrix}\)A
Applying R2→R2-2R3 and \(R_2 \rightarrow \frac{R_2}{-10}\)
\(\begin{bmatrix}1&0&0\\0&1&0\\0&6&2\end{bmatrix}\)=\(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\ \frac{1}{40}&\frac{-5}{40}&\frac{1}{5}\\\frac{3}{4}&\frac{-3}{4}&1\end{bmatrix}\)A
Applying R3→R3-6R2 and \(R_2 \rightarrow \frac{R_2}{2}\)
\(\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)=\(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\ \frac{1}{40}&\frac{-1}{8}&\frac{1}{5}\\ \frac{3}{40}&0&\frac{-1}{10}\end{bmatrix}\)A
A^-1=\(\begin{bmatrix}\frac{-1}{4}&\frac{1}{4}&0\\ \frac{1}{40}&\frac{-1}{8}&\frac{1}{5}\\ \frac{3}{40}&0&\frac{-1}{10}\end{bmatrix}\).