Correct answer is (c) \(\begin{bmatrix}3&5&-1\\-2&-9&1\\-2&-4&-5\end{bmatrix}\)
The best I can explain: For the given Matrix,
A=\(\begin{bmatrix}-1&-1&2\\0&1&-1\\2&2&1\end{bmatrix}\)
The characteristic polynomial is given by-
α^3-(Sum of diagonal elements) α^2+(Sum of minor of diagonal element)α-|A|=0
α^3-α^2+3α+5=0
The Cayley Hamilton’s Theorem states that every matrix satisfies its Characteristic Polynomial.
Thus,
A^3-A^2+3A+5I=0
A^3=A^2-3A-5I
A^3=\(\begin{bmatrix}5&2&5\\-2&-1&-2\\4&2&3\end{bmatrix}-3\begin{bmatrix}-1&-1&2\\0&1&-1\\2&2&1\end{bmatrix}-5\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)
A^3=\(\begin{bmatrix}5+3-5&2+3&5-6\\-2+0&-1-3-5&-2+3\\4-6&2-6&3-3-5\end{bmatrix}\)
A^3=\(\begin{bmatrix}3&5&-1\\-2&-9&1\\-2&-4&-5\end{bmatrix}\).