Right choice is (a) \(\begin{bmatrix}-200&0&-24\\24&-32&-24\\-72&96&-56\end{bmatrix}\)
The explanation is: Explanation: For the given Matrix,
A=\(\begin{bmatrix}5&0&-1\\1&2&-1\\-3&4&1\end{bmatrix}\)
The characteristic polynomial is given by-
α^3-(Sum of diagonal elements) α^2+(Sum of minor of diagonal element)α-|A|=0
α^3+2α^2-12α-40=0
The Cayley Hamilton’s Theorem states that every matrix satisfies its Characteristic Polynomial.
Thus,
A^3+2A^2-12A+40I=0
A^3+2A^2=12A-40I
A^3+2A^2=\(12\begin{bmatrix}5&0&-1\\1&2&-1\\-3&4&1\end{bmatrix}-40\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)
A^3+2A^2=\(\begin{bmatrix}-60-40&0&-12\\12&24-40&-12\\-36&48&12-40\end{bmatrix}\)
A^3+2A^2=\(\begin{bmatrix}-100&0&-12\\12&-16&-12\\-36&48&-28\end{bmatrix}\)
2A^3+4A^2=\(\begin{bmatrix}-200&0&-24\\24&-32&-24\\-72&96&-56\end{bmatrix}\).