The correct option is (b) \(\begin{bmatrix}80&-126&-504\\126&-172&-63\\252&-315&-46\end{bmatrix}\)
The best I can explain: For the given Matrix,
A=\(\begin{bmatrix}-1&2&8\\-2&3&0\\-4&5&1\end{bmatrix}\)
The characteristic polynomial is given by-
α^3-(Sum of diagonal elements) α^2+(Sum of minor of diagonal element)α-|A|=0
α^3-3α^2+35α-17=0
The Cayley Hamilton’s Theorem states that every matrix satisfies its Characteristic Polynomial.
Thus,
A^3-3A^2+35A-17I=0
On performing long division (α^3-3α^2+35α-17)/(α^2-7α)
Q=α+4 and R=63α-17
Using division properties,
α^3-3α^2+35α-17=(α^2-7α)×(α+4)+(63α-17)
α^3-3α^2+35α-17=(α^3-3α^2-28α)+( 63α-17)
0=(α^3-3α^2-28α)+(63α-17) ————— (From Characteristic Polynomial)
(α^3-3α^2-28α) = -63α+17
(A^3-3A^2-28A) = -63A+17I
(A^3-3A^2-28A) = \(-63\begin{bmatrix}-1&2&8\\-2&3&0\\-4&5&1\end{bmatrix}+17\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}\)
(A^3-3A^2-28A)=\(\begin{bmatrix}63+17&-126&-504\\126&17-189&-63\\252&-315&17-63\end{bmatrix}\)
(A^3-3A^2-28A)=\(\begin{bmatrix}80&-126&-504\\126&-172&-63\\252&-315&-46\end{bmatrix}\).