Right option is (d) -(a^3 + b^3)^2
The best I can explain: Given,\(\begin{vmatrix}2ab & a^2 & b^2 \\a^2 & b^2 & 2ab \\b^2 & 2ab & a^2 \end {vmatrix}\)
Using C1 = C1 + C2 + C3
= \(\begin{vmatrix}a^2 + b^2 + 2ab & a^2 & b^2 \\a^2 + b^2 + 2ab & b^2 & 2ab \\a^2 + b^2 + 2ab & 2ab & a^2 \end {vmatrix}\)
= (a + b)^2\(\begin{vmatrix}1 & a^2 & b^2 \\1 & b^2 & 2ab \\1 & 2ab & a^2 \end {vmatrix}\)
= (a + b)^2\(\begin{vmatrix}1 & a^2 & b^2 \\1 & b^2 – a^2 & 2ab – b^2 \\0 & 2ab – a^2 & a^2 – b^2 \end {vmatrix}\)
= (a + b)^2[(b^2 – a^2)(a^2 – b^2) – (2ab – b^2)( 2ab – a^2)]
= -(a + b)^2[(a^2 – b^2)^2 + 4a^2b^2 – 2ab(a^2 + b^2) + a^2 b^2)]
= –(a + b)^2[(a^2+b^2)^2 – 2(a^2+b^2) (ab)+(ab)^2]
= –(a + b)^2(a^2 + b^2 – ab)^2
= –[(a + b)^2(a^2 + b^2 – ab)^2]^2
= –(a^3 + b^3)^2