Correct answer is (a) 1
The best I can explain: We know that (a + b)^2 = a^2 + 2ab + b^2
(a + b)^3 = a^3 + 3ab^2 + 3a^2b + b^2
Using these formulae, we get
P(x) = (x^2 + 2xy + y^2)(x^3 + 3xy^2 + 3x^2y + y^2)
P(x) = 3xy^4 + 9x^2y^3 + 10x^3y^2 + 5x^4y + x^5 + y^4 + 2xy^3 + x^2y^2
The coefficient of x^2y^2 in (x + 1)^2 . (x + 1)^3 is 1.