Right choice is (b) e^x^2 (x^2-1)+C
The explanation: Let x^2=t
Differentiating w.r.t x, we get
2x dx=dt
∴∫ 2x^2 e^x^2 dx=∫ te^t dt
By using the formula, ∫ u.v dx=u∫ v dx-∫ u’ (∫ v dx) ,we get
∫ t e^t dt=t∫ e^t dt-∫ (t)’∫ e^t dt
=te^t-∫ e^t dt
=te^t-e^t=e^t (t-1)
Replacing t with x^2, we get
∫ 2x^3 e^x^2 dx=e^x^2 (x^2-1)+C