Right choice is (a) \(\frac{a^2 \,x^5}{5}+b^2 \,x+\frac{2abx^3}{3}+C\)
Explanation: To find (ax^2+b)^2
\(\int (ax^2+b)^2 dx=\int (a^2 \,x^4+b^2+2ax^2 \,b) dx\)
\(\int (ax^2+b)^2 dx=\int \,a^2 \,x^4 \,dx+\int \,b^2 \,dx+2\int \,ax^2 \,b \,dx\)
\(\int (ax^2+b)^2 dx=a^2 \,\int \,x^4 \,dx+b^2 \int \,dx+2ab\int \,x^2 \,dx\)
\(\int (ax^2+b)^2 dx=a^2 (\frac{x^5}{5})+b^2 x+2ab(\frac{x^3}{3})\)
\(\int (ax^2+b)^2 dx=\frac{a^2 \,x^5}{5}+b^2 x+\frac{2abx^3}{3}+C\)