Right option is (c) \(2[\frac{y^4}{4}-\frac{2}{3} (1-y^4)^{\frac{3}{2}}]\)
To explain: Given, f(x)=\(\int\int_0^y \frac{2xy^5}{\sqrt{1+x^2 y^2-y^4}} dxdy\)
=\(\int\int_0^y \frac{1}{y} \frac{2xy^5}{\sqrt{(\frac{1-y^4}{y^2})+x^2}} dxdy=\int 2y^4 \left |(\frac{1-y^4}{y^2})+x^2\right |_0^y dy\)
\(=2\int [y^3-\sqrt{1-y^4}y^3]dy=2[\frac{y^4}{4}-\frac{2}{3} (1-y^4)^{3/2}]\)