Right choice is (d) ^16⁄945
To explain: Given, f(x)=\(\int_0^1 \int_0^{1-y} xy\sqrt{1-x-y} dxdy\)
putting,t=x/(1-y)=>x=t(1-y)=>dx=(1-y)dt
\(\int_0^1 \int_0^1 t(1-y)y\sqrt{1-t(1-y)-y} (1-y)dtdy\)
=\(\int_0^1 \int_0^1 y(1-y)^{5/2} t(1-t)^{1/2} dtdy\)
=\(\int_0^1y(1-y)^{5/2} dy \int_0^1 t(1-t)^{1/2} dt\)
=\(\int_0^1 y^{2-1} (1-y)^{7/2-1} dy\int_0^1 t^{2-1} (1-t)^{3/2-1} dt=\beta(2,\frac{7}{2})\beta(2,\frac{3}{2})=\frac{16}{945}\)