Correct answer is (d) ^16⁄945
Best explanation: Given, f(x)=\(\int_1^0∫_0^{1-y} xy\sqrt{1-x-y} \,dxdy\)
putting, \(t=\frac{x}{1-y}\)=>x=t(1-y)=>dx=(1-y)dt
\(\int_1^0\int_0^1 t(1-y)y\sqrt{1-t(1-y)-y} (1-y)dtdy\)
=\(\int_1^0\int_0^1 y(1-y)^{5/2} t(1-t)^{1/2} dtdy\)
=\(\int_1^0y(1-y)^{5/2} dy \int_0^1 t(1-t)^{1/2} dt\)
=\(\int_0^1 y^{2-1} (1-y)^{7/2-1} dy \int_0^1 t^{2-1} (1-t)^{3/2-1} dt=β(2,\frac{7}{2})β(2,\frac{3}{2})=\frac{16}{945}\)