Right answer is (a) \(\frac{8\sqrt{2}-31}{6}\)
The best I can explain: Let \(I=\int_1^2 \sqrt{x}-3x \,dx\)
F(x)=\(\int \sqrt{x}-3x \,dx\)
=\(\frac{x^{1/2+1}}{1/2+1}-\frac{3x^2}{2}=\frac{2x^{\frac{3}{2}}}{3}-\frac{3x^2}{2}\)
By using the second fundamental theorem of calculus, we get
I=F(2)-F(1)=\(\left(\frac{2×2^{3/2}}{3}-\frac{3×2^2}{2}\right)-\left(\frac{2×1^{3/2}}{3}-\frac{3×1^2}{2}\right)\)
I=\(\frac{4\sqrt{2}}{3}-6-\frac{2}{3}+\frac{3}{2}=\frac{8\sqrt{2}-36-4+9}{6}=\frac{8\sqrt{2}-31}{6}\)