The correct option is (b) (1 – x^2)y” – xy’ = 0
The best I can explain: We have, y = a sin^-1x + b cos^-1x
Differentiating both sides with respect to x we get,
dy/dx = 12∫ e^2xcos3x dx – 5∫ e^2x sin3x dx
y’ = a* 1/√(1 – x^2) + b * (-1/√(1 – x^2))
or, y’√(1 – x^2) = a – b
now, squaring both sides,
or, y’^2(1 – x^2) = (a – b)^2
Differentiating again with respect to x we get,
(1 – x^2) * 2y’y” + y’^2 . (-2x) = 0
As, y’ ≠ 0,
(1 – x^2)y” – xy’ = 0