Right option is (b) \(\frac{z(z+1)}{(z-1)^3}\)
The best I can explain: Given x(n) = n^2 u(n)
We know that X(z) = Z[x(n)] = Z[u(n)] = \(\frac{z}{1-z}\)
The multiplication of n or differentiation in z-domain property of Z-transform states that
If x(n) ↔ X(z), then n^k x(n) ↔ (-1)^k z^k \(\frac{d^k X(z)}{dz^k}\)
Z[n^2 u(n)] = z^2 \(\frac{d^2 X(z)}{dz^2} = z^2 \frac{d^2}{dz^2}[\frac{z}{1-z}] = \frac{z(z+1)}{(z-1)^3}\).