Correct answer is (b) 1-a
To elaborate: We know that,
|H(ω)|=\(\frac{|b|}{\sqrt{1-2acosω+a^2}}\)
Since the parameter ‘a’ is positive, the denominator of |H(ω)| becomes minimum at ω=0. So, |H(ω)| attains its maximum value at ω=0. At this frequency we have,
\(\frac{|b|}{1-a}\) = 1 => b=±(1-a).