1.0 - Solving Recurrences

1.1 - Recurrences

• To be well-defined, a recurrence needs:

  • base case, and
  • recursive case(s) that converge on the base case

• Running time is usually bounded by a constant for constant-sized inputs, and so we often omit the base case, and assume that

  $$T(n)=\Theta(1) \ \ \ \ \ \ \text{for } n\le c$$

  where $c$ is a constant.

• Given a divide and conquer algorithm that:

  • Takes a problem of size $\color{lightblue} n$, and
  • Breaks it into $\color{lightblue} a$ parts, each of size $\color{lightblue} \frac nb$
  • Takes $\color{lightblue} D(n)$ time to divide the problem, and
  • Takes $\color{lightblue}C(n)$ time to combine solutions to sub problems, we get the following recurrence:

• Therefore, the time complexity of the algorithm is given by the following recurrence:

  $$T(n)= \begin{cases} a T(\frac nb) + D(n)+C(n), &\text{if } n>c\\ \in \Theta(1) &\text{if } n \le c \end{cases}$$

• There are three general methods for solving a recurrence

  • Substitution Guess an answer, and then prove by mathematical induction
  • Iteration Expand the recurrence to formulate a summation, then solve
    • Iterate out the recurrence, and develop a summation that describes the amount of work done by the algorithm
  • Master Method Remember 3 cases for solving recurrences of the form $T(n)=aT(\frac ab)+f(n)$

1.1 - Substitution Method

• Guess a solution, prove by induction (without asymptotic notation)

1.1.1 - Substitution Method - Example 1

Given the recurrence:

$T(n)=\begin{cases}2&\text{if } n=2\\2T(n/2)+n&\text{if }n=2^k, \text{for } k>1\end{cases}$

• We will solve this simply, without using asymptotic notation.

• We guess that $T(n)=n \lg n$ for all $n=2^k$ where $k\ge 1$ and prove by mathematical induction

• We start by proving the base case, when $k=1$ we know that $2^k=2^1=2$

  • $T(2)=2=2 \log_2 2$

• We then prove the inductive case:

  • Assume $T(n/2)=\frac n2 \lg(n/2)$ for $n/2=2^{k-1}$ and prove for $n=2^k$
  • Then:
  $$\begin{aligned} T(n)&=2{\color{pink}T(n/2)}+n\\ &=2{\color{pink}\frac n2 \lg (n/2)} + n &\text{by substitution of inductive assumption}\\ &=n \lg (n/2)+n &\text{as } 2\frac n2=n\\ &=n (\lg n-\lg 2) + n &\small\text{as } \lg (\frac ab)=\lg a-\lg b\\ &=n \lg n - n\times 1 + n &\text{as }\lg_2(2)=1\\ &=n \lg n-n+n &\text{as }n\times 1=1\\ &=n\lg n &\text{as }(-n+n)=0 \end{aligned}$$

1.1.2 - Substitution Method - Example 2

🌱 We can use this particular technique to prove that a recurrence has a certain asymptotic upper bound, say $O(n \lg n)$

Given the recurrence (includes floor):

$T(n)\begin{cases} 1&\text{if } n=1\ 2T(\lfloor\frac n2\rfloor) + n&\text{if }n>1 \end{cases}$

• We notice that this is like the previous proof, so we guess that:

  $$T(n)\in O(n \lg n)$$

• We need to prove (by mathematical induction), that there exists constants $n_0, c>0$ such that:

  $$\forall n\ge n_0 \bullet 0 \le T(n)\le c\cdot n \lg n$$

• For the given recurrence, $0\le T(n)$ for $n \ge1$, so we want to focus on the $T(n)\le cn \lg n$ component.

• We begin with the base case. This depends on our value of $n_0$.

  • We can’t choose $n_0=1$, since:

    $$T(1)=1\not{\le} c\cdot1\cdot \lg 1=0$$

  • If we choose $n_0=2$, then the only values of $2\le n$ directly dependent on $T(1)$ are $T(2)$ and $T(3)$

    • We want to now find constants such that the base case is correct.te
    $$\begin{aligned} T(2)=2T(1)+2=4 \le c\cdot 2\cdot \lg 2\ \ \ \ \ &\text{if }c \ge2\\ T(3)=2T(1)+3=5 \le c\cdot 3\cdot \lg 3 \ \ \ \ \ &\text{if } c\ge \frac{5}{3 \cdot \lg 3} \end{aligned}$$

  • So $T(n)\lt cn\lg n$ for base cases $n=2$ and $n=3$ if $c\ge 2$

• We continue with the inductive step.

  • Assume $T(n)\le cn \lg n$ for $\lfloor n/2 \rfloor$. That is,

    $$T(\lfloor n/2\rfloor)\le c\lfloor n/2\rfloor \lg \lfloor n/2\rfloor$$

  • We then prove $T(n)\le cn \lg n$ for some suitable $c>0$ (to find $c$)

    $$\begin{aligned} T(n)&=2{\color{pink}T(\lfloor n/2 \rfloor)} + n\\ &\le 2({\color{pink}c\lfloor n/2 \rfloor \lg \lfloor n/2 \rfloor}) + n \ \ \ \ \ \ &\text{substituting inductive assumption}\\ &= cn \lg (n/2) + n &\text{by removing floor, as }\lfloor n\rfloor \le n \\ &&\text{and we want to find upper bound}\\ &= cn \lg n - cn \lg 2 + n &\text{as }\lg(a/b)=\lg a - \lg b\\ &\le cn \lg n &\text{if } c \ge 1 \text{ and } n \ge 0 \end{aligned}$$
    • With respect to the last step, we know that $\color{lightblue}cn \lg n -cn \lg 2 + n\le cn \lg n$ if and only if $\color{lightblue}cn \lg 2+n$ is non-zero and positive.
    • Therefore, we add the constraint $c\ge1, n\ge 0$

  • We must also have $c\ge 2$ to satisfy the base cases, so $n_0=2$ and $c=2$ will suffice\

1.1.3 - Substitution Method - Guessing

Consider the following recurrence:

The difference between:

Since the difference is small, we can guess the same solution.

We can then guess loose upper and lower bounds, then refine:

Try to lower the upper bound and raise the lower bound until convergence to get a sufficiently tight bound

1.1.4 - Substitution Method - Strengthening the Guess

🌱 We can “Strengthen the Guess” by removing a lower-order term.

Consider:

We guess $T(n)=O(n)$

In our inductive step, we assume that $T(\lfloor n/2 \rfloor)\le c\lfloor n/2\rfloor$ and $T(\lceil n/2\rceil)\le c\lceil n/2\rceil$ and attempt to prove $T(n)\le cn$

• However, we haven’t shown that $T(n)\le cn$

We could try to prove that $T(n)=O(n^2)$, but that is too weak. Let’s strengthen the guess by subtracting a lower-order term. Guess that:

• Since our inductive assumption is now stronger, we can then prove a stronger bound

Inductive Step Assume for $\lfloor n/2 \rfloor$ and $\lceil n/2 \rceil$ and prove for $n$

Boundary Conditions Now find values for $c, n_0$ such that the boundary conditions are satisfied

• By making a stronger inductive assumption, we can prove a stronger result.

1.1.5 - Substitution Method - Incorrect Guess & Bugs in Proof

• Consider again:

  $$T(n)=2(T\lfloor n/2\rfloor) +n$$

• We make a (bad) guess that $T(n)\in O(n)$

• We then have the following incorrect inductive step:

  • Assume $T(\lfloor n/2\rfloor )\le c\lfloor n/2\rfloor$ and prove $T(n)\le cn$

    $$\begin{aligned} T(n)&=2{\color{pink}T(\lfloor n/2\rfloor)}+n\\ &\le 2{\color{pink}c\lfloor n/2 \rfloor} + n \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ &\text{substitute inductive assumption}\\ &\le cn+n\\ &=O(n) \end{aligned}$$

  • However, this does not prove that $T(n)\le cn$

  • Each line is mathematically correct, but this is not a valid inductive proof - we haven’t proven $T(n)\le cn$ for our chosen constant $c.$

1.1.6 - Change of Variables Technique

Consider

This looks hard to work with. Try the change of varaible technique, where $m=\lg n$ (or $n=2^m$)

Rename $S(m)=T(2^m)$

We know from before that the solution is $\Theta(m \lg m)$

We now change back the variables:

1.2 - Iteration Method

• Expand (iterate) out the recurrence to get a summation

• Evaluate a summation (develop a summation that describes the recurrence)

• More maths required, but no need to guess.

1.2.1 - Iteration Example

Consider the following recurrence:

We begin by taking the recurrence and expanding it out:

• We add the conditions (to the right) if __ > 1 to ensure that we don’t hit the base case and therefore can keep expanding.

We want to develop a generic form for the expression above, if we were to expand it some arbitrary amount of times, $i$.

• We can continue to expand, if the condition $\lfloor \frac{n}{4^{i-1}}\rfloor>1$ holds true.

• For which value of $i$ do we stop expanding the recurrence?

  • For the smallest value of $i$, such that $\lfloor n/4^i\rfloor\le 1$ (i.e., when we hit the base case)

• Imprecisely Assuming that $T(n)=\Theta(1)$ for $n\le 1$, we stop expanding when

  $$\begin{aligned} &\lfloor n/4^i\rfloor \approx 1\\ \equiv\ &4^i\approx n\\ \equiv\ &i \approx \log_4 n \end{aligned}$$
  • Therefore, we expand out the occurrence roughly $\log_4 n$ times.

• More precisely, we must have $i\le \lceil \log_4 n\rceil$ since $\lfloor n/4^{\lceil \log_4 n\rceil}\rfloor \le 1$

• From this, it follows that:

  $$\begin{aligned} \small T(n) &= n+3\lfloor n/4\rfloor +3^2\lfloor n/4^2\rfloor + \cdots + 3^i T(\lfloor n/4^i\rfloor) \ \ \ \ \ \ \ \ \ &\text{if } \lfloor \frac{n}{4^{i=1}}\rfloor>1\\ &\le n+3\lfloor n/4\rfloor +3^2\lfloor n/4^2\rfloor + \cdots + 3^{\lceil \log_4 n \rceil}\Theta(1) &\text{as } i\le \lceil \log_4 n\rceil\\ &&\scriptsize n=\lfloor n/4^i \rfloor\Rightarrow T(n)=\Theta(1)\\ &\le n+\frac{3n}{4}+\frac{3^2n}{4^2}+\frac{3n^3}{4^3}+\cdots+\Theta(3^{\log_4 n}) &\small\text{as finding upper bound}\\ &\le n \sum^\infty_{i=0} (\frac34)^i + \Theta(3^{\log_4 n}) &\text{as a summation}\\ &=\frac{1}{1-\frac 34}+O(3^{\log_4 n})&\text{using sum formula}\\ &= 4n+O(n^{\log_4 3})&\text{as } a^{\log_b(n)}=n^{\log_b (a)}\\ &=4n+O(n)&\text{as } \log_4 n<1\\ &= O(n) \end{aligned}$$

• Observe that when we are trying to find the upper bound, we can drop the floors and ceilings.

• Additionally, though the summation is not infinite, the infinite summation is an upper bound on it.

• The iteration tree for the recurrence $T(n)=n=3T(\lfloor n/4\rfloor)$ is given as shown below - we can perform the same iterative analysis as above by constructing a recurrence tree.

  • Once again, we can repeat this analysis until we see a pattern start to develop.

• To determine what summation describes, we need to determine two things:

  • When do we stop expanding the summation (described by the height of the tree) - in this case, this is given by $\lceil \log_4 n\rceil$
  • How much work is performed at each level of the tree.

• In this, the height of the tree is essentially how many iterations we perform, and the amount of work done at each level is the value we are summing in each iteration

1.2.2 - Iteration Method Notes

• This technique has a few issues:
  • Lots of math
  • You have to work out all of the terms, when to stop, and to sum what you get
  • Sometimes you can only start the iteration, and guess the answer
  • If you guess correctly, you can abandon the maths and revert to substitution

• Floors and ceilings can be troublesome
  • Workaround - Assume that $n$ has the correct form to delete them (as in the previous example, where $n=4^k$
  • Technical abuse often work well (OK if the function is well-behaved)
• Start trying to work out the solution if they weren’t there, and come back and ‘fix it up’

1.2.3 - Iteration Example 2

Solve the following recurrence using iteration:

assuming that $T(n)\in\Theta(1)$ for $n\le 0$

• How many terms are needed?

• Taking $n=i$, we have $T(n-i)=T(0)\in \Theta(1)$ (reached the base case) and so:

  $$\begin{aligned} T(n)&=\sum^{n-1}_{j=0}2^j+2^nT(0)\\ &=2^n-1+2^n\Theta(1)\\ &=\Theta(2^n) \end{aligned}$$

1.3 - Master Method

Need to memorise 3 cases for solving some recurrences of the form:

where $n/b$ can be $\lfloor n/b\rfloor$ or $\lceil n/b\rceil$

In each case, we compare $n^{\log_b a}$ with $f(n)$

1. $n^{\log_b a}$ is polynomially larger than $f(n)$ → Solution is $\Theta(n^{\log_b a})$
2. $n^{\log_b a}$ is same tight asymptotic bound as $f(n)$ → Solution is $\Theta(n^{\log_b a} \lg n)$
3. $f(n)$ is polynomially larger than $n^{\log_b a}$ and $f(n)$ is regular → Solution is $\Theta(f(n))$

1.3.1 - Polynomially Larger Than

A function $f(n)$ is polynomially larger than $g(n)$ when

Equivalently:

In which $f(n)$ is polynomially larger than $g(n)$ by polynomial $n^\epsilon$

Example:

Is $f(n)=n^2$ polynomially larger than $g(n)=n^{3/2}$?

$\frac{f(n)}{g(n)}=\frac{n^2}{n^{3/2}}=n^{2-3/2}=n^{1/2}$

Yes, as $n^2\in \Omega(n^{3/2}\times n^{1/2})$

Is $f(n)=\log_2 n$ polynomially larger than $g(n)=1$?

No, as $\log_2 n \not{\in}\Omega(1\times n^\epsilon)$ for any $\epsilon>0$

$\log_2 n$ is larger than, but it is not polynomially larger than

1.3.2 - Regularity

Function $f(n)$ is regular when:

• Most functions satisfy regularity, but still need to check

1.3.3 - Formal Definition of Master Method

Given recurrences of the form:

$\text{case 1}\ T(n)\in\Theta(n^{\log_b a})\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ f(n)\in O(n^{\log_b a-\epsilon})\ \ \ \ \text{for some } \epsilon>0\ \text{case 2}\ T(n)\in\Theta(n^{\log_b a }\lg n) \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ f(n)\in\Theta(n^{\log_b a})\ \ \ \ \ \ \ \text{for some }\epsilon>0\ \text{case 3}\ T(n)\in\Theta(f(n))\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ f(n)\in\Omega(n^{\log_b a + \epsilon}) \ \ \ \ \text{for some } \epsilon>1\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{and } \ \ \ \ \ \ \ \ \ \ af(\frac nb) \le cf(n) \ \ \ \ \ \ \ \ \ \ \text{for some } c<1$

We can re-define the master method, using the division syntax.

$\text{case 1}\ T(n)\in\Theta(n^{\log_b a})\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ {\small\frac{f(n)}{n^{\log_b a-\epsilon}}}\in O(n^{-\epsilon})\ \ \ \ \text{for some } \epsilon>0\ \text{case 2}\ T(n)\in\Theta(n^{\log_b a }\lg n) \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \frac{f(n)}{n^{\log_b a}}\in\Theta(1)\ \ \ \ \ \ \ \ \ \ \text{for some }\epsilon>0\ \text{case 3}\ T(n)\in\Theta(f(n))\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{if}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \frac{f(n)}{n^{\log_b a + \epsilon}}\in\Omega(n^\epsilon)\ \ \ \ \ \ \text{for some } \epsilon>1\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{and } \ \ \ \ \ \ \ \ \ \ af(\frac nb) \le cf(n) \ \ \ \ \ \ \text{for some } ​c<1$

1.3.4 - Application of the Master Theorem:

Recurrence: $\color{lightblue} T(n)=2T(\frac n2)+f(n)\ \ \ \ \ \text{where } f(n)\in \Theta(n)$

The master method works on recurreneces of the form

$\ \ \ \ a=2, b=2$

$\frac{f(n)}{n^{\log_b a}}\in\Theta(\frac{n}{n^{\log_2 2}})=\Theta(\frac{n}{n^1})=\Theta(1)$

Therefore, the second case applies, and the solution is $\Theta(n^{\log_2 2} \lg n)=\Theta(n \lg n)$

Recurrence: $\color{lightblue} T(n)=1T(\frac n2)+f(n)\ \ \ \ \ \text{where } f(n)\in\Theta(1)$

The master method works on recurreneces of the form

$\ \ \ \ a=1, b=2$

$\frac{f(n)}{n^{\log_b a}}\in \Theta(\frac{1}{n^{\log_2 1}})=\Theta(\frac 11)=\Theta(1)$

Therefore, the second case applies, and the solution is $\Theta(n^{\log_2 1 }\lg n)=\Theta(n^0 \lg n)=\Theta(\lg n)$

Recurrence: $\color{lightblue} T(n)=7T(\frac n2)+f(n)\ \ \ \ \ \ \text{where } f(n)\in \Theta(n^2)$

The master method works on recurreneces of the form

$\ \ \ \ a=7, b=2$

$\frac {f(n)}{n^{\log_b a}}\in\Theta(\frac{n^2}{n^{\log_2 7}})=\Theta(n^{-(\log_2 7-2)})$

Additionally, $\epsilon=\log_2 7-2>0$

Since $4\lt7\lt8$, we know that $\log_2 4<\log_2 7<\log_2 8$ and therefore, $2\lt \log_2 7\lt3$

Hence, case 1 applies and $T(n )\in \Theta(n^{log_b a})=\Theta(n^{log_2 7}) \approx\Theta(n^{2.81})$

• This is slightly better than the average solution of matrix multiplication, which is $\Theta(n^3)$

Recurrence: $\color{lightblue}T(n)=4T(\frac n2)+f(n)\ \ \ \ \ \text{where } f(n)\in\Theta(n^3)$

The master method works on recurrences of the form

$\ \ \ \ a=4, b=2$

$\frac {f(n)}{n^{\log_b a}}\in\Theta(\frac{n^3}{n^{\log_2 4}})=\Theta(\frac{n^3}{n^2})=\Theta(n)$

Case 3 of the master method applies, as long as we can satisfy the regularity condition.

That is,

For some choice of $c=\frac12<1$

1.3 - Extended Master Method

Fills in a gap in the Master Theorem

• If $f(n)\in\Theta(n^{\log_b a}\lg^kn)$
• $f(n)$ is larger, but not polynomially larger
• Solution is $\Theta(n^{\log_b a}\lg^{k+1}n)$

The above case 2 is a special case of this, when $k=0$