More binary search trees

Review of last time

Binary trees: example tree

Finding a value

node* find(node* root, int target) {
    if(!root || root->key == target)
        return root;
    else if(target < root->key)
        return find(root->left, target);
    else
        return find(root->right, target);
}

node* insert(node* root, int key) {
    if(root == nullptr) 
        return new node{key, nullptr, nullptr};
    else if(root->key == key)
        return root;
    else if(key < root->key) {
        root->left = insert(root->left, key);
    }
    else // key > root->key
        root->right = insert(root->right, key);

    return root;
}

Constructing a tree by repeated insert. This takes \(O(n \log n)\) time.

Treesort: construct a tree \(O(n \log n)\) and then use inorder traversal to write it out into an array (\(O(n)\)).

Finishing up remove

To delete a key from a tree we start by find-ing it. If find returns nullptr then the key did not exist in the tree in the first place, so we’re done. Otherwise, we need to handle the to-be-deleted node:

Deleting a node is easy if it’s a leaf node: just remove the node and set the pointer to it in its parent to nullptr. But what if it’s an internal node? E.g.

Tree removal case 1

Suppose we want to delete 9? We need to preserve the search tree ordering property. If a node has only one child, then we can simply replace the deleted node with that child (note that 7 could have children of its own; they get copied with it). If a node has more than one child, then the process is more complex:

Tree removal case 2

(Again, we want to remove node 9.) This case is tricky, because both 7 and 10 (the obvious choices to replace 9) might have children of their own. The problem is that we need a value that can take 9’s place, but we need it to not have two children of its own. What does the search tree ordering property tell us about the values that could replace 9? In order for the search property to be maintained, the replacement for 9 must be greater than all the values in its left subtree, and less than all the values in its right subtree. (Note that if we choose a replacement from either subtree, it’s OK as long as we remove it from the subtree.)

In order to find a value in the left subtree that could replace 9, we need to find the value that is

a) greater than all the other values in the left subtree but also

b) less than 9 (so that, transitively, it is less than all the values in the right subtree)

This value has a name: the predecessor. It is the value that comes right before 9, if we were to list them out in order. (We could also use the right subtree, and look for the successor.) Note that the predecessor is guaranteed to have zero or one children, never two (if it had two, it would have a right subtree, which would contain values greater than it, and we have asserted that the predecessor is the greatest value which is less than 9).

We’ll implement predecessor later, for now, we’ll just assume that it returns a reference to the node pointer, like all the other functions.

void remove(node*& root, int key) {
  node*& target = find(root,key);

  remove(target);
}

void remove(node*& target) {
  if(!target)
    return; // Not found

  if(!target->left && !target->right) {
    // No children, just remove
    delete target;
    target = nullptr; 
  }
  else if(target->left) {
    // One left child, remove and replace with left
    node*& tl = target->left;
    delete target;
    target = tl; 
  }
  else if(target->right) {
    // One child right, remove and replace with right
    node*& tr = target->right;
    delete target;
    target = tr;
  }
  else {
    // Both children, swap with predecessor
    node*& p = pred(root, key);
    std::swap(target->key, p->key);
    remove(p);
  }
}

What is the runtime complexity of remove? It doesn’t search through the tree at all, and even the recursive call will only ever be run once, so it takes \(O(1)\) time, assuming we already have a pointer to the node to be removed. The only tricky part is the complexity of finding the predecessor, which is still \(\log n\) in the height of the tree. So delete is \(O(\log n)\) in the height of the tree (it would be anyway, because we have to find the node to be deleted.)

Finding the predecessor for remove is relatively easy, because we only ever do it in the case where the node is known to have two children. To find the predecessor, we simply look for the largest value in the node’s left subtree (that is, the largest value which is still less than the target). In general, finding the predecessor/successor when those child nodes may not exist is more difficult. Here’s the remove-specific predecessor operation:

node*& pred(node*& target) {
  if(target->left)
      return largest(target->left);
  else
      ... // General pred. code left as an exercise for the reader
}

node*& largest(node*& root) {
  if(!root)
    return root;
  else if(!root->right)
    return root;
  else
    return largest(root->right);
}

largest finds the largest value in a (sub)tree, by simply going right as far as possible. A similar operation, smallest can easily be constructed:

node*& smallest(node*& root) {
  if(!root)
    return root;
  else if(!root->left)
    return root;
  else
    return smallest(root->right);
}

Balanced Trees

If we try constructing a BST by inserting the values

1 2 3 4 5 6 7 8 9

we will find that the resulting tree is highly unbalanced. Our estimates for the runtime of the various operations were based on the assumption that \(h \approx \log n\), but in this case, we have \(h = n\). Searching in an unbalanced tree takes closer to \(O(n)\) time, which is bad. In order to remedy this situation, we have to rebalance the tree. Usually this is done after each insert/remove. There are two ways to approach this:

A perfect rebalance can be slower, but it provides strong guarantees about the state of the tree, and hence the runtime of operations. An amortized rebalancing scheme can run faster, but the shape of the tree is likely to always be somewhat unbalanced.

Rotations

The fundamental rebalancing operation on a RB tree is a “rotation”; exchanging a node with its parent, while maintaining the tree order property.

If we have the tree

Rotate a with b, before

and we rotate a with its parent b, we get

Rotate a with b, after

Similarly, if we have

Rotate a with b, before

and we rotate a with b we get

Rotate a with b, after

(In these examples, X, Y and Z are arbitrary subtrees, while a and b are single nodes.)

If we sketch out the portions of the tree’s data that each subtree represensts, in ascending order, it should become clear why we can do a rotation:

... X b Y a Z ...

If we were building a tree for this sequence from scratch, we could choose either a or b to be the root of the subtree for this portion. If we choose a to be the root, then b becomes its left child and we have the original arrangement; if b is chosen to be the root, we get the rotated arrangement. Hence, rotations preserve the order of the elements within the tree.

Another way to look at this is the order in which elements would be visited in an in order traversal. Recall that an inorder traversal visits in the order left, root, right. In the original tree, we’d visit X, then b, then Y, then a, and finally Z. In the rotated tree, we’d visit X, then b (now as the root), then Y, then a, then Z. So rotation preserves the order of nodes in an inorder traversal, which is the order of the nodes as a sorted sequence.

Notice that the two rotations are opposite each other: if we rotate a with b, and then rotate b with a, we get right back where we started.

Some books describe rotations as “left” or “right” rotations of b, the parent node. Because I’ve targeted the rotation on the child node, which direction the rotation is in is implicit, based on whether its the left or right child of its parent.

To implement a rotation, we pass it the “target” node (a above) along with its parent (not necessary if we have parent pointers). We check to see whether a is a left or right child of its parent and then fill in the other nodes:

/* rotate(parent,child)
   Rotates child node with its parent.

   Returns: new parent node.
   Assumes: child is a child of parent, and neither is null.
*/
node* rotate(node* parent, node* child) {
    assert(parent != nullptr && child != nullptr);
    assert(child == parent->left || child == parent->right);

    if(child == parent->left) {
        node* y = child->right;

        // Rearrange nodes
        node* temp = parent;        
        child->right = temp;
        temp->left = y; 
    }
    else {
        node* y = child->left;

        node* temp = parent;        
        child->left = temp;
        temp->right = y;
    }
    return child;
}

We return a pointer to the new parent so that we can use this to rewrite the tree, by doing something like

// Rotate left grandchild with left child.
root->left = rotate(root->left, root->left->left);

If you have parent pointers, it’s possible to write a version of rotate that does the rotation “in place”, and does not need to return the new parent node.

AVL trees

An AVL tree is a binary search tree that is height balanced; for any node, the heights of its left and right child differ by no more than 1. That is,

abs(height(node->left) - height(node->right)) <= 1

The height of a node is defined, one again, as either 0 for empty trees, except that we don’t want to be constantly recalculating the heights from scratch, so we store them in the nodes themselves:

struct node {
  int key;
  node* left;
  node* right;
  node* parent;
  int height;
};

int height(node* root) {
  return root == nullptr ? -1 : root->height;
}

(As before, the empty tree, nullptr has a height of -1.)

Note, in particular, that if you start at a leaf and go upward, the height of the nodes you visit always increases.

In order to build an AVL tree, we have to store the height of each node in the node (technically we can get away with just storing the differences in heights between the left and right subtrees, which will always be -1, 0, or +1).

When we insert a node, we may break the height balance property, but we can use rotations to move things around to restore it. After an insert, we walk back up the path to the root, updating height values and doing rotations as necessary to fix things.

To understand how the different tree operations work on an AVL tree, let’s examine how the rotation affects the height of the subtrees by looking at an example:

Effect of single rotation on height

Here we have an imbalance at p, caused by c having a height of \(h+2\) while Z has a height of \(h\). This could be the result of an insertion into X, increasing its height, or maybe a removal from Z. (We’ll see in the next example that a removal from Y would result in a different situation.)

To fix the imbalance we simply rotate c with its parent p. This method is used whenever the imbalance is on the “outside”:

The other possibility is when Y rather than X is the taller of c‘s subtrees. This is an “inside” imbalance, and this rotation is not enough to fix it:

Effect of single rotation on height

Instead, we have to perform a double rotation: first, rotate c with b, then, rotate c with a:

Effect of double | rotation on height

Note that it doesn’t make any difference whether the X or Y is taller: the double-rotation works to balance things out if Y is 1 taller than X. Thus, we don’t actually care about X vs Y, we never need to look at them or their heights, they are handled automatically by the rotation of c.

(The right-inside case is just the mirror image of this.)

Insertion

We begin by inserting the new node as usual. Any violations of the height-balance property must be on the path from the root of the tree to the new node (because that is the only path that has been changed). So we walk up the path to the root (following parent pointers), checking heights as we go:

// node n has just been inserted into the tree
node* p = n->parent;

while(p && p->parent) {
  // Update this node's height
  p->height = 1 + max(height(p->left), height(p->right)); 

  if(abs(height(p->left) - height(p->right)) > 1) {
    // Unbalanced height, fix
    if(p == p->parent->left)
        p->parent->left = fix_height(p);
    else
        p->parent->right = fix_height(p);
  }

  p = p->parent;
}

The function fix_height(node*) (taking a pointer to the node at which the imbalance occurs) should check for the above situations and apply a rotation or double-rotation as needed to repair the height.

The code

    if(p == p->parent->left)
        p->parent->left = fix_height(p);
    else
        p->parent->right = fix_height(p);

is a fairly common pattern: in order to modify the tree in-place, we have to figure out how p is related to its parent: is it the left or right child? We then replace that child specifically.

(It’s possible to build AVL trees without parent pointers, by exploiting the fact that the same recursion that goes down into the tree must go back up, as it returns. We can do the fixup operation during the recursion, although getting the order right is tricky.)

Insertion example: insert 1 2 3 4 5 6 7 8 into an empty tree.

Deletion

Deletion is handled the same way: the only difference is the conceptual one: instead of thinking of the heights of the subtrees as \(h\) and \(h+1\), think of them as \(h-1\) and \(h\). The actual method for repairing the tree is the same: start at the modified node, and walk up to the root, fixing imbalanced nodes as you go. Note that in the two-child case, where we swap the target node with its predecessor/successor and then (recursively) delete the one-child node, we start the “fixing” at the one-child node, not at the original location.

Deletion example: delete 4 from the above tree. Delete 8, 1, 6, 2.

Splay trees

A splay tree is a BST which uses amortized rebalancing. After an insert or delete or a find we perform a number of splay operations, which tend to make the tree more balanced. Note that while AVL trees only rebalance the tree when its structure is modified, splay trees rebalance even when we are just find-ing a node; they take every opportunity to improve the structure of a tree, but they tend to not do very much at once.

A splay is divided into three cases:

To perform a full splay operation, we simply repeat the above until p is the root of the tree (has no parent). A full splay is performed after each find, and thus after every insert and remove, too. Each splay tends to make the tree slightly more balanced, while at the same time moving the node p up to the root (so that subsequent finds for it will take O(1) time). If p was a deep left child, then the full splay will have moved a good number of nodes over to the right side.

Deletion for splay trees splays the parent of the removed node, moving it all the way to the root. This is based on the assumption that if you delete x from the tree, you will probably delete or otherwise access other values close to x in the future, so it will be good to move them close to the root. (And, of course, it helps to balance the tree.)

Splay trees support a number of other interesting operations, all based on the fact that a splay lets us move an arbitrary value to the root of the tree.

Amortized analysis of Splay Trees

Because a single splay operation does not completely balance the tree, but merely makes it “more balanced” we need amortized analysis. We intend to show that the amortized cost of a find or insert operation is \(O(\log n)\); that is, that the tree is balanced “on average” over a series of operations. To do this, we will use a different amortized analysis, the potential method. Like the accounting method, the potential method associates with a data structure an extra amount of “resources”, called potential. An operation can have an amortized cost higher than its real cost, adding the extra to the structure’s potential, or it can spend some potential and have its amortized cost be lower than its real cost. As with the accounting method, the potential method serves as a way of smoothing out the differences in real cost over a series of operations.

The difference between the potential method and the accounting method is that while the accounting method focuses on the amount of “credit” accumulated in the structure — or, more often, in particular parts of the structure — in the potential method, the emphasis is usually on the change in potential caused by an operation. Similarly, potential is usually associated with the entire structure, rather than individual elements of it.

We define the potential of a tree to be a measurement of its “balanced-ness”: define

$$\text{size}(t) = \text{number of nodes in }t$$ $$\text{rank}(t) = \log_2 (\text{size}(t))$$ $$\text{Potential}: P(t) = \Sigma_{n \in t} \text{rank}(n)$$

P(t) will tend to be high for poorly-balanced trees, and low for well-balanced trees. To see why this is the case, note that for a well-balanced subtree, the rank will be roughly equal to its height. For an unbalanced tree, the height will be greater than the rank, closer to \(2^{\text{rank}(t)}\).

An important property of these definitions is that is c is a subtree of p, then \(\text{size}( c) \le \text{size}(p)\) and similarly \(\text{rank}( c) \le \text{rank}(p)\). It’s not possible for a subtree to have more nodes than the tree which contains it.

DIAGRAMS

A perfectly-balanced tree will have a rank exactly equal to the number of nodes in the tree.

To use the potential method, we calculate the change in potential ΔP for each of the cases in the splay operation. We are not yet looking at all the rotations done over the entire splay, just the change caused by a single Zig, Zig-Zig, or Zig-Zag step. If x is the target node, we have

Thus, we can conclude that in all three cases, the change in potential is

$$\Delta P \le 2 (\text{rank}(x’) - \text{rank}(x))$$

The total cost of any splay operation is the sum of all its steps, but since each step consists of \(\text{Ending P} - \text{Starting P}\), where the Ending P of one operation is the Starting P of the next, so the sum telescopes: all the interior terms cancel, and the total cost reduces to just

$$\le 2 (\text{rank}(\text{root_x}) - \text{rank}(x))$$

which is \(O(\log n)\). The amortized cost of m operations is \(O(m \log n)\) and each individual operation has an amortized cost of \(O(\log n)\).