Binary heaps

Motivation: Priority Queues

A priority queue is a queue in which items are enqueued with a numeric priority; when a dequeue is performed, the item removed is not the first- added item as in a standard FIFO queue, but the item with the highest (sometimes lowest) priority.

A simple implementation of a priority queue can be constructed using an array or vector:

template<typename Item>
class priority_queue {
  public:
    ⋮
  private:
    struct item 
    {
        Item i;
        int priority;
    };

    vector<item> q;
};

To enqueue an item, we simply push_back it into the vector:

void enqueue(Item i, int priority)
{
    q.push_back( item{ i, priority } ); 
}

To dequeue an item, we find the item with the highest priority, then (like with the bag data structure) swap it with the last element and pop_back:

Item dequeue()
{
    size_t index = 0;
    for(size_t i = 1; i < q.size(); ++i)
        if(q[i].priority > q[index].priority)
            index = i;

    swap(q[index], q.back());

    Item result = q.back().i; 
    q.pop_back();
    return result;
}

The complexity of enqueue is \(O(1)\), amortized; the complexity of dequeue is \(O(n)\).

An alternate implementation would be to store the items in ascending priority order (as in the ordered_array). enqueue-ing an item would insert it in the correct position (\(O(n)\)) while dequeue-ing would be just pop_back, as the highest-priority item would always be last. This swaps the complexities above, so that enqueue is \(O(1)\) and dequeue is \(O(1)\).

In either implementation, if we assume that enqueues and dequeues are performed equally frequently, then the overall complexity would be \(O(n)\). Can we do better?

Binary Heaps

A heap is like a binary tree, but instead of the search-order property, a heap has the heap order property:

The value in a node in a heap must be greater than the values of both its children.

(A heap like this is called a max heap; a min heap requires that every node’s value be less than its children. We’ll assume that we’re working with max-heaps; the change to min-heaps is trivial – just switch < for >.)

In addition, a heap is a complete binary tree: all the levels in the tree, except possibly the last one, are full, and the last level fills in from left to right.

This is an example of a binary (max) heap:

     8
   /   \
  7     6
 / \   /
5   3 1

While we can build a binary heap using nodes and pointers, finding the “last” node in the bottom level is somewhat tricky. Instead, we exploit the fact that a heap is always a complete binary tree to store it in an array.

Array implementation

The nice thing about this structure is that we don’t actually need to store it in a tree structure, with pointers and everything. Instead, we can just store it in an array or vector, listing off each row from left to right:

8 7 6 5 3 1 
-------------
1 2 3 4 5 6

(I’ve listed the array indices starting at 1, for convenience.) Given this representation, and the index of a node, how do we find it’s children? E.g., if we are at 7, what do we do to get to its left and right children?

Similarly, if we have a node at \(n\) then it’s parent is located at \(n / 2\) (using integer division).

A first sketch of a max-heap of ints looks like:

class heap {
public:
  heap() : _data(1) {}

  bool empty() { return size() == 0; }
  bool size() { return _data.size() - 1; }

  int max() { return _data.at(1); }

  int left(int n) { return 2*n; }
  int right(int n) { return 2*n + 1; }
  int parent(n) { return n / 2; }

  int& at(int n) {
    return _data.at(n-1);
  }

private:
  vector<int> _data;
}

The most useful operation on a heap is max which just returns the root. The idea of a heap is that it is a data structure in which we can add (and sometimes remove) element, and always have easy access to the maximal element. Note that getting the maximum, as long as the heap property is maintained, is always a \(O(1)\) operation.

Inserting new elements

To insert a new element, we begin by just adding it to the end of the array.

void heap::insert(int n) {
  _data.push_back(n);
  // ...
}

The new element might violate the heap property, it might be larger than its parent. If it is, we can just swap the element and its parent. We continue until the element is not out-of-order, or until we reach the root:

void heap::insert(int n) {
  _data.push_back(n);
  fix_up(_data.size());
}

void heap::fix_up(int n) {

  while(n != 1) {}
    if(at(n) > at(parent(n))) {
      std::swap(at(n), at(parent(n)));
      n = parent(n);
    }
    else
      break;
  }
}

This works by moving the new value up in the heap until it is in the right location. Because it has to move the value up the height of the heap, the runtime is \(O(\log n)\).

Example:

Extraction

If we want to remove (not just look at) the maximal element, then we have a hole at the root of the tree. We fix this by filling the hole with the last element in the heap, and then pushing it down until it’s in the right place. 

int heap::extract() 
{
  int r = max();

  // Put the last element at the root, and then remove the last element
  _data.at(1) = _data.at(_data.at(size-1));
  _data.pop_back();

  fix_down(1); 

  return r;
}

void heap::fix_down(int n) 
{
  while(n < size()) {
    int largest = n;
    if(at(n) < at(left(n))) 
      largest = left(n);
    else if(at(n) < at(right(n))) 
      largest = right(n);

    if(largest != n) {
      std::swap(at(n), at(largest));
      n = largest;
    }
    else
      break; //Found the right spot
  }
}

On average, extract has to walk the height of the heap, so it takes \(O(\log n)\) time.

Increase key

Although this operation might appear useless, we’ll see a use for it in a bit. We can increase the value in a particular node, by simply changing it and then running fix_up on it to repair the heap.

void heap::increase(int n, int value) {
  assert(at(n) <= value);

  at(n) = value;
  fix_up(n);
}

On average, this takes \(O(\log n)\) time.

Example:

Delete an item (other than the root)

The procedure is the same as for deleting the root: swap the deleted item with the end of the vector, and then use fix_down to repair the heap. This works because any node in a heap is actually the root of a sub-heap that begins with itself.

void heap::delete(int n) {
  std::swap(at(size()-1), at(n));
  _data.pop_back();

  fix_down(n);
}

Example:

Building a heap

We could build a heap from a sequence of values by inserting them one by one, but this is not the most efficient way. Instead, we just copy all the values into the heap array (ignoring, for the moment, the fact that this breaks the heap order property) and then run fix_down on every element from the bottom up. This is more efficient because as we move up the tree, the lower regions will already have the heap property, so our job gets easier as we go.

void heap::heap(vector<int> data) : _data(data) {
  for(int i = _data.size()-1; i >= 0; i--)
    fix_down(i);
}

The analysis is somewhat complex, but essentially, we can build a heap in \(O(n)\) time. This is because most of the fix_downs performed are near the bottom, of the heap, and thus take closer to \(O(1)\) time than the expected \(O(\log n)\) time. Only a few fix_downs are performed near the top of the heap; summing all the times needed for all the fix_downs results in a total time of \(O(n)\).

Merging two heaps

It’s possible to merge two heaps into a new one. An obvious way would be to just insert elements from both into a new empty heap, but this would take \(O(n \log n)\) time in the size of the larger heap. Instead, we’ll use the same trick as for building a heap: just take the contents of both heaps, put them into a new vector (not caring about order) and then fix the heap from the bottom up:

void heap::heap(vector<int> d1, vector<int> d2) : _data() {
  // Add both heap's data to ours
  data.insert(data.end(), d1.begin(), d1.end());
  data.insert(data.end(), d2.begin(), d2.end());

  // Fix the heap, bottom up
  for(int i = _data.size()-1; i >= 0; i--)
    fix_down(i);  
}

This is basically the same as “building” a heap from scratch, and runs in time \(O(m + n)\) time.

(There are heap implementations that can be merged faster than this, at an increase in the complexity of the code.)

Heap size

Here, we’ve implemented a heap on a vector, so it has unlimited size. It’s fairly common for heaps to be implemented on top of fixed-size arrays, giving the heap itself a maximum size.

Heapsort

Because we can build a heap in \(O(n)\) time, and extract the maximum in \(O(\log n)\) time, this suggests an algorithm for sorting:

  1. Transform the input into a heap

  2. As long as the heap is not empty, extract elements and insert them at the beginning of an initially-empty vector. (A max heap will output elements in descending order.)

vector<int> heapsort(const vector<int>& data) {
  heap h{data};

  vector<int> results(data.size());
  for(int i = data.size()-1; i >= 0; i--) {
    results.at(i) = h.extract();
  }

  return results;
}

If extract takes logarithmic time, and we have to do \(O(n)\) extracts, then the runtime of this algorithm is \(O(n \log n)\) time. It’s possible to modify the algorithm to sort in-place, by observing that as the heap shrinks by one element, the results grows by one element.

Disjoint Sets

A disjoint set is a specialized kind of tree data structure that has only parent pointers, no child pointers. This means that, given a child, we can work our way up to the root, but we can never go down; given a node, we can find it its ancestors, but not its descendants.

As an example application of when a disjoint set is useful, consider the following scenario: I want to take all your homework submissions and check to see if any of you have copied off each other. It’s possible that two, three, or more individuals may all have submitted identical work, so I really want to take all the n submissions, and group them into m unique submissions (with \(m \le n\)) while keeping track of which group each submission is in.

One way to do this would be to compare every possible pair of submissions (requiring \(O(n^2)\) comparisons), keeping track of the identical matches. The problem with this is that it does extra work. If I know that submission A is identical to submission B, and I know that submission B is identical to submission C, do I still need to check A against C? No; thanks to the transitive property, I already know the result of that comparison. Instead, we’re going to keep track of all the submissions that fall into an identical group, and avoid doing comparisons between submissions that are in the same group.

Initially, every submission is presumed unique. (Innocent until proven guilty.) This means that at the beginning, \(m = n\); the number of groups is equal to the number of submissions. I pick a pair of submissions which are not already in the same group, and I compare them. If they are the same, then I take those two submissions and put them into the same group. Now the number of groups is one less than the number of submissions. I continue with one of the submissions in the group (it doesn’t matter which; they are identical), comparing it to the other submissions/groups. Whenever I find a copy, I put it in the group.

Later on, I may need to merge two different groups together. Eventually, every submission will be in a group. Some will be in unique groups (group of size 1); these are the unique, non-copied submissions. Some will be in larger groups, indicating that all the submissions in that group are copies of each other.

In order to implement this system, I need several operations:

Let’s say our class for submissions looks like this:

struct submission {
    submission* parent = nullptr; 
    string text; 
};

We will say that two submissions are in the same group (are identical) if they share a common root. A “group” is just a pointer to a root submission: a submission whose parent pointer is nullptr. To find the group of a submission we just walk up its parent chain until we hit null:

submission* get_group(submission* s) {
    while(s->parent != nullptr)
        s = s->parent;

    return s;
}

Creating a group is just a matter of getting a pointer to a submission, and we can keep track of the collection of all groups in a vector of submission*. Similarly, to get an “example” of a group, we can just use the submission that it points to.

To merge two groups, I simply find the root of one, and then make it a child of the other:

DIAGRAM

void merge_groups(submission* g1, submission* g2) {
  g2 = get_group(g2); // Now g2->parent == nullptr
  g2->parent = g1;
}

Now, get_group on any element of g2 will return g1.

To count how many groups there are, we simply loop through the vector and count how many submissions still have nullptr parents:

std::vector<submission*> subs;

int c = 0;
for(submission* s : subs)
    c += (s->parent == nullptr);

Optimizations

There are a few optimizations we can do that will make things faster:

If both these optimizations are used, every disjoint set operation effectively runs in constant time. Technically, the operations run in \(O(\alpha(n))\) time, where \(\alpha(n)\) is a function that grows very slowly. \(\alpha(n)\ \le 5\) for all \(n \le \) the number of atoms in the universe.

Heap applications: Priority queues

Heaps become more interesting when we can attach values to the nodes. This allows us to attach priorities to items, and always be able to get the highest-priority item. In fact, a heap like this forms a priority queue: insert is like enqueue while extract is like dequeue. The difference between this and a traditional queue is that this one lets us prioritize elements; higher priority elements will be extracted sooner than lower ones (rather than in a string LIFO order).

The increase operation now actually has a use: we can use it to raise the priority of something that is already in the queue, pushing it towards the front of the queue.

Depending on how priorities are assigned, a priority queue can emulate a traditional queue or a stack.

Priority queues are used in operating systems for scheduling threads and processes. Because not all processes are equally important, some get higher priorities and thus come to the front of the queue more often.

Priority queues have other applications, however; basically, any situations where we want to keep track of a number of different choices and want to get access to the next best choice. For example, the A* pathfinding algorithm, used in many games, uses a priority queue to keep track of paths that need to be searched: higher priority paths correspond to those paths that are more likely to yield a solution. A number of graph theoretic algorithms rely on priority queues; we’ll look at some of these later in the semester.

It’s possible to create a double-ended priority queue, one where we have fast access to both the highest and the lowest elements. (A corresponding variant of the heap called a min-max heap provides \(O(1)\) access to both the min and max.)

One question we didn’t address above is what happens when we have entries with duplicate priorities. The normal way to deal with this is to state that if there are multiple entries in the queue whose priority is the min/max, then extract_min is allowed to return any of them (i.e., it is unpredictable which will be extracted). Note that this means that you cannot use a priority queue as a drop-in replacement for a normal FIFO queue, by simply setting all the priorities to 1 (or some other fixed values). If you want a priority queue that is also a FIFO queue on duplicates, then you can either use a combination of the original priority and the “insertion time” as the priority to order on, or you can modify the insert/extract operations so that they implicitly respect the original order. For extract, this means using <= rather than < when moving the substitute down into the heap, and prefering to swap it with the left child if both children are equal in priority.

Process simulation on priority queues

Let’s build a simulation of the way the operating system schedules processes and see what happens

#include <algorithm>
#include <iostream>
#include <string>
#include <vector>
using namespace std;

class process {
  public:
    process(string n, int p) {
      name = n;
      priority = orig_priority = p;
    }

    bool run() {
      cout << "Process " << name << " << running...\n";
      priority = orig_priority;
      return true;
    }

    bool operator< (const process& other) const {
      return priority < other.priority;
    }

    string name;
    int priority;
    int orig_priority;
};

using process_queue = vector<process>;
process_queue processes;


void enqueue(const process& p) {
    processes.push_back(p);
  push_heap(processes.begin(), processes.end());
}

process dequeue() {
  process p = processes.front();
  pop_heap(processes.begin(), processes.end());
  processes.pop_back();

  return p;
}

void simulate() {
  int i = 0;

  // Run for 1000 cycles
  while(!processes.empty() && i < 1000) {
    process p = dequeue();

    if(p.run())
      enqueue(p);

      ++i;
  }
}

int main() {
  enqueue(process("firefox.exe", 10));
  enqueue(process("steam.exe", 12));
  enqueue(process("photoshop.exe", 9));
  enqueue(process("norton.exe", 8));

  simulate();

  return 0;
}

What happens? After steam.exe runs, it just keeps running, the other processes never get another turn! This makes sense when you think about it: after the spooler process runs, it gets added back to the queue, again with a priority of 5, which means it immediately goes to the front of the queue. This is called starvation, when low-priority processes never get to run.

To combat this, we need to add priority aging: this means that the priorities of all processes that don’t run get decreased by 1, until they do. Basically, we decrease every node but the root and then fix the heap. Eventually, this will make those other processes have a lower priority than spooler and then they’ll get a turn. 

void simulate() {
  while(!processes.empty()) {
    process p = dequeue();

    // Increase all other processes priorities
    for(process& other : processes)
      other.priority++;    

    if(p.run())
      enqueue(p); // Add back to the queue at same priority
  }
}

Note that we don’t need to fix the elements in the heap if we are decreasing (or increasing) all of them. Increasing all the priorities does not change their order relative to each other and thus the heap order property still holds for the new priorities.

Normally, every process is assigned an amount of time called a quantum; under normal circumstances, every process gets the same amount of time. Let’s keep track of how much time each process gets, and then see how the priorities affect how much “CPU time” each process gets:

class process {
  public:
    string name;
    int priority;
    int time = 0;
    ...
}

void simulate() {
  int cycle = 0;
  while(!processes.empty() && cycle < 1000) {
    process p = dequeue();

    // Decrement all other processes priorities
    for(process& other : processes)
      other.priority--;

    p.time++; // 1 additional quantum
    if(p.run())
      enqueue(p); // Add back to the queue at same priority

    cycle++;
  }

Here we stop the simulation after 1000 cycles (because normally it never stops).

Then we can print out the priorities:

  int main() {
    using std::cout;
    using std::endl;
    ...
    simulate();

    for(process& p : processes)
        cout << "Process " << p.name 
             << " priority = " << p.priority 
             << ", CPU time = " << p.time << endl;

    return 0;
}