A Lightweight Windows Thread Library

Snip #6: A Lightweight Windows Thread Library

Introduction

Multicore processors are here to stay. To really take advantage of their benefits, software needs to be designed with more parallelism in mind. The old model of doing one things at a time and tweaking every last CPU cycle out of the code just doesn't go far enough in maximizing performance anymore (not that it ever did, mind you, but I digress); being able to do many things in parallel will go much farther, and allow for a more responsive and stable piece of software. And doing things in parallel is where threads come in.

The problem with parallelism (and thus threads) is complexity. The more things that are happening, the more complex the program becomes and thus the harder it is to debug. While Microsoft has provided a comprehensive set of API functions for utilizing threads, using the APIs directly can be cumbersome and error prone. Wrapping the library in a lightweight, easy to use C++ wrapper significantly reduces the risk of error and at the same time, simplifies the use of the APIs. For me, that's a winning combination. While there's plenty of libraries to choose from out there, I wasn't all that happy with any of them, and so set out to write my own. Here I'll present the library to you, for your use, education, and entertainment. Enjoy!

Elements of Threading

I'm not going to attempt to teach all the in's and out's of threading in this article, there's plenty of information out there already that does that. But I do want to briefly review the key elements of threading that I'll be including in the library, to set the stage for the library and it's intended uses. For any multithreaded program, there are three key considerations with respect to threads:

Starting and stopping the thread: If you can't start the thread, you're dead in the water before you even get started. When I started this library, my primary objective was to simplify this process as much as possible.
Synchronization between threads: A thread that goes off and does it's own thing is fine, but at some point it needs to do something that relates to the rest of the program. This generally requires access to some shared or global data, or providing access to some data that's owned by the thread. Synchronization mechanisms provide a means for threads to safely access common data.
Signalling threads: Signalling is a way for one thread to get the attention of another, to signify the completion of some task, the readiness of some piece of data, or some other thing that the thread needs to be aware of. In the world of threads, signalling is achieved through the use of events.

While there are many other elements to threading under Windows, these three considerations provide the bulk of what is commonly needed and so these are where I focused my efforts.

Starting and Stopping Threads

To me, this was the single biggest hurdle in using threads. The mechanisms that Windows provides are pretty straightforward, but they are inconvenient, especially when you're attempting follow an object-oriented methodology. I wanted to encapsulate thread lifetime in a class, and hide as much of the API as possible. This would allow threads to be a more natural outgrowth of the program design and relieve the burden of thread management from parts of the program that should be focused on other areas. Using the Windows API, there are two pieces to starting a thread:

First, you have to have a thread function. This function defines the thread - it is executed in it's own thread, and when it exits, the thread terminates. During it's execution, it does whatever it is the thread's job is to do. It is exactly analogous to the main() (or WinMain()) function.
Once you have your thread function, the API call ::CreateThread() is used to kick it off. Among other things, this API calls the thread function and allows you to specify whether to start the thread immediately, or start it in a suspended state.

The API works but it's clunky and doesn't fit into an object-oriented environment at all. I wanted to hide these details behind a library that provided useful abstractions for these, but at the same time weren't severely limiting in terms of the design of the rest of my software. At the same time, I wanted the design of each thread to be as independent as possible of the management of the thread. So, the library should provide separate abstractions for thread management and the actual threads. This is a little different from many libraries I've seen, which tend to glom these things together. Separating these aspects allowed me to more fully focus on each one independently, and (hopefully) arrive at a more complete and useful solution.

This library abstracts threads into two key classes:

The thread_holder template provides mechanisms for starting and stopping your threads. A create_thread template function simplifies the use of this template with your own thread_base derivatives.
The thread_base abstract base class provides a starting point for your own thread objects (i.e., the objects that will do the work that the thread is intended to do).

Thread Management Part 1: The thread_holder Class

Managing the thread requires providing means for starting a thread and querying various properties of the thread, such as whether it's still running or not. I also wanted each manager object to own the object representing the thread's task (henceforth called the thread object). This complicates access to the thread object for communications purposes, but I felt that this was a Good Thing, since it makes it very clear in the code when you're managing the object and when you're communicating directly with the object. Conceptually, it seemed that the thread manager object should own the thread that it was managing, and so it is in this library. Creating the manager object doesn't start the thread, however - all threads must be explicitly started.

The thread_holder class provides the primary interface for managing threads. thread_holder is a template, parameterized on the type of the thread object (which must be a descendant of thread_base - more on that later). Anything not directly dependent on the template parameter is abstracted into the thread_holder_base class - this is where most of the actual management functionality is located. The thread object itself is held in a thread_object_holder template object (also parameterized on the type of the thread object). This separation prevents code bloat by centralizing non-type-dependent code in a common base class, and also provides a bit of trickery for controlling initialization order. When you put it all together, you end up with this hierarchy:

This is all immaterial when it comes to actually using the library tho, as the thread_holder template gloms it all into a single object that provides a complete interface, as shown in this example:

thread_holder<scanner> manager;
manager.object().initialize();
manager.start();
while ( manager.wait_for_completion(100) )
{
    // do something else
}

It's not necessary to poll the object for completion in this way, I show that here only for convenience. In practice, I've found that I start a thread in response to something that happens in the GUI, then periodically check for completion in response to a timer event. But that's immaterial to this discussion. What is relevant here is the separation of the thread object from the management of the thread:

The manager object provides methods for starting the thread (start()), accessing the thread object (object()), and polling the thread to see if it's still running (wait_for_completion()). It also provides a few other polling methods, access to the Windows-assigned thread ID and handle, and a method for suspending the thread.
The object() method provides direct access to the thread object. The thread object can be accessed at any time, whether the thread is running or not. It's important to note that the object() method provides access without any attempt to provide synchronization. This puts the onus on the thread object implementer to provide synchronization where necessary, but eliminates the overhead when it's not (such as in this case, where the object is being initialized prior to it being started).
Although it's not apparent here, when manager is destructed, it will wait for the thread to complete execution. This was necessary because Windows keeps the process around until all threads complete anyway, and if a thread accesses data owned by another thread (such as the main thread) that has already been terminated, you end up with hard-to-track-down access violations. This significantly improved my shutdown behavior, but created another problem: I had to keep references around to threads that I no longer needed to directly access; in one application, this meant potentially hundreds of useless references to be kept around. In the next section I'll discuss my methodology for getting around this limitation.

This last point led me to creating an alternative mechanism for creating threads: the create_thread function template.

Thread Management Part 1A: The create_thread Function Template

So I needed a way to create threads that didn't require me to keep a reference to them hanging around, but I also wanted to force the main thread to stick around until all the threads I'd created had been terminated. These are not compatible goals, since the second requires the thread manager's destructor to wait on thread completion - but that prevents me from losing references to the thread. Fortunately, the solution is simple: let the main program lose references to the threads it starts, but keep references around within the library. This is really something that thread pools are designed to handle, but that's a topic for a future article.

To achieve this, I developed an alternative mechanism for creating threads: the create_thread function template. This template dynamically creates a thread manager object and returns a smart pointer to it (courtesy of the boost library). In the meantime, it tucks a copy of the smart pointer away in an internal thread list. When the thread terminates, the pointer is removed from the list and (if there's no other copies of the smart pointer around) the thread manager is destroyed.

Using the create_thread template is simple:

// If no initialization is needed ...
create_thread<thread_object>()->start();

// Or, if you need to access it:
shared_ptr< thread_holder<thread_object> > pthread = create_thread<thread_object>();
// do some stuff with pthread
pthread->start();

The shared_ptr declaration is a little unwieldy, but unfortunately C++ doesn't (yet) provide a mechanism for simplifying that syntax. A preprocessor macro has to be used instead to simplify that:

#define THREAD_PTR(THREAD_OBJ_TYPE) boost::shared_ptr< slug::thread_holder<THREAD_OBJ_TYPE> >

Thread Management Part 2: The thread_base Class

All of this management stuff is great, but the real point to all of this is to be able to have a thread. Fortunately, that's the easy part. If you want to have a thread object, simply inherit from thread_base and implement the execute() method. Then, create a thread management object using your thread_base descendant as the template parameter, and your thread is up and running:

class my_thread_object: public thread_base
{
public:
  virtual void execute()
    {
      // do some stuff in this thread
    }
};

// ...

create_thread<my_thread_object>()->start();

The execute() method can do whatever you want, it's analagous to your program's main() function.

So you may be asking how this all works, considering that so far there's been no API calls (or not, if so then you might want to skip this part). Well you can see for yourself in the code, but to summarize, here's what's happening behind the scenes:

The thread_holder_base::start() method is actually where the thread is created. It calls the API ::CreateThread(), passing a function internal to the library as the thread's execute function.
::CreateThread(), conveniently, lets you specify a parameter to pass to the thread's execute function when the thread is started. The library uses this to pass a pointer to a thread_base object (hence the reason your thread object must descend from thread_base).
The library's thread execution function simply calls thread_base::execute() to kick off the thread. When that returns, it calls the on_terminate method, an optional method that lets you specify what to do when the thread terminates. Finally, it also cleans up the internal thread list, if you happened to use the create_thread() template to start your thread.

I've spent way too much time discussing the mechanisms at work behind something that happens only once per thread. Unfortunately, thread theory gets significantly more complicated from this point on. Fortunately for us, the mechanisms involved get simpler and so won't need nearly as much discussion.

Synchronization Between Threads

Thread synchronization is one of the thornier problems of multithreaded development. Deadlocks and race conditions are all too easy to create, no matter how complete a library you develop. But like it or not, if you have multiple threads, at some point they're going to have to share some resource - the screen, some data, a socket, something - and so you're going to have to deal with synchronization. Debugging these types of issues is beyond the scope of this article, but I'll try to give you at least a few pointers before the end.

The Windows API provides several mechanisms for synchronizing threads, including semaphores and mutexes. Both of these are costly to use in terms of execution time, so I went with a less costly mechanism called a critical section. Strictly speaking, a critical section is a piece of code that accesses a shared resource that must not be accessed concurrently by other threads. The Windows API provides a critical section mechanism that lets you denote the entry and exit points of critical sections of your code, and prevents concurrent access to those sections of code. Thus, they serve the same purpose as a semaphore or mutex, but at significantly lower cost.

There are four parts to using a Windows critical section object: it must be created, entered, exited, and destroyed (all using separate API calls, of course). A critical section can be entered and exited multiple times during the course of program execution, but will only ever be created and destroyed once. When one thread has entered a critical section, Windows will prevent other threads from entering the same critical section until the first thread has exited. This mechanism can easily be wrapped into a class:

class critical_section
{
public:
  critical_section();
  ~critical_section();

  void enter();
  void leave();

private:
  CRITICAL_SECTION m_cs;
};

Using this class is a piece of cake:

In the object that owns some shared resource, create a critical section object.
Before accessing the resource, call enter(), and after accessing it, call leave().

This last point stuck in my craw a bit. Sometimes, it's necessary to directly control when you enter and exit a critical section. But what if an exception is thrown that prevents the explicit exit() call? This is exactly the sort of problem smart pointers solve for memory, so I borrowed that philosophy in developing a wrapper object for critical_section's:

class cs_holder
{
public:
    cs_holder(critical_section &cs):m_cs(cs) { m_cs.enter(); }
    ~cs_holder() { m_cs.leave(); }

private:
    critical_section &m_cs;
};

So how do you use this? Here's an example. The processing_queue class is used by threads to schedule jobs with some thread that processes them. Any time the queue is accessed, the access should be protected by a critical section. To simplify this process, processing_queue handles all the critical section details; other threads simply call processing_queue's methods directly without having to worry about them:

class processing_queue
{
public:
  void schedule_job(unsigned id)
    {
      cs_holder csh(q_protector);
      q.push_back(id);
    }

  unsigned get_next_job()
    {
      cs_holder csh(q_protector);
      unsigned id = q.front();
      q.pop_front();
      return id;
    }

  bool is_scheduled_jobs() const
    {
      cs_holder csh(q_protector);
      return q.size() != 0;
    }

private:
  critical_section q_protector;
  std::deque<unsigned> q;
};

Now the thread doing the processing can safely access the queue just by calling the methods of a processing_queue object. If another thread happens to be calling some method of the processing_queue object at the same time, the processing thread will block until the first thread completes. Simple!

Signalling Threads

Sometimes a thread needs to tell another thread that something happened. Sometimes a thread needs to wait for another thread to do something before it can continue. You could implement mechanisms for performing these types of things using critical sections in conjunction with some sort of signalling object that you devise, but a home-grown implementation of this would necessarily require a thread to enter and exit a critical section just to see if a notifcation has been set. Windows already provides a mechanism for providing these sorts of notifications that are cheap and easy: events.

Events are Windows objects that live in one of two states: signalled, or not signalled. An event is signalled to indicate that something interesting has happened. The event is cleared when the interesting thing that happened is no longer relevant. Events can be polled to see if they are signalled or not, and threads can wait on an event to be signalled before continuing with it's processing. All this is encapsulated in the event_object class:

class event_object
{
public:
  explicit event_object(const char *name = 0);
  ~event_object();
  void set();
  void clear();
  bool poll();
  bool wait_for_completion(DWORD ms = INFINITE);

private:
  HANDLE m_handle;
};

Using events, we could rewrite the processing_queue to set an event whenever there's a job placed on the queue and clear the event when the queue is empty.

class processing_queue
{
public:
  void schedule_job(unsigned id)
    {
      cs_holder csh(q_protector);
      q.push_back(id);
      q_event.set();
    }

  unsigned get_next_job()
    {
      cs_holder csh(q_protector);
      unsigned id = q.front();
      q.pop_front();
      if ( q.empty() ) q_event.clear();
      return id;
    }

  bool is_scheduled_jobs() const
    {
        return q_event.poll();
    }

  bool wait_for_job(DWORD ms = INFINITE)
    {
      return q_event.wait_for_completion(ms);
    }

private:
  critical_section q_protector;
  event_object q_event;
  std::deque<unsigned> q;
};

I've added a method here that allows the processing thread to go to sleep until it receives a job to do, thus improving the scheduling of other threads.

Conclusion

The thread library that I've presented here is far from complete; the Windows API is extensive and only getting bigger. But it encapsulates a sufficient portion of the API to make using threads in your programs very simple, and adds enough robustness to reduce the likelihood of common errors like forgetting to exit a critical section. With this library (or one like it), your multithreaded programs will be more responsive and more stable, and you'll never again have to be bothered with the gory details of Windows API threading.

Feel free to use this library in your programs, and if you make any interesting additions or changes to it, post a note to let me know. You can download the files here.

Games, Screensavers, and Photo UtilitiesTips for C++ and C++Builder