You are here: Start » Usage Tips » Processing Images in Worker Thread

Processing Images in Worker Thread

Introduction to the Problem

Adaptive Vision Library is a C++ library, that is designed for efficient image processing in C++ applications. A typical application uses a single primary thread for the user interface and can optionally use additional worker threads for data processing without freezing the main window of the application. Images processing can be a time-consuming task, so performing it in a separate worker thread is recommended, especially for processing performed in continuous mode.

Processing images in a worker thread is asynchronous and it means that accessing the resources by the worker thread and the main thread has to be coordinated. Otherwise, both threads could access the same resource at the same time, what would lead to unpredictable data corruption. The typical resource that has to be protected to be thread-safe is the image buffer. Typically, the worker thread of the vision application has a loop. In this loop it grabs images from a camera and does some kind of processing. Images are stored in memory of a buffer as avl::Image data. The main thread (UI thread) presents the results of the processing and/or images from the camera. It has to be ensured that the images are not read by the UI thread and processed by the worker thread at the same time.

Please note that the GUI controls should never be accessed directly from the worker thread. To display the results of the worker thread processing in the GUI, a resource access control has to be used.

Example Application and Image Buffer Synchronization

This article does not present the rules of multithreaded programming. It only focuses on the most typical aspects of it, that can be met when writing applications with Adaptive Vision Library. An example application that uses the main thread and the worker thread can be found among the examples distributed with Adaptive Vision Library. It is called MFC Simple Streaming and the easiest way to open it is by opening Examples directory of Adaptive Vision Library from the Start Menu. The application is located in 03 GigEVision tutorial subdirectory. It is a good template for other vision applications processing images in a separate thread. It is written using MFC, but the basics of multithreading stay the same for all other technologies.

There are many techniques of synchronization of a shared resources access in a multithreading environment. Each of them is good as long as it protects the resources in all states that the application can be in and as long as it properly handles thrown exceptions, application closing etc.

In the example application, the main form of the application has a private field called m_videoWorker that represents the worker thread:

class ExampleDlg : public CDialog
{
private:
	(...)
	GigEVideoWorker m_videoWorker;
	(...)
}

The GigEVideoWorker class contains the image buffer:

class GigEVideoWorker
{
	(...)
private:
	avl::Image m_imageBuffer;
	(...)
}

This is the image buffer that contains the image received from the camera that needs to be protected from parallel access from worker thread and from the main thread that displays the image in the main form. The access synchronization is internally achieved using critical section and EnterCriticalSection and LeaveCriticalSection functions of the Windows operating system. When one thread calls the GigEVideoWorker::LockResults() function, it enters the critical section and no other thread can access the image buffer until the thread that got the lock calls GigEVideoWorker::UnlockResults(). When one thread enters the critical section, other threads that try to enter the critical section will be suspended (blocked) until the one leaves the critical section.

Using functions like GigEVideoWorker::LockResults() and GigEVideoWorker::UnlockResults() is a good choice for protecting the image buffer from accessing by multiple threads, but what if due to an error in the code the resource is locked but never unlocked? It can happen for example in a situation when an exception is thrown inside the critical section and the code lacks the try/catch statement in the function that locks and should unlock the resource. In the example application this problem has been resolved using the RAII programming idiom. RAII stands for Resource Acquisition Is Initialization and in short it means that the resource is acquired by creating the synchronization object and is released by destroying it. In the example application being described here, there is the class called VideoWorkerResultsGuard. It exclusively calls the previously mentioned GigEVideoWorker::LockResults() and GigEVideoWorker::UnlockResults() functions in constructor and destructor. The instance of this VideoWorkerResultsGuard class is the synchronization object. The code of the class is listed below.

class VideoWorkerResultsGuard
{
private:
	GigEVideoWorker& m_object;

	VideoWorkerResultsGuard( const VideoWorkerResultsGuard& );	// = delete

public:
	explicit VideoWorkerResultsGuard( GigEVideoWorker& object )
	: m_object(object)
	{
		m_object.LockResults();
	}

	~VideoWorkerResultsGuard()
	{
		m_object.UnlockResults();
	}
};

It can be easily seen that when the object of VideoWorkerResultsGuard is created, the thread that creates it calls the LockResults() function and by that it enters the critical section protecting the image buffer. When the object is destroyed, the thread leaves the critical section. Please note that the destructor of every object is automatically called in C++ when the automatic variable goes out of scope. It also covers the cases, when the variable goes out of scope because of the exception thrown from within of the critical section. Using RAII pattern allows programmer to easily synchronize the access to shared resources from multiple threads. When a thread needs to access a shared image buffer, it has to create the VideoWorkerResultsGuard object and destroy it (or let it be destroyed automatically when the object goes out of scope) when the access to the image buffer is no longer needed. The example usage of this synchronization looks as follows:

// Retrieve the results under lock.
{
	VideoWorkerResultsGuard guard(m_videoWorker);
	(...)
	avl::AVLImageToCImage(m_videoWorker.GetLastResultData(), width, height, false, m_lastImage);
	(...)
}

The method GetLastResultData() returns the reference to the shared image buffer. It can be safely used thanks to the usage of VideoWorkerResultsGuard object.

Notifications about Image Ready to Display

Another issue that needs to be considered in a typical application that processes images and uses a worker thread is notifying the main thread that the image processed by the worker thread is ready to display. Such notifications can be implemented in several ways. The one that has been used in the example application is using system function PostMessage(). When the worker thread has the image ready for presentation, it copies it to the m_lastResultData buffer (this is the protected one) and posts the notification message to the main window of the application:

//
// TODO: Compute the result data and put them in the shared buffer (just copy the source image).
//
m_lastResultData = m_imageBuffer;

// Send notification message
if (PostMessage(m_hNotificationWindow, m_notificationMessage, 0, NULL))
{
	m_lastResultProcessed = false;
}

The message is received by the main (UI) thread. Once it's received, the main thread acquires the access to the shared image buffer by creating the VideoWorkerResultsGuard object. Then, the image can be safely displayed.

The worker thread has a flag called m_lastResultProcessed. The flag set to false indicates that the notification about image ready to display had been posted to the main thread but the main thread has not processed (displayed) the image yet. The flag is set to false just after posting the notification message. The main thread sets the flag back to true using NotificationGiveFeedback() function:

void GigEVideoWorker::NotificationGiveFeedback( void )
{
	VideoWorkerResultsGuard guard(*this);
	m_lastResultProcessed = true;
}

Once the worker thread has sent the notification message, it can acquire and perform the next frame from the camera, but there's no point in sending the next notification until the previous is performed by the UI thread. Sending the new notifications without performing the old ones could lead to cumulating them in the messages queue of the main window. This is why the worker thread of the example application checks if the previous notification message has been performed and sends the next one only if the processing of the previous is finished:

if (m_lastResultProcessed && NULL != m_hNotificationWindow)
{
	// Create the result in shared buffers under lock.
	VideoWorkerResultsGuard guard(*this);
	(...)
}

Please note that the flag is also protected by the VideoWorkerResultsGuard synchronization object, so the main thread cannot set it to true in the moment directly after the worker thread posted the notification message.

Issues of Multithreading

There are two primary issues to consider when using worker thread(s). The first one is destroying data by unsynchronized access from multiple threads and the second one is a deadlock that can appear when there are two (or more) resources to be synchronized.

Securing data integrity by the thread synchronization mechanisms has been shortly described in this article and is implemented in the example application distributed with Adaptive Vision Library. As a rule of a thumb, please assume that every image that can be accessed from more then one thread should be protected by some kind of synchronization. We recommend the standard C++ RAII pattern as an easy to use and secure solution.

The example application described in this article contains only one resource – a critical section represented by the VideoWorkerResultsGuard class, but of course there may exist some applications where there is more then one resource to share. In such cases, the synchronization of the threads has to be implemented very carefully because there is a danger of deadlock that can be a result of bad implementation. If your application freezes (stops responding) and you have more then one synchronized resource, please review the synchronization code.

Previous: Working with GenICam GenTL Devices Next: Troubleshooting