18. Concurrency

18.1 General

18.1.1 Do not use platform specific multi-threading facilities

Rather than using platform-specific facilities, the C++ standard library should be used as it is platform independent.

 // @@- Non-Compliant [email protected]@
#include <pthread.h>
void* thread1(void*);
void f1()
{
pthread_t t1;
pthread_create(&t1, nullptr, thread1, 0);
// ...
}
     
// @@+ Compliant [email protected]@
#include <thread>
void thread2();
void f2()
{
std::thread t1(thread2);
// ...
}

References

Williams Concurrency – 1.3.4
Use native handles to transcend the C++11 API Meyers Effective C++ ’11 (draft TOC)

View references >

18.2 Threads

18.2.1 Use high_integrity::thread in place of std::thread

The destructor of std::thread will call std::terminate if the thread owned by the class is still joinable. By using a wrapper class, a default behavior can be provided.

// high_integrity.h
#include <thread>
#include <cstdint>
                           
namespace high_integrity
{
enum ThreadExec : int32_t
{
DETACH,
JOIN,
};
                 
template <ThreadExec thread_exec>
class thread
{
public:
template <class F, class ...Args>
thread (F&& f, Args&&...args)
: m_thread(std::forward<F> (f), std::forward<Args>(args)...)
{
}
thread(thread const &) = delete;
thread(thread &&) = default;
                                       
~thread()
{
if(m_thread.joinable())
{
join_or_detach ();
}
}
               
inline void join () { m_thread.join (); } 
                                       
private:
inline void join_or_detach ();
                                       
private:
std::thread m_thread;
};
template <> void thread<ThreadExec::DETACH>::join_or_detach ()
{
m_thread.detach ();
}
   
template <> void thread<ThreadExec::JOIN>::join_or_detach ()
{
m_thread.join ();
}
}
               
using high_integrity::thread;
using high_integrity::ThreadExec;
                           
void f(int32_t);
int32_t main()
{
int32_t i;
                             
// @@- Non-Compliant: Potentially calls 'std::terminate' [email protected]@
std::thread t(f, i);
               
// @@+ Compliant: Will detach if required [email protected]@
thread<ThreadExec::DETACH> hi_t (f, i);
}

References

Williams Concurrency – 2.1.3
Make std::threads unjoinable on all paths Meyers Effective C++ ’11 (draft TOC)

View references >

18.2.2 Synchronize access to data shared between threads using a single lock

Using the same lock when accessing shared data makes it easier to verify the absence of problematic race conditions. To help achieve this goal, access to data should be encapsulated such that it is not possible to read or write to the variable without acquiring the appropriate lock. This will also help limit the amount of code executed in the scope of the lock. Note: Data may be referenced by more than one variable, therefore this requirement applies to the complete set of variables that could refer to the data.

#include <mutex>
#include <string>
#include <cstdint>
   
class some_data
{
public:
void do_something();
   
private:
int32_t a;
std::string b;
};
               
some_data* unprotected;
               
void malicious_function(some_data& protected_data)
{
// Suspicious, unprotected now refers to data protected by a mutex
unprotected=&protected_data;
}
               
class data_wrapper
{
public:
template<typename Function>
void process_data(Function func)
{
std::lock_guard<std::mutex> lk(m);
func(data);          // 'protected_data' assigned to 'unprotected' here
}
   
private:
some_data data;
mutable std::mutex m;
};
               
data_wrapper x;
               
void foo()
{
x.process_data(malicious_function);
   
// @@- Not Compliant: 'unprotected' accessed outside of 'data_wrapper::m' [email protected]@
unprotected->do_something();
}

Special attention needs to be made for const objects. The standard library expects operations on const objects to be thread-safe. Failing to ensure that this expectation is fulfilled may lead to problematic data races and undefined behavior. Therefore, operations on const objects of user defined types should consist of either reads entirely or internally synchronized writes.

#include <mutex>
#include <atomic>
#include <cstdint>
#include "high_integrity.h"
     
class A
{
public:
int32_t get1() const
{
++counter1;    // @@- Non-Compliant: unsynchronized write to a data [email protected]@
               // @@-                member of non atomic type      [email protected]@
                   
++counter2;    // @@+ Compliant: write to a data member of atomic type [email protected]@
}
int32_t get2() const
{
std::lock_guard<std::mutex> guard(mut);
++counter1;    // @@+ Compliant: synchronized write to data member of non atomic type [email protected]@
}
private:
mutable std::mutex           mut;
mutable int32_t              counter1;
mutable std::atomic<int32_t> counter2;
};
     
using high_integrity::thread;
using high_integrity::ThreadExec;
     
void worker(A & a);
void foo(A & a)
{
thread<ThreadExec::JOIN> thread (worker, std::ref (a));
}

References

Sutter Guru of the Week (GOTW) – 6a
Williams Concurrency – 3.2.2
Williams Concurrency – 3.2.8

View references >

18.2.3 Do not share volatile data between threads

Declaring a variable with the volatile keyword does not provide any of the required synchronization guarantees:

Atomicity
Visibility
Ordering

#include <functional>
#include <cstdint>
#include <unistd.h>
#include "high_integrity.h"
               
// @@- Non-Compliant - using volatile for synchronization [email protected]@
class DataWrapper
{
public:
DataWrapper ()
: flag (false)
, data (0)
{
}
                 
void incrementData()
{
while(flag)
{
sleep(1000);
}
flag = true;
++data;
flag = false;
}
                 
int32_t getData() const
{
while(flag)
{
sleep(1000);
}
flag = true;
int32_t result (data);
flag = false;
                   
return result;
}
     
private:
mutable volatile bool flag;
int32_t data;
};
     
using high_integrity::thread;
using high_integrity::ThreadExec;
               
void worker(DataWrapper & data);
void foo(DataWrapper & data)
{
thread<ThreadExec::JOIN> t (worker, std::ref (data));
}

Use mutex locks or ordered atomic variables, to safely communicate between threads and to prevent the compiler from optimizing the code incorrectly.

#include <functional>
#include <cstdint>
#include <mutex>
#include "high_integrity.h"
     
// @@+ Compliant - using locks [email protected]@
class DataWrapper
{
public:
DataWrapper ()
: data (0)
{
}
                 
void incrementData()
{
std::lock_guard<std::mutex> guard(mut);
++data;
}
                 
int32_t getData() const
{
std::lock_guard<std::mutex> guard(mut);
return data;
}
     
private:
mutable std::mutex mut;
int32_t data;
};
     
using high_integrity::thread;
using high_integrity::ThreadExec;
               
void worker(DataWrapper & data);
void foo(DataWrapper & data)
{
thread<ThreadExec::JOIN> t (worker, std::ref (data));
}

References

CERT C++ – CON01-CPP
Distinguish volatile from std::atomic<> Meyers Effective C++ ’11 (draft TOC)
Sutter Concurrency – 19

View references >

18.2.4 Use std::call_once rather than the Double-Checked Locking pattern

The Double-Checked Locking pattern can be used to correctly synchronize initializations.

#include <memory>
#include <atomic>
#include <mutex>
#include <cstdint>
                           
std::mutex mut;
static std::atomic<int32_t *> instance;
               
int & getInstance ()
{
// @@- Non-Compliant: Using double-checked locking pattern [email protected]@
if (! instance.load (std::memory_order_acquire))
{
std::lock_guard<std::mutex> lock (mut);
if (!instance.load (std::memory_order_acquire))
{
int32_t * i = new int32_t (0);
instance.store (i, std::memory_order_release);    
}    
}
               
return * instance.load (std::memory_order_relaxed);
}

However, the C++ standard library provides std::call_once which allows for a cleaner implementation:

#include <mutex>
#include <cstdint>
               
int32_t * instance;
std::once_flag initFlag;
               
void init ()
{
instance = new int32_t (0);
}
               
int32_t & getInstance ()
{
// @@+ Compliant: Using 'call_once' [email protected]@
std::call_once (initFlag, init);
return *instance;
}

Initialization of a local object with static storage duration is guaranteed by the C++ Language Standard to be re-entrant. However this conflicts with Rule <hicpp ref=”basic.stc.no-static-storage”/>, which takes precedence.

#include <cstdint>
               
int32_t & getInstance ()
{
// @@+ Non-Compliant: using a local static [email protected]@
static int32_t instance (0);
return instance;
}

References

Williams Concurrency – 3.3.1

View references >

18.3 Mutual Exclusion

18.3.1 Within the scope of a lock, ensure that no static path results in a lock of the same mutex

It is undefined behavior if a thread tries to lock a std::mutex it already owns, this should therefore be avoided.

#include <mutex>
#include <cstdint>
   
std::mutex mut;
int32_t i;
   
void f2(int32_t j);
   
void f1(int32_t j)
{
std::lock_guard<std::mutex> hold(mut);
if (j)
{
f2(j);
}
++i;
}
   
void f2(int32_t j)
{
if (! j)
{
std::lock_guard<std::mutex> hold(mut); // @@- Non-Compliant: "Static Path" Exists [email protected]@
                                       // @@-                to here from f1      [email protected]@
++i;
}
}

References

Williams Concurrency – 3.3.3

View references >

18.3.2 Ensure that order of nesting of locks in a project forms a DAG

Mutex locks are a common causes of deadlocks. Multiple threads trying to acquire the same lock but in a different order may end up blocking each other. When each lock operation is treated as a vertex, two consecutive vertices with no intervening lock operation in the source code are considered to be connected by a directed edge. The resulting graph should have no cycles, i.e. it should be a Directed Acyclic Graph (DAG).

#include <cstdint>
#include <mutex>
   
// @@- Non-Compliant:  Nesting of locks does not form a DAG: mut1->mut2 and then mut2->mut1 [email protected]@
class A
{
public:
void f1()
{
std::lock_guard<std::mutex> lock1(mut1);
std::lock_guard<std::mutex> lock2(mut2);
++i;
}
   
void f2()
{
std::lock_guard<std::mutex> lock2(mut2);
std::lock_guard<std::mutex> lock1(mut1);
++i;
}
   
private:
std::mutex mut1;
std::mutex mut2;
int32_t i;
};
   
// @@+ Compliant:  Nesting of locks forms a DAG: mut1->mut2 and then mut1->mut2 [email protected]@
class B
{
public:
void f1()
{
std::lock_guard<std::mutex> lock1(mut1);
std::lock_guard<std::mutex> lock2(mut2);
++i;
}
   
void f2()
{
std::lock_guard<std::mutex> lock1(mut1);
std::lock_guard<std::mutex> lock2(mut2);
++i;
}
   
private:
std::mutex mut1;
std::mutex mut2;
int32_t i;
};

References

Williams Concurrency – 3.2.4
Williams Concurrency – 3.2.5
Sutter Concurrency – 5
Sutter Concurrency – 6

View references >

18.3.3 Do not use std::recursive_mutex

Use of std::recursive_mutex is indicative of bad design: Some functionality is expecting the state to be consistent which may not be a correct assumption since the mutex protecting a resource is already locked.

// @@- Non-Compliant: Using recursive_mutex [email protected]@
#include <mutex>
#include <cstdint>
               
class DataWrapper
{
public:
int32_t incrementAndReturnData()
{
std::lock_guard<std::recursive_mutex> guard(mut);
incrementData();
return data;
}
                 
void incrementData()
{
std::lock_guard<std::recursive_mutex> guard(mut);
++data;
}
                 
// ...
private:
mutable std::recursive_mutex mut;
int32_t data;
};

Such situations should be solved by redesigning the code.

// @@+ Compliant: Not using mutex [email protected]@
#include <mutex>
#include <cstdint>
               
class DataWrapper
{
public:
int32_t incrementAndReturnData()
{
std::lock_guard<std::mutex> guard(mut);
inc();
return data;
}
                 
void incrementData()
{
std::lock_guard<std::mutex> guard(mut);
inc();
}
                 
// ...
private:
void inc()
{
// expects that the mutex has already been locked
++data;
}
               
mutable std::mutex mut;
int32_t data;
};

References

Williams Concurrency – 3.3.3

View references >

18.3.4 Only use std::unique_lock when std::lock_guard cannot be used

The std::unique_lock type provides additional features not available in std::lock_guard. There is an additional cost when using std::unique_lockand so it should only be used if the additional functionality is required.

#include <functional>
#include <mutex>
#include <cstdint>
   
std::unique_lock<std::mutex> getGlobalLock ();
               
void f1(int32_t val)
{
static std::mutex mut;
std::unique_lock<std::mutex> lock(mut); // @@- Non-Compliant [email protected]@
// ...
}
   
void f2()
{
auto lock = getGlobalLock ();           // @@+ Compliant [email protected]@
// ...
}

References

Williams Concurrency – 3.2.6

View references >

18.3.5 Do not access the members of std::mutex directly

A mutex object should only be managed by the std::lock_guard or std::unique_lock object that owns it.

#include <mutex>
               
std::mutex mut;
void f()
{
std::lock_guard<std::mutex> lock(mut);
mut.unlock ();                         // @@- Non-Compliant [email protected]@
}

References

Use RAII for resources – 3.4.3

View references >

18.3.6 Do not use relaxed atomics

Using non-sequentially consistent memory ordering for atomics allows the CPU to reorder memory operations resulting in a lack of total ordering of events across threads. This makes it extremely difficult to reason about the correctness of the code.

#include <atomic>
#include <cstdint>
   
template<typename T>
class CountingConsumer
{
public:
explicit CountingConsumer(T *ptr, int32_t counter)
: m_ptr(ptr), m_counter(counter)
{ }
                 
void consume (int data)
{
m_ptr->consume (data);
                 
// @@- Non-Compliant [email protected]@
if (m_counter.fetch_sub (1, std::memory_order_release) == 1)
{
delete m_ptr;
}
}
                 
T * m_ptr;
std::atomic<int32_t> m_counter;
};

References

Sutter Hardware – Part 2

View references >

18.4 Condition Variables

18.4.1 Do not use std::condition_variable_any on a std::mutex

When using std::condition_variable_any, there is potential for additional costs in terms of size, performance or operating system resources, because it is more general than std::condition_variable. std::condition_variable works with std::unique_lock<std::mutex>, while std::condition_variable_any can operate on any objects that have lock and unlock member functions.

#include <mutex>
#include <condition_variable>
#include <vector>
#include <cstdint>
               
std::mutex mut;
std::condition_variable_any cv;
std::vector<int32_t> container;
               
void producerThread()
{
int32_t i = 0;
std::lock_guard<std::mutex> guard(mut);
               
// critical section
container.push_back(i);
               
cv.notify_one();
}
               
void consumerThread()
{
std::unique_lock<std::mutex> guard(mut);
   
// @@- Non-Compliant: conditional_variable_any used with std::mutex based lock 'guard' [email protected]@
cv.wait(guard, []{ return !container.empty(); } );
               
// critical section
container.pop_back();
}

References

Williams Concurrency – 4.1.1

View references >

Request PDF Version

High Integrity C++ Coding Standard

18. Concurrency

18.1 General

18.2 Threads

18.3 Mutual Exclusion

18.4 Condition Variables