Two sources of reordering
Reordering is not only a hardware effect. The compiler also rearranges code as it optimizes, as long as a single threaded program would not notice.
What optimizers do
- They hoist a repeated read out of a loop into a register.
- They sink or delay a write that is overwritten later.
- They reorder independent statements to improve pipelining.
These transformations are perfectly legal under the as if rule, which only requires preserving single threaded observable behavior. But another thread is not part of that behavior, so a write you expected to be visible may never even be emitted.
Why source order is not enough
A naive spin loop reading a shared flag may be optimized into reading the flag once and looping forever on a cached register copy. The program text suggests it rereads memory, but the compiler proved it did not need to. To stop this you must tell the compiler the access is special, using atomics or barriers that limit reordering.
Key idea
Compilers reorder merge and eliminate memory accesses under the as if rule, so without atomics or barriers source order gives no guarantee about what another thread observes.