Invalidating query cache entries table replication dating to marriage timeline
Since then the replication process lags a lot behind the master with the line in the process list of: invalidating query cache entries (table).
but really, this looks mostly like a case of the actual thread status being misreported, and your issue is insufficient disk I/O bandwidth for the workload (or excessive flushing).
Platform: , with only difference in that the post-processed data is stored using separate INSERT and UPDATE queries rather than INSERT ON DUPLICATE KEY UPDATE.
When My SQL locks up, I execute SHOW PROCESSLIST command and I see the following queries: N User Time Status SQL query 1 system user XX update INSERT INTO `db A`.`table A` (...) VALUES (...) 2 ????
Have you been asked to diagnose issues caused by database nodes that constantly switch between online and offline?
I belive the issue is not related to long running queries.That is, even if the query cache is not enabled, the mutex (slow, by nature) to access the (not existing) query cache is acquired for every binlog event.The only way to not acquire the query cache mutex in My SQL pre-5.5 is to compile My SQL without query cache.It looks like some sort of dead lock, but I cannot understand why.) - and move your way up as the read performance increases without reaching "Waiting for query cache lock" on writes."Be cautious about sizing the query cache excessively large, which increases the overhead required to maintain the cache, possibly beyond the benefit of enabling it.Sizes in the hundreds of megabytes might not be." is not a useful answer.
Server3 will be done with query 3, replication delay jumps back to zero, and then back up to 10 as it processes the next query.