DDLWorker thread checks in a loop for new tasks by calling ZooKeeper GetChildren with a watch callback.

https://github.com/ClickHouse/ClickHouse/blob/5b0bc8a7fbf846cb45012e8009d58341656bf12c/src/Interpreters/DDLWorker.cpp#L374

Due to a recent change from #25373 instead of blocking on new events we retry the loop every 10 seconds instead of when watch callback is triggered.

https://github.com/ClickHouse/ClickHouse/blob/5b0bc8a7fbf846cb45012e8009d58341656bf12c/src/Interpreters/DDLWorker.cpp#L1140-L1142

This causes a zookeeper watch callback leak. It is probably not very important as all that happens is that a new std::function is added to the watches list for the ddl worker path every 10 seconds. What is 32 bytes (sizeof(std::function)?) every 10 seconds?

However, it is annoying to see that on metrics. The metric is somewhat useful for catching bugs and this isn't an important one.

I understand that we could probably fix this "temporary fix" in DDLWorker and remove the tryWait + retrying, but that pattern was seen in other places and is useful for implementing a non-blocking watch, also used in `waitForTableReplicaToProcessLogEntry`.

Is there a laconic fix for that? Maybe we could somehow remove the callback when not needed/before retry? Maybe we could change the behaviour slightly and add watchcallback only if it wasn't already added (this wouldn't work for getChildren which wraps a Poco Event in a watch)? Anything else beside just ignoring this?

---

I understand that all the callbacks will be cleared when the event is triggered.
if (!initializeMainThread())
                    break;
                LOG_DEBUG(log, "Initialized DDLWorker thread");
            }
            cleanup_event->set();
            scheduleTasks(reinitialized);
            LOG_DEBUG(log, "Waiting for queue updates (stat: {}, {}, {}, {})",
                      queue_node_stat.version, queue_node_stat.cversion, queue_node_stat.numChildren, queue_node_stat.pzxid);
            
            constexpr int queue_wait_timeout_ms = 10000;
            bool updated = queue_updated_event->tryWait(queue_wait_timeout_ms);
            if (!updated)
            {
                Coordination::Stat new_stat;
                tryGetZooKeeper()->get(queue_dir, &new_stat);
                bool queue_changed = memcmp(&queue_node_stat, &new_stat, sizeof(Coordination::Stat)) != 0;
                bool watch_triggered = queue_updated_event->tryWait(0);
                if (queue_changed && !watch_triggered)
                {
                    
                    
                    LOG_TRACE(
                        log,
                        "Queue was not updated (stat: {}, {}, {}, {})",
                        new_stat.version,
                        new_stat.cversion,
                        new_stat.numChildren,
                        new_stat.pzxid);
                    context->getZooKeeperLog()->flush();
                }