Under some circumstances, an attempt to `RESTORE REPLICA` will fail with the log message `It's a bug: replica is not readonly` even though the table shows as `read_only` in the `system.replicas` table. 

**Expected behavior**

The expected behavior is that RESTORE REPLICA will either execute as expected if the table is `read_only` in the system.replicas table, or the fail with the correct error/log message, such as `Replica path is present at {} -- nothing to restore. `


**Additional context**

There are two ways for this scenario to occur:

(1)  The mostly likely path:

The ClickHouse server starts, but zookeeper is not available, so the StorageReplicatedMergeTree field `is_read_only` is set`true` for this table.  However, the SRMT field `has_metadata_in_zookeeper` keeps its default value of `true`.

Zookeeper subsequently becomes available, but the expected zookeeper path does not exist, so the RESTORE REPLICA command appears to be valid, and passes the checks in function `InterpreterSystemQuery::restoreReplica()`

Because `has_metadata_in_zookeeper` retains its initial `true` value, this final check in `StorageReplicatedMergeTree::restoreMetadataInZooKeeper()` fails:

```
if (!is_readonly || has_metadata_in_zookeeper)
        throw Exception(ErrorCodes::LOGICAL_ERROR, "It's a bug: replica is not readonly");
```

(2)  Another path where the user is trying clearing zookeeper for some reason

ClickHouse connects to zookeeper, and finds some data about the replica table in the zookeeper path.  In that case the StorageReplicatedMergeTree member variable `has_metadata_in_zookeeper` will keep its default value of `true`, and the SRMT member variable `is_read_only` will keep its default value of `false`

The user, for whatever reason, manually deletes the replica path from zookeeper. 

Perhaps during the previous operation, ClickHouse loses its zookeeper connection.  The `ReplicatedMergeTreeRestartingThread ` discovers that zookeeper is down, and set the `is_read_only` flag to true.  However, again, `has_metadata_in_zookeeper` remains true.

In both cases the zookeeper path is indeed empty and the table is correctly marked `read_only`, but the `has_metadata_in_zookeeper` field has the incorrect `true` value.

The simplest workaround is to restart ClickHouse and `has_metadata_in_zookeeper` will be correctly set to false when attempting to initially attach the table.  However, this value could also be set correctly in the process of executing the RESTORE REPLICA query.

The other potential issue in this area is that the `InterpreterSystemQuery::restoreReplica()` function checks the system/context zookeeper and could return incorrect results if the replicated table is using an alternative zookeeper.
* - [FS] first write the part into a temporary directory on the filesystem;
  * - [FS] rename the temporary part to the result on the filesystem;
  * - [RAM] immediately afterwards add it to the `data_parts`, and remove from `data_parts` any parts covered by this one;
  * - [RAM] also set the `Transaction` object, which in case of an exception (in next point),
  *   rolls back the changes in `data_parts` (from the previous point) back;
  * - [ZK] then send a transaction (multi) to add a part to ZooKeeper (and some more actions);
  * - [FS, ZK] by the way, removing the covered (old) parts from filesystem, from ZooKeeper and from `all_data_parts`
  *   is delayed, after a few minutes.
  *
  * There is no atomicity here.
  * It could be possible to achieve atomicity using undo/redo logs and a flag in `DataPart` when it is completely ready.
  * But it would be inconvenient - I would have to write undo/redo logs for each `Part` in ZK, and this would increase already large number of interactions.
  *
  * Instead, we are forced to work in a situation where at any time
  *  (from another thread, or after server restart), there may be an unfinished transaction.
  *  (note - for this the part should be in RAM)
  * From these cases the most frequent one is when the part is already in the data_parts, but it's not yet in ZooKeeper.
  * This case must be distinguished from the case where such a situation is achieved due to some kind of damage to the state.
  *
  * Do this with the threshold for the time.
  * If the part is young enough, its lack in ZooKeeper will be perceived optimistically - as if it just did not have time to be added there
  *  - as if the transaction has not yet been executed, but will soon be executed.
  * And if the part is old, its absence in ZooKeeper will be perceived as an unfinished transaction that needs to be rolled back.
  *
  * PS. Perhaps it would be better to add a flag to the DataPart that a part is inserted into ZK.
  * But here it's too easy to get confused with the consistency of this flag.
  */
#define MAX_AGE_OF_LOCAL_PART_THAT_WASNT_ADDED_TO_ZOOKEEPER (5 * 60)
}