https://github.com/coreos/etcd
Revision c4c487eacaf914c53fb61c66c5d6822ffba31df1 authored by Gyuho Lee on 25 May 2018, 17:04:54 UTC, committed by Gyuho Lee on 31 May 2018, 18:42:25 UTC
This also happens without gRPC proxy. Fix panic when gRPC proxy leader watcher is restored: ``` go test -v -tags cluster_proxy -cpu 4 -race -run TestV3WatchRestoreSnapshotUnsync === RUN TestV3WatchRestoreSnapshotUnsync panic: watcher minimum revision 9223372036854775805 should not exceed current revision 16 goroutine 156 [running]: github.com/coreos/etcd/mvcc.(*watcherGroup).chooseAll(0xc4202b8720, 0x10, 0xffffffffffffffff, 0x1) /home/gyuho/go/src/github.com/coreos/etcd/mvcc/watcher_group.go:242 +0x3b5 github.com/coreos/etcd/mvcc.(*watcherGroup).choose(0xc4202b8720, 0x200, 0x10, 0xffffffffffffffff, 0xc420253378, 0xc420253378) /home/gyuho/go/src/github.com/coreos/etcd/mvcc/watcher_group.go:225 +0x289 github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchers(0xc4202b86e0, 0x0) /home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:340 +0x237 github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop(0xc4202b86e0) /home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:214 +0x280 created by github.com/coreos/etcd/mvcc.newWatchableStore /home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:90 +0x477 exit status 2 FAIL github.com/coreos/etcd/integration 2.551s ``` gRPC proxy spawns a watcher with a key "proxy-namespace__lostleader" and watch revision "int64(math.MaxInt64 - 2)" to detect leader loss. But, when the partitioned node restores, this watcher triggers panic with "watcher minimum revision ... should not exceed current ...". This check was added a long time ago, by my PR, when there was no gRPC proxy: https://github.com/coreos/etcd/pull/4043#discussion_r48457145 > we can remove this checking actually. it is impossible for a unsynced watching to have a future rev. or we should just panic here. However, now it's possible that a unsynced watcher has a future revision, when it was moved from a synced watcher group through restore operation. This PR adds "restore" flag to indicate that a watcher was moved from the synced watcher group with restore operation. Otherwise, the watcher with future revision in an unsynced watcher group would still panic. Example logs with future revision watcher from restore operation: ``` {"level":"info","ts":1527196358.9057755,"caller":"mvcc/watcher_group.go:261","msg":"choosing future revision watcher from restore operation","watch-key":"proxy-namespace__lostleader","watch-revision":9223372036854775805,"current-revision":16} {"level":"info","ts":1527196358.910349,"caller":"mvcc/watcher_group.go:261","msg":"choosing future revision watcher from restore operation","watch-key":"proxy-namespace__lostleader","watch-revision":9223372036854775805,"current-revision":16} ``` Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
1 parent 6bb88b9
Tip revision: c4c487eacaf914c53fb61c66c5d6822ffba31df1 authored by Gyuho Lee on 25 May 2018, 17:04:54 UTC
mvcc: fix panic by allowing future revision watcher from restore operation
mvcc: fix panic by allowing future revision watcher from restore operation
Tip revision: c4c487e
File | Mode | Size |
---|---|---|
.github | ||
Documentation | ||
alarm | ||
auth | ||
client | ||
clientv3 | ||
cmd | ||
compactor | ||
contrib | ||
discovery | ||
e2e | ||
embed | ||
error | ||
etcdctl | ||
etcdmain | ||
etcdserver | ||
hack | ||
integration | ||
lease | ||
logos | ||
mvcc | ||
pkg | ||
proxy | ||
raft | ||
rafthttp | ||
scripts | ||
snap | ||
store | ||
tests | ||
tools | ||
version | ||
wal | ||
.dockerignore | -rw-r--r-- | 5 bytes |
.gitignore | -rw-r--r-- | 177 bytes |
.godir | -rw-r--r-- | 23 bytes |
.header | -rw-r--r-- | 593 bytes |
.travis.yml | -rw-r--r-- | 3.4 KB |
CONTRIBUTING.md | -rw-r--r-- | 2.5 KB |
DCO | -rw-r--r-- | 1.4 KB |
Dockerfile | -rw-r--r-- | 177 bytes |
Dockerfile-release | -rw-r--r-- | 607 bytes |
Dockerfile-test | -rw-r--r-- | 1.6 KB |
LICENSE | -rw-r--r-- | 11.1 KB |
MAINTAINERS | -rw-r--r-- | 395 bytes |
Makefile | -rw-r--r-- | 17.5 KB |
NEWS | -rw-r--r-- | 2.9 KB |
NOTICE | -rw-r--r-- | 126 bytes |
Procfile | -rw-r--r-- | 1.5 KB |
README.md | -rw-r--r-- | 5.7 KB |
ROADMAP.md | -rw-r--r-- | 837 bytes |
V2Procfile | -rw-r--r-- | 1.5 KB |
build | -rwxr-xr-x | 1.7 KB |
build.bat | -rwxr-xr-x | 51 bytes |
build.ps1 | -rw-r--r-- | 2.7 KB |
cover | -rwxr-xr-x | 638 bytes |
etcd.conf.yml.sample | -rw-r--r-- | 3.4 KB |
glide.lock | -rw-r--r-- | 4.7 KB |
glide.yaml | -rw-r--r-- | 4.0 KB |
main.go | -rw-r--r-- | 1.1 KB |
test | -rwxr-xr-x | 7.8 KB |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...