https://github.com/coreos/etcd
Revision c4c487eacaf914c53fb61c66c5d6822ffba31df1 authored by Gyuho Lee on 25 May 2018, 17:04:54 UTC, committed by Gyuho Lee on 31 May 2018, 18:42:25 UTC
This also happens without gRPC proxy.

Fix panic when gRPC proxy leader watcher is restored:

```
go test -v -tags cluster_proxy -cpu 4 -race -run TestV3WatchRestoreSnapshotUnsync

=== RUN   TestV3WatchRestoreSnapshotUnsync
panic: watcher minimum revision 9223372036854775805 should not exceed current revision 16

goroutine 156 [running]:
github.com/coreos/etcd/mvcc.(*watcherGroup).chooseAll(0xc4202b8720, 0x10, 0xffffffffffffffff, 0x1)
	/home/gyuho/go/src/github.com/coreos/etcd/mvcc/watcher_group.go:242 +0x3b5
github.com/coreos/etcd/mvcc.(*watcherGroup).choose(0xc4202b8720, 0x200, 0x10, 0xffffffffffffffff, 0xc420253378, 0xc420253378)
	/home/gyuho/go/src/github.com/coreos/etcd/mvcc/watcher_group.go:225 +0x289
github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchers(0xc4202b86e0, 0x0)
	/home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:340 +0x237
github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop(0xc4202b86e0)
	/home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:214 +0x280
created by github.com/coreos/etcd/mvcc.newWatchableStore
	/home/gyuho/go/src/github.com/coreos/etcd/mvcc/watchable_store.go:90 +0x477
exit status 2
FAIL	github.com/coreos/etcd/integration	2.551s
```

gRPC proxy spawns a watcher with a key "proxy-namespace__lostleader"
and watch revision "int64(math.MaxInt64 - 2)" to detect leader loss.
But, when the partitioned node restores, this watcher triggers
panic with "watcher minimum revision ... should not exceed current ...".

This check was added a long time ago, by my PR, when there was no gRPC proxy:

https://github.com/coreos/etcd/pull/4043#discussion_r48457145

> we can remove this checking actually. it is impossible for a unsynced watching to have a future rev. or we should just panic here.

However, now it's possible that a unsynced watcher has a future
revision, when it was moved from a synced watcher group through
restore operation.

This PR adds "restore" flag to indicate that a watcher was moved
from the synced watcher group with restore operation. Otherwise,
the watcher with future revision in an unsynced watcher group
would still panic.

Example logs with future revision watcher from restore operation:

```
{"level":"info","ts":1527196358.9057755,"caller":"mvcc/watcher_group.go:261","msg":"choosing future revision watcher from restore operation","watch-key":"proxy-namespace__lostleader","watch-revision":9223372036854775805,"current-revision":16}
{"level":"info","ts":1527196358.910349,"caller":"mvcc/watcher_group.go:261","msg":"choosing future revision watcher from restore operation","watch-key":"proxy-namespace__lostleader","watch-revision":9223372036854775805,"current-revision":16}
```

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
1 parent 6bb88b9
History
Tip revision: c4c487eacaf914c53fb61c66c5d6822ffba31df1 authored by Gyuho Lee on 25 May 2018, 17:04:54 UTC
mvcc: fix panic by allowing future revision watcher from restore operation
Tip revision: c4c487e
File Mode Size
.github
Documentation
alarm
auth
client
clientv3
cmd
compactor
contrib
discovery
e2e
embed
error
etcdctl
etcdmain
etcdserver
hack
integration
lease
logos
mvcc
pkg
proxy
raft
rafthttp
scripts
snap
store
tests
tools
version
wal
.dockerignore -rw-r--r-- 5 bytes
.gitignore -rw-r--r-- 177 bytes
.godir -rw-r--r-- 23 bytes
.header -rw-r--r-- 593 bytes
.travis.yml -rw-r--r-- 3.4 KB
CONTRIBUTING.md -rw-r--r-- 2.5 KB
DCO -rw-r--r-- 1.4 KB
Dockerfile -rw-r--r-- 177 bytes
Dockerfile-release -rw-r--r-- 607 bytes
Dockerfile-test -rw-r--r-- 1.6 KB
LICENSE -rw-r--r-- 11.1 KB
MAINTAINERS -rw-r--r-- 395 bytes
Makefile -rw-r--r-- 17.5 KB
NEWS -rw-r--r-- 2.9 KB
NOTICE -rw-r--r-- 126 bytes
Procfile -rw-r--r-- 1.5 KB
README.md -rw-r--r-- 5.7 KB
ROADMAP.md -rw-r--r-- 837 bytes
V2Procfile -rw-r--r-- 1.5 KB
build -rwxr-xr-x 1.7 KB
build.bat -rwxr-xr-x 51 bytes
build.ps1 -rw-r--r-- 2.7 KB
cover -rwxr-xr-x 638 bytes
etcd.conf.yml.sample -rw-r--r-- 3.4 KB
glide.lock -rw-r--r-- 4.7 KB
glide.yaml -rw-r--r-- 4.0 KB
main.go -rw-r--r-- 1.1 KB
test -rwxr-xr-x 7.8 KB

README.md

back to top