https://github.com/cilium/cilium

sort by:
Revision Author Date Message Commit Date
5665303 Prepare for release v1.7.12 Signed-off-by: Joe Stringer <joe@cilium.io> 04 December 2020, 10:59:12 UTC
b8b817d vendor: Fix cilium/arping goroutine leak [ upstream commit 24d44500e40af599dfc1b932be0dac1b75504889 ] This fixes a privileged runtime test failure caused by leaked goroutines on arpings with no response. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 03 December 2020, 20:08:31 UTC
54444f3 metrics: add cilium_datapath_nat_gc_entries [ upstream commit 57784e318449e711ffc994ee88609397086330a4 ] [ Backporter's notes: Resolved conflict with gcFamily const types. ] Signed-off-by: ArthurChiao <arthurchiao@hotmail.com> Signed-off-by: Chris Tarazi <chris@isovalent.com> 30 November 2020, 10:10:15 UTC
2994bba metrics: replace replicated "direction" strings with LabelDirection constant [ upstream commit e4bf8ca149a95a611335bc82acd948995393a189 ] Signed-off-by: ArthurChiao <arthurchiao@hotmail.com> Signed-off-by: Chris Tarazi <chris@isovalent.com> 30 November 2020, 10:10:15 UTC
befd325 pkg/bpf: Wrap error to fix test failure [ upstream commit 0e220cfc3 ] [ Backporter's notes: The upstream commit is coming from v1.8 branch, as the fix was not needed on v1.9 and master. See https://github.com/cilium/cilium/pull/14022#issuecomment-727923791 and https://github.com/cilium/cilium/pull/13912#issuecomment-730185299 ] To fix the ctmap privileged test failure, the following needs to be applied (otherwise, if ...; errors.Is(err, unix.ENOENT) is always false in the PurgeOrphanNATEntries(); the change was introduced in v1.9 by 2283103): Signed-off-by: Tom Payne <tom@isovalent.com> 30 November 2020, 10:10:15 UTC
fe46860 ctmap: Iterate SNAT map once when doing GC [ upstream commit 0c83c28963cdc2af5514b9707cae42d815afbdc1 ] [ Backporter's notes: Resolved conflict with exporting MapType* consts which were unexported in master branch. ] Previously, after receiving the signal from the datapath, we iterated NAT map twice: first to compare against CT TCP map, second - against CT any map. Obviously, doing the iterations two times was inefficient. This commit fixes that by passing both CT {TCP,any} maps to the NAT GC routine. This allows the NAT GC to iterate once. Suggested-by: Joe Stringer <joe@cilium.io> Signed-off-by: Martynas Pumputis <m@lambda.lt> Signed-off-by: Chris Tarazi <chris@isovalent.com> 30 November 2020, 10:10:15 UTC
471dfd5 ctmap: GC orphan SNAT entries [ upstream commit c9810bf7b235ee371279c85f025535a86b1ea675 ] [ Backporter's note: Resolved slight conflicts with (*Map).DumpEntries and removed reference to NodePort hybrid mode which doesn't exist in v1.7 branch. ] This commit adds a mechanism to remove orphan SNAT entries. We call an SNAT entry orphan if it does not have either a corresponding CT entry or an SNAT entry in a reverse order. Both cases can happen due to LRU eviction heuristics (both CT and NAT maps are of the LRU type). The mechanism for the removal is based on the GC signaling in the datapath. When the datapath SNAT routine fails to find a free mapping after SNAT_SIGNAL_THRES attempts, it sends the signal via the perf ring buffer. The consumer of the buffer is the daemon. After receiving the signal it invokes the CT GC. The newly implemented GC addition iterates over all SNAT entries and checks whether a corresponding CT entry is found, and if not, it tries to remove both SNAT entries (for original and reverse flows). For now, I didn't add GC of orphan SNAT entries created by DSR to keep complexity of changes as low as possible. This will come as a follow up. Signed-off-by: Martynas Pumputis <m@lambda.lt> Signed-off-by: Chris Tarazi <chris@isovalent.com> 30 November 2020, 10:10:15 UTC
8473756 cilium: generalize and extend signal framework for CT fill-up [ upstream commit 24fcbe6144df229b302cddf045d9f229f6e69f2e ] Rework the 1:1 relationship with signal to channel and instead allow different signals from BPF datapath for the same go channel. This is useful so we can push the different BPF signals into the metric collection. Wire-up Signal{CT,Nat}FillUp signal into the SignalWakeGC channel. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Martynas Pumputis <m@lambda.lt> 30 November 2020, 10:10:15 UTC
a5fc85a bpf: signal agent on CT map update error that CT map is full [ upstream commit fc5a3bd73f39012cedb99cf4b32461d45eeca812 ] There are users on 4.9 kernels which are suffering connectivity loss since the CT map is full and GC doesn't trigger yet. We can help improving the situation with the same framework we set in place for NAT when under stress. Upon insertion error, send a signal to the agent in order to trigger GC so that it can free up old entries. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Martynas Pumputis <m@lambda.lt> 30 November 2020, 10:10:15 UTC
a628214 maps/ctmap: unexport NewMap, MapType type and related consts [ upstream commit b563284ac8910b5f26a98a539694fb652c8b56b9 ] All of these are not used outside the ctmap package. Also make the mapTypeIP* consts typed to avoid type conversions when using them. Signed-off-by: Tobias Klauser <tklauser@distanz.ch> Signed-off-by: Martynas Pumputis <m@lambda.lt> 30 November 2020, 10:10:15 UTC
553815e Register logging flags with operator main cmd [ upstream commit 54275be17ab694abc0dc39af6ddf3a87fb3b38c8 ] Signed-off-by: Vlad Ungureanu <vladu@palantir.com> Signed-off-by: André Martins <andre@cilium.io> 26 November 2020, 14:19:39 UTC
73c8d19 Support '--log-opt format=json' option to log in JSON format [ upstream commit 454eae136e7f5e4ba25b0901646bb87c70aa6ca0 ] Signed-off-by: Maxime VISONNEAU <maxime.visonneau@gmail.com> Signed-off-by: André Martins <andre@cilium.io> 26 November 2020, 14:19:39 UTC
491805f endpoint: Add DebugPolicy option Add endpoint DebugPolicy option that, if enabled, logs endpoint policy map update details to /var/run/cilium/state/endpoint-policy.log. The new DebugPolicy option is enabled if the new flag --debug-verbose=policy is set, but can be enabled also independently via: cilium endpoint config <EPID> DebugPolicy=true Signed-off-by: Martynas Pumputis <m@lambda.lt> Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 25 November 2020, 01:27:41 UTC
538223c daemon: Postpone ipcache upserts until after policy changes have been regenerated by endpoints. Move ipcache CIDR upserts and releases to the policy reaction queue, where upserts can be executed after regenerations have been completed, i.e. after endpoint policy maps have been updated. This way IP addresses are mapped to newly allocated identities only after endpoint policy maps are ready to classify them. Correspondingly, on deletes the to-be-deleted CIDR identities are first deleted from ipcache so that when they are deleted from endpoint policy maps they are no longer used in classification. Releases of CIDR identities must still be serialized with ipcache upserts via the policy reaction queue so that they are executed in the same order w.r.t. ipcache upserts as policy deletes and adds. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 24 November 2020, 19:15:19 UTC
451443d fqdn: Delay ipcache upserts until policies have been updated Add a map for newly allocated identities to ipcache.AllocateCIDR functions that the caller can use to upsert the IPs to ipcache later, after affected endpoint policy maps have been updated. Use this new functionality on the DNS proxy code path, that makes sure that new policy map entries are in place before an IP received from a DNS server is placed in ipcache. This is really straightforward as the logic for waiting was already in place for delaying the forwarding of the DNS response. Policy update path is still allowing ipcache upserts at policy ingestion time rather than waiting for the policy maps to be updated. This means that new, more specific CIDRs (e.g., 10.0.0/24) in policies can still cause momentary drops on traffic currently using a less specific CIDR (e.g., 10.0/16). Similarly the DNS poller path still upserts to ipcache before policies have been updated. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 24 November 2020, 19:15:19 UTC
8b35ae4 fqdn: Fix unit test Setting usedServers to nil caused write to nil map on other tests. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
f767866 fqdn: Only keep used IPs for restored DNS rules. [ upstream commit 61efa8fd0e58b65de9628de7e3ef8db0cd4df40c ] The DNS policy may allow a huge number of IPs, only some of which are actual DNS servers. Collect a set of DNS servers that have been allowed in the past and only store allowed IPs that have actually been used. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
2b03d55 fqdn: Fix confusion of ToFQDNs vs. DNS rules. [ upstream commit a218052444243b6e439e77675f5f5034d5e86ffe ] Restored DNS proxy rules are DNS rules, not ToFQDNs rules. Fixes: #13991 Fixes: #13992 Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
07f3dee dnsproxy: print total number of rules if too many [ upstream commit b115c544aadc9999b26964687bfebd893798db5f ] GetRules() will not process more than 1000 rules per port. Print how many are the total rules in the message. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
5f99a13 fqdn: Make maximum number of IPs per restored rule configurable [ upstream commit 871e7e128b406f0753d48196c8049689d863d798 ] Only count the number of IPs for each FQDN selector/rule when storing rules for restoration, rather than ignoring later rules on a port after previous rules have hit the maximum number of IPs. Make the maximum number of IPs per restored rule configurable with the new option '--tofqdns-max-ips-per-restored-rule' (default 1000). Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
3b0f646 Add Registry Credentials to Tests [ upstream commit e8605f88baabceebef52170b7337c793436492aa ] In order to get around image registry pull limits, credentials can be set. Signed-off-by: Nate Sweet <nathanjsweet@pm.me> Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 21 November 2020, 00:58:27 UTC
6585c1d k8s: update k8s libraries to 1.17.14 Also update k8s test versions to 1.17.14 Signed-off-by: André Martins <andre@cilium.io> 17 November 2020, 17:24:57 UTC
e0fc12f ci: log in to docker in vagrant boxes [ upstream commit 238262f15a681674c2c02b4650e311b947ab44fc ] Signed-off-by: Maciej Kwiek <maciej@isovalent.com> Signed-off-by: Tom Payne <tom@isovalent.com> 13 November 2020, 10:04:26 UTC
307ff61 change default docker image repository from docker.io to quay.io [ upstream commit 434b056f7c2f25487068458684247063e2a52452 ] With the recent limitations introduced by Docker, docker hub has been rate limiting docker pulls on images for non-registered users. To avoid this limitation we should switch the default container image repositories to quay.io. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Tom Payne <tom@isovalent.com> 13 November 2020, 10:04:26 UTC
a968618 fqdn: keep IPs alive if their name is alive [ upstream commit 5923dafd88be2cef9f292479aff1eb0ea1ff3d05 ] There are applications that when a DNS name resolves to multiple IPs, they will store the IPs and use them past their TTL point. For example: - name resolves to IP1,IP2 - app connects to IP1 - protocol error forces disconnect - app connects to IP2 This patch keeps the IPs that map to a name alive as long as one of the IPs for the given name is alive, so that applications like the one above will not fail. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 10 November 2020, 08:25:42 UTC
e2e3e6b fqdn: Add a nil check for security id lookup [ upstream commit af95561ff00e815a6509b5fc6fccd17756c7e896 ] The security id lookup could return nil if the identity cache isn't initialized during endpoints restore time, resulting in a crash. Hence, add a nil check before populating log record values. Signed-off-by: Aditi Ghag <aditi@cilium.io> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 10 November 2020, 08:25:42 UTC
4789bef operator: increase GC Rate limit of identities to 2500 per minute [ upstream commit 86e419ead4568c972cf67088441857d8beda07bf ] In cluster that have some high churn of pods being created and deleted with different security identities, garbage collecting 250 identities per minute might not be sufficient. Thus, we are increasing the default limit to 2500 identities per minute. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 10 November 2020, 08:25:42 UTC
5273db1 release: add script to check presence of docker images [ upstream commit 73be2c15ced2d54aecdfa462b38d2aac2e6f631f ] To check if images are published across all repositories the `check-docker-images.sh` script will be able to perform this check of a particular release. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 10 November 2020, 08:25:42 UTC
f3ef8f3 api-limiter: Add documentation for log fields [ upstream commit cd6f6c47a8982d2711ab14e22af75a8926337c20 ] This commit contains no functional changes. It is meant to provide more context and ease understanding of the log msgs from the rate limiter to help debugging, and so on. Signed-off-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
04d3160 docs: Explain the rate limiter log messages [ upstream commit e9a536c48073225e49d0c96ac49dff9f27e44804 ] Signed-off-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
c41f7be docs: Fix format of rate limiter config table [ upstream commit 8fd20e820bd516d856b87266856138002a5bdced ] This commit simply fixes the format by making it consistent. Signed-off-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
92c8adc k8s: clarify CRD schema versioning and its update process [ upstream commit 72ec75fe0961bb3f106abc96d38f097867546465 ] Add steps on and when the CRD validation version should be updated. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
9826c18 backporting: Escape commit message when used as regex [ upstream commit 1f178f092c155c728d6fa9822ee062b2cfa9821b ] This commit escapes the commit message when it is used as a regular expression. Any special character of the posix extended regex (`.^$*+?()[{\|`) is prefixed with a backslash. This fixes an issue where `git log` would crash due to to the string being passed to `--grep` not being a valid regex, such as e.g. in the commit messages found in PR #13674 which contain `$`, `*` and `{`. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
edee82a backporting: Clean tmp files after backport with conflicts [ upstream commit 7b4d70015a88437e34476a2360dc14333a7dfd2b ] During the backport process, when cherry-picking commits to backport, the cherry-pick script may fail if there are conflicts. In that case, the temporary file holding the backported commit is not clean up. This commit fixes it to clean the temporary file even in case of failure. Suggested-by: Quentin Monnet <quentin@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> 03 November 2020, 19:56:28 UTC
1cb7b5e contrib: Sort authors without depending on locale [ upstream commit d8be57743e34bd1a8350122e40b47e1aaa6ad0f1 ] [ Backporter's notes: The above upstream commit is from the v1.8 branch as this commit was never in master. ] This is a partial backport of https://github.com/cilium/cilium/pull/13106. Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 November 2020, 18:34:42 UTC
6258a70 Prepare for release v1.7.11 Signed-off-by: Chris Tarazi <chris@isovalent.com> 28 October 2020, 00:43:10 UTC
97f4483 Dockerfile: Bump cilium-runtime image Signed-off-by: Chris Tarazi <chris@isovalent.com> 28 October 2020, 00:21:39 UTC
c5480cf docs: Add a note about systemd 245 rp_filter issue [ upstream commit 61100c50b8fece5cac963c67f71c259ca1b05052 ] Signed-off-by: Martynas Pumputis <m@lambda.lt> Signed-off-by: Chris Tarazi <chris@isovalent.com> 24 October 2020, 10:26:31 UTC
709b6a8 pkg/endpoint: calculate Kube API-Server lag from pod events [ upstream commit 4e29130043408063d62e91d21cfcf00f9ef837ed ] Since Cilium receives CNI events when a pod is created, Cilium can calculate the lag for kube-apiserver events by checking the time an ADD event for that Pod was received and subtracting by the time the CNI event for that pod was received. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 24 October 2020, 10:26:31 UTC
792414c pkg/labelsfilter: add more unit test and rewrite docs [ upstream commit 5733f6af771c127ecc2777af4da3d2bcbb870154 ] This change is only adding more unit tests to better understand the behavior of the labelsfilter as well as improving the documentation for the expectation of filtering labels. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> 23 October 2020, 18:39:23 UTC
e752015 lbmap: Optimize lbmap byte order related code [ upstream commit 682f6826bce735056a5e0d285ec0cdbb1e6cd9c8 ] Gets rid of ToNetwork() in DumpParser(). Signed-off-by: Jianlin Lv <Jianlin.Lv@arm.com> Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> 23 October 2020, 18:39:23 UTC
4819438 lbmap: Fixed lb cmd byte order issue [ upstream commit c5b709b22cfc0c127c7bc92d7ea2eacdc6b59179 ] The port/RevNat info is stored in the bpf map in network byte order; When displaying the given lbmap content, the port needs to be converted to host byte order. Add ToHost() function for lbmap ToHost converts fields to host byte order. Signed-off-by: Jianlin Lv <Jianlin.Lv@arm.com> Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> 23 October 2020, 18:39:23 UTC
e13de04 metrics: fix negative identity count [ upstream commit 9673c485a72ec93c10e2db1f4fdc8feab45d3d98 ] Identity allocation uses cache and refcnt mechanisms, if the identity info is already in remote kvstore and localkeys store, it will just increase the refcnt, then notify the caller that this identity is reused. The caller will then not bump up the identity counter. However, there is a corner case that not get handled: refcnt from 0 to 1, which will result to negative identity count in the metrics output. This patch fixes the problem by returning another flag to indicate whether the identity is first-time referenced (refcnt from 0 to 1) or not. The caller then uses this information to determine whether or not to increase the counter. Signed-off-by: arthurchiao <arthurchiao@hotmail.com> Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> 23 October 2020, 18:39:23 UTC
984f7bf test: Display BPF map content on fail [ upstream commit ec2c18a074de2446186854188c4853c5c5664ffc ] Signed-off-by: Paul Chaignon <paul@cilium.io> Signed-off-by: Nate Sweet <nathanjsweet@pm.me> 22 October 2020, 23:23:11 UTC
fda8422 pkg/endpoint: reduce cardinality of prometheus labels [ upstream commit ec16cab361309155d012ce12b93750fc5b876c9d ] If the controller that is used for label resolution fails, the prometheus metrics will increase its cardinality since the uniquely controller name was being used as a prometheus label. To avoid this, we will reference these warnings with a common subsystem name, 'resolve-labels'. Fixes: a31ab29f57b2 ("endpoint: Run labels controller under ep manager") Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Paul Chaignon <paul@cilium.io> 22 October 2020, 22:04:10 UTC
9494169 backporting: Update labels by default when submitting backport [ upstream commit a8e67f1139aa9f384d2c0365aecfacd3532bda10 ] When submitting the backport PR using submit-backport, the script proposes to update the labels (i.e., remove needs-backport/X and add backport-pending/X): Sending pull request... Everything up-to-date https://github.com/cilium/cilium/pull/13700 Updating labels for PRs 13383 13608 12975 Set labels for all PRs above? [y/N] y Setting labels for PR 13383... ✓ Setting labels for PR 13608... ✓ Setting labels for PR 12975... ✓ The choice defaults to not updating the labels. That may give the wrong impression that it is an optional step---and if you're like me, when you're unsure what an optional step does, you skip it. We should default to setting the labels because we later rely on the labels being set (e.g., when we update them after the PR is merged). This commit changes it to the following: Sending pull request... Everything up-to-date https://github.com/cilium/cilium/pull/13700 Updating labels for PRs 13383 13608 12975 Set labels for all PRs above? [Y/n] Setting labels for PR 13383... ✓ Setting labels for PR 13608... ✓ Setting labels for PR 12975... ✓ Signed-off-by: Paul Chaignon <paul@cilium.io> 22 October 2020, 22:04:10 UTC
bb49619 endpoint: Avoid benign error messages on restoration [ upstream commit 228a485f2a5441f506ba9c0f357321c060f93590 ] During the endpoint restoration process, when we parse the endpoints, we assign them a reserved init identity if they don't already have an identity [0]. If we later remove the endpoint (because the corresponding K8s pod or interface are missing), we attempt to remove the identity from the identity manager. That last operation results in the following error message because the init identity was never added to the manager. level=error msg="removing identity not added to the identity manager!" identity=5 subsys=identitymanager This commit fixes it by skipping the removal attempt from the manager in the case of identity init. 0 - https://github.com/cilium/cilium/blob/80a71791320df34df5b6252b9680553e38d88d20/pkg/endpoint/endpoint.go#L819 Signed-off-by: Paul Chaignon <paul@cilium.io> 22 October 2020, 22:04:10 UTC
a2c9ee2 endpoint: Avoid unnecessary warning logs [ upstream commit df6ede7fa5555984d5b60d961cf3c90a965a6cdb ] Do not log a warning when can't release the ID of a disconnected endpoint. These changes remove warning logs like: msg="Unable to restore endpoint, ignoring" endpointID=1925 error="interface lxc18d62e89ea16 could not be found" k8sPodName=default/spaceship-d5d56b59-6c582 subsys=daemon msg="Unable to release endpoint ID" error="Unable to release endpoint ID 1925" state=disconnected subsys=endpoint Signed-off-by: Jarno Rajahalme <jarno@covalent.io> Signed-off-by: Paul Chaignon <paul@cilium.io> 22 October 2020, 22:04:10 UTC
87017ae endpoint: Remove interval for metadata resolver [ upstream commit cb9aa3e73c72f7a9c1bf65609d9db85f2c493e5f ] This controller's run interval is actually unnecessary as this controller is intended to only run once as long as it succeeds. This is evidenced by the fact that we have a `done` channel that's closed when the controller succeeds and a goroutine waiting `done` to be closed to remove the controller, thereby stopping it. In addition, in the very unlikely case that the controller is scheduled again to run _and_ the goroutine waiting for the `done` channel to be closed has not run yet, the controller could panic because it would close the channel twice. This commit prevents that as the interval defaults to 10m if not provided. If the goroutine waiting on the channel doesn't run within 10m, then we very likely have much worse problems at hand. Signed-off-by: Chris Tarazi <chris@isovalent.com> 22 October 2020, 16:33:58 UTC
44d7996 endpoint: Fix goroutine leak when EP is deleted [ upstream commit 6747a845542b38015917760d9905883170b5496c ] The metadata resolver controller can be put in a scenario where it can never succeed if the Cilium K8s pod cache is out-of-sync with the current view of pods in the cluster. The cache can be out-of-sync if there's heavy load on the cluster and especially on the apiserver. This leaves the Cilium pod cache starved. When the pod cache is starved, Cilium may never have an up-to-date view of the pods running in the cluster, and therefore may never be able to resolve labels for pods because the fetch will return "pod not found". Eventually, kubelet (or maybe even the user) will give up on the pod and remove it, thereby Cilium begins removing the endpoint. The controller is stopped, but the goroutine waiting on the `done` channel will never receive, hence the leak. This commit fixes the goroutine leak when the controller never succeeds and therefore, never closes the `done` channel. The fix is to add a `select` statement to also watch the endpoint's `aliveCtx` which is cancelled (and the context's `Done` channel is closed) when the endpoint is deleted. This commit was validated by forcing the metadata resolver to never find a pod (manually hardcode the wrong pod name), and ensuring that the `gops stack` output does not contain an entry like: ``` goroutine 1244259 [chan receive, 1434 minutes]: github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver.func1(0xc0055d2a20, 0xc0049ff600, 0xc0055d2ae0, 0x5d) /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1531 +0x34 created by github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1530 +0x11e ``` Fixes: https://github.com/cilium/cilium/issues/13680 Signed-off-by: Chris Tarazi <chris@isovalent.com> 22 October 2020, 16:33:58 UTC
05b5cd3 k8s: update k8s libraries to 1.17.13 Also update k8s test versions to 1.17.13 Signed-off-by: André Martins <andre@cilium.io> 21 October 2020, 14:33:16 UTC
8603523 test: Clean up etcd-deployment after test Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:46:34 UTC
7fa76d0 test: Refactor DatapathConfiguration etcd test This commit separates out the preamble of deploying etcd and reconfiguring Cilium to use it into a BeforeEach. This improves the readability of the test. Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:46:34 UTC
f7f3913 test: Avoid reinstalling Cilium after test suite This commit removes an unnecessary step which installs Cilium after every Context block in this test suite. It is unnecessary because each Context will install Cilium and configure it accordingly. This will also speed up the test suite execution time. Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:46:34 UTC
7eef9ed test: Remove leftover etcd operator related code [ upstream commit 43a7bba3bf543eba6deebf35c57f63856f44f819 ] Signed-off-by: Thomas Graf <thomas@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:46:34 UTC
070c410 test: Replace managed etcd test with etcd test [ upstream commit bf3a6c585ce05d83fc52224c602192d547d84302 ] [ Backporter's notes: The deployment manager code doesn't exist in v1.7, so that was omitted in this backport commit. The functional changes that the deployment manager provided were translated for the current code in v1.7. Also, "synchronizeK8sNodes" is not under the "config" Helm subchart in v1.7, but rather under "operator". ] The managed etcd feature is being deprecated soon. The test has been unreliable due to etcd operator not being able to bring up etcd clusters reliably. We want to ensure test coverage for the etcd backing though. Convert the managed etcd test to a more generic test which uses stateless etcd for a reliable outcome. Fixes: #11181 Signed-off-by: Thomas Graf <thomas@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:46:34 UTC
e1c9152 identity: Fix nil pointer panic in LookupIdentityByID [ upstream commit 005291cf60e0c48d3968093091c090349e623955 ] Because the identity allocator is initialized asychronously via `InitIdentityAllocator`, the local identitiy allocator might not have been initialized yet when the lookup functions are called. This can cause nil pointer panics, as observed in #13479. Before b194612c004c3e69289286e9a35d337b2645fc50, this nil pointer panic could not occur in `LookupIdentityByID` as the function checked for `m.IdentityAllocator != nil` which also implies `m.localIdentities != nil`. This commit adds an explict check for `m.localIdentities` and fixes a potential data race by checking the initialization channels before accessing `m.localIdentities` or `m.IdentityAllocator`. Fixes: #13479 Fixes: b194612c004c ("identity: Avoid kvstore lookup for local identities") Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:33:56 UTC
ee1b3e3 service: Use initNextID in acquireLocalID() [ upstream commit b34e5d80ad6929e735e9486f1f38c77df99043cc ] When rolling over, it should use initNextID instead of FirstFreeServiceID, which doesn't belong to the IDAllocator. This would create problems if FirstFreeServiceID and FirstFreeBackendID have different values although now they happen to be the same. Fixes: ab9cf4ba4206 ("service: Make local ID allocator more service agnostic") Signed-off-by: Han Zhou <hzhou8@ebay.com> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:33:56 UTC
1454ca3 bpf: only clean up XDP from devices with XDP attached [ upstream commit 8f0e7fad3aff07cf0b00528080fb42dffe15e1af ] [ Backporter's notes: Includes update to bpf.sha. ] Currently, during agent startup, cilium removes XDP from all interfaces except for `cilium_host`, `cilium_net` and `$XDP_DEV` regardless of whether there is an XDP program attached to it. For some drivers, e.g. Mellanox mlx5, the following command will cause device reset regardless of whether there is an XDP program attached to it, which introduces node and pod network interruption: `ip link set dev $DEV xdpdrv off`. This patch adds a check of XDP program existence to avoid such network interruption. Fixes: #13526 Reported-by: ArthurChiao <arthurchiao@hotmail.com> Signed-off-by: Jaff Cheng <jaff.cheng.sh@gmail.com> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:33:56 UTC
54fdd63 bpf: clean up XDP from prior devices when cilium configuration changes [ upstream commit 1afb536cd61b5a817c472f761979fd08e0611ae9 ] Avoid having to leave around stale XDP programs when the config changes. Therefore do the same as we do in tc which is to clean up prior state. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Chris Tarazi <chris@isovalent.com> 20 October 2020, 08:33:56 UTC
98210c4 contrib: match commit subject exactly when searching for upstream commit [ upstream commit 6557f7557ca7ef65f8097743d4fdb935967d686e ] In generate_commit_list_for_pr, the commit subject is used to determine the upstream commit ID from $REMOTE/master. However, if in the meantime another commit with e.g. a Fixes tag that mentions this commit subject, it appears first and leads to the original commit not being found. This can be demonstrated using #13383: ``` * PR: 13383 -- daemon: Enable configuration of iptables --random-fully (@kh34) -- https://github.com/cilium/cilium/pull/13383 Merge with 2 commit(s) merged at: Wed, 14 Oct 2020 11:41:51 +0200! Branch: master (!) refs/pull/13383/head ---------- ------------------- v (start) | Warning: No commit correlation found! via dbac86cffc6d57e8c093d2821e0d794f4c13d284 ("daemon: Enable configuration of iptables --random-fully") | 350f0b36fd9b4cf23ebc11f4365c5c89591d0ff4 via 22d4554e963e2d8029ff95087ac03e55e90a7377 ("test: Test iptables masquerading with --random-fully") v (end) $ # this is the git log command (with the subject added) from $ # contrib/backporting/check-stable that should extract a single $ # upstream commit $ git log -F --since="1year" --pretty="%H %s" --no-merges --grep "daemon: Enable configuration of iptables --random-fully" origin/master 078ec543d36a8f5d6caed5c4649c74c72090ae20 install/kubernetes: consistent case spelling of iptables related values 4e39def13bca568a21087238877fbc60f8751567 daemon: Enable configuration of iptables --random-fully $ git show 078ec543d36a8f5d6caed5c4649c74c72090ae20 commit 078ec543d36a8f5d6caed5c4649c74c72090ae20 Author: Tobias Klauser <tklauser@distanz.ch> Date: Wed Oct 14 11:58:29 2020 +0200 install/kubernetes: consistent case spelling of iptables related values Make the case spelling of the newly introduced "ipTablesRandomFully" value consistent with other iptables option values which use the "iptables" spelling. Fixes: 4e39def13bca ("daemon: Enable configuration of iptables --random-fully") Signed-off-by: Tobias Klauser <tklauser@distanz.ch> ``` Note the `Fixes: ...` line in commit 078ec543d36a8f5d6caed5c4649c74c72090ae20 above. Fix this behavior by grepping for the subject line from start of line: ``` $ git log -F --since="1year" --pretty="%H %s" --no-merges --extended-regexp --grep "^daemon: Enable configuration of iptables --random-fully" origin/master 4e39def13bca568a21087238877fbc60f8751567 daemon: Enable configuration of iptables --random-fully * PR: 13383 -- daemon: Enable configuration of iptables --random-fully (@kh34) -- https://github.com/cilium/cilium/pull/13383 Merge with 2 commit(s) merged at: Wed, 14 Oct 2020 11:41:51 +0200! Branch: master (!) refs/pull/13383/head ---------- ------------------- v (start) | 4e39def13bca568a21087238877fbc60f8751567 via dbac86cffc6d57e8c093d2821e0d794f4c13d284 ("daemon: Enable configuration of iptables --random-fully") | 350f0b36fd9b4cf23ebc11f4365c5c89591d0ff4 via 22d4554e963e2d8029ff95087ac03e55e90a7377 ("test: Test iptables masquerading with --random-fully") v (end) ``` Reported-by: Robin Hahling <robin.hahling@gw-computing.net> Signed-off-by: Tobias Klauser <tklauser@distanz.ch> Signed-off-by: Nate Sweet <nathanjsweet@pm.me> 20 October 2020, 08:26:58 UTC
2e29740 docs: Fix TLS visibility GSG [ upstream commit 837caf8a649d11d879726415fc61d872b4def62e ] - Use k8s secret key name 'ca.crt' for 'ca-certificates.crt' so that the example CNP works - Change the example monitor output from lyft.com to artii.herokuapp.com - Add deletion of the secrets to clean-up - Fix indentation and white-spacing in enumerated lists so that they are rendered properly Signed-off-by: Jarno Rajahalme <jarno@covalent.io> Signed-off-by: Quentin Monnet <quentin@isovalent.com> 13 October 2020, 07:40:53 UTC
b79349f test: fix escaping in l7 tls vis policy [ upstream commit 8ed026ada6b01bf06b05b3cb392c2210eabbbb71 ] Signed-off-by: Maciej Kwiek <maciej@isovalent.com> Signed-off-by: Quentin Monnet <quentin@isovalent.com> 13 October 2020, 07:40:53 UTC
da73428 docs, test: replace swapi by artii.herokuapp.com for TLS visibility [ upstream commit 92aa025e90127c8fb5d00bb832882783d2cca318 ] The STAR WARS API used as an example for TLS visibility documentation and tests is no longer maintained [0], and the website is down. Let's switch to something else instead: https://artii.herokuapp.com/, which turns arguments into ASCII art snippets. [0] https://github.com/phalt/swapi Signed-off-by: Quentin Monnet <quentin@isovalent.com> 13 October 2020, 07:40:53 UTC
fbe6d7d cilium.io/v2: fix CNP Deep Equal function [ upstream commit bafc8813f5b044bd396deefbfe50f6426d7024a5 ] An equal function should never modify its fields, by doing it so it might cause race conditions. [ Backport note: Change applied to pkg/k8s/factory_functions.go instead of pkg/k8s/apis/cilium.io/v2/cnp_types.go where the related code is located in v1.9. Also keep the check on reflect.deepEqual(...) added to the value returned by the function, as is the case in branch v1.7 (changed in v1.9 with commit c813a15 ("k8s: Explicitly embed CRD types into CCNP"). ] Fixes: 100d83c27535 ("k8s: ignore kubectl.kubernetes.io/last-applied-configuration annotation") Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Quentin Monnet <quentin@isovalent.com> 12 October 2020, 20:00:40 UTC
ff72ce1 pkg/comparator: add function to compare maps ignoring keys [ upstream commit 94ddcd35946007afb7cf8e513c50b7b2afa16b61 ] MapStringEqualsIgnoreKeys will be used to compare maps while ignoring certain keys. Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Quentin Monnet <quentin@isovalent.com> 12 October 2020, 20:00:40 UTC
55f0c5a api-limiter: Add test covering cancelation of rate limiter [ upstream commit 269aa1b69786eed0649162bf319e10ebf5f4b601 ] Test that failing requests will not impact the rate limiter. This is already working correctly so this test only adds additional coverage. Signed-off-by: Thomas Graf <thomas@cilium.io> 09 October 2020, 07:22:43 UTC
61f2aa9 api-limiter: Add stress test [ upstream commit 75159c64565c59175cdae7c127e0394fde148f19 ] Simulate stress by feeding many parallel requests through the API rate limiter. The maximum wait duration will soon be hit for requests causing them to fail. Without the fix, this test would often fail. If it succeeded, the number of retries is in the range of 60-80K. With the fix, the test succeeds consistently and retries are in the range of 6-8K. That's a 10x reduction of retries. Signed-off-by: Thomas Graf <thomas@cilium.io> 09 October 2020, 07:22:43 UTC
8019191 api-limiter: Enforce ParallelRequests before applying rate limiting [ upstream commit 208c69ceddb127d811c7dc011c69bd51b14896e1 ] So far, the rate limiting has been enforced before enforcing parallel requests. In theory, this is better because we can return error code 429 earlier to tell callers to slow down. In practice, many callers will retry immediately anyway which means that all the limiting will happen by the rate limiter. The rate limiter relies on reservations that need to be canceled. If too many requests happen in parallel, reservations can't be canceled quickly enough. By swapping the enforcement of parallel requests with the rate limiter, all requests will block for at least MaxWaitDuration if more than the allowed number of parallel requests are pending which will naturally pace the callers. Swapping the enforcement order requires the acquired semaphore to be released in error cases of the rate limiter. This requires to change the structure of Wait() to have a single error handling structure. By reusing finishRequest(), the metrics handler has to be adjusted slightly to account for new outcomes as it now bumps the metric for canceled requests as well. What remains unchanged is that only successful API requests are used to calculate the mean processing duration. Fixes: 3141e6581e0 ("rate: Add API rate limiting system") Signed-off-by: Thomas Graf <thomas@cilium.io> 09 October 2020, 07:22:43 UTC
09fee20 api-limiter: Fix duplicate log message [ upstream commit ecb2369afc1660ae19406a32168f1a838aa06554 ] The following log message was duplicated and printed twice for two different meanings. Fixes: 3141e6581e0 ("rate: Add API rate limiting system") Signed-off-by: Thomas Graf <thomas@cilium.io> 09 October 2020, 07:22:43 UTC
b462f52 contexthelpers: Fix deadlock when nobody recvs on success channel [ upstream commit 4a66961b5ab8857d77bf1be13d56776049b2ca5f ] It has been observed that the "sessionSuccess <- true" statement can block forever. E.g. Cilium v1.7.9: goroutine 742 [chan send, 547 minutes]: github.com/cilium/cilium/pkg/kvstore.(*etcdClient).renewLockSession(0xc000f66000, 0x2790b40, 0xc000e247c0, 0x0, 0x0) /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:657 +0x3e2 github.com/cilium/cilium/pkg/kvstore.connectEtcdClient.func6(0x2790b40, 0xc000e247c0, 0x3aae820, 0x2) /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:819 +0x3e github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc000f42500) /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xa2a created by github.com/cilium/cilium/pkg/controller.(*Manager).updateController /go/src/github.com/cilium/cilium/pkg/controller/manager.go:120 +0xb09 This can happen when the context cancellation timer fires after a function which consumes the context has returned, but before "sessionSuccess <- true" is executed. After the timeout, the receiving goroutine is closed, making the sending block forever. Fix this by making the sessionSuccess channel buffered. Note that after sending we don't check whether the context has been cancelled, as we expect that any subsequent functions which consume the context will check for the cancellation. Fixes: 02628547 ("etcd: Fix incorrect context usage in session renewal") Signed-off-by: Martynas Pumputis <m@lambda.lt> Signed-off-by: Quentin Monnet <quentin@isovalent.com> 07 October 2020, 16:38:01 UTC
b98f285 doc: document option --enable-api-rate-limit and default to disabled As a backported feature, API rate limiting is disabled by default and gated behind a dedicated option. Let's document it. Fixes: #13392 Signed-off-by: Quentin Monnet <quentin@isovalent.com> 06 October 2020, 19:23:03 UTC
30b59c1 install: re-generate quick-install to account for "enableAPIRateLimit" Following backport PR #13392 for API rate limiting, and the addition of a specific option for the agent and to the Helm chart, re-generate the quick-install.yaml file: make -C install/kubernetes Fixes: #13392 Signed-off-by: Quentin Monnet <quentin@isovalent.com> 06 October 2020, 19:23:03 UTC
58c92a0 vagrant: Only set K8S_NODE_NAME if K8S=1 [ upstream commit fff6d6c6b02d0d2aaa2d45edb1d370ce70f8cba1 ] Fixes: #11021 Signed-off-by: Jarno Rajahalme <jarno@covalent.io> Signed-off-by: Paul Chaignon <paul@cilium.io> 06 October 2020, 08:58:44 UTC
7e1c9a8 fsnotify: correctly check for event operation [ upstream commit 28b4c96871df76c84c9afd2f91058b78d87f7c03 ] fsnotify Event.Op is a bit mask and testing for strict equality might not detect the event operation correctly. This patch make it so we check for fsnotify event operation consistently as documented at https://github.com/fsnotify/fsnotify. Signed-off-by: Alexandre Perrin <alex@kaworu.ch> Signed-off-by: Paul Chaignon <paul@cilium.io> 06 October 2020, 08:58:44 UTC
7d3c3a7 contrib: Improve start-release.sh script [ upstream commit b71cf0d0fff4d17f788a01c55affd79715731d46 ] Due to an extra `v` in the branch name, this script would fail with: $ ~/git/cilium/contrib/release/start-release.sh v1.6.12 128 fatal: 'origin/vv1.6' is not a commit and a branch 'pr/prepare-v1.6.12' cannot be created from it Signal ERR caught! Traceback (line function script): 62 main /home/joe/git/cilium/contrib/release/start-release.sh Fix it. While we're at it, update the instructions at the end for next steps, since there's also now a `submit-backport.sh` script to send the PR from the CLI. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Paul Chaignon <paul@cilium.io> 06 October 2020, 08:58:44 UTC
fbf7d4a bugtool: get bpffs mountpoint from /proc/self/mounts [ upstream commit 57d3473e7b8dff3181dd7b4394c6bd452ec60399 ] Rather then hardcoding the /sys/fs/bpf value in bugtool, use the `mountinfo` package (which exposes the information in /proc/self/mounts) to determine the correct mountpoint for the BPF filesystem. Fixes: #13218 Signed-off-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io> 06 October 2020, 08:58:44 UTC
b976923 doc: Document API rate limiting [ upstream commit d315ec3a4b26cd77d2edb4e2f4b79a148d42d246 ] [ Backporter's notes: `operations/metrics.rst doesn't exist in v1.7, it's `configuration/metrics.rst. Moved the upstream changes there to resolve conflicts. ] Signed-off-by: Thomas Graf <thomas@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
6bf7090 install: Add enableAPIRateLimit option Following the commit to add the flag to the agent, this commit adds it to the Helm chart. Note, that the api rate limiting itself is not configurable through the Helm chart. It must be done directly on the agent. Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
550f925 agent, rate: Add option to control api rate limit This commit introduces `--enable-api-rate-limit` to enable the use of api rate limiting. Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
8eb90e6 agent: Add rate limiting to endpoint API calls [ upstream commit 6333eaac4d602cb65dc6a7544767399d40e81714 ] [ Backporter's notes: * v1.7 tree doesn't have `daemon/cmd`, so renamed upstream `daemon/cmd/api_limits.go` -> `daemon/api_limits.go`. This includes changing the file's package from cmd -> main. * Small conflict with parameters to `createEndpoint()` ] Add default rate limiting for all endpoint related API calls with automatic adjustment based on estimated processing duration. Metrics are provided to monitor the rate limiting system: ``` cilium_api_limiter_adjustment_factor api_call="endpoint-create" 0.695787 cilium_api_limiter_processed_requests_total api_call="endpoint-create" outcome="success" 7.000000 cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="estimated" 2.000000 cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="mean" 2.874443 cilium_api_limiter_rate_limit api_call="endpoint-create" value="burst" 4.000000 cilium_api_limiter_rate_limit api_call="endpoint-create" value="limit" 0.347894 cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="in-flight" 0.000000 cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="limit" 0.000000 cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="max" 15.000000 cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="mean" 0.000000 cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="min" 0.000000 ``` Signed-off-by: Thomas Graf <thomas@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
489a24d rate: Add API rate limiting system [ upstream commit 3141e6581e02a895d9f7ed93f77850cbc48b9b9d ] The API rate limiting system is capable to enforce both rate limiting and maximum parallel requests. Instead of enforcing static limits, the system is capable to automatically adjust rate limits and allowed parallel requests by comparing the provided estimated processing duration with the mean processing duration observed. Usage: ``` var requestLimiter = rate.NewAPILimiter("myRequest", rate.APILimiterParameters{ rate.SkipInitial: 5, rate.RateLimit: 1.0, // 1 request/s rate.ParallelRequests: 2, }, nil) func myRequestHandler() error { req, err := requestLimiter.Wait(context.Background()) if err != nil { // request timed out whie waiting return err } defer req.Done() // Signal that request has been processed // process request .... return nil } ``` Configuration parameters: - EstimatedProcessingDuration time.Duration EstimatedProcessingDuration is the estimated duration an API call will take. This value is used if AutoAdjust is enabled to automatically adjust rate limits to stay as close as possible to the estimated processing duration. - AutoAdjust bool AutoAdjust enables automatic adjustment of the values ParallelRequests, RateLimit, and RateBurst in order to keep the mean processing duration close to EstimatedProcessingDuration - MeanOver int MeanOver is the number of entries to keep in order to calculate the mean processing and wait duration - ParallelRequests int ParallelRequests is the parallel requests allowed. If AutoAdjust is enabled, the value will adjust automatically. - MaxParallelRequests int MaxParallelRequests is the maximum parallel requests allowed. If AutoAdjust is enabled, then the ParalelRequests will never grow above MaxParallelRequests. - MinParallelRequests int MinParallelRequests is the minimum parallel requests allowed. If AutoAdjust is enabled, then the ParallelRequests will never fall below MinParallelRequests. - RateLimit rate.Limit RateLimit is the initial number of API requests allowed per second. If AutoAdjust is enabled, the value will adjust automatically. - RateBurst int RateBurst is the initial allowed burst of API requests allowed. If AutoAdjust is enabled, the value will adjust automatically. - MinWaitDuration time.Duration MinWaitDuration is the minimum time an API request always has to wait before the Wait() function returns an error. - MaxWaitDuration time.Duration MaxWaitDuration is the maximum time an API request is allowed to wait before the Wait() function returns an error. - Log bool Log enables info logging of processed API requests. This should only be used for low frequency API calls. Example: ``` level="info" msg="Processing API request with rate limiter" maxWaitDuration=10ms name=foo parallelRequests=2 subsys=rate uuid=933267c5-01db-11eb-93bb-08002720ea43 level="info" msg="API call has been processed" name=foo processingDuration=10.020573ms subsys=rate totalDuration=10.047051ms uuid=933265c7-01db-11eb-93bb-08002720ea43 waitDurationTotal="18.665µs" level=warning msg="Not processing API request. Wait duration for maximum parallel requests exceeds maximum" maxWaitDuration=10ms maxWaitDurationParallel=10ms name=foo parallelRequests=2 subsys=rate uuid=933269d2-01db-11eb-93bb-08002720ea43 ``` - DelayedAdjustmentFactor float64 DelayedAdjustmentFactor is percentage of the AdjustmentFactor to be applied to RateBurst and MaxWaitDuration defined as a value between 0.0..1.0. This is used to steer a slower reaction of the RateBurst and ParallelRequests compared to RateLimit. - SkipInitial int SkipInitial is the number of API calls to skip before applying rate limiting. This is useful to define a learning phase in the beginning to allow for auto adjustment before imposing wait durations on API calls. - MaxAdjustmentFactor float64 MaxAdjustmentFactor is the maximum adjustment factor when AutoAdjust is enabled. Base values will not adjust more than by this factor. The configuration of API rate limiters is typically provided as-code to establish defaults. A string based configuration option can then be used to adjust defaults. This allows to expose configuration of rate limiting using a single option flag: ``` go l, err = NewAPILimiterSet(map[string]string{ "foo": "rate-limit:2/m,rate-burst:2", }, map[string]APILimiterParameters{ "foo": { RateLimit: rate.Limit(1.0 / 60.0), AutoAdjust: true, }, }, nil) ``` Signed-off-by: Thomas Graf <thomas@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
549868e vendor: Bump golang.org/x/time This commit bumps the package to v0.0.0-20191024005414-555d28b269f0, which brings in support for setting bursts. This is required for an upcoming commit for api rate limiting. Signed-off-by: Chris Tarazi <chris@isovalent.com> 03 October 2020, 00:33:39 UTC
faa658a Prepare for release v1.7.10 Signed-off-by: Joe Stringer <joe@cilium.io> 30 September 2020, 23:05:17 UTC
2c3f4d7 envoy: Stop using deprecated filter names Stop using deprecated Envoy filter names in order to get rid of deprecation warning logs. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 30 September 2020, 17:38:45 UTC
863a98f Envoy: Update to release 1.14.5 [ upstream commit 3cf224e99523273d01835fb83876d92765c90b6b ] Signed-off-by: Jarno Rajahalme <jarno@covalent.io> Signed-off-by: Joe Stringer <joe@cilium.io> 30 September 2020, 17:38:45 UTC
0e9b244 envoy: Require Node only on the first request of a stream [ upstream commit 89f654d44351aa3fad671ee298f5beaadd2e5704 ] xDS request's Node field has grown huge and Envoy optionally only sends it in the first request. Use this option an adapt xDS stream processing accordingly. Partially also: [ upstream commit df0c9bbb1f3d25c1c1a534f66d28083e741354aa ] Signed-off-by: Jarno Rajahalme <jarno@covalent.io> 30 September 2020, 17:38:45 UTC
cd5c6c5 k8s: Remove CRD deleting functionality [ upstream commit 5c6aad6d5c78d80283c8d5b0238c180576ae1416 ] This commit removes the ability to delete CRDs from Cilium because that would delete all the CRs in the cluster. Follow-up from: https://github.com/cilium/cilium/pull/11477#discussion_r487816729 Updates: https://github.com/cilium/cilium/issues/12737 Signed-off-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
aa927f9 iptables: comment on xt_connmark requirement for EKS rules [ upstream commit 815be6aa8d1660b37fd33c492ec8db39b07f5d78 ] EKS requires some specific rules for asymmetric routing with multi-node NodePort traffic. These rules relies on the xt_connmark kernel module, which is usually loaded by iptables when necessary. The rules are installed when the selected IPAM is ENI, meaning they are installed on AWS (but not only EKS). The xt_connmark module should be loaded in a similar way, unless loading modules after boot has been disabled, in which case the setup fails and the agent crashes. Add a comment to at least help debug the issue. Longer term, we may want to add more explicit hints to the logs if too many users hit the issue, but that would require parsing iptables' output for the specific error, so let's see how it goes with a simple comment in the code for now. Signed-off-by: Quentin Monnet <quentin@isovalent.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
a0d754b iptables, loader: use interface with default route for EKS rules [ upstream commit a301853023ef79b36649dc985e0e4b0d9db0edbe ] Multi-node NodePort traffic on EKS needs specific rules regarding asymmetric routing. These rules were implemented for the eth0 interface (namely), because this is what EKS uses. With the default Amazon Linux 2 distribution. But EKS can also run with Ubuntu for example, and the name of the interface is not the same in that case. Instead of "eth0", use the interface with the dafault route. This is a quick fix, and longer term we want to add the rules to all relevant interfaces, as discussed in #12770. Fixes: #12770 Fixes: #13143 Signed-off-by: Quentin Monnet <quentin@isovalent.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
2a0308c iptables, loader: skip rules for EKS asymmetric routing if !IPv4 [ upstream commit 09e9a469f667b877bdcedbe4137d8919c7420aa5 ] EKS needs some specific rules for asymmetric routing with multi-node NodePort traffic. These rules are implemented only for IPv4, so we can avoid installing them when IPv4 is disabled. This is what this commit does. Note that this check is, in fact, not necessary at the moment, because as the config package says: "IPv6 cannot be enabled in ENI IPAM mode". So we always run with IPv4. But let's have it for good measure, to avoid issues if IPv6 support comes in the future. For the same reason, we also do not have to implement equivalent rules for IPv6 at the moment. Signed-off-by: Quentin Monnet <quentin@isovalent.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
1c4c3c1 loader: move ENI rules for asymmetric routing to dedicated function [ upstream commit 01f8dcc51c84e1cab269f84e782e09d8261ac495 ] EKS needs some specific rules for NodePort traffic (see PR #12770, or comments in the code, for details). The addition of part of these rules were added to the body of the Reinitialize() function in the loader. To make them easier to maintain or extend, let's move them to a dedicated function called by Reinitialize(). No functional change. Signed-off-by: Quentin Monnet <quentin@isovalent.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
de660c3 cilium: encrypt-node creates two IPsec tunnels but only uses one [ upstream commit 86858947d0e52d48c0b31fd496467b07bbed3c79 ] When we enable encrypt-node to encrypt all host traffic instead of just the pod traffic we setup two tunnels. The first tunnel is the typical one used in the case where encrypt-node is not set. This tunnel uses the cilium_host ip address range. The second tunnel uses the nodes IP per the encrypt-interface parameter or in the auto-discovered case the IP found after inspecting the route tables. On new kernels having duplicate entires in the 'ip xfrm policy' table that matches to multiple sates appears to be causing packets to be dropped. This was working on 4.19 kernels, but broke on upgrade to 5.4 kernels. We have not found the exact patch or fixed the regression. But, it is actually better/simpler to only have a single tunnel running in the encrypt-node case. This means fewer xfrm rules and is much easier to understand when only one rule can match a packet. Before this patch the xfrm policy/state looked like this with states and policies for both the 44.* xfrm tunnel and the 10.* xfrm tunnel. # ip x s src 0.0.0.0 dst 44.0.0.20 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0xd00/0xf00 output-mark 0xd00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 src 0.0.0.0 dst 10.0.0.200 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0xd00/0xf00 output-mark 0xd00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0x3e00/0xff00 output-mark 0xe00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x25d, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 src 10.0.0.200 dst 10.0.1.135 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0x3e00/0xff00 output-mark 0xe00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x589, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 # ip x p src 0.0.0.0/0 dst 44.0.0.20/32 dir fwd priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 44.0.0.20 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 44.0.0.20/32 dir in priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 44.0.0.20 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 10.0.0.0/24 dir fwd priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 10.0.0.200 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 10.0.0.0/24 dir in priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 10.0.0.200 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 10.0.1.0/24 dir out priority 0 mark 0x3e00/0xff00 tmpl src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel src 0.0.0.0/0 dst 44.0.0.10/32 dir out priority 0 mark 0x3e00/0xff00 tmpl src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel src 10.0.0.0/24 dst 10.0.1.0/24 dir out priority 0 mark 0x3e00/0xff00 tmpl src 10.0.0.200 dst 10.0.1.135 proto esp spi 0x00000003 reqid 1 mode tunnel Now we get only the single 44* xfrm tunnel, # ip x s src 0.0.0.0 dst 44.0.0.20 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0xd00/0xf00 output-mark 0xd00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel replay-window 0 mark 0x3e00/0xff00 output-mark 0xe00 aead rfc4106(gcm(aes)) 0xd0561beea6ab84e073edf1e76c49e9c0c6531d52 128 anti-replay context: seq 0x0, oseq 0x4423, bitmap 0x00000000 sel src 0.0.0.0/0 dst 0.0.0.0/0 # ip x p src 0.0.0.0/0 dst 44.0.0.20/32 dir fwd priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 44.0.0.20 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 44.0.0.20/32 dir in priority 0 mark 0xd00/0xf00 tmpl src 0.0.0.0 dst 44.0.0.20 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 10.0.1.0/24 dir out priority 0 mark 0x3e00/0xff00 tmpl src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel src 0.0.0.0/0 dst 44.0.0.10/32 dir out priority 0 mark 0x3e00/0xff00 tmpl src 44.0.0.20 dst 44.0.0.10 proto esp spi 0x00000003 reqid 1 mode tunnel There are two key pieces here. First, pod traffic is encrypted by the policy with dst 10.0.1.0/24 which is the peer nodes pod IP ranges. If we had multiple nodes we would have multiple entries here. And then second, because we now encrypt always with the 44.* IP tunnel we can eliminate a set of state rules. Considering this is both cleaner and will work on all kernels regardless of fix for kernel regression lets use this for encrypt-node use cases instead. Signed-off-by: John Fastabend <john.fastabend@gmail.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 29 September 2020, 21:54:19 UTC
fb10a31 identity: Avoid kvstore lookup for local identities [ upstream commit f3424a3690b4aca315a7b5e9c40b5fa8ed270e36 ] Reserved and CIDR identities are local to the agent and not stored in the kvstore. This commit changes the identity cache to avoid performing a kvstore lookup for CIDR entries (which is currently done when a CIDR identity is released). This is a follow-up to https://github.com/cilium/cilium/pull/13205#discussion_r490306171 Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> 29 September 2020, 21:31:11 UTC
4bae20d operator: fix operator behaviour when kube-apiserver is down [ upstream commit 30835c7a69dc863076d24708ad004d8ccd1f9f7f ] * This commit fixes an issue in Cilium dev environment wherein if kube-apiserver is stopped then cilium-operator does not restarts after loosing leader election. This was happening because we were returning exit code 0 on loosing leader election. This was causing systemd to not restart cilium-operator as the case is not regarded as failure. This was working fine with Kubernetes deployment of operator as we have restart policy set to always for operator deployment. * One edge case is fixed where now we do an exit if an error is returned when updating K8s capabilities. Earlier this could lead to an inconsistent behaviour in the cluster as we can misinterpret capabilities if kube-apiserver was down. Fixes: df90c99905ad "operator: support HA mode for operator using k8s leaderelection library" Fixes #13185 Signed-off-by: Deepesh Pathak <deepshpathak@gmail.com> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 24 September 2020, 09:28:31 UTC
b8d50d9 doc: typo fix in gettingstarted clustermesh [ upstream commit e2a935d06fb0883e42d09bcbb0c65b9b8306771a ] Signed-off-by: Alexandre Perrin <alex@kaworu.ch> Signed-off-by: Michal Rostecki <mrostecki@opensuse.org> 24 September 2020, 09:28:31 UTC
153dccd docs: Fix cilium-operator cmdref This was missed during an earlier backport, fix it up. Fixes: 8c23646845d1 ("operator: Encase CNP status handover in CLI option") Signed-off-by: Joe Stringer <joe@cilium.io> 18 September 2020, 08:43:05 UTC
3b91443 identity: Perform identity lookup in remote kvstore caches too [ upstream commit 72f6685ddb0b69b312cca9652147c16484229f11 ] When looking up a security identity (either by its numeric id or by labels) in user-space (e.g. in Hubble or the API), we want to ensure to also include identities owned by remote clusters in cluster mesh too. Before this commit, `GetIdentities` function of the identity allocator (e.g. used for `cilium identity list`) would return all global identities (i.e. including the ones from remote clusters as well), while `LookupIdentity{,ByID}` would only return identities found the main kvstore, ignoring any indentities cached from remote kvstores. This fixes multiple missed annotations which can occur in cluster-mesh setups: - Hubble failed to annotate identities from remote clusters (#13076). - While the API would list remote identities in `/v1/identities`, performing a lookup on identities from remote clusters via API would fail with a "not found error". This 404 could be observed in `cilium identity get <remote-id>` or in `cilium bpf policy get`. - DNS proxy logrecords would not have the destination endpoint labels populated. - The `CiliumEndpoint.Status.Policy` CRD field would not contain labels for identities for remote clusters (if CEP status updates were enabled). Fixes: #13076 Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> Signed-off-by: Joe Stringer <joe@cilium.io> 18 September 2020, 08:43:05 UTC
2d56d0b endpoint: Update proxy policy after adding redirects [ upstream commit 407c4157e0eafd95316b1ba9b1f7ca2bf40bade5 ] Proxy policy must be updated after redirects have been added, as waiting for ACKs for the current version from the proxy is disabled when there are no redirects (yet). This leads to the Cilium API updating the realized policy too soon, and may lead to test flakes due to traffic being denied because policy has not been initialized yet. Found in local testing with dev VM proxy smoke test. Signed-off-by: Jarno Rajahalme <jarno@covalent.io> Signed-off-by: Joe Stringer <joe@cilium.io> 18 September 2020, 08:43:05 UTC
8fd63f5 service: Add unit test for HealthCheckNodePort=false [ upstream commit 584318fb77ecb8d09137dcb2891a772cb970c1e2 ] [ Backporter's notes: pkg/service/service_test.go had a few conflicts: * v1.7 UpsertService() takes all parameters directly * GetServiceNameByAddr() doesn't exist * TestHealthCheckNodePort doesn't create a service with id2 Resolve by reworking the code against the v1.7 version to avoid backporting large refactoring commits. ] When HealthCheckNodePort is false, then the service health server will be nil. This commit adds a unit test to ensure that the code does not crash if the service health server is nil. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com> Signed-off-by: Joe Stringer <joe@cilium.io> 18 September 2020, 08:43:05 UTC
back to top