| 3bf2e87 | brharrington | 06 October 2021, 20:12:44 UTC | add jdk17 to PR validation (#1353) Drop jdk16 and add jdk17 for PR builds. | 06 October 2021, 20:12:44 UTC |
| 8f716f9 | brharrington | 06 October 2021, 19:25:31 UTC | update dependencies (#1352) | 06 October 2021, 19:25:31 UTC |
| e486dfd | Brian Harrington | 06 October 2021, 19:07:50 UTC | sbt-scalafmt 2.4.3 | 06 October 2021, 19:07:50 UTC |
| 94b5a86 | Brian Harrington | 06 October 2021, 19:02:43 UTC | sbt-sonatype 3.9.10 | 06 October 2021, 19:02:43 UTC |
| 5108c11 | Brian Harrington | 06 October 2021, 19:01:03 UTC | RoaringBitmap 0.9.22 | 06 October 2021, 19:01:03 UTC |
| 8aca728 | Brian Harrington | 06 October 2021, 18:57:50 UTC | datasketches-java 3.0.0 | 06 October 2021, 18:57:50 UTC |
| 0eb4732 | Brian Harrington | 06 October 2021, 18:54:55 UTC | iep 3.0.7 | 06 October 2021, 18:54:55 UTC |
| c2bbb02 | Brian Harrington | 06 October 2021, 18:54:14 UTC | postgresql 42.2.24 | 06 October 2021, 18:54:14 UTC |
| 45eb489 | Brian Harrington | 06 October 2021, 18:49:04 UTC | jackson 2.13.0 | 06 October 2021, 18:49:04 UTC |
| 61a51ab | Brian Harrington | 06 October 2021, 18:43:33 UTC | scalatest 3.2.10 | 06 October 2021, 18:43:33 UTC |
| ca9d805 | Brian Harrington | 06 October 2021, 18:42:13 UTC | spectator 1.0.3 | 06 October 2021, 18:42:13 UTC |
| cceef6d | brharrington | 06 October 2021, 17:48:52 UTC | ignore entity if present on pre-flight check (#1351) Some clients send OPTIONS requests with an empty chunked payload. This would result in warnings in the log because the response was sent before consuming the request entity. | 06 October 2021, 17:48:52 UTC |
| 1a341b9 | brharrington | 04 October 2021, 18:56:19 UTC | helpers for working with Postgres COPY (#1350) Adds buffer types that can be used to collect data in memory in a format that can be passed to the Postgres COPY operation. | 04 October 2021, 18:56:19 UTC |
| 09914fa | brharrington | 30 September 2021, 15:37:52 UTC | optimize validation when using SortedTagMap (#1349) Add validate variant that allows checking this type without first converting to SmallHashMap. Benchmark was updated and fixed so that validation would pass which is the most common case in actual use. Before it would fail due to the reserved key check. That made the benchmark results misleading as the composite check seemed faster since it was able to short-circuit earlier. For the successful case it was actually slower. Also adds a flag that can be used to choose whether or not to group simple tag rules into a composite. By default it will not as that seems to generally be faster. | 30 September 2021, 15:37:52 UTC |
| f72dcbf | brharrington | 30 September 2021, 15:17:19 UTC | avoid option allocation for contains (#1348) Update SmallHashMap and SortedTagMap to override the contains method and reduce the overhead. | 30 September 2021, 15:17:19 UTC |
| e791973 | brharrington | 29 September 2021, 17:37:03 UTC | add stream wrappers for nio buffers (#1347) Add simple InputStream and Reader wrappers for use with ByteBuffer and CharBuffer respectively. Useful for cases where we have data in a NIO buffer, but need to pass it to something that expects an InputStream or Reader. | 29 September 2021, 17:37:03 UTC |
| f013eed | brharrington | 21 September 2021, 17:57:48 UTC | move flatMap to after the broadcast hub (#1346) Doing it before results in a lot more message passing. By moving it after fusing happens and it is fairly cheap. | 21 September 2021, 17:57:48 UTC |
| 020a121 | brharrington | 21 September 2021, 14:35:08 UTC | batch messages going evaluation queue (#1345) There is some synchronization needed to offer an item to the source queue. With a high throughput this can add up to a lot of contention across threads. Batch the incoming datapoints to reduce the number of items that have to be offered to the queue. | 21 September 2021, 14:35:08 UTC |
| 034df25 | brharrington | 21 September 2021, 13:55:42 UTC | reduce overhead of expressions etag check (#1344) Before it would compute the etag by computing a sha1 hash of a string representation of the sorted expression list. With 100k+ expressions that is quite expensive both in terms of the memory use and CPU. This change introduces a cache for the encoded entity so it can be returned directly if there are frequent calls within a short period. The ETag is computed over the encoded and compressed response using CRC32. This reduces the memory and CPU overhead considerably. | 21 September 2021, 13:55:42 UTC |
| 7ab1c6d | brharrington | 20 September 2021, 12:06:09 UTC | add simplified tag index (#1343) Experiment for a more restrictive tag index that only supports finding items overall and not projecting keys or values. The index is also setup so that it doesn't need to reference the items or tag maps to give more flexibility to the database implementation over how that data is stored and structured. The implementation is heavily based on the corresponding parts of the RoaringTagIndex implementation. If the overall experiments go well, then this will replace the various implementations of TagIndex. | 20 September 2021, 12:06:09 UTC |
| 823eb31 | brharrington | 19 September 2021, 17:14:51 UTC | tune id calculation when using sorted map (#1342) Avoid the duplicate sort and support a direct copy into the buffer array. | 19 September 2021, 17:14:51 UTC |
| c6f2759 | brharrington | 19 September 2021, 16:28:35 UTC | use sortPairs for id calculator (#1341) Update the id calculator util to use the common helper function for sorting the pairs to reduce code duplication. | 19 September 2021, 16:28:35 UTC |
| cc6e818 | brharrington | 19 September 2021, 16:17:51 UTC | avoid allocation for computing size (#1340) The Java Map view for SmallHashMap was using the default implementation of `size` that calls `entrySet().size()`. This change overrides the size to avoid creating the entry set unnecessarily. | 19 September 2021, 16:17:51 UTC |
| f7db74b | brharrington | 16 September 2021, 15:35:21 UTC | handle AuthorizationFailedRejection (#1339) Update the default handler so it will use a 401 code rather than 400. Also adds a custom rejection type that can be used for custom directives to provide more context into why the authorization failed. | 16 September 2021, 15:35:21 UTC |
| bf98d56 | brharrington | 13 September 2021, 16:53:36 UTC | update access logger to record sizes (#1338) Record the request/response entity sizes to the ipc logger when it is known. | 13 September 2021, 16:53:36 UTC |
| b52e94e | brharrington | 11 September 2021, 16:04:03 UTC | update access log to use warn for failures (#1337) Adjust the log level for failed requests to warn instead of the default of debug. | 11 September 2021, 16:04:03 UTC |
| 556309e | brharrington | 09 September 2021, 16:39:44 UTC | add SortedTagMap implementation (#1336) For some of the storage experiments it is useful to be able to receive and maintain the tags sorted by the keys. This change adds an implementation of Map that is based on a sorted string array. The memory overhead and general performance is similar to SmallHashMap, though lookups for an individual key can be slower, logarithmic rather than typically constant, compared to a non-compact small hash map instance. | 09 September 2021, 16:39:44 UTC |
| 9433bdd | brharrington | 09 September 2021, 00:58:40 UTC | eval: properly handle non-strict messages (#1335) In cases where the message from the websocket was not strict, the source from the parsed batch wasn't getting flattend properly and the batch would get ignored later because it wasn't a recognized message type. | 09 September 2021, 00:58:40 UTC |
| 92b4620 | brharrington | 07 September 2021, 15:01:41 UTC | update dependencies (#1334) | 07 September 2021, 15:01:41 UTC |
| 54a19e6 | Brian Harrington | 07 September 2021, 14:49:17 UTC | akka 2.6.16 | 07 September 2021, 14:49:17 UTC |
| af0e14a | Brian Harrington | 07 September 2021, 14:46:49 UTC | jackson 2.12.5 | 07 September 2021, 14:46:49 UTC |
| 9949035 | Brian Harrington | 07 September 2021, 14:46:18 UTC | iep 3.0.5 | 07 September 2021, 14:46:18 UTC |
| efd704d | Brian Harrington | 07 September 2021, 14:42:59 UTC | spectator 0.135.0 | 07 September 2021, 14:42:59 UTC |
| 8d1b3c1 | Brian Harrington | 07 September 2021, 14:41:46 UTC | remove unused cloudwatch dependency | 07 September 2021, 14:41:46 UTC |
| d8decb1 | brharrington | 01 September 2021, 19:31:23 UTC | remove unused ConnectionManager class (#1333) This class is no longer used after change to websocket a while back. | 01 September 2021, 19:31:23 UTC |
| dc9e819 | brharrington | 20 August 2021, 17:53:23 UTC | improve performance for computing id (#1332) Refactors the logic to create the complete id string before feeding into the UTF-8 encoder and then message digest. For the existing benchmark this change improves the throughput by ~33%. | 20 August 2021, 17:53:23 UTC |
| e4e7d48 | brharrington | 19 August 2021, 23:33:14 UTC | reduce allocations for computing ids (#1331) Eliminates most of the allocations for computing ids from the tag maps. In particular, the buffers, message digest, UTF-8 encoder, and string array now get reused. It also uses a custom sort so the tuple allocations can be avoided and just use pairs of strings in an array. | 19 August 2021, 23:33:14 UTC |
| 4971630 | brharrington | 19 August 2021, 21:12:48 UTC | more efficient empty check for hash map (#1330) Updates the SmallHashMap implementation to override the `isEmpty` method with an efficient check. The default that is inherited will allocate an iterator. | 19 August 2021, 21:12:48 UTC |
| ae27b98 | brharrington | 16 August 2021, 19:30:24 UTC | update dependencies (#1329) | 16 August 2021, 19:30:24 UTC |
| 4f20d73 | Brian Harrington | 16 August 2021, 19:22:13 UTC | sbt-updates 0.6.0 | 16 August 2021, 19:22:13 UTC |
| 8822aed | Brian Harrington | 16 August 2021, 19:20:32 UTC | sbt 1.5.5 | 16 August 2021, 19:20:32 UTC |
| 3bd0e75 | Brian Harrington | 16 August 2021, 19:17:06 UTC | RoaringBitmap 0.9.16 | 16 August 2021, 19:17:06 UTC |
| 56adc7f | Brian Harrington | 16 August 2021, 19:15:09 UTC | equalsverifier 3.7.1 | 16 August 2021, 19:15:09 UTC |
| 82b6488 | Brian Harrington | 16 August 2021, 19:13:50 UTC | akka-http 10.2.6 | 16 August 2021, 19:13:50 UTC |
| 8d944a7 | Brian Harrington | 16 August 2021, 19:12:43 UTC | iep 3.0.3 | 16 August 2021, 19:12:43 UTC |
| acabe23 | Brian Harrington | 16 August 2021, 19:11:44 UTC | spectator 0.133.0 | 16 August 2021, 19:11:44 UTC |
| 16302f1 | Brian Harrington | 16 August 2021, 19:09:54 UTC | slf4j 1.7.32 | 16 August 2021, 19:09:54 UTC |
| 02c21ff | Matthew Johnson | 27 July 2021, 20:52:53 UTC | limit gh-actions to the main repo (#1328) Previously, when maintaining a fork of this repo, merging updates from the upstream would trigger GitHub actions for the snapshot workflow, which failed because the secrets are missing. We do not want forks to run either the snapshot or the release actions. | 27 July 2021, 20:52:53 UTC |
| c369bf2 | brharrington | 15 July 2021, 17:51:46 UTC | remove cloudwatch projects (#1327) The cloudwatch polling logic has been moved to the iep-apps repo. | 15 July 2021, 17:51:46 UTC |
| 8735c5a | brharrington | 09 July 2021, 17:59:45 UTC | update dependencies (#1326) | 09 July 2021, 17:59:45 UTC |
| d6f15a7 | Brian Harrington | 09 July 2021, 17:48:09 UTC | sbt-release 1.1.0 | 09 July 2021, 17:48:09 UTC |
| 012e655 | Brian Harrington | 09 July 2021, 17:47:24 UTC | sbt 1.5.4 | 09 July 2021, 17:47:24 UTC |
| 3181fd7 | Brian Harrington | 09 July 2021, 17:46:47 UTC | RoaringBitmap 0.9.15 | 09 July 2021, 17:46:47 UTC |
| e3eb55e | Brian Harrington | 09 July 2021, 17:39:35 UTC | equalsverifier 3.7 | 09 July 2021, 17:39:35 UTC |
| b2c4756 | Brian Harrington | 09 July 2021, 17:38:16 UTC | slf4j 1.7.31 | 09 July 2021, 17:38:16 UTC |
| 152e693 | Brian Harrington | 09 July 2021, 17:36:23 UTC | scala-collection-compat 2.5.0 | 09 July 2021, 17:36:23 UTC |
| 602e875 | Brian Harrington | 09 July 2021, 16:42:25 UTC | scala-logging 3.9.4 | 09 July 2021, 16:42:25 UTC |
| 18e2023 | Brian Harrington | 09 July 2021, 16:41:24 UTC | iep 3.0.2 | 09 July 2021, 16:41:24 UTC |
| 87f08f4 | Brian Harrington | 09 July 2021, 16:33:17 UTC | spectator 0.132.0 | 09 July 2021, 16:33:17 UTC |
| 535fbef | Brian Harrington | 09 July 2021, 16:32:32 UTC | jackson 2.12.4 | 09 July 2021, 16:32:32 UTC |
| 72dfc52 | Brian Harrington | 09 July 2021, 16:31:01 UTC | caffeine 2.9.2 | 09 July 2021, 16:31:01 UTC |
| 4a480e3 | Brian Harrington | 09 July 2021, 16:29:00 UTC | aws-sdk-java-v2 2.16.92 | 09 July 2021, 16:29:00 UTC |
| 44ab442 | brharrington | 10 June 2021, 14:45:39 UTC | reduce array allocations for encoding batch (#1325) Use `fromArrayUnsafe` to avoid copying the byte array used by the ByteString object. | 10 June 2021, 14:45:39 UTC |
| 03593c9 | Matthew Johnson | 09 June 2021, 21:20:28 UTC | add nlb cloudwatch metrics (#1324) https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html * ProcessedPackets, statistic: sum * PeakPacketsPerSecond, statistic: max * PacketsPerSecondCapacity, statistic: min | 09 June 2021, 21:20:28 UTC |
| 8682431 | brharrington | 09 June 2021, 21:12:26 UTC | update dependencies (#1323) | 09 June 2021, 21:12:26 UTC |
| fc5b377 | Brian Harrington | 09 June 2021, 20:29:45 UTC | aws-sdk-java-v2 2.16.78 | 09 June 2021, 20:29:45 UTC |
| 66c7e2c | Brian Harrington | 09 June 2021, 20:28:45 UTC | jol-core 0.16 | 09 June 2021, 20:28:45 UTC |
| 35cf750 | Brian Harrington | 09 June 2021, 20:27:15 UTC | caffeine 2.9.1 | 09 June 2021, 20:27:15 UTC |
| 1ccf14b | Brian Harrington | 09 June 2021, 20:26:13 UTC | scalatest 3.2.9 | 09 June 2021, 20:26:13 UTC |
| 5b4c081 | Brian Harrington | 09 June 2021, 20:12:57 UTC | equalsverifier 3.6.1 | 09 June 2021, 20:12:57 UTC |
| 8feaeca | Brian Harrington | 09 June 2021, 20:09:12 UTC | scala-collection-compat 2.4.4 | 09 June 2021, 20:09:12 UTC |
| 96a3c25 | Brian Harrington | 09 June 2021, 20:08:13 UTC | iep 3.0.0 | 09 June 2021, 20:08:13 UTC |
| 73ed254 | Brian Harrington | 09 June 2021, 20:06:44 UTC | spectator 0.131.0 | 09 June 2021, 20:06:44 UTC |
| 251cb45 | Brian Harrington | 09 June 2021, 20:05:55 UTC | scala 2.13.6 | 09 June 2021, 20:05:55 UTC |
| a850473 | Brian Harrington | 09 June 2021, 19:59:33 UTC | sbt-git 1.0.1 | 09 June 2021, 19:59:33 UTC |
| bf983be | Brian Harrington | 09 June 2021, 19:58:44 UTC | sbt-jmh 0.4.3 | 09 June 2021, 19:58:44 UTC |
| 3256ece | Brian Harrington | 09 June 2021, 19:57:35 UTC | sbt 1.5.3 | 09 June 2021, 19:57:35 UTC |
| 6b52384 | brharrington | 09 June 2021, 15:47:53 UTC | allow out of order updates for blocks (#1322) In #1297, normalized values were buffered and would only get output at the end. This led to some user complaints that data was delayed. It was still within SLA, but it wouldn't show up until a bit later than before if the end time was set to now instead of offset by the SLA for data to be actionable. With the compressed mutable block type introduced in #1320, it is possible to update the older blocks as they are not immutable. In this change, the block store checks have been updated so the previous block can be updated to account for out of order values that may arrive. This should make the data visible earlier as it was before. | 09 June 2021, 15:47:53 UTC |
| 76496e2 | brharrington | 09 June 2021, 14:37:00 UTC | fix block size calculation (#1321) The compressed array block was always using buffer member rather than the passed in array. This caused resizes to be ignored since the delta would always be 0. | 09 June 2021, 14:37:00 UTC |
| 8520de5 | brharrington | 08 June 2021, 22:09:17 UTC | add compressed mutable block type (#1320) Adds a new block type that is mutable, allows random access, and tries to compress data as it is coming in. This should reduce the overhead for the data that is filling in and help avoid issue for out of order updates since the historical blocks will not be compressed to an immutable type. | 08 June 2021, 22:09:17 UTC |
| d888bf2 | brharrington | 05 June 2021, 18:39:46 UTC | update CI builds to use jdk16 (#1319) Jdk16 strongly encapsulates the internals by default causing problems for older versions of Guice. When running on a newer JDK, have the build use Guice 5. For the published library still need to use 4.1.0 to avoid creating problems internally for other users stuck on old versions. | 05 June 2021, 18:39:46 UTC |
| 9a87b4e | brharrington | 04 June 2021, 18:41:48 UTC | experimental v2/subscribe api for lwc (#1318) Adds an experimental api for the LWC websocket that should allow much higher throughput. The two main changes are: 1. Uses a binary encoding which is more efficient to process. 2. Allow output messages to be batched to reduce the number of small messages that are sent over the network. This makes a big difference in isolated tests we have tried. | 04 June 2021, 18:41:48 UTC |
| 5d572ad | brharrington | 04 June 2021, 13:08:38 UTC | remove deprecated lwcapi endpoints (#1317) Eval client no longer relies on them and they have a number of critical bugs. | 04 June 2021, 13:08:38 UTC |
| d4cf25f | brharrington | 03 June 2021, 15:49:40 UTC | eval: reduce parsing of data exprs (#1316) Make the parsed expression a lazy val and reuse the first instance for a given id to minimize the number of times a data expression needs to be fully processed. Before it occur for each expression from each server. | 03 June 2021, 15:49:40 UTC |
| d6f4ed2 | brharrington | 03 June 2021, 14:08:54 UTC | eval: move message parsing earlier (#1315) Moves the message parsing out of LwcToAggrDatapoint to just after the data is received. This allows us to avoid some intermediate conversions from String to ByteString. | 03 June 2021, 14:08:54 UTC |
| 2873c3b | brharrington | 03 June 2021, 02:52:34 UTC | eval: avoid redundant parsing of expressions (#1313) Refactor so that the expressions passed in are only processed once rather than once per server instance. | 03 June 2021, 02:52:34 UTC |
| 0bfcc5a | brharrington | 02 June 2021, 20:51:51 UTC | fix CCE for RoaringTagIndex.empty (#1312) Fix ClassCastException for casting the array based on the generic type. | 02 June 2021, 20:51:51 UTC |
| cebb24f | brharrington | 02 June 2021, 15:13:09 UTC | delay heartbeat timestamp (#1311) Update the heartbeat timestamp to be one interval behind the current time. This gives more time for data flowing in from real systems to arrive when using a small number of buffers on the eval client. | 02 June 2021, 15:13:09 UTC |
| 2ba3edd | brharrington | 02 June 2021, 15:02:55 UTC | avoid fold for strict messages (#1310) When there are many small strict messages the fold step adds a significant overhead. | 02 June 2021, 15:02:55 UTC |
| 7ef6895 | brharrington | 21 May 2021, 21:22:51 UTC | reduce usage of Manifest (#1308) Manifest is not supported in scala 3. Removes the usage of Manifest for everything but JSON type detection. | 21 May 2021, 21:22:51 UTC |
| bee70a0 | brharrington | 21 May 2021, 20:48:12 UTC | avoid boxing for computing hash code (#1307) Compute the hash code on the long value directly and avoid boxing with a `java.lang.Long`. | 21 May 2021, 20:48:12 UTC |
| f22dd4e | brharrington | 19 May 2021, 15:45:46 UTC | fix build deprecation warnings (#1305) Replace in with [slash syntax]. [slash syntax]: https://www.scala-sbt.org/1.x/docs/Migrating-from-sbt-013x.html#slash | 19 May 2021, 15:45:46 UTC |
| 7a3f620 | brharrington | 18 May 2021, 23:39:21 UTC | rewrite do-while to while loop (#1304) The [do-while] loop is not supported in scala 3. [do-while]: http://dotty.epfl.ch/docs/reference/dropped-features/do-while.html | 18 May 2021, 23:39:21 UTC |
| 39f1a41 | brharrington | 18 May 2021, 19:16:42 UTC | remove coverage target (#1303) The plugin for SBT was removed a long time ago. Remove the target from the makefile since it doesn't work. | 18 May 2021, 19:16:42 UTC |
| 7f0249d | brharrington | 14 May 2021, 20:34:17 UTC | remove CaseClassDeserializer (#1302) Since version 2.12.2, the Jackson Scala module supports default values for case class constructor parameters. The main limitation is that special field names like `$` may not work correctly. That can be worked around by using the `@JsonProperty` annotation. | 14 May 2021, 20:34:17 UTC |
| c08bf76 | brharrington | 05 May 2021, 00:03:33 UTC | add timeout for ec2 cloudwatch metrics (#1301) Ensures that they will stop getting published after an ASG goes away. Otherwise they will stick around until the poller service is redeployed. | 05 May 2021, 00:03:33 UTC |
| 873ee4c | Tim Jiang | 28 April 2021, 21:26:57 UTC | increase delay for alb cloudwatch metrics to avoid partially reported data points (#1300) Increase forward delay for alb cloudwatch metrics to avoid partially reported data points. ALB Cloudwatch metrics data for the most recent minute can be incomplete and volatile depending on the timing of polling, increase offset to 2 min to avoid this issue. | 28 April 2021, 21:26:57 UTC |
| 939e202 | brharrington | 27 April 2021, 20:24:33 UTC | update dependencies (#1299) | 27 April 2021, 20:24:33 UTC |
| 90176f4 | Brian Harrington | 27 April 2021, 20:14:21 UTC | sbt-updates 0.5.3 | 27 April 2021, 20:14:21 UTC |
| bf30c08 | Brian Harrington | 27 April 2021, 20:13:40 UTC | sbt-release 1.0.15 | 27 April 2021, 20:13:40 UTC |
