| e283528 | brharrington | 13 July 2017, 22:17:14 UTC | improve error message when missing uri param (#630) Fixes #574. Improves the error message returned if the input URI is missing required parameters. It has also been changed to get returned as a diagnostic message as part of the stream so that a single bad URI will not stop the processor for all evaluations. | 13 July 2017, 22:17:14 UTC |
| 6c6204c | brharrington | 04 July 2017, 17:17:45 UTC | refresh dependencies (#627) | 04 July 2017, 17:17:45 UTC |
| 3250f95 | Brian Harrington | 04 July 2017, 17:08:37 UTC | jsr305 3.0.2 | 04 July 2017, 17:08:37 UTC |
| c69cd42 | Brian Harrington | 04 July 2017, 17:01:36 UTC | spectator 0.56.0 | 04 July 2017, 17:01:36 UTC |
| 1fb621c | Brian Harrington | 04 July 2017, 16:59:54 UTC | akka-http 10.0.9 | 04 July 2017, 16:59:54 UTC |
| 93c2236 | Brian Harrington | 04 July 2017, 16:55:15 UTC | akka 2.5.3 | 04 July 2017, 16:55:15 UTC |
| 695c429 | Brian Harrington | 04 July 2017, 16:52:29 UTC | aws-java-sdk 1.11.158 | 04 July 2017, 16:52:29 UTC |
| 775706b | Brian Harrington | 04 July 2017, 16:51:39 UTC | roaringbitmap 0.6.45 | 04 July 2017, 16:51:39 UTC |
| dbc4453 | Brian Harrington | 04 July 2017, 16:50:42 UTC | equalsverifier 2.3.1 | 04 July 2017, 16:50:42 UTC |
| ef312e7 | Brian Harrington | 04 July 2017, 16:49:05 UTC | iep 1.0.2 | 04 July 2017, 16:49:05 UTC |
| 6dc5e10 | brharrington | 29 June 2017, 22:24:03 UTC | update merge to also dedup the list (#626) For the use-cases we care about the input list will already be sorted and uniqued. This fixes the merge to also remove duplicate values across the lists. | 29 June 2017, 22:24:03 UTC |
| 2e98374 | brharrington | 29 June 2017, 21:29:14 UTC | helper for merging sorted lists (#625) Broken off from larger effort that is still in-progress. Basic helper utility for merging a collection of sorted lists. One such use-case is merging tag list results coming back from many shards. | 29 June 2017, 21:29:14 UTC |
| 7e7d093 | brharrington | 29 June 2017, 16:34:41 UTC | fix error when rendering with a long time range (#624) When trying to render a chart with a long time range a NoSuchElementException would result if the graph width was too small. Sample exception below: ``` java.util.NoSuchElementException: head of empty list at scala.collection.immutable.Nil$.head(List.scala:428) ~[scala-library-2.12.2.jar:?] at scala.collection.immutable.Nil$.head(List.scala:425) ~[scala-library-2.12.2.jar:?] at com.netflix.atlas.chart.graphics.Ticks$.time(Ticks.scala:265) ~[atlas-chart_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] ``` The cause is that there were no options for meeting the desired number of major ticks given the options for timeTickSizes. This change uses a simpler selection method with just major ticks for longer ranges setting it to the name of the month or year. | 29 June 2017, 16:34:41 UTC |
| be4cd8b | brharrington | 20 June 2017, 18:05:17 UTC | add :clamp-min and :clamp-max (#623) Allows the user to restrict the min and max values of the input respectively. The most common use-case motivating this is to be able to support a bounded auto-scaling for data on an axis. The axis lower and upper limits are either explicit or automatic. These operators give more flexiblity and can be set for a given line to tune its behavior. Name was chosen to match the same operators in Promoetheus. | 20 June 2017, 18:05:17 UTC |
| 1b4d0aa | brharrington | 19 June 2017, 15:17:12 UTC | cache most recent datapoints from cloudwatch (#621) In some cases if there are a lot of cloudwatch metrics, then the polling can start to get throttled and cannot keep up at the desired rate. This can cause spotty reporting until the limit can get raised or the poller config is changed to be less aggressive. To minimize the visibility pf the problem for a user consuming the metric the last datapoint is now cached and the reported value comes from the local cache. This does mean that when such a problem is happening the wrong value might be propagated for several minutes. From recent testing though in the more typical case it avoids confusion more often until there is a better way to access cloudwatch data. If it is a big concern the ttl can be configured to a smaller value. | 19 June 2017, 15:17:12 UTC |
| 346e9c6 | brharrington | 19 June 2017, 14:48:36 UTC | add mapping for ApproximateAgeOfOldestMessage (#620) New SQS metric that was added a while back: https://aws.amazon.com/about-aws/whats-new/2016/08/new-amazon-cloudwatch-metric-for-amazon-sqs-monitors-the-age-of-the-oldest-message/ | 19 June 2017, 14:48:36 UTC |
| 66439f1 | brharrington | 15 June 2017, 20:18:20 UTC | fix IllegalFormatConversionException (#619) Illegal conversion when trying to access the group by key for a data expression. ``` java.util.IllegalFormatConversionException: x != com.netflix.atlas.core.model.ItemId at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4302) at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2793) at java.util.Formatter$FormatSpecifier.print(Formatter.java:2747) at java.util.Formatter.format(Formatter.java:2520) at java.util.Formatter.format(Formatter.java:2455) at java.lang.String.format(String.java:2940) at scala.collection.immutable.StringLike.format(StringLike.scala:351) at scala.collection.immutable.StringLike.format$(StringLike.scala:350) at scala.collection.immutable.StringOps.format(StringOps.scala:29) at com.netflix.atlas.core.model.DataExpr.groupByKey(DataExpr.scala:35) ``` | 15 June 2017, 20:18:20 UTC |
| 938edd6 | brharrington | 13 June 2017, 17:36:25 UTC | docs: add Roy's presentation from Monitorama (#618) | 13 June 2017, 17:36:25 UTC |
| 4e73843 | brharrington | 06 June 2017, 16:38:46 UTC | refresh dependencies (#617) | 06 June 2017, 16:38:46 UTC |
| 45b0f76 | Brian Harrington | 06 June 2017, 16:27:19 UTC | iep 1.0.1 | 06 June 2017, 16:27:19 UTC |
| 65d70d1 | Brian Harrington | 06 June 2017, 15:09:32 UTC | akka 2.5.2 Note, there was a signature change to `FileIO.toPath` causing the build to break. Looks like the type for the set of options was changed leading to: ``` Error:(115, 34) type mismatch; found : scala.collection.immutable.Set[java.nio.file.StandardOpenOption] required: Set[java.nio.file.OpenOption] Note: java.nio.file.StandardOpenOption <: java.nio.file.OpenOption, but trait Set is invariant in type A. You may wish to investigate a wildcard type such as `_ <: java.nio.file.OpenOption`. (SLS 3.2.10) .toMat(FileIO.toPath(file, options))(Keep.right) ``` | 06 June 2017, 16:14:56 UTC |
| d080359 | Brian Harrington | 06 June 2017, 15:08:59 UTC | akka-http 10.0.7 | 06 June 2017, 15:08:59 UTC |
| ea8db2b | brharrington | 06 June 2017, 15:08:36 UTC | fix spelling in publish-test script (#616) s/anwser/answer/ | 06 June 2017, 15:08:36 UTC |
| 8a75a59 | Brian Harrington | 06 June 2017, 15:07:49 UTC | jol-core 0.8 | 06 June 2017, 15:07:49 UTC |
| bbd4a62 | Brian Harrington | 06 June 2017, 15:06:49 UTC | sbt-jmh 0.2.25 | 06 June 2017, 15:06:49 UTC |
| 975785a | Brian Harrington | 06 June 2017, 15:05:31 UTC | caffeine 2.5.2 | 06 June 2017, 15:05:31 UTC |
| 0e83dac | Brian Harrington | 06 June 2017, 15:04:46 UTC | equalsverifier 2.3 | 06 June 2017, 15:04:46 UTC |
| 4138533 | Brian Harrington | 06 June 2017, 15:04:05 UTC | log4j 2.8.2 | 06 June 2017, 15:04:05 UTC |
| 81e3e37 | Brian Harrington | 06 June 2017, 15:01:27 UTC | aws-java-sdk 1.11.133 | 06 June 2017, 15:01:27 UTC |
| 4985299 | brharrington | 06 June 2017, 14:43:39 UTC | add more test cases for cors (#615) Add some test cases for cors on error responses. There was a report CORS headers were not added to errors. That does not seem to be the case. Most likely they did not handle the rejections and the filter was not applied to the error. | 06 June 2017, 14:43:39 UTC |
| c3964ca | brharrington | 06 June 2017, 04:40:21 UTC | docs: update to recommend 1.5.3 (#614) | 06 June 2017, 04:40:21 UTC |
| 0dfb445 | brharrington | 03 June 2017, 17:00:09 UTC | fix warnings about adapting argument list to tuple (#611) The usage of the parameters directive was giving warnings like: ``` Warning:(48, 17) Adapting argument list by creating a 3-tuple: this may not be what you want. signature: ParameterDirectives.parameters(pdm: akka.http.scaladsl.server.directives.ParameterDirectives.ParamMagnet): pdm.Out given arguments: scala.Symbol("name").$qmark, scala.Symbol("expression").$qmark, scala.Symbol("frequency").$qmark after adaptation: ParameterDirectives.parameters((scala.Symbol("name").$qmark, scala.Symbol("expression").$qmark, scala.Symbol("frequency").$qmark): akka.http.scaladsl.server.directives.ParameterDirectives.ParamMagnet{type Out = akka.http.scaladsl.server.Directive[(Option[String], Option[String], Option[String])]}) parameters('name.?, 'expression.?, 'frequency.?) { (name, expr, frequency) => ``` It is easy to fix, though previous usage does seem to be inline with examples from docs: http://doc.akka.io/docs/akka-http/10.0.7/scala/http/routing-dsl/directives/parameter-directives/parameters.html#optional-parameter-with-default-value | 03 June 2017, 17:00:09 UTC |
| 43df9ee | brharrington | 03 June 2017, 16:48:13 UTC | fix deprecation warning for use of actor publisher (#610) There are a few more of these in other parts of the code, but they are a bit more involved to cleanup. Will look at those later. | 03 June 2017, 16:48:13 UTC |
| a08e04c | brharrington | 03 June 2017, 04:21:56 UTC | use hash code to short-circuit equals check (#609) Since the hash code is precomputed it can be a cheap way to short circuit the equality check on the array. | 03 June 2017, 04:21:56 UTC |
| 5b91291 | brharrington | 02 June 2017, 22:57:24 UTC | add test case for decoding datapoint (#608) Make sure encoding/decoding of datapoint works as expected. | 02 June 2017, 22:57:24 UTC |
| c0bb87a | brharrington | 02 June 2017, 22:47:33 UTC | skip value associated with unknown fields (#607) If an unknown object field was used in a datapoint it would result in a NullPointerException by prematurely closing hitting the end of object. | 02 June 2017, 22:47:33 UTC |
| edaaaac | brharrington | 02 June 2017, 20:56:36 UTC | change ids to custom class (#606) Before the ids were just a BigInteger. This was mostly for convenience and it was an easy starting point for some modifications. Flame graphs of stateful clusters for some stacks are showing a significant about of time being spent computing the hash code for the ids when looking up the blocks. The custom class in this change uses a precomputed hash code so that it will be constant time. | 02 June 2017, 20:56:36 UTC |
| 4f54d00 | brharrington | 02 June 2017, 20:19:12 UTC | fix regex matching when prefix is an exact match (#605) When searching for the prefix it was looking for the postion of the first match that was strictly greater than the prefix. For cases where the prefix is an exact match to the full value they would not match. The search now checks for the first match that is greater than or equal to the prefix. | 02 June 2017, 20:19:12 UTC |
| 9cc4292 | brharrington | 31 May 2017, 01:50:06 UTC | fix warning for offset (#604) s/var/val/. | 31 May 2017, 01:50:06 UTC |
| 7cfcc2c | brharrington | 31 May 2017, 01:28:26 UTC | fix multi-threaded reads from long set/map (#603) Several use-cases of the primitive collections assume multi-threaded read-only access is safe after the collection is built and effectively immutable. That is nothing will ever write to it again. The variants with a Long key type use an internal buffer to avoid allocations when passing the long value to `MurmurHash3.bytesHash`. The multi-threaded access could result in errors such as BufferOverflowException on the buffer. This changes those collections to use a ThreadLocal for keeping track of the buffers. | 31 May 2017, 01:28:26 UTC |
| 36f537c | brharrington | 30 May 2017, 15:41:04 UTC | update findKeys to just return the strings (#602) Before it was wrapping the string in a TagKey type that could also provide counts for each key. Since this isn't supported or used anymore we can avoid the additional allocations. Also most use-cases were just mapping it back to strings after getting the response. | 30 May 2017, 15:41:04 UTC |
| 3864010 | brharrington | 29 May 2017, 21:51:01 UTC | improve tag query performance (#601) Drops support for counts in the index and does some initial work to simplify and improve performance. For a lot of the common use-cases we see a 2-3x improvement: **Before** ``` Benchmark Mode Cnt Score Error Units create thrpt 10 13.210 ± 1.041 ops/s findKeysAll thrpt 10 4646.344 ± 176.685 ops/s findKeysQuery thrpt 10 450.728 ± 39.296 ops/s findValuesAllMany thrpt 10 739.919 ± 41.530 ops/s findValuesAllOne thrpt 10 2194.050 ± 248.351 ops/s ``` **After** ``` Benchmark Mode Cnt Score Error Units create thrpt 10 20.270 ± 0.798 ops/s findKeysAll thrpt 10 6382544.499 ± 353321.426 ops/s findKeysQuery thrpt 10 1118.262 ± 370.960 ops/s findValuesAllMany thrpt 10 3849.128 ± 416.156 ops/s findValuesAllOne thrpt 10 4485.419 ± 424.741 ops/s ``` Some of the primitive collections have also been adjusted to use murmur hash instead of `Long.hashCode(v)` because the collision rate was way to high. This adds a bit of memory overhead because we copy the value into a pre-allocated byte buffer to reuse the murmur hash implementation provided in the scala std library. | 29 May 2017, 21:51:01 UTC |
| 51f676b | brharrington | 27 May 2017, 18:10:52 UTC | drop cross build for 2.11 (#600) We have been running on 2.12 for a while now and internal builds and deployments are mostly on 2.12 now. The few stragglers are using the 1.5.x Atlas builds which is 2.11.x only and that will not change. | 27 May 2017, 18:10:52 UTC |
| 73416e3 | brharrington | 27 May 2017, 18:01:53 UTC | always sort NaN values to the end (#599) This change alters the behvaior of sorting to always move NaN values to the end regardless of the order. Before, NaN values were at the end for ascending order and descending order would just reverse the list causing NaN values to show at the top. Fixes #586. | 27 May 2017, 18:01:53 UTC |
| 1765646 | brharrington | 23 May 2017, 21:37:51 UTC | fix unused variable warnings (#595) | 23 May 2017, 21:37:51 UTC |
| 46a5fb2 | brharrington | 23 May 2017, 17:01:54 UTC | avoid unused method warnings in guice modules (#594) Since scala 2.12 the private provider methods in the module cause warnings about unused methods. Unfortunately scala doesn't provide a way to suppress these on individual methods (SI-1781). For now the work around is to mark them as protected rather than private. Note the module class is final so there will not be any subclasses. | 23 May 2017, 17:01:54 UTC |
| 42de1e4 | brharrington | 23 May 2017, 16:50:43 UTC | update flow types to allow DiagnosticMessages (#593) Updates the high level processor to pass JsonSupport messages instead of TimeSeriesMessage specifically. This gives a lot of flexibility to send through other types like DiagnosticMessage or other helpful information when processing the stream. | 23 May 2017, 16:50:43 UTC |
| 39d481d | Matthew Johnson | 19 May 2017, 16:58:38 UTC | docs: discuss low volume conditions for alerting (#592) With traffic failovers internally low volume conditions cause a number of false positives. Update the philosophy doc to discuss that use-case and compensating with a check on the absolute volume. | 19 May 2017, 16:58:38 UTC |
| 0d953a3 | brharrington | 18 May 2017, 14:16:49 UTC | suppress chunks that are all NaN (#591) If there is no data for an expression in a given chunk window, then drop that data. | 18 May 2017, 14:16:49 UTC |
| 162c392 | brharrington | 17 May 2017, 22:35:51 UTC | docs: expand description for legend (#590) In particular discuss the use of variables in the legends. | 17 May 2017, 22:35:51 UTC |
| 95665c5 | brharrington | 17 May 2017, 21:22:41 UTC | fix unit used when paritioning a context (#588) The context was ignoring the unit parameter passed into partition. This would result in an IAE under some situations. See test case for more details. | 17 May 2017, 21:22:41 UTC |
| 39f9d30 | brharrington | 16 May 2017, 16:13:11 UTC | initial port of fetch api (#585) Starting work on #547. This adds an endpoint compatible with the internal fetch API to the oss repo. There are still a number of things left to do, but this should unblock some other efforts. To be addressed in later PRs: - Test coverage, there are some basic test cases but I still need to debug some issues with one of them and clean them up a bit to remove use of some test data that was dumped from internal sources. - The throttling/limiter logic still needs to be moved over, but is a bit more intertwined. - In general need to move the actors away from the actor publisher that was deprecated in akka 2.5. | 16 May 2017, 16:13:11 UTC |
| e4640c8 | brharrington | 07 May 2017, 16:07:43 UTC | docs: update to refer to 1.5.2 (#583) | 07 May 2017, 16:07:43 UTC |
| 749e214 | brharrington | 07 May 2017, 15:47:19 UTC | akka-http 10.0.6 (#582) Fixes security vulnerability: http://doc.akka.io/docs/akka-http/10.0.6/security/2017-05-03-illegal-media-range-in-accept-header-causes-stackoverflowerror.html | 07 May 2017, 15:47:19 UTC |
| e311d28 | brharrington | 07 May 2017, 04:24:46 UTC | allow step size for datapoint to be set explicitly (#581) Before the step would always come from the global config setting. This made it impossible to use the datapoint class with multiple step sizes in play. | 07 May 2017, 04:24:46 UTC |
| cafa8ce | brharrington | 07 May 2017, 04:07:25 UTC | fix #578, incorrect formatting for links (#580) Looks like github no longer accepts the a line break between the display text and href for the link. | 07 May 2017, 04:07:25 UTC |
| c52f00c | brharrington | 06 May 2017, 18:13:18 UTC | search and replace over legend text (#579) This allows some simple manipulation of the legend text using a sed like search and replace. The main use-case is for massaging the results of a group by to be more presentable. | 06 May 2017, 18:13:18 UTC |
| 26239b2 | brharrington | 05 May 2017, 04:02:25 UTC | create bitmap directly instead of through int set (#573) The use of the int set was a carry over from the lazy set implementation. This removes some steps saving memory and computation cost when building the index. | 05 May 2017, 04:02:25 UTC |
| 8c2a809 | brharrington | 05 May 2017, 03:53:56 UTC | use int iterator for bitmaps (#572) Avoids boxing the values to Integer. | 05 May 2017, 03:53:56 UTC |
| eeb92af | Brian Harrington | 04 May 2017, 15:46:58 UTC | add test case for createStreamsProcessor() Provides a simple example of usage and verifies some of the basic functionality with the file based uri. | 04 May 2017, 15:46:58 UTC |
| f2edd36 | brharrington | 01 May 2017, 22:19:52 UTC | included rendered image in v2.json response (#568) Fixes #564. We may move this behind a flag before 1.6 final, but for now it is enabled by default. By including in the v2.json format the UI/dashboard can use the static image and switch to a dynamic rendering without making another request. The message has a type of `graph-image` with a data field that is a data uri of the image without the legend. Sample usage: ```html <html> <body> <div id="content"></div> <script> var content = document.getElementById('content'); fetch('http://localhost:7101/api/v1/graph?q=name,sps,:eq,(,nf.cluster,),:by&format=v2.json') .then(function(response) { return response.json(); }) .then(function(json) { var html = ''; json.forEach(function(msg) { if (msg.type === 'graph-image') { html += '<div><img src="' + msg.data + '"/></div>'; } else if (msg.type === 'timeseries') { html += '<div>' + msg.label + '</div>'; } }); content.innerHTML = html; }); </script> </body> </html> ``` | 01 May 2017, 22:19:52 UTC |
| 16a734c | brharrington | 01 May 2017, 21:23:55 UTC | remove ExpressionDatabaseActor from sample config (#567) Updates the lwcapi.conf sample config to remove the actor since it went away in #561. Also updates to spectator 0.55.0. | 01 May 2017, 21:23:55 UTC |
| 1481015 | brharrington | 29 April 2017, 00:23:17 UTC | add additional graph api tests (#566) Add some additional graph api test images that were being used for the dynamic rendering component. Trying to keep them in sync. | 29 April 2017, 00:23:17 UTC |
| f766add | brharrington | 28 April 2017, 04:35:15 UTC | fix #560, axis params now work with axis_per_line (#563) When using `axis_per_line` flag the specific url params for other y-axes would get ignored because it was just copying from the main axis. Now overrides will take effect just like they do for explicit multi-y. | 28 April 2017, 04:35:15 UTC |
| 9d0db91 | brharrington | 28 April 2017, 01:39:26 UTC | make axisPerLine with explicit axis a warning (#562) Instead of giving an error to the user make it a warning and disable the axisPerLine flag. | 28 April 2017, 01:39:26 UTC |
| cbb0c6c | brharrington | 27 April 2017, 23:39:59 UTC | fix #467, remove the use of redis for lwcapi (#561) The only use-case was for ensuring the set of expressions was sync'd up across instances so that clients would not see expressions come and go very quickly due to timing issues for the subscribers. To simplify the overall setup the clients will do a merge and maintain the expression with a TTL. | 27 April 2017, 23:39:59 UTC |
| 210322e | brharrington | 26 April 2017, 15:07:56 UTC | update to frigga 0.18.0 (#558) Adds support for `~` in the detail for group names (Netflix/frigga#20). | 26 April 2017, 15:07:56 UTC |
| 91049e7 | brharrington | 26 April 2017, 14:54:06 UTC | remove LazyTagIndex implementation (#557) We have been using the implementation based on roaring bitmaps (#479) for a few months internally and it is working well. | 26 April 2017, 14:54:06 UTC |
| f848278 | brharrington | 26 April 2017, 14:38:44 UTC | switch to Files.new{Input,Output}Stream (#556) They avoid finalize making them a bit more GC friendly. https://www.cloudbees.com/blog/fileinputstream-fileoutputstream-considered-harmful | 26 April 2017, 14:38:44 UTC |
| 650d555 | brharrington | 26 April 2017, 00:14:23 UTC | fix spacing on sidebar header (#555) Fixes the spacing so that it will render correctly on the wiki. | 26 April 2017, 00:14:23 UTC |
| 76ed570 | brharrington | 25 April 2017, 23:32:36 UTC | update to sbt 0.13.15 (#554) This version seems to fix `formatLicenseHeaders` so it will work when run from the root. In #553 it was noticed that it worked when running for a particular subproject, but not when run for the root project. | 25 April 2017, 23:32:36 UTC |
| f56a60f | brharrington | 25 April 2017, 15:44:52 UTC | add eval lib to root project (#553) It wasn't getting published to jfrog/jcenter since it wasn't part of the root. Also fixes license header for SourceRef. | 25 April 2017, 15:44:52 UTC |
| d68cf46 | brharrington | 25 April 2017, 15:03:51 UTC | docs: fix link to sorting modes (#552) | 25 April 2017, 15:03:51 UTC |
| 8a1b68e | brharrington | 25 April 2017, 14:59:58 UTC | add reference config for eval lib (#550) This was accidentally left out of a previous commit. | 25 April 2017, 14:59:58 UTC |
| 5af80e1 | brharrington | 24 April 2017, 20:08:59 UTC | add module for configuring evaluator (#549) Minimal module that can setup an evaluator instance that can be injected. | 24 April 2017, 20:08:59 UTC |
| bb30335 | brharrington | 24 April 2017, 16:01:58 UTC | simplify eval usage for common use-cases (#548) Adds an `Evaluator` class with a simple public API for evaluating the stream. It is a java class that exposes the reactive streams types so it is easy to use with various implementations. There are two main methods: **createPublisher(uri)** Creates a publisher for a graph uri so the same data can be accessed as a stream. This makes it really easy for a user to transition from one view to another. ``` +------------+ | URI | | Eval | -------- [TimeSeriesMessage] ---> | Source | +------------+ ``` **createStreamsProcessor** Takes a set of id/URI pairs as inputs and evaluates all of the expressions. This allows for better reuse when evaluating many expressions. There is an id for each input that will be added to the output so that corresponding entries can be correlated. The complete set of current data sources must be supplied as part of each input event. ``` +------+ | | --- [DataSources] ---> | Eval | --- [MessageEnvelope] ---> Set((id,URI), ...) | | (id, TimeSeriesMessage) +------+ ``` | 24 April 2017, 16:01:58 UTC |
| f0ea676 | brharrington | 19 April 2017, 15:20:02 UTC | update to scala 2.12.2 and fix warnings (#545) Updates to scala 2.12.2 (and 2.11.11) and fixes a bunch of new compiler warnings. | 19 April 2017, 15:20:02 UTC |
| e46a321 | brharrington | 19 April 2017, 14:36:00 UTC | fix rewrite to use new display expr (#544) Fixes a regression introduced in #540. The non-query case was re-evaluating the rewrite command on the original query rather than the new rewritten display expression. | 19 April 2017, 14:36:00 UTC |
| a37ee20 | brharrington | 18 April 2017, 16:47:38 UTC | add util for wrapping ByteString as InputStream (#543) This makes it easier to quickly pass the ByteString to jackson and keep the number of additional allocations minimal. | 18 April 2017, 16:47:38 UTC |
| 924a076 | brharrington | 18 April 2017, 15:25:02 UTC | update to akka 2.5.0 (#542) Switch to akka 2.5.0 to get improved materialization performance for streams. Throughput for the simple benchmark in #541 improved by 26%. | 18 April 2017, 15:25:02 UTC |
| 6ed04f0 | brharrington | 18 April 2017, 14:46:13 UTC | reduce allocations for processing messages (#541) Some initial performance checks on the streaming evaluation show a lot of allocations. The biggest contributor was converting from ByteString to a form that jackson could parse. This has been reduced by reusing buffers. Some other minor changes have been made to reduce allocations for fields and to go direct to SmallHashMap for the tag data. Biggest bottlenecks right now are parsing the json messages from the SSE stream followed by the materialization speed. The materialization speed should be less of an issue in practice because the streams will be longer lived and is supposed to be improved in akka 2.5.0. | 18 April 2017, 14:46:13 UTC |
| fed398a | brharrington | 15 April 2017, 04:32:01 UTC | named rewrite expr string now preserves offset (#540) Fixes a bug with the expression string for named rewrites that contain offsets. If the rewrite is being used similar to an aggregate function and the display expression is a query, then the offset was getting lost because that is an attribute of the aggregate functions that are only present in the expanded evaluation expression. Now the named rewrite will extract the offset and group by information if the display type is a query. For others the rewrite will be re-evaluated using the same interpreter. | 15 April 2017, 04:32:01 UTC |
| 8063495 | brharrington | 13 April 2017, 20:45:27 UTC | initial streaming eval library (#538) This isn't yet fully baked or hardened, but is a good enough check point. This is an initial version of a library to evaluate expressions using the LWC service as a source. Still left to do: 1. Expose simple and stable java API for accessing the stream for a corresponding graph uri. This is to make it really easy to transition between the backend graph and a streaming source. 2. Source identification in LWC so we can reliably dedup messages. 3. Improving test coverage of Eureka source and evaluator helpers. This is really sparse right now. 4. Refactor Datapoint so that the step size is provided rather than coming from a static config. 5. Performance testing and tuning. These follow up items will be addressed in later PRs. | 13 April 2017, 20:45:27 UTC |
| 19a9a1a | brharrington | 11 April 2017, 16:18:45 UTC | enable self-reporting for standalone example (#537) The standalone example will not report data to itself when running with the sample `memory.conf`. A similar thing can be done with the `lwcapi.conf` by uncommenting the enabled flag. It is disabled by default since there are some ordering issues on startup that cause exceptions in the log. | 11 April 2017, 16:18:45 UTC |
| 0e77e68 | brharrington | 11 April 2017, 15:20:09 UTC | disable log4j shutdown hook on default config (#536) Updates the default logging config for standalone so that it disables the log4j shutdown hook. This is to avoid missing log messages for the app shutdown: https://github.com/Netflix/iep/tree/master/iep-guice#shutdown | 11 April 2017, 15:20:09 UTC |
| d4b8470 | brharrington | 11 April 2017, 15:16:34 UTC | only run lwc when using the lwcapi.conf (#535) Fixes the standalone example to only run the lwc api and services when using the lwcapi config. The webapi portions have been restricted in a similar way. This avoids the need for having redis to run the basic standalone example. | 11 April 2017, 15:16:34 UTC |
| cd03716 | brharrington | 10 April 2017, 23:07:41 UTC | add limit operation that can be used with sort (#534) Adds a `:limit` operation that can be used with the existing `:sort` and `:order` to restrict the output set. The `:head` operator has been removed as a data operator and changed to be an alias for limit. Limit was chosen because it is more familiar to those with a SQL background. The `:head` operator was never really optimized in the data layer and a desire to use sort with a limit for top-N use-cases is more prominent than the sampling use-cases for which `:head` was originally intended, but that did not really materialize in practice. | 10 April 2017, 23:07:41 UTC |
| d02eb51 | brharrington | 09 April 2017, 15:45:54 UTC | support merging rollup blocks (#533) With inline rollups the the blocks from a single mirror can contain a rollup. For now it follows a simple strategy of just merging the individual blocks for the rollup. | 09 April 2017, 15:45:54 UTC |
| fee4182 | brharrington | 05 April 2017, 18:00:41 UTC | fix prefix for cloudwatch sqs metrics (#532) It was incorrectly specified as `s3`. | 05 April 2017, 18:00:41 UTC |
| d34e90d | brharrington | 31 March 2017, 06:29:46 UTC | store the interpreted expression in the Request (#531) For some use-cases we access `GraphApi.Request.exprs` many times, there is no reason to interpret it each time. | 31 March 2017, 06:29:46 UTC |
| 30fbbf0 | brharrington | 30 March 2017, 17:47:12 UTC | add support for listing all expressions (#530) Modifies the `/lwc/api/v1/expressions` endpoint to support listing all expressions. Before it would only list out expresssions for a particular cluster. | 30 March 2017, 17:47:12 UTC |
| a2e8d20 | brharrington | 30 March 2017, 14:49:50 UTC | fix warnings for lwcapi (#529) Cleanup minor warnings such as unused references in lwcapi. Also simplifies to get rid of separate impl classes. The only api considered public for this module is the interaction with HTTP interface. The separate classes were just extra boilerplate. | 30 March 2017, 14:49:50 UTC |
| 4bb0ecf | brharrington | 27 March 2017, 15:51:39 UTC | add `atlas.rollup` to allowed keys (#528) Updates the default allowed keys list for `atlas.` prefix to include `atlas.rollup`. | 27 March 2017, 15:51:39 UTC |
| 1995eb4 | brharrington | 26 March 2017, 03:36:22 UTC | add additional build metadata to jar (#527) Add the build date and number to the jar manifest to aid in debugging. | 26 March 2017, 03:36:22 UTC |
| 57516f5 | brharrington | 25 March 2017, 20:16:39 UTC | improve performance of SmallHashMap.{+,-} (#526) Changes the small hash map implementation of `+` and `-` to retain the type. This is primarily to avoid using a less memory efficient map type if the map is modified somewhere along the way. A side benefit is better performance and reduced allocations. **Before** ``` Benchmark Mode Cnt Score Error Units addPair thrpt 10 781344.330 ± 25173.790 ops/s removePair thrpt 10 738796.252 ± 44849.997 ops/s ``` **After** ``` Benchmark Mode Cnt Score Error Units addPair thrpt 10 1637470.772 ± 29948.315 ops/s removePair thrpt 10 2520939.662 ± 32320.759 ops/s ``` | 25 March 2017, 20:16:39 UTC |
| c68cda0 | brharrington | 24 March 2017, 21:32:42 UTC | support inline rollup of datapoints (#525) This change allows datapoints to be rolled up as they are flowing into the in memory storage. Typically this has been done later as part of offline processing using EMR or via a separate aggregator cluster that would perform the aggregations prior to being sent to the in memory storage. To be eligible for the rollup the data must have already been normalized to the step boundary and rate conversion for counters should have already been performed. The inline rollup will create a rollup block so it will have more overhead in terms of memory use unless the reduction multiple is over 4. Data will only go through the rollup path if it is tagged with `atlas.rollup=true`. The publish payload can now take an `id` field which will get used to dedup messages. If used, then it should avoid over counting if retries are present in the publishing pipeline as long as the updates for a given datapoint are always mapped to the same id for a given interval. The message id should not be used for multiple intervals or subsequent messages would incorrectly get lost. | 24 March 2017, 21:32:42 UTC |
| 51eb66f | brharrington | 23 March 2017, 16:50:23 UTC | fix poller client to discard entity bytes (#524) The poller client was not properly discarding the entity bytes for success or unexpected status codes. | 23 March 2017, 16:50:23 UTC |
| d97cf2d | brharrington | 18 March 2017, 20:50:01 UTC | update dependencies (#523) | 18 March 2017, 20:50:01 UTC |
| 3c0eca7 | Brian Harrington | 18 March 2017, 20:09:26 UTC | remove unused spray dependencies | 18 March 2017, 20:09:26 UTC |
| 828db9e | Brian Harrington | 18 March 2017, 20:08:20 UTC | sbt-jmh 0.2.24 | 18 March 2017, 20:08:20 UTC |
