https://github.com/twitter-archive/kestrel

sort by:
Revision Author Date Message Commit Date
cc280ec Merge pull request #150 from ReadmeCritic/master Update README URLs based on HTTP redirects 22 January 2016, 07:54:28 UTC
47b8842 Merge pull request #147 from alex-hofsteede/patch-1 s/itmes/items/ 22 January 2016, 07:53:24 UTC
4340534 Update README.md 22 January 2016, 07:47:46 UTC
2cce69e Update README.md 22 January 2016, 07:44:00 UTC
57192d4 Update README URLs based on HTTP redirects 19 November 2015, 15:44:08 UTC
5279aeb Fix bad Future.sleep substitute 21 September 2015, 03:50:18 UTC
1d3ad64 Update resolvers, specify SBT version, and add status note 08 September 2015, 19:39:35 UTC
c4fabda Revert to finagle 6.4, fix build 07 September 2015, 22:21:21 UTC
30128e4 s/itmes/items/ typo in docs 25 March 2015, 21:15:38 UTC
f32e7f3 Remove PeriodicSyncFile 10 October 2014, 18:29:59 UTC
8ee3dd0 - Kestrel: replace mutable.Queue with java.util.LinkedList for gc reasons - Fix linkedhashmap by using java.util - Kestrel-core changes for durability - Kestrel: dont fsync on a closed writer - Add sampling stats to persistent queue - Kestrel: Close Reader during replay - After replaying a file, close the reader - Local File reader should close the FileChannel as well as the FileInputStream. - Kestrel: Trace on NoSuchElementException - shutdown sequence fix fix - state change needs to be transient -> we want the last persisted state to be restored on restart - change gracefulshutdown bit for writeavoid to false since we dont need to wait to transition to this state - Kestrel: LogEmptyException in length() Inline with the behavior of AppendOnlyStreamWriter/Reader, when a stream is empty, its length should be treated as zero. Currently this call may throw an exception. The behavior is not common as we never explicitly call .length as the first operation once a stream is created. This happens only as a race condition between background pack and rotate. - Kestrel: Change PersistentStream, PersistentStreamWriter, PersistentStreamReader to be traits This ensure that only implementations of PersistentStreamContainer and PersistentMetadataStore need to provide public constructors. This helps with exception handling and retries especially if state that spans the entire container has to be reset because of shared data structures. - Kestrel: Recover Metadata before accessing the directory The constructor of Journal calls a getStream(queueName). When PersistentQueue is used as a library, this call may precede any call to listStreams or listQueues which would therefore create the stream before the metadata has been recovered. In case of the Kestrel service though, QueueCollection first loads queues, this recovers metadata before any other operation has a chance to access the directory map. - Kestrel: Log client description on client errors - IndexOutofBoundsException indicates that the request was incorrectly formed. Translate that to a CLIENT_ERROR - Log the clientDescription when we encounter a client error - add a metric for total fsync time, count some viz changes - Kestrel: Support for whitelisting clientIds that are allowed to access queues The goal of this change is to allow the users to specify the client id that is allowed to enqueue to or dequeue from a queue. The purpose of the feature is to allow incorrectly deployed code or config from enqueing to or dequeing from queues in production. This is *NOT* intended as a security feature. - Kestrel: Refactor ServerStatus so that the storage can be made pluggable - Introduce PersistentMetadataStore that abstracts the storage of the server status. - Provide an implementation (LocalMetadataStore) that stores the persistent metadata in a local file (similar to current implementation) - Kestrel: Gauge for max items and max size Both based on configuration parameters - Kestrel: Change to Rewrite sequence and tests The change includes i) Split the delete and rename as two separate steps in the rewrite sequence. ii) Add fail point based tests to verify rewrite behavior in the presence of failures - Kestrel: Remove Deprecated methods from tests Remove Deprecated methods on util.Future and use Await.result instead - helps eliminate ~100 warnings that are produced in the build - Remove waiters on monitorUntil in KestrelHandler and not just those from get() - Use eventually on kestrel flaky handoff test - Kestrel: Journal Replay Time This is particularly useful in debugging slow restarts - or misconfigured queues (incorrect journal parameters). I did not add a metric as replay only runs once per restart and hence there wont be enough samples to get any meaningful statistics - Kestrel: Refactor Journal Storage Refactor the Journal Storage as follows a) PersistentStreamContainer - the abstraction that manages all the persistent streams associated with Kestrel queue(s). b) PersistentStream - an instance of an append only durable stream, each Kestrel queue's persistent state is represented as one or more persistent streams c) PersistentStreamReader/Writer - interfaces for reading from and writing to the PersistentStream. Instead of supporting all the semantics of a generic InputStream/OutputStream, these interfaces only provide the methods that are required by Kestrel - Kestrel: Improve Discard by minimizing calls to FillReadBehind For large queues that are in read behind - discarding expired items need not issue fillReadBehind for every item expired. Reducing the calls to the case where the queue in memory is empty an once after the current discard loop has been completed, should improve efficiency. - kestrel : bump util/finagle version to 6.x version, since 5.x and 6.x has different interfaces on Future definition. - Fix leaking transactions on kestrel The future could be canceled before the addPendingRead() is executed. This would result in abortAnyOpenRead() not being aware of the newly added value which would linger around as an open transaction. - Kestrel: Implement Open Transaction Timeout In Kestrel. - Kestrel: Some cleanup based on canary testing Cleanup the graceful shutdown logic and update the rewrite message so we can differentiate rewrites coming from clean shutdown - Kestrel Canary: Turn off the new behavior of aggressive rewrites The new logic to disableAggressiveRewrites should be off by default so that we can canary Zoo - Bound number of items to be expired at add/remove/peek. Allow PersistentQueue.discardExpired() to have a configurable limit of items to be discarded Currently, add/remove/peek will cause an unbounded number of items to expire. This is bad. It should bound number of items to be expired by config.maxExpireSweep instead. 10 October 2014, 18:03:59 UTC
cfac185 Kestrel 2.4.7 10 October 2014, 18:02:42 UTC
ac8a224 Revert update to Finagle 6+ until dependencies can be advanced to the compatible versions naggati still uses Util version of 5.x, this causes memcache clients to behave incorrectly. I am reverting the change to update the finagle (and util) version until I can update naggati. I will move this to birdcage manually once the repo is ready in birdcage. RB_ID=141014 19 April 2013, 00:46:46 UTC
a4decc5 Update Kestrel to Finagle 6 and friends (Util and Servo) The changes include - Using the setInterruptHandler on Promise - Getting rid of the onCancellation call on Future - handling the Cancelled exception instead - Stop overriding finagle.Service.release () use close instead. RB_ID=129914 11 April 2013, 22:46:19 UTC
074fb67 Additional Tests for aggressive rewrites Tests for exercising the following scenarios - Enabling aggressive rewrites - Ensure that we don't stop rewriting if the journal space is freeing up - Ensure that rewrites are delayed for compact delay if rewrite resulted in insufficient compaction RB_ID=137648 04 April 2013, 19:20:06 UTC
294fd76 Additional tests for graceful shutdown Adding tests for graceful shutdown - with non empty queue - open transactions RB_ID=137623 04 April 2013, 18:04:42 UTC
c4e4eca Rewrite journals on graceful shutdown When we take a machine out of service, we first set it readonly to drain the queues. In order to minimize the startup time, we should also attempt to rewrite journals for empty queues. In general we want to keep shutdown very light weight so this graceful shutdown option is limited to the case when Kestrel is in readonly and quiescent state (states used for planned outages) Added a test to exercise the rewrite RB_ID=137141 03 April 2013, 22:03:40 UTC
71ede57 Disable Aggressive Rewrites by default as we prepare to canary RB_ID=137154 03 April 2013, 21:18:46 UTC
453b755 Tests the replay fixes for remove tentative Added new test that verifies the replay fixes when remove tentative/confirm remove is used in place of unconditional removes. This also exercises the open transactions logic in the checkpoint creation and packing of journals. RB_ID=132406 17 March 2013, 14:05:13 UTC
ea5fee3 Kestrel Replay Fix Part IV: Concurrency between pack and rotate Pack and Rotations can happen concurrently. They both by definition deal with separate parts of the queue - pack for an old checkpoint while rotate will operate on the latest journal file. They both share the logic that does the clean-up of .pack files (called through calculateArchiveSize) created by the checkpoint. If the clean-up runs concurrently from two separate threads, then we can have the following race between the two threads List of files before the pack journal.1 journal.2 journal.2.pack Thread 1 reads the set journal.1 and journal.2 for cleanup Thread 2 cleans up journal.1 and journal.2 and renames journal.2.pack to journal.2 Thread 1 uses the list it generates to delete the newly packed journal.2 => data loss This change addresses the synchronization Note: 1. Pack and Rewrite never run concurrently (rewrite *does not* apply to queues that are in read behind) 2. Rotate and Rewrite are never issued simultaneously so also they are synchronized by virtue of them being inline in the queue operations which synchronize in persistent queue implementation (only one operation at a time per queue) RB_ID=132415 15 March 2013, 22:42:54 UTC
3d04f6a Fix ThriftHandlerSpec test by avoiding mocking of InetSocketAddress RB_ID=132616 15 March 2013, 17:45:25 UTC
20488ba KEST-402: Kestrel replay fix Part III: Recovering journal contents after pack - Provide a method that journal replay can use to keep the removesSinceReadBehind count in sync. - Update the count when replaying remove/confirm-remove/continue journal records (items) if the replaying queue is in read-behind. - Add a test that exercises the scenario by ensuring that repeated restarts correctly restore the count of items RB_ID=130998 14 March 2013, 20:45:51 UTC
0a2f3ee Update foreground.sh script to handle mesos launch. 13 March 2013, 18:38:28 UTC
e7b1258 KEST-402: Kestrel replay fix - Part II Part II: Transactional Add (Continue) should update removesSinceReadBehind only if the queue is in read behind. While this logic doesn't cause incorrect behavior as startReadBehind resets removesSinceReadBehind unconditionally, this makes it hard to reason and verify the value of the counter. RB_ID=130997 12 March 2013, 00:44:49 UTC
3262960 Remove the reverse lookup in the log message Use getHostString instead of getHostName RB_ID=130001 08 March 2013, 02:10:05 UTC
606786f Kestrel replay fix Kestrel replay fix - Part 1: Problem: The pack request is triggered by the fillReadBehind where as the journal.remove updates removesSinceReadBehind after the fillReadBehind call. This ordering leaves two windows in which either we will lose an item or deliver an extra (potentially dummy/invalid item). This change addresses the problem as follows 1. In the normal mode - fillReadBehind is called after the remove has already been logged in the journal - this makes sure that the removesSinceReadBehind count has already been updated to reflect the fact that the current item has been removed from the in-memory queue which is used as the source of the pack. The same is done for the discard/expiration workflows 2. In the replay mode - Since journal removes are already logged (or are not being newly logged) the fillReadBehind is done as before. 3. The pack logic calls a fsync on the journal file before the packed file replaces the old journal files on disk to ensure that the removes that were accounted for in the pack have been indeed logged in the journal. 4. The change to PersistentQueue#setup is just for the tests This doesn't fully address KEST-402. However this is necessary for the second part of the fix. RB_ID=121133 28 February 2013, 07:31:27 UTC
96b6719 Change the Journal Rewrite/Rotation logic The condition that aggressively rewrites the journal when the queue is empty is expected to be a cheap operation that would shrink the journal to a very small size. When there are a large number of open transactions, even after the rewrite the journal size may not shrink. Since checkRotateJournal is called on each get/set, it effectively means we will rewrite the journal on each get and set without ever shrinking the journal size. This repeated rewrites affect latency of writes to the queue (as journal rewrite is synchronized) and also affects the I/O to the journal disk which can affect other queues on the same machine. Once in the past we have tried to remove this logic and resulted in not compacting the journal when it should have been compacted - so I don't want to completely remove, just reduce the frequency so we don't have repeated re-writes. Here the logic is changed as follows: 1. When the queue is empty, if the journal rewrite resulted in the journal size being larger than half the default size (what triggered the rewrite), this indicates that the number of open transactions is large, so we delay subsequent compaction by the compaction delay interval - this should provide an opportunity for the open transactions to drain and subsequent rewrite to successfully shrink the journal 2. Before attempting to rotate the journal, if the queue is empty first attempt to rewrite. If the journal still does not shrink, then proceed with the rotate. RB_ID=116056 28 February 2013, 07:05:50 UTC
acb6d56 Merge branch 'master' of https://git.twitter.biz/kestrel into kestrel-pidfile 16 January 2013, 00:35:40 UTC
32e0d33 'release commit for net.lag:kestrel:2.4.3-SNAPSHOT' 10 January 2013, 23:15:02 UTC
ba94b2d 'release commit for net.lag:kestrel:2.4.2' 10 January 2013, 23:14:51 UTC
65bdfc0 Change Log for the 2.4.2 Release 10 January 2013, 22:23:29 UTC
6548b22 Tracing operation on individual queues This change allows tracing operations to an individual queue and identify misbehaving clients. By limiting the logging to an individual queue, we can track down queue backups or leaking transactions on a single queue without affecting other queues on the instance of Kestrel. Also the trace out put produced per queue will be much more manageable compared to enabling this for all queues on an instance. The main motivation for this is to track down the open transactions leak. RB_ID=116677 08 January 2013, 17:06:13 UTC
5f506a2 Trace client connections and session lifetime - Since the queue that a session will operate on is only determined after the session has been established, its cleaner to enable session trace separately through Kestrel Config - This will help us determine all the clients connecting to a machine when the number of connections are high. Also it helps track down the cases when the session did not clean-up upon termination There are clusters that are seeing a large connection churn so I did not want to enable this by default - as it may bloat the log for no reason. Since config deployment doesnt require a Kestrel restart, enabling and disabling this for the duration of the investigation makes more sense. RB_ID=116866 08 January 2013, 08:40:43 UTC
7b59b3f Merge branch 'master' of https://git.twitter.biz/kestrel into kestrel-pidfile 03 January 2013, 19:11:51 UTC
d20cc9c Add Journal Rotation Metric - Stat counter for number of times this journal has been rewritten - Separate stat counter for the number of times this journal has been rotated and a checkpoint was optionally generated RB_ID=116055 02 January 2013, 22:03:26 UTC
8319bd5 Half of KEST-223 Remove hooks for legacy kestrel-daemon.pid file. 21 December 2012, 19:35:42 UTC
76992d7 Revert "P=cwatson" This reverts commit 45eb4f2659864f514abc358b9bf08a5ac81f1f5a. 21 December 2012, 19:29:10 UTC
45eb4f2 P=cwatson G=kestrel Modified to remove legacy hooks for kestrel-daemon.pid file. Half of KEST-223 21 December 2012, 19:13:09 UTC
dccb65b KEST-361: Add status command to kcluster for getting and setting status on servers RB_ID=101136 05 December 2012, 17:09:40 UTC
63ba805 Bump finagle version to 5.3.23 RB_ID=100909 04 December 2012, 21:48:37 UTC
b17ebe2 Merge branch 'master' into KEST-239-more-mesos 04 December 2012, 20:54:36 UTC
076a76e Remove redirect of stdout and stderr. Mesos already handles this and being in the foreground implies that such things are the invoker's problem. RB_ID=100842 04 December 2012, 20:54:22 UTC
857263e Fix mkdir in foreground. 30 November 2012, 22:36:57 UTC
0889d1f Merge branch 'master' into KEST-239-more-mesos 30 November 2012, 17:48:12 UTC
379ed61 reinstate journal rewriting under old conditions; but limit the frequency RB_ID=98094 19 November 2012, 16:59:41 UTC
342628d ksniff: payload percentile size and reprocess capture file RB_ID=97943 19 November 2012, 16:58:13 UTC
56ce9f5 Fix comment atop foreground script. 15 November 2012, 18:14:41 UTC
69bf7b4 ksniff: report time span over which packets were collected RB_ID=95762 07 November 2012, 23:14:46 UTC
975fd1c switch to form of markdown supported by site publishing tools 02 November 2012, 23:39:13 UTC
227e0f0 'release commit for net.lag:kestrel:2.4.2-SNAPSHOT' 02 November 2012, 15:54:55 UTC
ef6409f 'release commit for net.lag:kestrel:2.4.1' 02 November 2012, 15:54:48 UTC
eaf7fe7 update Changelog for 2.4.1 02 November 2012, 15:53:09 UTC
f3f5a4a never rewrite journals unless the queue is empty or in read-behind RB_ID=94977 02 November 2012, 15:47:25 UTC
34e37ae 'release commit for net.lag:kestrel:2.4.1-SNAPSHOT' 26 October 2012, 20:56:30 UTC
50eba65 'release commit for net.lag:kestrel:2.4.0' 26 October 2012, 20:56:23 UTC
88b204a removed loadtest; will consider re-publishing after iago matures RB_ID=93826 26 October 2012, 19:26:39 UTC
431c29e move gc log to a logs subdirectory. RB_ID=93786 26 October 2012, 18:13:30 UTC
bfb8217 return SERVER ERROR when availability blocks an operation RB_ID=93662 26 October 2012, 17:31:34 UTC
13cef6b Add foreground script for launching kestrel in a not-so-daemony way. Specifically for mesos. Note that the $@ in the final line is for passing in the -f option that lets us override the config file. RB_ID=93359 25 October 2012, 15:17:09 UTC
b942244 ksniff: introduce .ksniffrc and make flag options reversible RB_ID=93075 23 October 2012, 18:57:47 UTC
94bae3e better lookup of external IP address for ZK endpoints; don't advertise external IP when listening on loopback; docs RB_ID=92556 22 October 2012, 19:31:30 UTC
ce02205 Merge branch 'github_master' 18 October 2012, 18:04:02 UTC
892e008 bump to finagle-5.3.19; ostrich 8.2.9 RB_ID=92238 18 October 2012, 17:50:20 UTC
dfddc3f Clean up commented out line. 11 October 2012, 16:53:06 UTC
5b3fdaa Use kill -0 to detect a running process rather than relying on Ostrich HTTP checking. RB_ID=90376 11 October 2012, 16:52:33 UTC
defea76 fix shutdown bugs and add DeadlineWaitQueueSpec Kestrel now evicts connections waiting on PersistentQueue operations (akak waiters) to allow finagle to drain connections. Provides explicit start/shutdown for journal packer thread to prevent stray shutdown log messages. RB_ID=89058 01 October 2012, 15:55:05 UTC
59e3d33 modernize kestrel libraries RB_ID=88730 27 September 2012, 18:44:49 UTC
962c65c Version bump. 24 September 2012, 16:56:53 UTC
d421715 Add support for managed/unmanaged flag. Refs KEST-271 RB_ID=86284 24 September 2012, 16:56:24 UTC
9ffcd5c minor guide edit 24 September 2012, 16:43:59 UTC
fcabfac fix angle brackets 24 September 2012, 16:38:53 UTC
173e5b0 Merge branch 'github_master' Conflicts: loadtest/project/Build.scala project/Build.scala 24 September 2012, 16:26:32 UTC
6f74c3e missed one item in changelog 24 September 2012, 16:03:03 UTC
7d4f153 'release commit for net.lag:kestrel:2.3.5-SNAPSHOT' 21 September 2012, 23:05:56 UTC
09ec415 'release commit for net.lag:kestrel:2.3.4' 21 September 2012, 23:05:51 UTC
9548817 update iago to latest version 21 September 2012, 23:01:49 UTC
b5e007d Merge branch 'release-2.3.4' 21 September 2012, 21:41:08 UTC
bc93c87 update changelog 21 September 2012, 21:40:46 UTC
38d71d0 Merge branch 'github_master' into release-2.3.4 21 September 2012, 21:34:40 UTC
f935206 Merge branch 'release-2.3.4' Conflicts: ChangeLog project/Build.scala src/main/scala/net/lag/kestrel/KestrelHandler.scala src/main/scala/net/lag/kestrel/config/KestrelConfig.scala src/test/scala/net/lag/kestrel/ThriftHandlerSpec.scala 21 September 2012, 21:01:55 UTC
3e1c887 KEST-71: log creation/deletion/flush of queues with client description RB_ID=87714 21 September 2012, 20:19:13 UTC
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload RB_ID=87497 21 September 2012, 16:12:57 UTC
ea9d4da Merge branch 'release-2.3.3' into release-2.3.4 Conflicts: project/Build.scala 20 September 2012, 18:59:44 UTC
939af21 Version number bump for kcluster. 17 September 2012, 18:15:01 UTC
7c22c7c Add -R (reverse, biggest first) and -c (count, only show X items) options to kcluster. RB_ID=86471 17 September 2012, 18:14:36 UTC
7413ed4 Fix kestrel zookeeper membership changes Never update status from DEAD to DEAD (which finagle-serverset sees as a live server). Avoid updating status when it has not changed. Changing form DEAD to ALIVE requires re-joining the server set. RB_ID=86301 13 September 2012, 20:46:32 UTC
4aec80a fix kestrel serverset startup RB_ID=85412 10 September 2012, 18:43:59 UTC
f3a47af 'release commit for net.lag:kestrel:2.3.4-SNAPSHOT' 07 September 2012, 18:47:29 UTC
f9269e0 'release commit for net.lag:kestrel:2.3.3' 07 September 2012, 18:47:23 UTC
eea6849 version to 2.3.3-snapshot 07 September 2012, 18:46:12 UTC
4379608 Use old pid file to remain compatible with 2.3.1. RB_ID=84949 07 September 2012, 18:43:53 UTC
2103bc9 Use old pid file to remain compatible with 2.3.1. RB_ID=84949 07 September 2012, 14:23:34 UTC
dad717e Merge branch 'github_master' Conflicts: src/test/scala/net/lag/kestrel/KestrelHandlerSpec.scala 06 September 2012, 22:12:07 UTC
f34342f remove ensime-sbt-cmd -- keep IDE junk external 06 September 2012, 22:08:34 UTC
d3dbb89 Merge pull request #110 from rvoicilas/master Cleanup ItemIdList test && ensime-sbt-cmd added 06 September 2012, 21:33:11 UTC
996bb04 Add more tests to ItemIdListSpec 06 September 2012, 21:12:22 UTC
9b12920 Got rid of doBefore and small code fixes 06 September 2012, 21:04:18 UTC
a45e2ce Move ItemIdListSpec out of KestrelHandlerSpec - have ItemIdListSpec in its own file - make tests more simple and readable 31 August 2012, 23:06:35 UTC
7a13809 Add ensime-sbt-cmd as a sbt plugin This automatically adds 'ensime generate' command to sbt (which will write an .ensime file to the project's root directory) 31 August 2012, 23:01:53 UTC
b39d0e4 update Changelog 29 August 2012, 22:38:35 UTC
back to top