8135279335
Reduce serialize/deserialize in shred recovery ( #5887 )
2019-09-12 21:52:13 -07:00
385086359c
Reduce serializations/deserializations of shreds ( #5879 )
2019-09-12 10:10:25 -07:00
7682db4826
Generate coding shreds on the fly based on erasure limits ( #5852 )
...
* Generate coding shreds on the fly based on erasure limits
* fix uncle
2019-09-09 17:26:51 -07:00
df205f8752
Use ureq instead of influx_db_client ( #5839 )
2019-09-07 12:48:45 -07:00
e8d88f3237
Split SDK's timing.rs ( #5823 )
2019-09-06 14:30:56 -07:00
3d3b03a123
Verify signature of recovered shred before adding them to blocktree ( #5811 )
...
* Verify signature of recovered shred before adding them to blocktree
* fix failing tests, and review comments
2019-09-05 18:20:30 -07:00
f78b865cba
Cleanup shreds to remove FirstShred data structure ( #5789 )
...
* Cleanup shreds to remove FirstShred data structure
* Also reduce size used by parent slot information in shred header
* clippy
* fixes
* fix chacha test
2019-09-04 21:06:47 -07:00
7062fe4b47
Refactor Blocktree for clarity and correctness ( #5700 )
...
* Refactor shreds to prevent insertion of any metadata on bad shreds
* Refactor fetching Index in blocktree
* Refactor get_slot_meta_entry
* Re-enable local cluster test
* cleanup
* Add tests for success/fail insertion of coding/data shreds
* Remove assert
* Fix and add tests for should_insert coding and data blobs
2019-09-04 17:14:42 -07:00
02ee2a601c
Further cleanup of blocktree after Blob deprecation ( #5780 )
2019-09-04 12:47:09 -07:00
3b0d48e3b8
Remove blocktree blob references ( #5691 )
...
* Remove blocktree blob references
* fixes and cleanup
* replace uninitialized() call with MaybeUninit
* fix bench
2019-09-03 21:32:51 -07:00
8adac30c05
Integrate shreds to the replicators ( #5711 )
...
* Integrate shreds to the replicators
* fix cuda stuff
* fix cuda tests
2019-08-28 22:34:47 -07:00
47535b9ff1
Use serialize_into to fill in shreds instead of writing one byte at a time ( #5695 )
...
automerge
2019-08-27 17:11:24 -07:00
8b9c3a2561
Blocktree last_root to enforce a slot floor ( #5593 )
...
* Add last_root to blocktree
* Don't repair earlier than last_root
* Add integration test to make sure blocktree floor is enforced
2019-08-27 15:09:41 -07:00
12ad95eb5e
Erasure statistics for shreds ( #5676 )
2019-08-27 11:22:06 -07:00
4ac1213c9c
Integrate coding shreds and recovery ( #5625 )
...
* Integrate coding shreds and recovery
* More tests for shreds and some fixes
* address review comments
* fixes to code shred generation
* unignore tests
* fixes to recovery
2019-08-26 18:27:45 -07:00
615da845cd
remove replicode in run_purge_batch() ( #5630 )
...
* remove replicode
* bigger timeout
* backout 90 min timeout
2019-08-26 09:47:48 -07:00
c18ea3ccc9
Fix ignored tests in blocktree ( #5591 )
2019-08-21 20:07:51 -07:00
faafee6b42
to to/the the ( #5590 )
2019-08-21 17:46:59 -07:00
0dc0594aaa
Fixes to repair and orphan logic for data shreds ( #5587 )
2019-08-21 15:27:42 -07:00
4798e7fa73
Integrate data shreds ( #5541 )
...
* Insert data shreds in blocktree and database
* Integrate data shreds with rest of the code base
* address review comments, and some clippy fixes
* Fixes to some tests
* more test fixes
* ignore some local cluster tests
* ignore replicator local cluster tests
2019-08-20 17:16:06 -07:00
b1d43ace14
Add columns for data and code shreds ( #5461 )
2019-08-12 10:03:57 -07:00
c78db6a94b
ledger path reform: use Path/PathBuf instead of strings ( #5344 )
2019-07-30 15:53:41 -07:00
535df0026d
Fixes for Blocktree space amplification and slot deletion ( #5266 )
...
* Fixes for Blocktree space amplification and slot deletion
2019-07-24 17:28:08 -07:00
a07b17b9b5
Drop older slots in the ledger ( #5188 )
...
* Add facility to delete blocktree columns in range
* Add ledger cleanup service
* Add local_cluster test
2019-07-20 13:13:55 -07:00
8b69998379
Lower recovery messages ( #5181 )
2019-07-19 09:20:14 -07:00
6ad9dc18d8
Add ability to prune ledger ( #5128 )
...
* Add utility to prune the ledger
* Add tests
* Fix clippy
* Fix off by one
* Rework to force delete every column
* Minor fixup
2019-07-17 14:42:29 -07:00
027ebb6670
no more OUT_DIR ( #5139 )
...
* no more OUT_DIR
* no more OUT_DIR
* more information about failure
2019-07-17 14:27:58 -07:00
0d27515d09
tmp dirs target to farf ( #5079 )
2019-07-12 18:28:42 -07:00
ebcdc06dc3
Restore ledger-tool print and json commands ( #5048 )
...
* Restore ledger-tool print and json commands
* Remove obsolete read_ledger()
2019-07-11 20:33:36 -07:00
4c90898f0b
Dynamic erasure set configuration ( #5018 )
...
* Use local erasure session to create/broadcast coding blobs
* Individual session for each recovery (as the config might be different)
* address review comments
* new constructors for session and coding generator
* unit test for dynamic erasure config
2019-07-11 13:58:33 -07:00
a383ea532f
Implement new Index Column ( #4827 )
...
* Implement new Index Column
* Correct slicing of blobs
* Mark coding blobs as coding when they're recovered
* Prevent broadcast stages from mixing coding and data blobs in blocktree
* Mark recovered blobs as present in the index
* Fix indexing error in recovery
* Fix broken tests, and some bug fixes
* increase min stack size for coverage runs
2019-07-10 11:08:17 -07:00
22ef3c7c54
Blob verify ( #4951 )
...
* Ensure signable data is not out of range
* Add a broadcast stage that puts bad sizes in blobs
* Resign blob after modifyign size
* Remove assertions that fail when size != meta.size
2019-07-08 18:21:52 -07:00
2aac094f63
Ensure blobs are deserializable without unwrapping ( #4948 )
...
* Return result from deserializing blobs in blocktree instead of assuming deserialization will succeed
* Mark bad deserialization as dead fork
* Add test for corrupted blobs in blocktree and replay_stage
2019-07-07 14:37:12 -07:00
36c9e22e3d
Revert "Dynamic erasure ( #4653 )"
...
This reverts commit ada4d16c4c
2019-06-20 20:53:03 -07:00
ada4d16c4c
Dynamic erasure ( #4653 )
...
Remove erasure-related constants
Remove unneeded `Iterator::collect` call
Document erasure module
Randomize coding blobs used for repair
2019-06-20 20:27:41 -05:00
aacb38864c
Mark dead forks in replay stage ( #4715 )
...
* Add DeadSlots column family
* Filter dead forks from get_slots_since
* Mark erroring slots as dead in replay stage, add test
* Mark dead forks in progress instead of removing them
* Fix logging process_entries failures in replay_stage
* Unignore test_fail_entry_verification_leader
2019-06-20 15:50:41 -07:00
182096dc1a
Create bank snapshots ( #4244 )
...
* Revert "Revert "Create bank snapshots (#3671 )" (#4243 )"
This reverts commit 81fa69d347
2019-05-30 21:31:35 -07:00
6ed071c4dd
Fix storage stage operating on empty slots ( #4474 )
...
* Fix storage stage operating on empty slots
* Reduce fn argument count
* Fix tests
2019-05-29 15:01:20 -07:00
534244b322
Fix set_roots to use cached bank parents instead of searching blocktree ( #4466 )
2019-05-29 09:43:22 -07:00
591fd72e0b
Implement listener for serving repairs through Repairman protocol ( #4306 )
...
* Make listener for serving repairs through Repairman protocol
2019-05-23 03:10:16 -07:00
1a77486f8e
Make RootedSlotsIterator for traversing slots on the root fork ( #4361 )
2019-05-20 23:09:00 -07:00
f1e5edee14
Modify Roots Column To Support Multiple Roots ( #4321 )
...
* Fix 1) Roots column family to handle storing multiple slots, 2) Store all slots on the rooted path in the roots column family
2019-05-20 19:04:18 -07:00
9476fe5ce3
Use log levels for datapoint metrics ( #4335 )
...
* Use log levels for datapoint metrics
* address review comments
* fix cyclomatic complexity
2019-05-17 17:34:05 -07:00
e920191de0
Rate limit metrics per log level ( #4313 )
...
* Rate limit metrics per log level
* fix tests
2019-05-16 22:27:05 -07:00
abd7f6b090
Change slot_meta_iterator() to return an iterator not a cursor ( #4303 )
...
automerge
2019-05-15 18:28:23 -07:00
a79fbbafc9
SlotMeta is incorrectly updated on insertion of erasure blobs ( #4289 )
...
* Fix put_coding_blob_bytes to properly update slotmetas and chaining
2019-05-15 00:28:31 -07:00
e20a8329d3
Add API to iterate over slot's blobs ( #4276 )
2019-05-13 22:04:54 -07:00
796000e96f
Improve erasure metrics ( #4268 )
...
* Improve erasure metrics
* Simplify metrics submission
2019-05-13 16:04:43 -05:00
f567877d1d
Cleanup metrics ( #4230 )
2019-05-10 08:33:58 -07:00
9881820444
RepairService saves db_iterator instead of reconstructing on every search ( #4242 )
2019-05-09 19:57:51 -07:00
Pankaj Garg
Michael Vines
Jack May
carllin
Rob Walker
Sagar Dhawan
sakridge
Mark E. Sinclair
Sathish