Commit Graph

106 Commits

Author SHA1 Message Date
Richard Patel d3910780d5 compactindex: add initial format and query
- Adds compactindex file format
- Adds memory binary search query strategy
2022-11-12 23:06:09 +00:00
Richard Patel 928fefea9d ipld: simplify entry type 2022-10-31 11:36:15 +01:00
Richard Patel 180578ea85 car: add sanity checks to tests 2022-10-31 11:35:43 +01:00
Richard Patel 2742cd8867 fix Go tests
- merkletree: fix broken test case
- fixtures: fix compatibility with Bazel
2022-10-30 08:19:30 +01:00
Richard Patel 4ddc64e2c0 bazel: add Go rules 2022-10-30 08:17:36 +01:00
Richard Patel bcf9eb553b genesis: fix test 2022-10-29 20:18:46 +02:00
Richard Patel 582a566dc5 blockstore: don't depend on exp/constraints
Gazelle hates exp/constraints for some inexplicable reason.

> build aborted: no such package '@org_golang_x_exp//constraints': BUILD file not found in directory 'constraints' of external repository @org_golang_x_exp.
2022-10-29 20:18:46 +02:00
Richard Patel fb579ba6fb cargen: add unit test
- Adds unit test running cargen over a fake blockstore.
- Covers the CAR splitting code path.
- Ensures resulting CARs decode and contain expected CIDs.
2022-10-29 14:35:09 +02:00
Richard Patel 56a7065d79 ipldgen: fix shredding serialization
Fixes regression in d4c1dfe1
2022-10-29 14:30:12 +02:00
Richard Patel a6f33dd77d cargen: bug fixes
- Fix last CAR not being flushed
- Fix wrong epoch number in file name
- Print error message in edge case where block exceeds CAR length (splitting fails)
2022-10-29 14:24:57 +02:00
Richard Patel 2f5954278c blockstore: simplify BlockWalkI.Entries()
Shredding is already implied by SlotMeta,
thus no need to return shreds and entries in the iterator.
2022-10-29 13:50:25 +02:00
Richard Patel 776c1e88f1 sort Go imports 2022-10-29 13:30:26 +02:00
Richard Patel b8c9810620 cargen: default to 32GB sectors, make it configurable 2022-10-29 13:29:19 +02:00
Richard Patel aafccecfcc car: add parsing unit test 2022-10-29 13:22:03 +02:00
Richard Patel 1f38306312 car: fix leb128 length estimation 2022-10-29 12:18:26 +02:00
Richard Patel d4c1dfe18c ipld: optimize entry schema
- use tuple CBOR representation
- use efficient entry/shred mapping
- remove RadianceLedger placeholder multicodec type
2022-10-28 20:51:11 +00:00
Richard Patel 9a647f9aee cargen: use blockstore abstraction
Uses the blockstore.BlockWalkI abstraction introduced in 4644c3f
to decouple cargen from RocksDB.
2022-10-28 20:21:32 +00:00
Richard Patel 4644c3fa8e blockstore: split RocksDB-specific code into _rocks.go files
Splitting pure Go and Cgo code into separate Go files
allows other packages to depend on blockstore mocks.

This is useful for writing test scenarios for external
components that consume data from blockstore.
2022-10-28 18:31:15 +00:00
Richard Patel a739dd5879 poh: add mainnet block 0 and 1 tests 2022-10-20 01:02:47 +00:00
Richard Patel dc8246dcfd poh: move package from runtime/poh to poh, add tests 2022-10-20 01:02:47 +00:00
Richard Patel 5d7f351484 replay: mainnet epoch 0 PoH 🏁
- Implements transaction mixin
- Replays fine for a few thousand slots so far
2022-10-19 01:44:30 +00:00
Richard Patel 53bbce1304 add merkletree package
Ports the construction algorithm of solana-merkle-tree.
2022-10-18 23:23:09 +00:00
Richard Patel 14594ded1b
replay: mainnet block 0 🏁 2022-10-18 16:48:39 -05:00
Richard Patel 0eeb85b874 add runtime package 2022-10-16 04:10:12 +00:00
Richard Patel 8278a5dc00
cargen: open as O_RDWR
Copying files between CARs needs read/write-file mode.
Embarrassing bug. Adding unit tests next time.
2022-09-28 23:01:56 +02:00
Richard Patel f9acf4f518
cargen: fix cache bug while splitting CARs
When splitHandle moves data from old to new CAR,
it fails to flush the write cache of the old car,
and seeks past the end of the file.

Regression introduced by a319031b
2022-09-26 16:44:19 +02:00
Leo 55c6b113af Make rocksdb optional and document build 2022-09-26 14:07:11 +02:00
Leo 0d5dd26722 radiance: optional build tag for rocksdb commands
Compiling the rocksdb binding isn't as straight-forward
as installing rocksdb headers:

$ go build ./...
# github.com/linxGnu/grocksdb
../../go/pkg/mod/github.com/linx!gnu/grocksdb@v1.7.7/cache.go:84:2: could not determine kind of name for C.rocksdb_lru_cache_options_set_num_shard_bits

Adding an optional build tag to compile radiance with rocksdb
support, otherwise remove the blockstore command. This keeps
everything else working with CGO_ENABLED=0.
2022-09-26 13:29:59 +02:00
Richard Patel a319031bd8
cargen: use buffered writer (+120% perf) 2022-09-26 00:41:26 +02:00
Richard Patel bb94b7e6c0
blockstore: let BlockWalk iterate over rooted slots only 2022-09-25 15:43:57 +02:00
Richard Patel 2d8b42f33a
cargen: improve godoc 2022-09-25 15:35:04 +02:00
Richard Patel 473624867a
cargen: implement CAR file splitting 2022-09-25 15:35:03 +02:00
Richard Patel 5fb1dfa576
car: refactor into cargen 2022-09-25 15:35:03 +02:00
Richard Patel 6b5feeba9a
ipldgen: add IPLD ledger stream generator 2022-09-25 15:35:03 +02:00
Richard Patel d3b2c3e9a5
ipldsch: add ledger IPLD schema 2022-09-25 14:25:03 +02:00
Richard Patel 758ede5690 Move Go module to go.firedancer.io/radiance
Change-Id: I76845d19e864d6c7cd0fff77dbe63fc722bbf808
Reviewed-on: https://git.firedancer.io/c/radiance/+/131
2022-09-21 23:40:06 +00:00
gagliardetto f6a9e60f31
shred: improve DataShredsToEntries performance
* Don't use reflection for shred decoding
* Add shred decoding benchmark
2022-09-19 13:46:29 +02:00
Richard Patel bb6d4ce6f6
shred: skip empty shreds
Encountered in mainnet slot 102763321 data shred index 484
2022-09-18 01:49:01 +02:00
Richard Patel 0f7098a795 shred: support recent legacy shred
Some time along Solana mainnet-beta history,
the legacy data shred header was extended
from 86 bytes to 88 bytes.

We don't know when exactly this change has happened.
Or whether the shred version is self-describing.

For now, add a simple switch by slot number.
2022-09-18 01:21:42 +02:00
Richard Patel 0585231e9f
blockstore: add genesis deshred test 2022-09-17 20:42:34 +02:00
Richard Patel 64f4cbec93
cmd/blockstore/verify-data: log txn and byte count 2022-09-17 20:26:16 +02:00
Richard Patel 047585136b
cmd/blockstore: add verify-data tool
Adds tool that scans over the ledger entries in a blockstore.
2022-09-17 18:51:24 +02:00
Richard Patel b1052fb69a
genesis: support genesis.bin
Adds handwritten serde for mainnet genesis config.
(Binary doesn't properly support bincode-fixedint-le)
2022-09-17 16:47:47 +02:00
Richard Patel fd64bc1d6a
runtime: add core ledger data structures
Adds structs that appear in the GenesisConfig object

Adds handwritten serde for runtime structs because
binary doesn't properly support bincode-fixedint-le
2022-09-17 16:46:26 +02:00
Richard Patel 452d388b5c
archiveutil: add OpenTar() for `.tar`, `.tar.gz`, `.tar.bz2`
Adds utility function that reads tar streams
with optional compression handling.
2022-09-17 16:39:47 +02:00
Pires ffd44286a8 pkg/util: Schroeder wants testify/assert
This will come in handy for sure very soon
2022-09-15 09:45:53 -05:00
Pires 6db3220d26 cmd/solrays: move IsValidHostname to pkg/util 2022-09-15 09:45:53 -05:00
Richard Patel d7cd878bc9 blockstore: accurately encode entry batches as YAML
Updates blockstore dump YAML schema to reflect that
slices of shreds map to slices of entries.

Adds a new "entry_batches" wrapper list that annotates
each slice of entries with shred range and encoded size.
2022-09-11 08:56:03 +02:00
Richard Patel 3c69cf0eea blockstore: decode mainnet genesis shreds 2022-09-10 19:56:59 +02:00
Richard Patel 62357507ff blockstore: add shreds, slot meta client 2022-09-10 14:30:49 +02:00