zcashd/README.md

175 lines
8.0 KiB
Markdown
Raw Normal View History

**LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.**
Squashed 'src/leveldb/' changes from 20ca81f08..a31c8aa40 a31c8aa40 Add NewAppendableFile for win32 environment 1913d718e Merge upstream LevelDB 1.19 3080a45b6 Increase leveldb version to 1.19. fa6dc010a A zippy change broke test assumptions about the size of compressed output. Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472 06a191b8d fix problems in LevelDB's caching code a7bff697b Fix LevelDB build when asserts are enabled in release builds. (#367) ea992b467 Change std::uint64_t to uint64_t (#354) e84b5bdb5 This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split. 321134390 Deleted redundant null ptr check prior to delete. 7306ef856 Merge pull request #348 from randomascii/master 6b18316d0 Fix signed/unsigned mismatch on VC++ builds adbe3eb07 Putting build artifacts in subdirectory. 2d0320a45 Merge pull request #329 from ralphtheninja/travis-badge dd1c3c357 add travis build badge 43fcf23af Merge pull request #328 from cmumford/master 9fcae6164 Added a Travis CI build file. dac40d25f Merge pull request #284 from ideawu/master 8ec241a3b Merge pull request #317 from falvojr/patch-1 5d36bedd1 Merge pull request #272 from vapier/master 4753c9b61 Added a contributors section to README.md e2446d084 Merge pull request #275 from paulirish/patch-1 706b7f8d4 Resolve race when getting approximate-memory-usage property 3c9ff3c69 Only compiling TrimSpace on linux. f8d205cf8 Including atomic_pointer.h in port_posix 889de31a5 Let LevelDB use xcrun to determine Xcode.app path instead of using a hardcoded path. 528c2bc6a Add "approximate-memory-usage" property to leveldb::DB::GetProperty 359b6bcec Add leveldb::Cache::Prune 50e77a826 Fix size_t/int comparison/conversion issues in leveldb. 5208e7952 Added leveldb::Status::IsInvalidArgument() method. ce45404bb Suppress error reporting after seeking but before a valid First or Full record is encountered. b9afa1f2e include <assert> -> <cassert> edf2939c0 Update README.md 65190ac48 Will not reuse manifest if reuse_logs options is false. ac1d69da3 LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened. 76bba139c fix indent 8fcceb2a6 log compaction output file's level along with number 0e0f07417 documentation. improved link c85addcdf readme: improved documentation link ceff6f121 Fix Android/MIPS build. 77948e7ee Add benchmark that measures cost of repeatedly opening the database. 34ad72e3e Move header guard below copyright banner. a75d435d1 Clean up layering of storage/leveldb/... b234f65b3 Added a new fault injection test. c4c38f9c1 Add arm64 support to leveldb. cea9b10e5 Fixed incorrect comment wording for Iterator::Seek. c00c569f2 Deleted old README file. git-subtree-dir: src/leveldb git-subtree-split: a31c8aa408d5594830f7cb20ead1ef1dff51b79e
2020-09-22 16:04:47 -07:00
[![Build Status](https://travis-ci.org/google/leveldb.svg?branch=master)](https://travis-ci.org/google/leveldb)
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
# Features
* Keys and values are arbitrary byte arrays.
* Data is stored sorted by key.
* Callers can provide a custom comparison function to override the sort order.
* The basic operations are `Put(key,value)`, `Get(key)`, `Delete(key)`.
* Multiple changes can be made in one atomic batch.
* Users can create a transient snapshot to get a consistent view of data.
* Forward and backward iteration is supported over the data.
Squashed 'src/leveldb/' changes from 20ca81f08..a31c8aa40 a31c8aa40 Add NewAppendableFile for win32 environment 1913d718e Merge upstream LevelDB 1.19 3080a45b6 Increase leveldb version to 1.19. fa6dc010a A zippy change broke test assumptions about the size of compressed output. Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472 06a191b8d fix problems in LevelDB's caching code a7bff697b Fix LevelDB build when asserts are enabled in release builds. (#367) ea992b467 Change std::uint64_t to uint64_t (#354) e84b5bdb5 This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split. 321134390 Deleted redundant null ptr check prior to delete. 7306ef856 Merge pull request #348 from randomascii/master 6b18316d0 Fix signed/unsigned mismatch on VC++ builds adbe3eb07 Putting build artifacts in subdirectory. 2d0320a45 Merge pull request #329 from ralphtheninja/travis-badge dd1c3c357 add travis build badge 43fcf23af Merge pull request #328 from cmumford/master 9fcae6164 Added a Travis CI build file. dac40d25f Merge pull request #284 from ideawu/master 8ec241a3b Merge pull request #317 from falvojr/patch-1 5d36bedd1 Merge pull request #272 from vapier/master 4753c9b61 Added a contributors section to README.md e2446d084 Merge pull request #275 from paulirish/patch-1 706b7f8d4 Resolve race when getting approximate-memory-usage property 3c9ff3c69 Only compiling TrimSpace on linux. f8d205cf8 Including atomic_pointer.h in port_posix 889de31a5 Let LevelDB use xcrun to determine Xcode.app path instead of using a hardcoded path. 528c2bc6a Add "approximate-memory-usage" property to leveldb::DB::GetProperty 359b6bcec Add leveldb::Cache::Prune 50e77a826 Fix size_t/int comparison/conversion issues in leveldb. 5208e7952 Added leveldb::Status::IsInvalidArgument() method. ce45404bb Suppress error reporting after seeking but before a valid First or Full record is encountered. b9afa1f2e include <assert> -> <cassert> edf2939c0 Update README.md 65190ac48 Will not reuse manifest if reuse_logs options is false. ac1d69da3 LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened. 76bba139c fix indent 8fcceb2a6 log compaction output file's level along with number 0e0f07417 documentation. improved link c85addcdf readme: improved documentation link ceff6f121 Fix Android/MIPS build. 77948e7ee Add benchmark that measures cost of repeatedly opening the database. 34ad72e3e Move header guard below copyright banner. a75d435d1 Clean up layering of storage/leveldb/... b234f65b3 Added a new fault injection test. c4c38f9c1 Add arm64 support to leveldb. cea9b10e5 Fixed incorrect comment wording for Iterator::Seek. c00c569f2 Deleted old README file. git-subtree-dir: src/leveldb git-subtree-split: a31c8aa408d5594830f7cb20ead1ef1dff51b79e
2020-09-22 16:04:47 -07:00
* Data is automatically compressed using the [Snappy compression library](http://google.github.io/snappy/).
* External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.
Squashed 'src/leveldb/' changes from 20ca81f08..a31c8aa40 a31c8aa40 Add NewAppendableFile for win32 environment 1913d718e Merge upstream LevelDB 1.19 3080a45b6 Increase leveldb version to 1.19. fa6dc010a A zippy change broke test assumptions about the size of compressed output. Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472 06a191b8d fix problems in LevelDB's caching code a7bff697b Fix LevelDB build when asserts are enabled in release builds. (#367) ea992b467 Change std::uint64_t to uint64_t (#354) e84b5bdb5 This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split. 321134390 Deleted redundant null ptr check prior to delete. 7306ef856 Merge pull request #348 from randomascii/master 6b18316d0 Fix signed/unsigned mismatch on VC++ builds adbe3eb07 Putting build artifacts in subdirectory. 2d0320a45 Merge pull request #329 from ralphtheninja/travis-badge dd1c3c357 add travis build badge 43fcf23af Merge pull request #328 from cmumford/master 9fcae6164 Added a Travis CI build file. dac40d25f Merge pull request #284 from ideawu/master 8ec241a3b Merge pull request #317 from falvojr/patch-1 5d36bedd1 Merge pull request #272 from vapier/master 4753c9b61 Added a contributors section to README.md e2446d084 Merge pull request #275 from paulirish/patch-1 706b7f8d4 Resolve race when getting approximate-memory-usage property 3c9ff3c69 Only compiling TrimSpace on linux. f8d205cf8 Including atomic_pointer.h in port_posix 889de31a5 Let LevelDB use xcrun to determine Xcode.app path instead of using a hardcoded path. 528c2bc6a Add "approximate-memory-usage" property to leveldb::DB::GetProperty 359b6bcec Add leveldb::Cache::Prune 50e77a826 Fix size_t/int comparison/conversion issues in leveldb. 5208e7952 Added leveldb::Status::IsInvalidArgument() method. ce45404bb Suppress error reporting after seeking but before a valid First or Full record is encountered. b9afa1f2e include <assert> -> <cassert> edf2939c0 Update README.md 65190ac48 Will not reuse manifest if reuse_logs options is false. ac1d69da3 LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened. 76bba139c fix indent 8fcceb2a6 log compaction output file's level along with number 0e0f07417 documentation. improved link c85addcdf readme: improved documentation link ceff6f121 Fix Android/MIPS build. 77948e7ee Add benchmark that measures cost of repeatedly opening the database. 34ad72e3e Move header guard below copyright banner. a75d435d1 Clean up layering of storage/leveldb/... b234f65b3 Added a new fault injection test. c4c38f9c1 Add arm64 support to leveldb. cea9b10e5 Fixed incorrect comment wording for Iterator::Seek. c00c569f2 Deleted old README file. git-subtree-dir: src/leveldb git-subtree-split: a31c8aa408d5594830f7cb20ead1ef1dff51b79e
2020-09-22 16:04:47 -07:00
# Documentation
[LevelDB library documentation](https://github.com/google/leveldb/blob/master/doc/index.md) is online and bundled with the source code.
# Limitations
* This is not a SQL database. It does not have a relational data model, it does not support SQL queries, and it has no support for indexes.
* Only a single process (possibly multi-threaded) can access a particular database at a time.
* There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library.
Squashed 'src/leveldb/' changes from 20ca81f08..a31c8aa40 a31c8aa40 Add NewAppendableFile for win32 environment 1913d718e Merge upstream LevelDB 1.19 3080a45b6 Increase leveldb version to 1.19. fa6dc010a A zippy change broke test assumptions about the size of compressed output. Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472 06a191b8d fix problems in LevelDB's caching code a7bff697b Fix LevelDB build when asserts are enabled in release builds. (#367) ea992b467 Change std::uint64_t to uint64_t (#354) e84b5bdb5 This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split. 321134390 Deleted redundant null ptr check prior to delete. 7306ef856 Merge pull request #348 from randomascii/master 6b18316d0 Fix signed/unsigned mismatch on VC++ builds adbe3eb07 Putting build artifacts in subdirectory. 2d0320a45 Merge pull request #329 from ralphtheninja/travis-badge dd1c3c357 add travis build badge 43fcf23af Merge pull request #328 from cmumford/master 9fcae6164 Added a Travis CI build file. dac40d25f Merge pull request #284 from ideawu/master 8ec241a3b Merge pull request #317 from falvojr/patch-1 5d36bedd1 Merge pull request #272 from vapier/master 4753c9b61 Added a contributors section to README.md e2446d084 Merge pull request #275 from paulirish/patch-1 706b7f8d4 Resolve race when getting approximate-memory-usage property 3c9ff3c69 Only compiling TrimSpace on linux. f8d205cf8 Including atomic_pointer.h in port_posix 889de31a5 Let LevelDB use xcrun to determine Xcode.app path instead of using a hardcoded path. 528c2bc6a Add "approximate-memory-usage" property to leveldb::DB::GetProperty 359b6bcec Add leveldb::Cache::Prune 50e77a826 Fix size_t/int comparison/conversion issues in leveldb. 5208e7952 Added leveldb::Status::IsInvalidArgument() method. ce45404bb Suppress error reporting after seeking but before a valid First or Full record is encountered. b9afa1f2e include <assert> -> <cassert> edf2939c0 Update README.md 65190ac48 Will not reuse manifest if reuse_logs options is false. ac1d69da3 LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened. 76bba139c fix indent 8fcceb2a6 log compaction output file's level along with number 0e0f07417 documentation. improved link c85addcdf readme: improved documentation link ceff6f121 Fix Android/MIPS build. 77948e7ee Add benchmark that measures cost of repeatedly opening the database. 34ad72e3e Move header guard below copyright banner. a75d435d1 Clean up layering of storage/leveldb/... b234f65b3 Added a new fault injection test. c4c38f9c1 Add arm64 support to leveldb. cea9b10e5 Fixed incorrect comment wording for Iterator::Seek. c00c569f2 Deleted old README file. git-subtree-dir: src/leveldb git-subtree-split: a31c8aa408d5594830f7cb20ead1ef1dff51b79e
2020-09-22 16:04:47 -07:00
# Contributing to the leveldb Project
The leveldb project welcomes contributions. leveldb's primary goal is to be
a reliable and fast key/value store. Changes that are in line with the
features/limitations outlined above, and meet the requirements below,
will be considered.
Contribution requirements:
1. **POSIX only**. We _generally_ will only accept changes that are both
compiled, and tested on a POSIX platform - usually Linux. Very small
changes will sometimes be accepted, but consider that more of an
exception than the rule.
2. **Stable API**. We strive very hard to maintain a stable API. Changes that
require changes for projects using leveldb _might_ be rejected without
sufficient benefit to the project.
3. **Tests**: All changes must be accompanied by a new (or changed) test, or
a sufficient explanation as to why a new (or changed) test is not required.
## Submitting a Pull Request
Before any pull request will be accepted the author must first sign a
Contributor License Agreement (CLA) at https://cla.developers.google.com/.
In order to keep the commit timeline linear
[squash](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Squashing-Commits)
your changes down to a single commit and [rebase](https://git-scm.com/docs/git-rebase)
on google/leveldb/master. This keeps the commit timeline linear and more easily sync'ed
with the internal repository at Google. More information at GitHub's
[About Git rebase](https://help.github.com/articles/about-git-rebase/) page.
# Performance
Here is a performance report (with explanations) from the run of the
included db_bench program. The results are somewhat noisy, but should
be enough to get a ballpark performance estimate.
## Setup
We use a database with a million entries. Each entry has a 16 byte
key, and a 100 byte value. Values used by the benchmark compress to
about half their original size.
LevelDB: version 1.1
Date: Sun May 1 12:11:26 2011
CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache: 4096 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Raw Size: 110.6 MB (estimated)
File Size: 62.9 MB (estimated)
## Write performance
The "fill" benchmarks create a brand new database, in either
sequential, or random order. The "fillsync" benchmark flushes data
from the operating system to the disk after every operation; the other
write operations leave the data sitting in the operating system buffer
cache for a while. The "overwrite" benchmark does random writes that
update existing keys in the database.
fillseq : 1.765 micros/op; 62.7 MB/s
fillsync : 268.409 micros/op; 0.4 MB/s (10000 ops)
fillrandom : 2.460 micros/op; 45.0 MB/s
overwrite : 2.380 micros/op; 46.5 MB/s
Each "op" above corresponds to a write of a single key/value pair.
I.e., a random write benchmark goes at approximately 400,000 writes per second.
Each "fillsync" operation costs much less (0.3 millisecond)
than a disk seek (typically 10 milliseconds). We suspect that this is
because the hard disk itself is buffering the update in its memory and
responding before the data has been written to the platter. This may
or may not be safe based on whether or not the hard disk has enough
power to save its memory in the event of a power failure.
## Read performance
We list the performance of reading sequentially in both the forward
and reverse direction, and also the performance of a random lookup.
Note that the database created by the benchmark is quite small.
Therefore the report characterizes the performance of leveldb when the
working set fits in memory. The cost of reading a piece of data that
is not present in the operating system buffer cache will be dominated
by the one or two disk seeks needed to fetch the data from disk.
Write performance will be mostly unaffected by whether or not the
working set fits in memory.
readrandom : 16.677 micros/op; (approximately 60,000 reads per second)
readseq : 0.476 micros/op; 232.3 MB/s
readreverse : 0.724 micros/op; 152.9 MB/s
LevelDB compacts its underlying storage data in the background to
improve read performance. The results listed above were done
immediately after a lot of random writes. The results after
compactions (which are usually triggered automatically) are better.
readrandom : 11.602 micros/op; (approximately 85,000 reads per second)
readseq : 0.423 micros/op; 261.8 MB/s
readreverse : 0.663 micros/op; 166.9 MB/s
Some of the high cost of reads comes from repeated decompression of blocks
read from disk. If we supply enough cache to the leveldb so it can hold the
uncompressed blocks in memory, the read performance improves again:
readrandom : 9.775 micros/op; (approximately 100,000 reads per second before compaction)
readrandom : 5.215 micros/op; (approximately 190,000 reads per second after compaction)
## Repository contents
See [doc/index.md](doc/index.md) for more explanation. See
[doc/impl.md](doc/impl.md) for a brief overview of the implementation.
The public interface is in include/*.h. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Guide to header files:
* **include/db.h**: Main interface to the DB: Start here
* **include/options.h**: Control over the behavior of an entire database,
and also control over the behavior of individual reads and writes.
* **include/comparator.h**: Abstraction for user-specified comparison function.
If you want just bytewise comparison of keys, you can use the default
comparator, but clients can write their own comparator implementations if they
want custom ordering (e.g. to handle different character encodings, etc.)
* **include/iterator.h**: Interface for iterating over data. You can get
an iterator from a DB object.
* **include/write_batch.h**: Interface for atomically applying multiple
updates to a database.
* **include/slice.h**: A simple module for maintaining a pointer and a
length into some other byte array.
* **include/status.h**: Status is returned from many of the public interfaces
and is used to report success and various kinds of errors.
* **include/env.h**:
Abstraction of the OS environment. A posix implementation of this interface is
in util/env_posix.cc
* **include/table.h, include/table_builder.h**: Lower-level modules that most
clients probably won't use directly