Add deps.sh and redo install guide
Change-Id: I1d85acb3f4dc85dc88846fd331e07d15a712b35f
This commit is contained in:
Richard Patel
ripatel-jump
parent
382e723242
commit
b87a298f72
|
|
@ -19,6 +19,9 @@ OPTIONS-*
|
|||
# Nix
|
||||
/result
|
||||
|
||||
# deps.sh
|
||||
opt/
|
||||
|
||||
# Editor junk files
|
||||
# Please just use a global gitignore conf, I'm tired of this
|
||||
.idea
|
||||
|
|
|
|||
122
INSTALL.md
122
INSTALL.md
|
|
@ -1,18 +1,69 @@
|
|||
# Building Radiance
|
||||
|
||||
Radiance depends on many external dependencies including optional C libraries.
|
||||
This makes the build procedure slightly more complicated than a pure Go project.
|
||||
|
||||
For installing C deps, the following options are available, each with their own tradeoffs:
|
||||
- [Setup Script](#building-with-depssh) **(recommended)**
|
||||
- Stable, one-time ~3 minute installation
|
||||
- Supported on Debian, Ubuntu 20.04+, Fedora, RHEL 8+, Alpine 3, and macOS
|
||||
- [Nixpkgs](#building-with-nix)
|
||||
- Somewhat stable, one-time 60 second installation
|
||||
- Supported on any platform with a pre-existing installation of Nix (Linux, macOS)
|
||||
- [Without Cgo](#without-cgo)
|
||||
- Plain old pure Go build
|
||||
- Missing a lot of functionality (no Solana Labs compatibility layer)
|
||||
- Manual Installation: If you know what you're doing, feel free to install C deps manually.
|
||||
- Good luck. Plan in 30 minutes of debugging time
|
||||
|
||||
Radiance requires further requires Go 1.19. Using Go 1.18 or Go 1.20 will not work.
|
||||
|
||||
Here's a trick to download another Go version in case you have the wrong one.
|
||||
(See [Managing Go versions](https://golang.org/doc/manage-install))
|
||||
|
||||
go get golang.org/dl/go1.19.8
|
||||
go1.19.8 download
|
||||
alias go=go1.19.8
|
||||
|
||||
Once your Go toolchain and build dependencies are installed, you can build Radiance as usual:
|
||||
|
||||
go mod download
|
||||
go run ./cmd/radiance
|
||||
|
||||
## Building with deps.sh
|
||||
|
||||
`deps.sh` fetches deps using Git, compiles them, and installs them into the `opt` dir of your Radiance checkout.
|
||||
|
||||
To complete the one-time installation of deps, run
|
||||
|
||||
./deps.sh
|
||||
|
||||
You may get prompted to install basic system packages like `pkg-config` or `cmake` if they are missing.
|
||||
|
||||
To add the installed deps to your shell env (and Go build tools), each time you open a new shell, run:
|
||||
|
||||
source activate-opt
|
||||
|
||||
To clean up, simply run `./deps.sh nuke` or `rm -rf opt`
|
||||
|
||||
### How deps.sh works
|
||||
|
||||
Managing C deps is only as complicated as you want it to be.
|
||||
|
||||
`deps.sh` is a ~500 line long handwritten shell script. It just calls `git clone`, `make`, and `make install` or their
|
||||
respective equivalents for each dependency. And yet it is more reliable than any tool that tries to be smart.
|
||||
|
||||
It will not pollute your system -- The C deps are not installed globally.
|
||||
|
||||
## Building with Nix
|
||||
|
||||
The recommended and most stable way to build Radiance is with [Nix](https://nixos.org/).
|
||||
This requires an existing Nix installation on your machine.
|
||||
Radiance provides a [Nix](https://nixos.org/) package.
|
||||
|
||||
nix-build
|
||||
./result/bin/radiance --help
|
||||
|
||||
However, note that the resulting binary will only run under the same Nix environment.
|
||||
|
||||
## Building with Go
|
||||
|
||||
Radiance commands can be built with standard Go 1.19 tooling.
|
||||
|
||||
go run ./cmd/radiance
|
||||
Note that the resulting binary is not freestanding.
|
||||
It is linked against libraries provided by Nix and will not run on non-Nix hosts.
|
||||
|
||||
### Without Cgo
|
||||
|
||||
|
|
@ -20,56 +71,3 @@ The full set of functionality requires C dependencies via Cgo.
|
|||
To create a pure-Go build use the `lite` tag.
|
||||
|
||||
go build -tags=lite ./cmd/radiance
|
||||
|
||||
### With Cgo dependencies
|
||||
|
||||
Radiance tools that require direct access to the blockstore (such as `blockstore` and `car`)
|
||||
require a working C toolchain and extra compiler arguments to link against rocksdb.
|
||||
|
||||
You'll need a working C compiler toolchain as well as prerequisites listed
|
||||
by [grocksdb](https://github.com/linxGnu/grocksdb#prerequisite) and
|
||||
[RocksDB itself](https://github.com/linxGnu/grocksdb#prerequisite).
|
||||
|
||||
**RHEL/Centos/Fedora**
|
||||
|
||||
dnf -y install "@Development Tools" cmake zlib zlib-devel bzip2 bzip2-devel lz4-devel libzstd-devel
|
||||
|
||||
**Debian**
|
||||
|
||||
# With package manager RocksDB
|
||||
apt install -y librocksdb-dev
|
||||
|
||||
# With RocksDB from source
|
||||
apt install -y build-essential cmake zlib1g-dev libbz2-dev liblz4-dev libzstd-dev
|
||||
|
||||
**Building RocksDB**
|
||||
|
||||
To build RocksDB from source, run the following commands:
|
||||
|
||||
git clone https://github.com/facebook/rocksdb --branch v7.10.2 --depth 1
|
||||
cd rocksdb
|
||||
mkdir -p build && cd build
|
||||
cmake .. \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DROCKSDB_BUILD_SHARED=OFF \
|
||||
-DWITH_GFLAGS=OFF \
|
||||
-DWITH_BZ2=ON \
|
||||
-DWITH_SNAPPY=OFF \
|
||||
-DWITH_ZLIB=ON \
|
||||
-DWITH_ZSTD=ON \
|
||||
-DWITH_ALL_TESTS=OFF \
|
||||
-DWITH_BENCHMARK_TOOLS=OFF \
|
||||
-DWITH_CORE_TOOLS=OFF \
|
||||
-DWITH_RUNTIME_DEBUG=OFF \
|
||||
-DWITH_TESTS=OFF \
|
||||
-DWITH_TOOLS=OFF \
|
||||
-DWITH_TRACE_TOOLS=OFF
|
||||
make -j
|
||||
cd ../..
|
||||
|
||||
Finally, rebuild RocksDB with the appropriate Cgo flags.
|
||||
|
||||
export CGO_CFLAGS="-I$(pwd)/rocksdb/include"
|
||||
export CGO_LDFLAGS="-L$(pwd)/rocksdb/build"
|
||||
|
||||
go run ./cmd/radiance
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Set pkg-config path to source libraries from ./opt
|
||||
export PKG_CONFIG_LIBDIR=""
|
||||
export PKG_CONFIG_PATH="$(pwd)/opt/lib/pkgconfig:$(pwd)/opt/lib64/pkgconfig"
|
||||
|
|
@ -0,0 +1,387 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Derived from Firedancer:
|
||||
# https://github.com/firedancer-io/firedancer/blob/main/deps.sh
|
||||
|
||||
# Change into Radiance root directory
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
REPO_ROOT="$(pwd)"
|
||||
|
||||
# Fix pkg-config path and environment
|
||||
# shellcheck source=./activate-opt
|
||||
source activate-opt
|
||||
|
||||
# Load OS information
|
||||
OS="$(uname -s)"
|
||||
case "$OS" in
|
||||
Darwin)
|
||||
MAKE=( make -j )
|
||||
ID=macos
|
||||
;;
|
||||
Linux)
|
||||
MAKE=( make -j --output-target=sync )
|
||||
# Load distro information
|
||||
if [[ -f /etc/os-release ]]; then
|
||||
source /etc/os-release
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
echo "[!] Unsupported OS $OS"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Figure out how to escalate privileges
|
||||
SUDO=""
|
||||
if [[ ! "$(id -u)" -eq "0" ]]; then
|
||||
SUDO="sudo "
|
||||
fi
|
||||
|
||||
# Install prefix
|
||||
PREFIX="$(pwd)/opt"
|
||||
|
||||
help () {
|
||||
cat <<EOF
|
||||
|
||||
Usage: $0 [cmd] [args...]
|
||||
|
||||
deps.sh is a lightweight dependency manager for Radiance.
|
||||
|
||||
It locally installs
|
||||
- build tools using the system package manager
|
||||
- third party dependencies from Git repo sources
|
||||
|
||||
If cmd is ommitted, default is 'install'.
|
||||
|
||||
Commands are:
|
||||
|
||||
help
|
||||
- Prints this message
|
||||
|
||||
check
|
||||
- Runs system requirement checks for dep build/install
|
||||
- Exits with code 0 on success
|
||||
|
||||
nuke
|
||||
- Get rid of dependency checkouts
|
||||
- Get rid of all third party dependency files
|
||||
- Same as 'rm -rf $(pwd)/opt'
|
||||
|
||||
fetch
|
||||
- Fetches dependencies from Git repos into $(pwd)/opt/git
|
||||
|
||||
install
|
||||
- Runs 'fetch'
|
||||
- Runs 'check'
|
||||
- Builds dependencies
|
||||
- Re-installs all project dependencies into prefix $(pwd)/opt
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
nuke () {
|
||||
rm -rf ./opt
|
||||
echo "[-] Nuked $(pwd)/opt"
|
||||
exit 0
|
||||
}
|
||||
|
||||
checkout_repo () {
|
||||
# Skip if dir already exists
|
||||
if [[ -d ./opt/git/"$1" ]]; then
|
||||
echo "[~] Skipping $1 fetch as \"$(pwd)/opt/git/$1\" already exists"
|
||||
else
|
||||
echo "[+] Cloning $1 from $2"
|
||||
git -c advice.detachedHead=false clone "$2" "./opt/git/$1" --branch "$3" --depth=1
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "[~] Checking out $1 $3"
|
||||
(
|
||||
cd ./opt/git/"$1"
|
||||
git fetch origin "$3" --depth=1
|
||||
git -c advice.detachedHead=false checkout "$3"
|
||||
)
|
||||
echo
|
||||
}
|
||||
|
||||
fetch () {
|
||||
mkdir -pv ./opt/git
|
||||
|
||||
checkout_repo zlib https://github.com/madler/zlib "v1.2.13"
|
||||
checkout_repo zstd https://github.com/facebook/zstd "v1.5.4"
|
||||
checkout_repo rocksdb https://github.com/facebook/rocksdb "v7.10.2"
|
||||
}
|
||||
|
||||
check_fedora_pkgs () {
|
||||
local REQUIRED_RPMS=( cmake pkgconf )
|
||||
|
||||
echo "[~] Checking for required RPM packages"
|
||||
|
||||
local MISSING_RPMS=( )
|
||||
for rpm in "${REQUIRED_RPMS[@]}"; do
|
||||
if ! rpm -q "$rpm" >/dev/null; then
|
||||
MISSING_RPMS+=( "$rpm" )
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "${#MISSING_RPMS[@]}" -eq 0 ]]; then
|
||||
echo "[~] OK: RPM packages required for build are installed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "[!] Found missing packages"
|
||||
echo "[?] This is fixed by the following command:"
|
||||
echo " ${SUDO}dnf install -y ${MISSING_RPMS[*]}"
|
||||
read -r -p "[?] Install missing packages with superuser privileges? (y/N) " choice
|
||||
case "$choice" in
|
||||
y|Y)
|
||||
echo "[+] Installing missing RPMs"
|
||||
${SUDO}dnf install -y "${MISSING_RPMS[@]}"
|
||||
echo "[+] Installed missing RPMs"
|
||||
;;
|
||||
*)
|
||||
echo "[-] Skipping package install"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
check_debian_pkgs () {
|
||||
local REQUIRED_DEBS=( build-essential pkg-config )
|
||||
|
||||
echo "[~] Checking for required DEB packages"
|
||||
|
||||
local MISSING_DEBS=( )
|
||||
for deb in "${REQUIRED_DEBS[@]}"; do
|
||||
if ! dpkg -s "$deb" >/dev/null 2>/dev/null; then
|
||||
MISSING_DEBS+=( "$deb" )
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ ${#MISSING_DEBS[@]} -eq 0 ]]; then
|
||||
echo "[~] OK: DEB packages required for build are installed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "[!] Found missing packages"
|
||||
echo "[?] This is fixed by the following command:"
|
||||
echo " ${SUDO}apt-get install -y ${MISSING_DEBS[*]}"
|
||||
read -r -p "[?] Install missing packages with superuser privileges? (y/N) " choice
|
||||
case "$choice" in
|
||||
y|Y)
|
||||
echo "[+] Installing missing DEBs"
|
||||
${SUDO}apt-get install -y "${MISSING_DEBS[@]}"
|
||||
echo "[+] Installed missing DEBs"
|
||||
;;
|
||||
*)
|
||||
echo "[-] Skipping package install"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
check_alpine_pkgs () {
|
||||
local REQUIRED_APKS=( build-base pkgconf cmake )
|
||||
|
||||
echo "[~] Checking for required APK packages"
|
||||
|
||||
local MISSING_APKS=( )
|
||||
for deb in "${REQUIRED_APKS[@]}"; do
|
||||
if ! apk info -e "$deb" >/dev/null; then
|
||||
MISSING_APKS+=( "$deb" )
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ ${#MISSING_APKS[@]} -eq 0 ]]; then
|
||||
echo "[~] OK: APK packages required for build are installed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "[!] Found missing packages"
|
||||
echo "[?] This is fixed by the following command:"
|
||||
echo " ${SUDO}apk add ${MISSING_APKS[*]}"
|
||||
read -r -p "[?] Install missing packages with superuser privileges? (y/N) " choice
|
||||
case "$choice" in
|
||||
y|Y)
|
||||
echo "[+] Installing missing APKs"
|
||||
${SUDO}apk add "${MISSING_APKS[@]}"
|
||||
echo "[+] Installed missing APKs"
|
||||
;;
|
||||
*)
|
||||
echo "[-] Skipping package install"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
check_macos_pkgs () {
|
||||
local REQUIRED_FORMULAE=( pkg-config cmake )
|
||||
|
||||
echo "[~] Checking for required brew formulae"
|
||||
|
||||
local MISSING_FORMULAE=( )
|
||||
for formula in "${REQUIRED_FORMULAE[@]}"; do
|
||||
if [[ ! -d "/usr/local/Cellar/$formula" ]]; then
|
||||
MISSING_FORMULAE+=( "$formula" )
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ ${#MISSING_FORMULAE[@]} -eq 0 ]]; then
|
||||
echo "[~] OK: brew formulae required for build are installed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "[!] Found missing formulae"
|
||||
echo "[?] This is fixed by the following command:"
|
||||
echo " brew install ${MISSING_FORMULAE[*]}"
|
||||
read -r -p "[?] Install missing formulae with brew? (y/N) " choice
|
||||
case "$choice" in
|
||||
y|Y)
|
||||
echo "[+] Installing missing formulae"
|
||||
brew install "${MISSING_FORMULAE[@]}"
|
||||
echo "[+] Installed missing formulae"
|
||||
;;
|
||||
*)
|
||||
echo "[-] Skipping formula install"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
check () {
|
||||
DISTRO="${ID_LIKE:-${ID:-}}"
|
||||
case "$DISTRO" in
|
||||
fedora)
|
||||
check_fedora_pkgs
|
||||
;;
|
||||
debian)
|
||||
check_debian_pkgs
|
||||
;;
|
||||
alpine)
|
||||
check_alpine_pkgs
|
||||
;;
|
||||
macos)
|
||||
check_macos_pkgs
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported distro $DISTRO. Your mileage may vary."
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
install_zlib () {
|
||||
cd ./opt/git/zlib
|
||||
|
||||
echo "[+] Configuring zlib"
|
||||
./configure \
|
||||
--prefix="$PREFIX"
|
||||
echo "[+] Configured zlib"
|
||||
|
||||
echo "[+] Building zlib"
|
||||
"${MAKE[@]}" libz.a
|
||||
echo "[+] Successfully built zlib"
|
||||
|
||||
echo "[+] Installing zlib to $PREFIX"
|
||||
make install -j
|
||||
echo "[+] Successfully installed zlib"
|
||||
}
|
||||
|
||||
install_zstd () {
|
||||
cd ./opt/git/zstd/lib
|
||||
|
||||
echo "[+] Installing zstd to $PREFIX"
|
||||
"${MAKE[@]}" DESTDIR="$PREFIX" PREFIX="" install-pc install-static install-includes
|
||||
echo "[+] Successfully installed zstd"
|
||||
}
|
||||
|
||||
install_rocksdb () {
|
||||
cd ./opt/git/rocksdb
|
||||
mkdir -p build
|
||||
cd build
|
||||
cmake .. \
|
||||
-G"Unix Makefiles" \
|
||||
-DCMAKE_INSTALL_PREFIX:PATH="" \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DROCKSDB_BUILD_SHARED=OFF \
|
||||
-DWITH_GFLAGS=OFF \
|
||||
-DWITH_BZ2=OFF \
|
||||
-DWITH_SNAPPY=OFF \
|
||||
-DWITH_ZLIB=ON \
|
||||
-DWITH_ZSTD=ON \
|
||||
-DWITH_ALL_TESTS=OFF \
|
||||
-DWITH_BENCHMARK_TOOLS=OFF \
|
||||
-DWITH_CORE_TOOLS=OFF \
|
||||
-DWITH_RUNTIME_DEBUG=OFF \
|
||||
-DWITH_TESTS=OFF \
|
||||
-DWITH_TOOLS=OFF \
|
||||
-DWITH_TRACE_TOOLS=OFF
|
||||
make -j
|
||||
make install DESTDIR="$PREFIX"
|
||||
}
|
||||
|
||||
install () {
|
||||
echo "Cleaning install dir"
|
||||
for dir in bin include lib lib64 share usr; do
|
||||
rm -rf "$PREFIX/$dir"
|
||||
done
|
||||
|
||||
# Compression algorithms. Prerequisites of RocksDB, but also used in
|
||||
# the Solana protocol.
|
||||
( install_zlib )
|
||||
( install_zstd )
|
||||
|
||||
# RocksDB (imported by grocksdb)
|
||||
# See https://github.com/linxGnu/grocksdb#prerequisite
|
||||
( install_rocksdb )
|
||||
|
||||
echo "[~] Done! To wire up $(pwd)/opt with Go, run:"
|
||||
echo " source activate-opt"
|
||||
echo
|
||||
}
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "[~] This will fetch, build, and install Radiance dependencies into $(pwd)/opt"
|
||||
echo "[~] For help, run: $0 help"
|
||||
echo
|
||||
echo "[~] Running $0 install"
|
||||
|
||||
read -r -p "[?] Continue? (y/N) " choice
|
||||
case "$choice" in
|
||||
y|Y)
|
||||
echo
|
||||
fetch
|
||||
check
|
||||
install
|
||||
;;
|
||||
*)
|
||||
echo "[!] Stopping." >&2
|
||||
exit 1
|
||||
esac
|
||||
fi
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
-h|--help|help)
|
||||
help
|
||||
;;
|
||||
nuke)
|
||||
shift
|
||||
nuke
|
||||
;;
|
||||
fetch)
|
||||
shift
|
||||
fetch
|
||||
;;
|
||||
check)
|
||||
shift
|
||||
check
|
||||
;;
|
||||
install)
|
||||
shift
|
||||
fetch
|
||||
check
|
||||
install
|
||||
;;
|
||||
*)
|
||||
echo "Unknown command: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
Loading…
Reference in New Issue