Merge pull request #6422

c73a8ea Clarifying offline build process using gbuild --url and noting it is not done automatically. (Midnight Magic)
This commit is contained in:
Wladimir J. van der Laan 2015-09-18 05:43:20 +02:00
commit 8bc1b3a1f3
No known key found for this signature in database
GPG Key ID: 74810B012346C9A6
2 changed files with 107 additions and 22 deletions

View File

@ -330,10 +330,11 @@ There will be a lot of warnings printed during the build of the image. These can
Getting and building the inputs Getting and building the inputs
-------------------------------- --------------------------------
Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change) Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change)
in the bitcoin repository to install sources which require manual intervention. Also follow in the bitcoin repository under 'Fetch and build inputs' to install sources which require
the next step: 'Seed the Gitian sources cache', which will fetch all the necessary source manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache
files to allow gitian to work offline. and offline git repositories' which will fetch the remaining files required for building
offline.
Building Bitcoin Building Bitcoin
---------------- ----------------
@ -391,6 +392,57 @@ COMMIT=2014_03_windows_unicode_path
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml ./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
``` ```
Building fully offline
-----------------------
For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository
and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be
told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or
manually disabling gitian-builder's use of apt-get to update the VM build environment.
To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant
files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
plain archive.ubuntu.com: us.archive.ubuntu.com works.
So, if you use LXC:
```bash
export PATH="$PATH":/path/to/gitian-builder/libexec
export USE_LXC=1
cd /path/to/gitian-builder
./libexec/make-clean-vm --suite precise --arch amd64
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get update
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root \
-e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
$( sed -ne '/^packages:/,/[^-] .*/ {/^- .*/{s/"//g;s/- //;p}}' ../bitcoin/contrib/gitian-descriptors/*|sort|uniq )
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get -q -y purge grub
LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
```
And then set offline mode for apt-cacher-ng:
```
/etc/apt-cacher-ng/acng.conf
[...]
Offlinemode: 1
[...]
service apt-cacher-ng restart
```
Then when building, override the remote URLs that gbuild would otherwise pull from the gitian descriptors::
```bash
cd /some/root/path/
git clone https://github.com/bitcoin/bitcoin-detached-sigs.git
BTCPATH=/some/root/path/bitcoin.git
SIGPATH=/some/root/path/bitcoin-detached-sigs.git
./bin/gbuild --url bitcoin=${BTCPATH},signature=${SIGPATH} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
```
Signing externally Signing externally
------------------- -------------------

View File

@ -6,39 +6,54 @@ Release Process
* * * * * *
###update (commit) version in sources ###first time only or for new builders, check out the source in the following directory hierarchy
cd /path/to/your/toplevel/build
git clone https://github.com/bitcoin/gitian.sigs.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/bitcoin/bitcoin.git
###for bitcoin maintainers/release engineers, update (commit) version in sources
pushd ./bitcoin
contrib/verifysfbinaries/verify.sh contrib/verifysfbinaries/verify.sh
doc/README* doc/README*
share/setup.nsi share/setup.nsi
src/clientversion.h (change CLIENT_VERSION_IS_RELEASE to true) src/clientversion.h (change CLIENT_VERSION_IS_RELEASE to true)
###tag version in git ###for bitcoin maintainers/release engineers, tag version in git
git tag -s v(new version, e.g. 0.8.0) git tag -s v(new version, e.g. 0.8.0)
###write release notes. git shortlog helps a lot, for example: ###for bitcoin maintainers/release engineers, write release notes. git shortlog helps a lot, for example:
git shortlog --no-merges v(current version, e.g. 0.7.2)..v(new version, e.g. 0.8.0) git shortlog --no-merges v(current version, e.g. 0.7.2)..v(new version, e.g. 0.8.0)
popd
* * * * * *
###update gitian ###update gitian, gitian.sigs, checkout bitcoin version, and perform gitian builds
In order to take advantage of the new caching features in gitian, be sure to update to a recent version (`e9741525c` or later is recommended) To ensure your gitian descriptors are accurate for direct reference for gbuild, below, run the following from a directory containing the bitcoin source:
###perform gitian builds
From a directory containing the bitcoin source, gitian-builder and gitian.sigs
pushd ./bitcoin
export SIGNER=(your gitian key, ie bluematt, sipa, etc) export SIGNER=(your gitian key, ie bluematt, sipa, etc)
export VERSION=(new version, e.g. 0.8.0) export VERSION=(new version, e.g. 0.8.0)
pushd ./bitcoin
git checkout v${VERSION} git checkout v${VERSION}
popd popd
pushd ./gitian-builder
###fetch and build inputs: (first time, or when dependency versions change) Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other gitian signatures:
pushd ./gitian.sigs
git pull
popd
Ensure your gitian-builder sources are up-to-date to take advantage of the new caching features of gitian (`e9741525c` or later is recommended)
pushd ./gitian-builder
git pull
###fetch and create inputs: (first time, or when dependency versions change)
mkdir -p inputs mkdir -p inputs
wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
@ -52,28 +67,44 @@ Release Process
tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk
###Optional: Seed the Gitian sources cache ###Optional: Seed the Gitian sources cache and offline git repositories
By default, gitian will fetch source files as needed. For offline builds, they can be fetched ahead of time: By default, gitian will fetch source files as needed. To cache them ahead of time:
make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common
Only missing files will be fetched, so this is safe to re-run for each build. Only missing files will be fetched, so this is safe to re-run for each build.
###Build Bitcoin Core for Linux, Windows, and OS X: Clone the detached-sigs repository:
popd
git clone https://github.com/bitcoin/bitcoin-detached-sigs.git
pushd ./bitcoin-builder
NOTE: Offline builds must use the --url flag to ensure gitian fetches only from local URLs.
For example: ./bin/bguild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments}
The following gbuild invocations DO NOT DO THIS by default.
###Build (and optionally verify) Bitcoin Core for Linux, Windows, and OS X:
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../ mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../ mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../ mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
popd popd
Build output expected: Build output expected:
1. source tarball (bitcoin-${VERSION}.tar.gz) 1. source tarball (bitcoin-${VERSION}.tar.gz)
@ -98,19 +129,21 @@ Commit your signature to gitian.sigs:
Once the Windows/OSX builds each have 3 matching signatures, they will be signed with their respective release keys. Once the Windows/OSX builds each have 3 matching signatures, they will be signed with their respective release keys.
Detached signatures will then be committed to the bitcoin-detached-sigs repository, which can be combined with the unsigned apps to create signed binaries. Detached signatures will then be committed to the bitcoin-detached-sigs repository, which can be combined with the unsigned apps to create signed binaries.
Create the signed OSX binary: Create (and optionally verify) the signed OSX binary:
pushd ./gitian-builder pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
popd popd
Create the signed Windows binaries: Create (and optionally verify) the signed Windows binaries:
pushd ./gitian-builder pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
mv build/out/bitcoin-*win32-setup.exe ../bitcoin-${VERSION}-win32-setup.exe mv build/out/bitcoin-*win32-setup.exe ../bitcoin-${VERSION}-win32-setup.exe
popd popd