Merge branch 'master' into master

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,2 @@
+<!-- Note: This website is for bug reports, not general questions.
+Do not post issues about non-bitcoin versions of Electrum. -->

.travis.yml

@@ -4,11 +4,28 @@ python:
     - 3.5
     - 3.6
 install:
-  - pip install -r requirements_travis.txt
+  - pip install -r contrib/requirements/requirements-travis.txt
 cache:
-  - pip
+  - pip: true
+  - directories:
+    - /tmp/electrum-build
 script:
     - tox
 after_success:
     - if [ "$TRAVIS_BRANCH" = "master" ]; then pip install pycurl requests && contrib/make_locale; fi
     - coveralls
+jobs:
+  include:
+    - stage: windows build
+      sudo: true
+      python: 3.5
+      install:
+        - sudo dpkg --add-architecture i386
+        - wget -nc https://dl.winehq.org/wine-builds/Release.key
+        - sudo apt-key add Release.key
+        - sudo apt-add-repository https://dl.winehq.org/wine-builds/ubuntu/
+        - sudo apt-get update -qq
+        - sudo apt-get install -qq winehq-stable dirmngr gnupg2 p7zip-full
+      before_script: ls -lah /tmp/electrum-build
+      script: ./contrib/build-wine/build.sh
+      after_success: true

Dockerfile

@@ -1,6 +1,6 @@
 FROM ubuntu:18.04
 
-ENV VERSION 1.0.6
+ENV VERSION 1.0.7
 
 RUN set -x \
     && apt-get update \

README.rst

@@ -101,30 +101,18 @@ Run the docker image::
     ./run-docker.sh
 
 
-Building Releases
-=================
-
-
-MacOS
-------
-
-Simply - ::
-
-    ./setup-mac.sh
-
-    sudo ./create-dmg.sh
+See `contrib/build-osx/`.
 
 Windows
 -------
 
-See `contrib/build-wine/README` file.
+See `contrib/build-wine/`.
 
 
 Android
 -------
 
 See `gui/kivy/Readme.txt` file.
-UPSTREAM PATCH: https://github.com/spesmilo/electrum/blob/master/gui/kivy/Readme.md
 
 ---
 

RELEASE-NOTES

@@ -1,3 +1,80 @@
+
+# Release 3.1 - (March 5, 2018)
+
+ * Memory-pool based fee estimation. Dynamic fees can target a desired
+   depth in the memory pool. This feature is optional, and ETA-based
+   estimates from Bitcoin Core are still available. Note that miners
+   could exploit this feature, if they conspired and filled the memory
+   pool with expensive transactions that never get mined. However,
+   since the Electrum client already trusts an Electrum server with
+   fee estimates, activating this feature does not introduce any new
+   vulnerability. In addition, the client uses a hard threshold to
+   protect itself from servers sending excessive fee estimates. In
+   practice, ETA-based estimates have resulted in sticky fees, and
+   caused many users to overpay for transactions. Advanced users tend
+   to visit (and trust) websites that display memory-pool data in
+   order to set their fees.
+ * Capital gains: For each outgoing transaction, the difference
+   between the acquisition and liquidation prices of outgoing coins is
+   displayed in the wallet history. By default, historical exchange
+   rates are used to compute acquisition and liquidation prices. These
+   values can also be entered manually, in order to match the actual
+   price realized by the user. The order of liquidation of coins is
+   the natural order defined by the blockchain; this results in
+   capital gain values that are invariant to changes in the set of
+   addresses that are in the wallet. Any other ordering strategy (such
+   as FIFO, LIFO) would result in capital gain values that depend on
+   the presence of other addresses in the wallet.
+ * Local transactions: Transactions can be saved in the wallet without
+   being broadcast. The inputs of local transactions are considered as
+   spent, and their change outputs can be re-used in subsequent
+   transactions. This can be combined with cold storage, in order to
+   create several transactions before broadcasting them. Outgoing
+   transactions that have been removed from the memory pool are also
+   saved in the wallet, and can be broadcast again.
+ * Checkpoints: The initial download of a headers file was replaced
+   with hardcoded checkpoints. The wallet uses one checkpoint per
+   retargeting period. The headers for a retargeting period are
+   downloaded only if transactions need to be verified in this period.
+ * The 'privacy' and 'priority' coin selection policies have been
+   merged into one. Previously, the 'privacy' policy has been unusable
+   because it was was not prioritizing confirmed coins. The new policy
+   is similar to 'privacy', except that it de-prioritizes addresses
+   that have unconfirmed coins.
+ * The 'Send' tab of the Qt GUI displays how transaction fees are
+   computed from transaction size.
+ * The wallet history can be filtered by time interval.
+ * Replace-by-fee is enabled by default. Note that this might cause
+   some issues with wallets that do not display RBF transactions until
+   they are confirmed.
+ * Watching-only wallets and hardware wallets can be encrypted.
+ * Semi-automated crash reporting
+ * The SSL checkbox option was removed from the GUI.
+ * The Trezor T hardware wallet is now supported.
+ * BIP84: native segwit p2wpkh scripts for bip39 seeds and hardware
+   wallets can now be created when specifying a BIP84 derivation
+   path. This is usable with Trezor and Ledger.
+ * Windows: the binaries now include ZBar, and QR code scanning should work.
+ * The Wallet Import Format (WIF) for private keys that was extended in 3.0
+   is changed. Keys in the previous format can be imported, compatibility
+   is maintained. Newly exported keys will be serialized as
+   "script_type:original_wif_format_key".
+ * BIP32 master keys for testnet once again have different version bytes than
+   on mainnet. For the mainnet prefixes {x,y,Y,z,Z}|{pub,prv}, the
+   corresponding testnet prefixes are   {t,u,U,v,V}|{pub,prv}.
+   More details and exact version bytes are specified at:
+   https://github.com/spesmilo/electrum-docs/blob/master/xpub_version_bytes.rst
+   Note that due to this change, testnet wallet files created with previous
+   versions of Electrum must be considered broken, and they need to be
+   recreated from seed words.
+ * A new version of the Electrum protocol is required by the client
+   (version 1.2). Servers using older versions of the protocol will
+   not be displayed in the GUI.
+
+
+# Release 3.0.6 :
+  * Fix transaction parsing bug #3788
+
 # Release 3.0.5 : (Security update)
 
 This is a follow-up to the 3.0.4 release, which did not completely fix

contrib/build-osx/README.md

@@ -0,0 +1,17 @@
+Building Mac OS binaries
+========================
+
+This guide explains how to build Electrum binaries for macOS systems.
+We build our binaries on El Capitan (10.11.6) as building it on High Sierra
+makes the binaries incompatible with older versions.
+
+This assumes that the Xcode command line tools (and thus git) are already installed. 
+
+
+## 1. Run the script
+
+
+
+    ./make_osx
+
+## 2. Done

contrib/build-osx/make_osx

@@ -0,0 +1,84 @@
+#!/bin/bash
+RED='\033[0;31m'
+BLUE='\033[0,34m'
+NC='\033[0m' # No Color
+function info {
+	printf "\r💬 ${BLUE}INFO:${NC} ${1}\n"
+}
+function fail {
+    printf "\r🗯 ${RED}ERROR:${NC} ${1}\n"
+    exit 1
+}
+
+build_dir=$(dirname "$0")
+test -n "$build_dir" -a -d "$build_dir" || exit
+cd $build_dir/../..
+
+export PYTHONHASHSEED=22
+VERSION=`git describe --tags`
+
+# Paramterize
+PYTHON_VERSION=3.6.4
+BUILDDIR=/tmp/electrum-build
+PACKAGE=Electrum-ZCL
+GIT_REPO=https://github.com/z-classic/electrum-zcl
+
+
+info "Installing Python $PYTHON_VERSION"
+export PATH="~/.pyenv/bin:~/.pyenv/shims:~/Library/Python/3.6/bin:$PATH"
+if [ -d "~/.pyenv" ]; then
+  pyenv update
+else
+  curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash > /dev/null 2>&1
+fi
+PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -s $PYTHON_VERSION && \
+pyenv global $PYTHON_VERSION || \
+fail "Unable to use Python $PYTHON_VERSION"
+
+
+info "Installing pyinstaller"
+python3 -m pip install git+https://github.com/ecdsa/pyinstaller@fix_2952 -I --user || fail "Could not install pyinstaller"
+
+info "Using these versions for building $PACKAGE:"
+sw_vers
+python3 --version
+echo -n "Pyinstaller "
+pyinstaller --version
+
+rm -rf ./dist
+
+
+rm  -rf $BUILDDIR > /dev/null 2>&1
+mkdir $BUILDDIR
+
+info "Downloading icons and locale..."
+for repo in icons locale; do
+  git clone $GIT_REPO-$repo $BUILDDIR/electrum-$repo
+done
+
+cp -R $BUILDDIR/electrum-locale/locale/ ./lib/locale/
+cp    $BUILDDIR/electrum-icons/icons_rc.py ./gui/qt/
+
+
+info "Downloading libusb..."
+curl https://homebrew.bintray.com/bottles/libusb-1.0.21.el_capitan.bottle.tar.gz | \
+tar xz --directory $BUILDDIR
+cp $BUILDDIR/libusb/1.0.21/lib/libusb-1.0.dylib contrib/build-osx
+
+info "Installing requirements..."
+python3 -m pip install -Ir ./contrib/deterministic-build/requirements.txt --user && \
+python3 -m pip install -Ir ./contrib/deterministic-build/requirements-binaries.txt --user || \
+fail "Could not install requirements"
+
+info "Installing hardware wallet requirements..."
+python3 -m pip install -Ir ./contrib/deterministic-build/requirements-hw.txt --user || \
+fail "Could not install hardware wallet requirements"
+
+info "Building $PACKAGE..."
+python3 setup.py install --user > /dev/null || fail "Could not build $PACKAGE"
+
+info "Building binary"
+pyinstaller --noconfirm --ascii --name $VERSION contrib/build-osx/osx.spec || fail "Could not build binary"
+
+info "Creating .DMG"
+hdiutil create -fs HFS+ -volname $PACKAGE -srcfolder dist/$PACKAGE.app dist/Electrum-ZCL-$VERSION.dmg || fail "Could not create .DMG"

contrib/build-osx/osx.spec

@@ -0,0 +1,99 @@
+# -*- mode: python -*-
+
+from PyInstaller.utils.hooks import collect_data_files, collect_submodules, collect_dynamic_libs
+
+import sys
+import os
+
+PACKAGE='Electrum-ZCL'
+PYPKG='electrum'
+MAIN_SCRIPT='electrum'
+ICONS_FILE='electrum.icns'
+
+for i, x in enumerate(sys.argv):
+    if x == '--name':
+        VERSION = sys.argv[i+1]
+        break
+else:
+    raise BaseException('no version')
+
+electrum = os.path.abspath(".") + "/"
+block_cipher = None
+
+# see https://github.com/pyinstaller/pyinstaller/issues/2005
+hiddenimports = []
+hiddenimports += collect_submodules('trezorlib')
+hiddenimports += collect_submodules('btchip')
+hiddenimports += collect_submodules('keepkeylib')
+hiddenimports += collect_submodules('websocket')
+
+datas = [
+    (electrum+'lib/currencies.json', PYPKG),
+    (electrum+'lib/servers.json', PYPKG),
+    (electrum+'lib/checkpoints.json', PYPKG),
+    (electrum+'lib/servers_testnet.json', PYPKG),
+    (electrum+'lib/checkpoints_testnet.json', PYPKG),
+    (electrum+'lib/wordlist/english.txt', PYPKG + '/wordlist'),
+    (electrum+'lib/locale', PYPKG + '/locale'),
+    (electrum+'plugins', PYPKG + '_plugins'),
+]
+datas += collect_data_files('trezorlib')
+datas += collect_data_files('btchip')
+datas += collect_data_files('keepkeylib')
+
+# Add libusb so Trezor will work
+binaries = [(electrum + "contrib/build-osx/libusb-1.0.dylib", ".")]
+
+# Workaround for "Retro Look":
+binaries += [b for b in collect_dynamic_libs('PyQt5') if 'macstyle' in b[0]]
+
+# We don't put these files in to actually include them in the script but to make the Analysis method scan them for imports
+a = Analysis([electrum+MAIN_SCRIPT,
+              electrum+'gui/qt/main_window.py',
+              electrum+'gui/text.py',
+              electrum+'lib/util.py',
+              electrum+'lib/wallet.py',
+              electrum+'lib/simple_config.py',
+              electrum+'lib/bitcoin.py',
+              electrum+'lib/dnssec.py',
+              electrum+'lib/commands.py',
+              electrum+'plugins/cosigner_pool/qt.py',
+              electrum+'plugins/email_requests/qt.py',
+              electrum+'plugins/trezor/client.py',
+              electrum+'plugins/trezor/qt.py',
+              electrum+'plugins/keepkey/qt.py',
+              electrum+'plugins/ledger/qt.py',
+              ],
+             binaries=binaries,
+             datas=datas,
+             hiddenimports=hiddenimports,
+             hookspath=[])
+
+# http://stackoverflow.com/questions/19055089/pyinstaller-onefile-warning-pyconfig-h-when-importing-scipy-or-scipy-signal
+for d in a.datas:
+    if 'pyconfig' in d[0]:
+        a.datas.remove(d)
+        break
+
+pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
+
+exe = EXE(pyz,
+          a.scripts,
+          a.binaries,
+          a.datas,
+          name=PACKAGE,
+          debug=False,
+          strip=False,
+          upx=True,
+          icon=electrum+ICONS_FILE,
+          console=False)
+
+app = BUNDLE(exe,
+             version = VERSION,
+             name=PACKAGE + '.app',
+             icon=electrum+ICONS_FILE,
+             bundle_identifier=None,
+             info_plist = {
+                 'NSHighResolutionCapable':'True'
+             }
+)

contrib/build-wine/README.md

@@ -2,7 +2,7 @@ Windows Binary Builds
 =====================
 
 These scripts can be used for cross-compilation of Windows Electrum executables from Linux/Wine.
-Produced binaries are deterministic so you should be able to generate binaries that match the official releases.
+Produced binaries are deterministic, so you should be able to generate binaries that match the official releases. 
 
 
 Usage:
@@ -12,6 +12,7 @@ Usage:
 
  - dirmngr
  - gpg
+ - 7Zip
  - Wine (>= v2)
 
 
@@ -19,8 +20,7 @@ For example:
 
 
 ```
-$ sudo apt-get install wine-development dirmngr gnupg2
-$ sudo ln -sf /usr/bin/wine-development /usr/local/bin/wine
+$ sudo apt-get install wine-development dirmngr gnupg2 p7zip-full
 $ wine --version
  wine-2.0 (Debian 2.0-3+b2)
 ```

contrib/build-wine/build-electrum-git.sh

@@ -16,6 +16,7 @@ PYTHON="wine $PYHOME/python.exe -OO -B"
 cd `dirname $0`
 set -e
 
+mkdir -p tmp
 cd tmp
 
 for repo in electrum electrum-locale electrum-icons; do
@@ -55,7 +56,9 @@ cp -r electrum-locale/locale $WINEPREFIX/drive_c/electrum/lib/
 cp electrum-icons/icons_rc.py $WINEPREFIX/drive_c/electrum/gui/qt/
 
 # Install frozen dependencies
-$PYTHON -m pip install -r ../../requirements.txt
+$PYTHON -m pip install -r ../../deterministic-build/requirements.txt
+
+$PYTHON -m pip install -r ../../deterministic-build/requirements-hw.txt
 
 pushd $WINEPREFIX/drive_c/electrum
 $PYTHON setup.py install

contrib/build-wine/build.sh

@@ -7,14 +7,13 @@ if [ ! -z "$1" ]; then
 fi
 
 here=$(dirname "$0")
+test -n "$here" -a -d "$here" || exit
 
 echo "Clearing $here/build and $here/dist..."
-rm $here/build/* -rf
-rm $here/dist/* -rf
+rm "$here"/build/* -rf
+rm "$here"/dist/* -rf
 
-$here/prepare-wine.sh && \
-$here/prepare-pyinstaller.sh && \
-$here/prepare-hw.sh || exit 1
+$here/prepare-wine.sh || exit 1
 
 echo "Resetting modification time in C:\Python..."
 # (Because of some bugs in pyinstaller)

contrib/build-wine/deterministic.spec

@@ -1,9 +1,8 @@
 # -*- mode: python -*-
 
-from PyInstaller.utils.hooks import collect_data_files, collect_submodules
+from PyInstaller.utils.hooks import collect_data_files, collect_submodules, collect_dynamic_libs
 
 import sys
-import os
 for i, x in enumerate(sys.argv):
     if x == '--name':
         cmdline_name = sys.argv[i+1]
@@ -11,13 +10,21 @@ for i, x in enumerate(sys.argv):
 else:
     raise BaseException('no name')
 
-home = os.getcwd()+'\\'
+
+home = 'C:\\electrum-zcl\\'
 
 # see https://github.com/pyinstaller/pyinstaller/issues/2005
 hiddenimports = []
-# hiddenimports += collect_submodules('trezorlib')
-# hiddenimports += collect_submodules('btchip')
-# hiddenimports += collect_submodules('keepkeylib')
+hiddenimports += collect_submodules('trezorlib')
+hiddenimports += collect_submodules('btchip')
+hiddenimports += collect_submodules('keepkeylib')
+hiddenimports += collect_submodules('websocket')
+
+# Add libusb binary
+binaries = [("c:/python3.5.4/libusb-1.0.dll", ".")]
+
+# Workaround for "Retro Look":
+binaries += [b for b in collect_dynamic_libs('PyQt5') if 'qwindowsvista' in b[0]]
 
 datas = [
     (home+'lib/currencies.json', 'electrum'),
@@ -26,12 +33,13 @@ datas = [
     (home+'lib/servers_testnet.json', 'electrum'),
     (home+'lib/checkpoints_testnet.json', 'electrum'),
     (home+'lib/wordlist/english.txt', 'electrum/wordlist'),
-#    (home+'lib/locale', 'electrum/locale'),
+    (home+'lib/locale', 'electrum-zcl/locale'),
     (home+'plugins', 'electrum_plugins'),
+    ('C:\\Program Files (x86)\\ZBar\\bin\\', '.')
 ]
-# datas += collect_data_files('trezorlib')
-# datas += collect_data_files('btchip')
-# datas += collect_data_files('keepkeylib')
+datas += collect_data_files('trezorlib')
+datas += collect_data_files('btchip')
+datas += collect_data_files('keepkeylib')
 
 # We don't put these files in to actually include them in the script but to make the Analysis method scan them for imports
 a = Analysis([home+'electrum-zcl',
@@ -45,12 +53,13 @@ a = Analysis([home+'electrum-zcl',
               home+'lib/commands.py',
               home+'plugins/cosigner_pool/qt.py',
               home+'plugins/email_requests/qt.py',
-              #home+'plugins/trezor/client.py',
-              #home+'plugins/trezor/qt.py',
-              #home+'plugins/keepkey/qt.py',
-              #home+'plugins/ledger/qt.py',
+              home+'plugins/trezor/client.py',
+              home+'plugins/trezor/qt.py',
+              home+'plugins/keepkey/qt.py',
+              home+'plugins/ledger/qt.py',
               #home+'packages/requests/utils.py'
               ],
+             binaries=binaries,
              datas=datas,
              #pathex=[home+'lib', home+'gui', home+'plugins'],
              hiddenimports=hiddenimports,
@@ -85,40 +94,40 @@ exe_standalone = EXE(
     console=False)
     # console=True makes an annoying black box pop up, but it does make Electrum output command line commands, with this turned off no output will be given but commands can still be used
 
-# exe_portable = EXE(
-    # pyz,
-    # a.scripts,
-    # a.binaries,
-    # a.datas,
-    # name=os.path.join('build\\pyi.win32\\electrum', cmdline_name + "-portable.exe"),
-    # debug=False,
-    # strip=None,
-    # upx=False,
-    # icon=home+'icons/electrum.ico',
-    # console=False)
+exe_portable = EXE(
+    pyz,
+    a.scripts,
+    a.binaries,
+    a.datas + [ ('is_portable', 'README.md', 'DATA' ) ],
+    name=os.path.join('build\\pyi.win32\\electrum', cmdline_name + "-portable.exe"),
+    debug=False,
+    strip=None,
+    upx=False,
+    icon=home+'icons/electrum.ico',
+    console=False)
 
-# #####
-# # exe and separate files that NSIS uses to build installer "setup" exe
+#####
+# exe and separate files that NSIS uses to build installer "setup" exe
 
-# exe_dependent = EXE(
-    # pyz,
-    # a.scripts,
-    # exclude_binaries=True,
-    # name=os.path.join('build\\pyi.win32\\electrum', cmdline_name),
-    # debug=False,
-    # strip=None,
-    # upx=False,
-    # icon=home+'icons/electrum.ico',
-    # console=False)
+exe_dependent = EXE(
+    pyz,
+    a.scripts,
+    exclude_binaries=True,
+    name=os.path.join('build\\pyi.win32\\electrum', cmdline_name),
+    debug=False,
+    strip=None,
+    upx=False,
+    icon=home+'icons/electrum.ico',
+    console=False)
 
-# coll = COLLECT(
-    # exe_dependent,
-    # a.binaries,
-    # a.zipfiles,
-    # a.datas,
-    # strip=None,
-    # upx=True,
-    # debug=False,
-    # icon=home+'icons/electrum.ico',
-    # console=False,
-    # name=os.path.join('dist', 'electrum'))
+coll = COLLECT(
+    exe_dependent,
+    a.binaries,
+    a.zipfiles,
+    a.datas,
+    strip=None,
+    upx=True,
+    debug=False,
+    icon=home+'icons/electrum.ico',
+    console=False,
+    name=os.path.join('dist', 'electrum'))

contrib/build-wine/prepare-hw.sh

@@ -1,28 +0,0 @@
-#!/bin/bash
-
-TREZOR_GIT_URL=https://github.com/trezor/python-trezor.git
-KEEPKEY_GIT_URL=https://github.com/keepkey/python-keepkey.git
-BTCHIP_GIT_URL=https://github.com/LedgerHQ/btchip-python.git
-
-BRANCH=master
-
-PYTHON_VERSION=3.5.4
-
-# These settings probably don't need any change
-export WINEPREFIX=/opt/wine64
-
-PYHOME=c:/python$PYTHON_VERSION
-PYTHON="wine $PYHOME/python.exe -OO -B"
-
-# Let's begin!
-cd `dirname $0`
-set -e
-
-cd tmp
-
-$PYTHON -m pip install setuptools --upgrade
-$PYTHON -m pip install cython --upgrade
-$PYTHON -m pip install trezor==0.7.16 --upgrade
-$PYTHON -m pip install keepkey==4.0.0 --upgrade
-$PYTHON -m pip install btchip-python==0.1.23 --upgrade
-

contrib/build-wine/prepare-pyinstaller.sh

@@ -1,24 +0,0 @@
-#!/bin/bash
-PYTHON_VERSION=3.5.4
-
-PYINSTALLER_GIT_URL=https://github.com/ecdsa/pyinstaller.git
-BRANCH=fix_2952
-
-export WINEPREFIX=/opt/wine64
-PYHOME=c:/python$PYTHON_VERSION
-PYTHON="wine $PYHOME/python.exe -OO -B"
-
-cd `dirname $0`
-set -e
-cd tmp
-if [ ! -d "pyinstaller" ]; then
-    git clone -b $BRANCH $PYINSTALLER_GIT_URL pyinstaller
-fi
-
-cd pyinstaller
-git pull
-git checkout $BRANCH
-$PYTHON setup.py install
-cd ..
-
-wine "C:/python$PYTHON_VERSION/scripts/pyinstaller.exe" -v

contrib/build-wine/prepare-wine.sh

@@ -1,8 +1,18 @@
 #!/bin/bash
 
 # Please update these carefully, some versions won't work under Wine
-NSIS_URL=https://prdownloads.sourceforge.net/nsis/nsis-3.02.1-setup.exe?download
+NSIS_FILENAME=nsis-3.02.1-setup.exe
+NSIS_URL=https://prdownloads.sourceforge.net/nsis/$NSIS_FILENAME?download
 NSIS_SHA256=736c9062a02e297e335f82252e648a883171c98e0d5120439f538c81d429552e
+
+ZBAR_FILENAME=zbarw-20121031-setup.exe
+ZBAR_URL=https://sourceforge.net/projects/zbarw/files/$ZBAR_FILENAME/download
+ZBAR_SHA256=177e32b272fa76528a3af486b74e9cb356707be1c5ace4ed3fcee9723e2c2c02
+
+LIBUSB_FILENAME=libusb-1.0.21.7z
+LIBUSB_URL=https://prdownloads.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.21/$LIBUSB_FILENAME?download
+LIBUSB_SHA256=acdde63a40b1477898aee6153f9d91d1a2e8a5d93f832ca8ab876498f3a6d2b8
+
 PYTHON_VERSION=3.5.4
 
 ## These settings probably don't need change
@@ -21,23 +31,31 @@ verify_signature() {
         return 0
     else
         echo "$out" >&2
-        exit 0
+        exit 1
     fi
 }
 
 verify_hash() {
-    local file=$1 expected_hash=$2 out=
+    local file=$1 expected_hash=$2
     actual_hash=$(sha256sum $file | awk '{print $1}')
     if [ "$actual_hash" == "$expected_hash" ]; then
         return 0
     else
         echo "$file $actual_hash (unexpected hash)" >&2
-        exit 0
+        rm "$file"
+        exit 1
+    fi
+}
+
+download_if_not_exist() {
+    local file_name=$1 url=$2
+    if [ ! -e $file_name ] ; then
+        wget -O $PWD/$file_name "$url"
     fi
 }
 
 # Let's begin!
-cd `dirname $0`
+here=$(dirname $(readlink -e $0))
 set -e
 
 # Clean up Wine environment
@@ -47,22 +65,21 @@ echo "done"
 
 wine 'wineboot'
 
-echo "Cleaning tmp"
-rm -rf tmp
-mkdir -p tmp
-echo "done"
+mkdir -p /tmp/electrum-build
 
-cd tmp
+cd /tmp/electrum-build
 
 # Install Python
 # note: you might need "sudo apt-get install dirmngr" for the following
 # keys from https://www.python.org/downloads/#pubkeys
-KEYRING_PYTHON_DEV=keyring-electrum-build-python-dev.gpg
-gpg --no-default-keyring --keyring $KEYRING_PYTHON_DEV --recv-keys 531F072D39700991925FED0C0EDDC5F26A45C816 26DEA9D4613391EF3E25C9FF0A5B101836580288 CBC547978A3964D14B9AB36A6AF053F07D9DC8D2 C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF 12EF3DC38047DA382D18A5B999CDEA9DA4135B38 8417157EDBE73D9EAC1E539B126EB563A74B06BF DBBF2EEBF925FAADCF1F3FFFD9866941EA5BBD71 2BA0DB82515BBB9EFFAC71C5C9BE28DEE6DF025C 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D C9B104B3DD3AA72D7CCB1066FB9921286F5E1540 97FC712E4C024BBEA48A61ED3A5CA953F73C700D 7ED10B6531D7C8E1BC296021FC624643487034E5
+KEYLIST_PYTHON_DEV="531F072D39700991925FED0C0EDDC5F26A45C816 26DEA9D4613391EF3E25C9FF0A5B101836580288 CBC547978A3964D14B9AB36A6AF053F07D9DC8D2 C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF 12EF3DC38047DA382D18A5B999CDEA9DA4135B38 8417157EDBE73D9EAC1E539B126EB563A74B06BF DBBF2EEBF925FAADCF1F3FFFD9866941EA5BBD71 2BA0DB82515BBB9EFFAC71C5C9BE28DEE6DF025C 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D C9B104B3DD3AA72D7CCB1066FB9921286F5E1540 97FC712E4C024BBEA48A61ED3A5CA953F73C700D 7ED10B6531D7C8E1BC296021FC624643487034E5"
+KEYRING_PYTHON_DEV="keyring-electrum-build-python-dev.gpg"
+KEYSERVER_PYTHON_DEV="hkp://keys.gnupg.net"
+gpg --no-default-keyring --keyring $KEYRING_PYTHON_DEV --keyserver $KEYSERVER_PYTHON_DEV --recv-keys $KEYLIST_PYTHON_DEV
 for msifile in core dev exe lib pip tools; do
     echo "Installing $msifile..."
-    wget "https://www.python.org/ftp/python/$PYTHON_VERSION/win32/${msifile}.msi"
-    wget "https://www.python.org/ftp/python/$PYTHON_VERSION/win32/${msifile}.msi.asc"
+    wget -nc "https://www.python.org/ftp/python/$PYTHON_VERSION/win32/${msifile}.msi"
+    wget -nc "https://www.python.org/ftp/python/$PYTHON_VERSION/win32/${msifile}.msi.asc"
     verify_signature "${msifile}.msi.asc" $KEYRING_PYTHON_DEV
     wine msiexec /i "${msifile}.msi" /qb TARGETDIR=C:/python$PYTHON_VERSION
 done
@@ -70,36 +87,35 @@ done
 # upgrade pip
 $PYTHON -m pip install pip --upgrade
 
-# Install PyWin32
-$PYTHON -m pip install pypiwin32
-
-# Install PyQt
-$PYTHON -m pip install PyQt5
-
-## Install pyinstaller
-#$PYTHON -m pip install pyinstaller==3.3
-
-
-# Install ZBar
-#wget -q -O zbar.exe "https://sourceforge.net/projects/zbar/files/zbar/0.10/zbar-0.10-setup.exe/download"
-#wine zbar.exe
-
-# install Cryptodome
-$PYTHON -m pip install pycryptodomex
+# Install pywin32-ctypes (needed by pyinstaller)
+$PYTHON -m pip install pywin32-ctypes==0.1.2
 
 # install PySocks
-$PYTHON -m pip install win_inet_pton
+$PYTHON -m pip install win_inet_pton==1.0.1
 
-# install websocket (python2)
-$PYTHON -m pip install websocket-client
+$PYTHON -m pip install -r $here/../deterministic-build/requirements-binaries.txt
+
+# Install PyInstaller
+$PYTHON -m pip install https://github.com/ecdsa/pyinstaller/archive/fix_2952.zip
+
+# Install ZBar
+download_if_not_exist $ZBAR_FILENAME "$ZBAR_URL"
+verify_hash $ZBAR_FILENAME "$ZBAR_SHA256"
+wine "$PWD/$ZBAR_FILENAME" /S
 
 # Upgrade setuptools (so Electrum can be installed later)
 $PYTHON -m pip install setuptools --upgrade
 
 # Install NSIS installer
-wget -q -O nsis.exe "$NSIS_URL"
-verify_hash nsis.exe $NSIS_SHA256
-wine nsis.exe /S
+download_if_not_exist $NSIS_FILENAME "$NSIS_URL"
+verify_hash $NSIS_FILENAME "$NSIS_SHA256"
+wine "$PWD/$NSIS_FILENAME" /S
+
+download_if_not_exist $LIBUSB_FILENAME "$LIBUSB_URL"
+verify_hash $LIBUSB_FILENAME "$LIBUSB_SHA256"
+7z x -olibusb $LIBUSB_FILENAME -aos
+
+cp libusb/MS32/dll/libusb-1.0.dll $WINEPREFIX/drive_c/python$PYTHON_VERSION/
 
 # Install UPX
 #wget -O upx.zip "https://downloads.sourceforge.net/project/upx/upx/3.08/upx308w.zip"
@@ -109,5 +125,4 @@ wine nsis.exe /S
 # add dlls needed for pyinstaller:
 cp $WINEPREFIX/drive_c/python$PYTHON_VERSION/Lib/site-packages/PyQt5/Qt/bin/* $WINEPREFIX/drive_c/python$PYTHON_VERSION/
 
-
-echo "Wine is configured. Please run prepare-pyinstaller.sh"
+echo "Wine is configured."

contrib/deterministic-build/requirements-binaries.txt

@@ -0,0 +1,5 @@
+pycryptodomex==3.4.12
+PyQt5==5.10
+sip==4.19.7
+six==1.11.0
+websocket-client==0.46.0

contrib/deterministic-build/requirements-hw.txt

@@ -0,0 +1,18 @@
+btchip-python==0.1.24
+certifi==2018.1.18
+chardet==3.0.4
+click==6.7
+Cython==0.27.3
+ecdsa==0.13
+hidapi==0.7.99.post21
+idna==2.6
+keepkey==4.0.2
+libusb1==1.6.4
+mnemonic==0.18
+pbkdf2==1.3
+protobuf==3.5.1
+pyblake2==1.1.0
+requests==2.18.4
+six==1.11.0
+trezor==0.9.0
+urllib3==1.22

contrib/deterministic-build/requirements.txt

@@ -1,13 +1,13 @@
-certifi==2017.11.5
+certifi==2018.1.18
 chardet==3.0.4
 dnspython==1.15.0
 ecdsa==0.13
 idna==2.6
 jsonrpclib-pelix==0.3.1
 pbkdf2==1.3
-protobuf==3.5.0.post1
+protobuf==3.5.1
 pyaes==1.6.1
-PySocks==1.6.7
+PySocks==1.6.8
 qrcode==5.3
 requests==2.18.4
 six==1.11.0

contrib/freeze_packages.sh

@@ -6,17 +6,17 @@ contrib=$(dirname "$0")
 
 which virtualenv > /dev/null 2>&1 || { echo "Please install virtualenv" && exit 1; }
 
-rm $venv_dir -rf
-virtualenv $venv_dir
+for i in '' '-hw' '-binaries'; do
+    rm "$venv_dir" -rf
+    virtualenv -p $(which python3) $venv_dir
 
-source $venv_dir/bin/activate
+    source $venv_dir/bin/activate
 
-echo "Installing dependencies"
+    echo "Installing $i dependencies"
 
-pushd $contrib/..
-python setup.py install
-popd
+    python -m pip install -r $contrib/requirements/requirements${i}.txt --upgrade
 
-pip freeze | sed '/^Electrum/ d' > $contrib/requirements.txt
+    pip freeze | sed '/^Electrum/ d' > $contrib/deterministic-build/requirements${i}.txt
+done
 
-echo "Updated requirements"
+echo "Done. Updated requirements"

contrib/make_packages

@@ -1,12 +1,13 @@
 #!/bin/bash
 
 contrib=$(dirname "$0")
+test -n "$contrib" -a -d "$contrib" || exit
 
 whereis pip3
 if [ $? -ne 0 ] ; then echo "Install pip3" ; exit ; fi
 
-rm $contrib/../packages/ -r
+rm "$contrib"/../packages/ -r
 
 #Install pure python modules in electrum directory
-pip3 install -r $contrib/requirements.txt -t $contrib/../packages
+pip3 install -r $contrib/deterministic-build/requirements.txt -t $contrib/../packages
 

contrib/requirements/requirements-binaries.txt

@@ -0,0 +1,3 @@
+PyQt5
+pycryptodomex
+websocket-client

contrib/requirements/requirements-hw.txt

@@ -0,0 +1,4 @@
+Cython>=0.27
+trezor>=0.9.0
+keepkey
+btchip-python

contrib/requirements/requirements.txt

@@ -0,0 +1,9 @@
+pyaes>=0.1a1
+ecdsa>=0.9
+pbkdf2
+requests
+qrcode
+protobuf
+dnspython
+jsonrpclib-pelix
+PySocks>=1.6.6

electrum-env

@@ -9,6 +9,8 @@
 # python-qt and its dependencies will still need to be installed with
 # your package manager.
 
+PYTHON_VER="$(python3 -c 'import sys; print(sys.version[:3])')"
+
 if [ -e ./env/bin/activate ]; then
     source ./env/bin/activate
 else
@@ -17,7 +19,7 @@ else
     python3 setup.py install
 fi
 
-export PYTHONPATH="/usr/local/lib/python3.5/site-packages:$PYTHONPATH"
+export PYTHONPATH="/usr/local/lib/python${PYTHON_VER}/site-packages:$PYTHONPATH"
 
 ./electrum "$@"
 

electrum-zcl

@@ -90,10 +90,12 @@ if is_local or is_android or is_macOS:
     imp.load_module('electrum_plugins', *imp.find_module('plugins'))
 
 
-from electrum import bitcoin
+
+from electrum import bitcoin, util
+from electrum import constants
 from electrum import SimpleConfig, Network
 from electrum.wallet import Wallet, Imported_Wallet
-from electrum.storage import WalletStorage
+from electrum.storage import WalletStorage, get_derivation_used_for_hw_device_encryption
 from electrum.util import print_msg, print_stderr, json_encode, json_decode
 from electrum.util import set_verbosity, InvalidPassword
 from electrum.commands import get_parser, known_commands, Commands, config_variables
@@ -163,6 +165,8 @@ def run_non_RPC(config):
             print_msg("Recovering wallet...")
             wallet.synchronize()
             wallet.wait_until_synchronized()
+            wallet.stop_threads()
+            # note: we don't wait for SPV
             msg = "Recovery successful" if wallet.is_found() else "Found no history for this wallet"
         else:
             msg = "This wallet was restored offline. It may contain more addresses than displayed."
@@ -195,7 +199,10 @@ def init_daemon(config_options):
         print_msg("Type 'electrum create' to create a new wallet, or provide a path to a wallet with the -w option")
         sys.exit(0)
     if storage.is_encrypted():
-        if config.get('password'):
+        if storage.is_encrypted_with_hw_device():
+            plugins = init_plugins(config, 'cmdline')
+            password = get_password_for_hw_device_encrypted_storage(plugins)
+        elif config.get('password'):
             password = config.get('password')
         else:
             password = prompt_password('Password:', False)
@@ -222,7 +229,7 @@ def init_cmdline(config_options, server):
     if cmdname in ['payto', 'paytomany'] and config.get('broadcast'):
         cmd.requires_network = True
 
-    # instanciate wallet for command-line
+    # instantiate wallet for command-line
     storage = WalletStorage(config.get_wallet_path())
 
     if cmd.requires_wallet and not storage.file_exists():
@@ -239,7 +246,10 @@ def init_cmdline(config_options, server):
     # commands needing password
     if (cmd.requires_wallet and storage.is_encrypted() and server is None)\
        or (cmd.requires_password and (storage.get('use_encryption') or storage.is_encrypted())):
-        if config.get('password'):
+        if storage.is_encrypted_with_hw_device():
+            # this case is handled later in the control flow
+            password = None
+        elif config.get('password'):
             password = config.get('password')
         else:
             password = prompt_password('Password:', False)
@@ -258,19 +268,57 @@ def init_cmdline(config_options, server):
     return cmd, password
 
 
-def run_offline_command(config, config_options):
+def get_connected_hw_devices(plugins):
+    support = plugins.get_hardware_support()
+    if not support:
+        print_msg('No hardware wallet support found on your system.')
+        sys.exit(1)
+    # scan devices
+    devices = []
+    devmgr = plugins.device_manager
+    for name, description, plugin in support:
+        try:
+            u = devmgr.unpaired_device_infos(None, plugin)
+        except:
+            devmgr.print_error("error", name)
+            continue
+        devices += list(map(lambda x: (name, x), u))
+    return devices
+
+
+def get_password_for_hw_device_encrypted_storage(plugins):
+    devices = get_connected_hw_devices(plugins)
+    if len(devices) == 0:
+        print_msg("Error: No connected hw device found. Can not decrypt this wallet.")
+        sys.exit(1)
+    elif len(devices) > 1:
+        print_msg("Warning: multiple hardware devices detected. "
+                  "The first one will be used to decrypt the wallet.")
+    # FIXME we use the "first" device, in case of multiple ones
+    name, device_info = devices[0]
+    plugin = plugins.get_plugin(name)
+    derivation = get_derivation_used_for_hw_device_encryption()
+    xpub = plugin.get_xpub(device_info.device.id_, derivation, 'standard', plugin.handler)
+    password = keystore.Xpub.get_pubkey_from_xpub(xpub, ())
+    return password
+
+
+def run_offline_command(config, config_options, plugins):
     cmdname = config.get('cmd')
     cmd = known_commands[cmdname]
     password = config_options.get('password')
     if cmd.requires_wallet:
         storage = WalletStorage(config.get_wallet_path())
         if storage.is_encrypted():
+            if storage.is_encrypted_with_hw_device():
+                password = get_password_for_hw_device_encrypted_storage(plugins)
+                config_options['password'] = password
             storage.decrypt(password)
         wallet = Wallet(storage)
     else:
         wallet = None
     # check password
-    if cmd.requires_password and storage.get('use_encryption'):
+    if cmd.requires_password and wallet.has_password():
         try:
             seed = wallet.check_password(password)
         except InvalidPassword:
@@ -281,7 +329,8 @@ def run_offline_command(config, config_options):
     # arguments passed to function
     args = [config.get(x) for x in cmd.params]
     # decode json arguments
-    args = list(map(json_decode, args))
+    if cmdname not in ('setconfig',):
+        args = list(map(json_decode, args))
     # options
     kwargs = {}
     for x in cmd.options:
@@ -298,8 +347,10 @@ def init_plugins(config, gui_name):
     from electrum.plugins import Plugins
     return Plugins(config, is_local or is_android, gui_name)
 
-if __name__ == '__main__':
 
+if __name__ == '__main__':
+    # The hook will only be used in the Qt GUI right now
+    util.setup_thread_excepthook()
     # on osx, delete Process Serial Number arg generated for apps launched in Finder
     sys.argv = list(filter(lambda x: not x.startswith('-psn'), sys.argv))
 
@@ -364,7 +415,7 @@ if __name__ == '__main__':
     cmdname = config.get('cmd')
 
     if config.get('testnet'):
-        bitcoin.NetworkConstants.set_testnet()
+        constants.set_testnet()
 
     # run non-RPC commands separately
     if cmdname in ['create', 'restore']:
@@ -430,8 +481,8 @@ if __name__ == '__main__':
                 print_msg("Daemon not running; try 'electrum daemon start'")
                 sys.exit(1)
             else:
-                init_plugins(config, 'cmdline')
-                result = run_offline_command(config, config_options)
+                plugins = init_plugins(config, 'cmdline')
+                result = run_offline_command(config, config_options, plugins)
                 # print result
     if isinstance(result, str):
         print_msg(result)

gui/kivy/Readme.md

@@ -0,0 +1,80 @@
+# Kivy GUI
+
+The Kivy GUI is used with Electrum on Android devices. To generate an APK file, follow these instructions.
+
+## 1. Install python-for-android (p4a)
+p4a is used to package Electrum, Python, SDL and a bootstrap Java app into an APK file. 
+We patched p4a to add some functionality we need for Electrum. Until those changes are
+merged into p4a, you need to merge them locally (into the master branch):
+
+2. [kivy/python-for-android#1217](https://github.com/kivy/python-for-android/pull/1217)
+
+Something like this should work:
+
+```sh
+cd /opt
+git clone https://github.com/kivy/python-for-android
+cd python-for-android
+git remote add agilewalker https://github.com/agilewalker/python-for-android
+git checkout a036f4442b6a23
+git fetch agilewalker
+git merge agilewalker/master
+```
+
+## 2. Install buildozer
+2.1 Buildozer is a frontend to p4a. Luckily we don't need to patch it:
+
+```sh
+cd /opt
+git clone https://github.com/kivy/buildozer
+cd buildozer
+sudo python3 setup.py install
+```
+
+2.2 Download the [Crystax NDK](https://www.crystax.net/en/download) manually.
+Extract into `/opt/crystax-ndk-10.3.2`
+
+## 3. Update the Android SDK build tools
+
+### Method 1: Using the GUI
+
+  Start the Android SDK manager in GUI mode:
+
+      ~/.buildozer/android/platform/android-sdk-20/tools/android
+
+  Check the latest SDK available and install it.
+  Close the SDK manager.
+  Reopen the SDK manager, scroll to the bottom and install the latest build tools (probably v27)
+  Install "Android Support Library Repository" from the SDK manager.
+
+### Method 2: Using the command line:
+
+  Repeat the following command until there is nothing to install:
+
+      ~/.buildozer/android/platform/android-sdk-20/tools/android update sdk -u -t tools,platform-tools
+
+  Install Build Tools, android API 19 and Android Support Library:
+
+      ~/.buildozer/android/platform/android-sdk-20/tools/android update sdk -u -t build-tools-27.0.3,android-19,extra-android-m2repository
+
+
+
+## 5. Create the UI Atlas
+In the `gui/kivy` directory of Electrum, run `make theming`.
+
+## 6. Download Electrum dependencies
+Run `contrib/make_packages`.
+
+## 7. Build the APK
+Run `contrib/make_apk`.
+
+# FAQ
+## Why do I get errors like `package me.dm7.barcodescanner.zxing does not exist` while compiling?
+Update your Android build tools to version 27 like described above.
+
+## Why do I get errors like  `(use -source 7 or higher to enable multi-catch statement)` while compiling?
+Make sure that your p4a installation includes commit a3cc78a6d1a107cd3b6bd28db8b80f89e3ecddd2.
+Also make sure you have recent SDK tools and platform-tools
+
+## I changed something but I don't see any differences on the phone. What did I do wrong?
+You probably need to clear the cache: `rm -rf .buildozer/android/platform/build/{build,dists}`

gui/kivy/Readme.txt

@@ -1,24 +0,0 @@
-Before compiling, create packages: `contrib/make_packages`
-
-Commands::
-
-    `make theming` to make a atlas out of a list of pngs
-
-    `make apk` to make a apk
-
-
-If something in included modules like kivy or any other module changes
-then you need to rebuild the distribution. To do so:
-
-  rm -rf .buildozer/android/platform/python-for-android/dist
-
-
-how to build with ssl:
-
-  rm -rf .buildozer/android/platform/build/
-  ./contrib/make_apk
-  pushd /opt/electrum/.buildozer/android/platform/build/build/libs_collections/Electrum/armeabi-v7a
-  cp libssl1.0.2g.so /opt/crystax-ndk-10.3.2/sources/openssl/1.0.2g/libs/armeabi-v7a/libssl.so
-  cp libcrypto1.0.2g.so /opt/crystax-ndk-10.3.2/sources/openssl/1.0.2g/libs/armeabi-v7a/libcrypto.so
-  popd
-  ./contrib/make_apk

gui/kivy/i18n.py

@@ -1,21 +1,22 @@
 import gettext
 
+
 class _(str):
 
     observers = set()
     lang = None
 
-    def __new__(cls, s, *args, **kwargs):
+    def __new__(cls, s):
         if _.lang is None:
             _.switch_lang('en')
-        t = _.translate(s, *args, **kwargs)
+        t = _.translate(s)
         o = super(_, cls).__new__(cls, t)
         o.source_text = s
         return o
 
     @staticmethod
     def translate(s, *args, **kwargs):
-        return _.lang(s).format(args, kwargs)
+        return _.lang(s)
 
     @staticmethod
     def bind(label):

gui/kivy/main.kv

@@ -239,7 +239,7 @@
         self.screen.show_menu(args[0]) if self.state == 'down' else self.screen.hide_menu()
     canvas.before:
         Color:
-            rgba: (0.192, .498, 0.745, 1) if self.state == 'down' else (0.3, 0.3, 0.3, 1)
+            rgba: (0.192, .498, 0.745, 1) if self.state == 'down' else (0.15, 0.15, 0.17, 1)
         Rectangle:
             size: self.size
             pos: self.pos
@@ -397,7 +397,7 @@
             slide: 1
         CleanHeader:
             id: history_tab
-            text: _('History')
+            text: _('Balance')
             slide: 2
         CleanHeader:
             id: receive_tab

gui/kivy/main_window.py

@@ -82,6 +82,10 @@ class ElectrumWindow(App):
     server_port = StringProperty('')
     num_chains = NumericProperty(0)
     blockchain_name = StringProperty('')
+    fee_status = StringProperty('Fee')
+    balance = StringProperty('')
+    fiat_balance = StringProperty('')
+    is_fiat = BooleanProperty(False)
     blockchain_checkpoint = NumericProperty(0)
 
     auto_connect = BooleanProperty(False)
@@ -95,8 +99,8 @@ class ElectrumWindow(App):
         from .uix.dialogs.choice_dialog import ChoiceDialog
         protocol = 's'
         def cb2(host):
-            from electrum.bitcoin import NetworkConstants
-            pp = servers.get(host, NetworkConstants.DEFAULT_PORTS)
+            from electrum import constants
+            pp = servers.get(host, constants.net.DEFAULT_PORTS)
             port = pp.get(protocol, '')
             popup.ids.host.text = host
             popup.ids.port.text = port
@@ -175,8 +179,10 @@ class ElectrumWindow(App):
     def btc_to_fiat(self, amount_str):
         if not amount_str:
             return ''
+        if not self.fx.is_enabled():
+            return ''
         rate = self.fx.exchange_rate()
-        if not rate:
+        if rate.is_nan():
             return ''
         fiat_amount = self.get_amount(amount_str + ' ' + self.base_unit) * rate / pow(10, 8)
         return "{:.2f}".format(fiat_amount).rstrip('0').rstrip('.')
@@ -185,7 +191,7 @@ class ElectrumWindow(App):
         if not fiat_amount:
             return ''
         rate = self.fx.exchange_rate()
-        if not rate:
+        if rate.is_nan():
             return ''
         satoshis = int(pow(10,8) * Decimal(fiat_amount) / Decimal(rate))
         return format_satoshis_plain(satoshis, self.decimal_point())
@@ -271,6 +277,7 @@ class ElectrumWindow(App):
         # cached dialogs
         self._settings_dialog = None
         self._password_dialog = None
+        self.fee_status = self.electrum_config.get_fee_status()
 
     def wallet_name(self):
         return os.path.basename(self.wallet.storage.path) if self.wallet else ' '
@@ -457,6 +464,7 @@ class ElectrumWindow(App):
         if self.network:
             interests = ['updated', 'status', 'new_transaction', 'verified', 'interfaces']
             self.network.register_callback(self.on_network_event, interests)
+            self.network.register_callback(self.on_fee, ['fee'])
             self.network.register_callback(self.on_quotes, ['on_quotes'])
             self.network.register_callback(self.on_history, ['on_history'])
         # URI passed in config
@@ -631,17 +639,17 @@ class ElectrumWindow(App):
             if not self.wallet.up_to_date or server_height == 0:
                 status = _("Synchronizing...")
             elif server_lag > 1:
-                status = _("Server lagging (%d blocks)"%server_lag)
+                status = _("Server lagging")
             else:
-                c, u, x = self.wallet.get_balance()
-                text = self.format_amount(c+x+u)
-                status = str(text.strip() + ' ' + self.base_unit)
+                status = ''
         else:
             status = _("Disconnected")
-
-        n = self.wallet.basename()
-        self.status = '[size=15dp]%s[/size]\n%s' %(n, status)
-        #fiat_balance = self.fx.format_amount_and_units(c+u+x) or ''
+        self.status = self.wallet.basename() + (' [size=15dp](%s)[/size]'%status if status else '')
+        # balance
+        c, u, x = self.wallet.get_balance()
+        text = self.format_amount(c+x+u)
+        self.balance = str(text.strip()) + ' [size=22dp]%s[/size]'% self.base_unit
+        self.fiat_balance = self.fx.format_amount(c+u+x) + ' [size=22dp]%s[/size]'% self.fx.ccy
 
     def get_max_amount(self):
         inputs = self.wallet.get_spendable_coins(None, self.electrum_config)
@@ -684,9 +692,6 @@ class ElectrumWindow(App):
     def on_resume(self):
         if self.nfcscanner:
             self.nfcscanner.nfc_enable()
-        # workaround p4a bug:
-        # show an empty info bubble, to refresh the display
-        self.show_info_bubble('', duration=0.1, pos=(0,0), width=1, arrow_pos=None)
 
     def on_size(self, instance, value):
         width, height = value
@@ -831,6 +836,16 @@ class ElectrumWindow(App):
         popup = AmountDialog(show_max, amount, cb)
         popup.open()
 
+    def fee_dialog(self, label, dt):
+        from .uix.dialogs.fee_dialog import FeeDialog
+        def cb():
+            self.fee_status = self.electrum_config.get_fee_status()
+        fee_dialog = FeeDialog(self, self.electrum_config, cb)
+        fee_dialog.open()
+
+    def on_fee(self, event, *arg):
+        self.fee_status = self.electrum_config.get_fee_status()
+
     def protected(self, msg, f, args):
         if self.wallet.has_password():
             self.password_dialog(msg, f, args)
@@ -846,7 +861,7 @@ class ElectrumWindow(App):
     def _delete_wallet(self, b):
         if b:
             basename = os.path.basename(self.wallet.storage.path)
-            self.protected(_("Enter your PIN code to confirm deletion of %s") % basename, self.__delete_wallet, ())
+            self.protected(_("Enter your PIN code to confirm deletion of {}").format(basename), self.__delete_wallet, ())
 
     def __delete_wallet(self, pw):
         wallet_path = self.get_wallet_path()
@@ -928,6 +943,10 @@ class ElectrumWindow(App):
                 return
             if not self.wallet.can_export():
                 return
-            key = str(self.wallet.export_private_key(addr, password)[0])
-            pk_label.data = key
+            try:
+                key = str(self.wallet.export_private_key(addr, password)[0])
+                pk_label.data = key
+            except InvalidPassword:
+                self.show_error("Invalid PIN")
+                return
         self.protected(_("Enter your PIN code in order to decrypt your private key"), show_private_key, (addr, pk_label))

gui/kivy/tools/buildozer.spec

@@ -31,7 +31,7 @@ version.filename = %(source.dir)s/contrib/versions.py
 #version = 1.9.8
 
 # (list) Application requirements
-requirements = python3crystax, android, openssl, plyer, kivy==master
+requirements = python3crystax==3.6, android, openssl, plyer, kivy==master
 
 # (str) Presplash of the application
 #presplash.filename = %(source.dir)s/gui/kivy/theming/splash.png

gui/kivy/uix/dialogs/amount_dialog.py

@@ -13,21 +13,35 @@ Builder.load_string('''
         anchor_x: 'center'
         BoxLayout:
             orientation: 'vertical'
-            size_hint: 0.8, 1
+            size_hint: 0.9, 1
+            Widget:
+                size_hint: 1, 0.2
             BoxLayout:
                 size_hint: 1, None
                 height: '80dp'
-                Label:
-                    id: a
-                    btc_text: (kb.amount + ' ' + app.base_unit) if kb.amount else ''
-                    fiat_text: (kb.fiat_amount + ' ' + app.fiat_unit) if kb.fiat_amount else ''
-                    text1: ((self.fiat_text if kb.is_fiat else self.btc_text) if app.fiat_unit else self.btc_text) if self.btc_text else ''
-                    text2: ((self.btc_text if kb.is_fiat else self.fiat_text) if app.fiat_unit else '') if self.btc_text else ''
-                    text: self.text1 + "\\n" + "[color=#8888ff]" + self.text2 + "[/color]"
+                Button:
+                    background_color: 0, 0, 0, 0
+                    id: btc
+                    text: kb.amount + ' ' + app.base_unit
+                    color: (0.7, 0.7, 1, 1) if kb.is_fiat else (1, 1, 1, 1)
                     halign: 'right'
                     size_hint: 1, None
-                    font_size: '22dp'
-                    height: '80dp'
+                    font_size: '20dp'
+                    height: '48dp'
+                    on_release:
+                        kb.is_fiat = False
+                Button:
+                    background_color: 0, 0, 0, 0
+                    id: fiat
+                    text: kb.fiat_amount + ' ' + app.fiat_unit
+                    color: (1, 1, 1, 1) if kb.is_fiat else (0.7, 0.7, 1, 1)
+                    halign: 'right'
+                    size_hint: 1, None
+                    font_size: '20dp'
+                    height: '48dp'
+                    disabled: not app.fx.is_enabled()
+                    on_release:
+                        kb.is_fiat = True
             Widget:
                 size_hint: 1, 0.2
             GridLayout:
@@ -65,6 +79,9 @@ Builder.load_string('''
                     text: '0'
                 KButton:
                     text: '<'
+                Widget:
+                    size_hint: 1, None
+                    height: '48dp'
                 Button:
                     id: but_max
                     opacity: 1 if root.show_max else 0
@@ -75,13 +92,6 @@ Builder.load_string('''
                     on_release:
                         kb.is_fiat = False
                         kb.amount = app.get_max_amount()
-                Button:
-                    id: button_fiat
-                    size_hint: 1, None
-                    height: '48dp'
-                    text: (app.base_unit if not kb.is_fiat else app.fiat_unit) if app.fiat_unit else ''
-                    on_release:
-                        if app.fiat_unit: popup.toggle_fiat(kb)
                 Button:
                     size_hint: 1, None
                     height: '48dp'
@@ -102,7 +112,7 @@ Builder.load_string('''
                     height: '48dp'
                     text: _('OK')
                     on_release:
-                        root.callback(a.btc_text)
+                        root.callback(btc.text if kb.amount else '')
                         popup.dismiss()
 ''')
 
@@ -117,9 +127,6 @@ class AmountDialog(Factory.Popup):
         if amount:
             self.ids.kb.amount = amount
 
-    def toggle_fiat(self, a):
-        a.is_fiat = not a.is_fiat
-
     def update_amount(self, c):
         kb = self.ids.kb
         amount = kb.fiat_amount if kb.is_fiat else kb.amount

gui/kivy/uix/dialogs/bump_fee_dialog.py

@@ -3,7 +3,6 @@ from kivy.factory import Factory
 from kivy.properties import ObjectProperty
 from kivy.lang import Builder
 
-from electrum.util import fee_levels
 from electrum_gui.kivy.i18n import _
 
 Builder.load_string('''
@@ -29,7 +28,11 @@ Builder.load_string('''
                 text: _('New Fee')
                 value: ''
         Label:
-            id: tooltip
+            id: tooltip1
+            text: ''
+            size_hint_y: None
+        Label:
+            id: tooltip2
             text: ''
             size_hint_y: None
         Slider:
@@ -72,39 +75,39 @@ class BumpFeeDialog(Factory.Popup):
         self.tx_size = size
         self.callback = callback
         self.config = app.electrum_config
-        self.fee_step = self.config.max_fee_rate() / 10
-        self.dynfees = self.config.get('dynamic_fees', True) and self.app.network
+        self.mempool = self.config.use_mempool_fees()
+        self.dynfees = self.config.is_dynfee() and self.app.network and self.config.has_dynamic_fees_ready()
         self.ids.old_fee.value = self.app.format_amount_and_units(self.init_fee)
         self.update_slider()
         self.update_text()
 
     def update_text(self):
-        value = int(self.ids.slider.value)
-        self.ids.new_fee.value = self.app.format_amount_and_units(self.get_fee())
-        if self.dynfees:
-            value = int(self.ids.slider.value)
-            self.ids.tooltip.text = fee_levels[value]
+        fee = self.get_fee()
+        self.ids.new_fee.value = self.app.format_amount_and_units(fee)
+        pos = int(self.ids.slider.value)
+        fee_rate = self.get_fee_rate()
+        text, tooltip = self.config.get_fee_text(pos, self.dynfees, self.mempool, fee_rate)
+        self.ids.tooltip1.text = text
+        self.ids.tooltip2.text = tooltip
 
     def update_slider(self):
         slider = self.ids.slider
+        maxp, pos, fee_rate = self.config.get_fee_slider(self.dynfees, self.mempool)
+        slider.range = (0, maxp)
+        slider.step = 1
+        slider.value = pos
+
+    def get_fee_rate(self):
+        pos = int(self.ids.slider.value)
         if self.dynfees:
-            slider.range = (0, 4)
-            slider.step = 1
-            slider.value = 3
+            fee_rate = self.config.depth_to_fee(pos) if self.mempool else self.config.eta_to_fee(pos)
         else:
-            slider.range = (1, 10)
-            slider.step = 1
-            rate = self.init_fee*1000//self.tx_size
-            slider.value = min( rate * 2 // self.fee_step, 10)
+            fee_rate = self.config.static_fee(pos)
+        return fee_rate
 
     def get_fee(self):
-        value = int(self.ids.slider.value)
-        if self.dynfees:
-            if self.config.has_fee_estimates():
-                dynfee = self.config.dynfee(value)
-                return int(dynfee * self.tx_size // 1000)
-        else:
-            return int(value*self.fee_step * self.tx_size // 1000)
+        fee_rate = self.get_fee_rate()
+        return int(fee_rate * self.tx_size // 1000)
 
     def on_ok(self):
         new_fee = self.get_fee()

gui/kivy/uix/dialogs/fee_dialog.py

@@ -3,7 +3,6 @@ from kivy.factory import Factory
 from kivy.properties import ObjectProperty
 from kivy.lang import Builder
 
-from electrum.util import fee_levels
 from electrum_gui.kivy.i18n import _
 
 Builder.load_string('''
@@ -12,29 +11,46 @@ Builder.load_string('''
     title: _('Transaction Fees')
     size_hint: 0.8, 0.8
     pos_hint: {'top':0.9}
+    method: 0
     BoxLayout:
         orientation: 'vertical'
         BoxLayout:
             orientation: 'horizontal'
             size_hint: 1, 0.5
             Label:
-                id: fee_per_kb
+                text: _('Method') + ':'
+            Button:
+                text: _('Mempool') if root.method == 2 else _('ETA') if root.method == 1 else _('Static')
+                background_color: (0,0,0,0)
+                bold: True
+                on_release:
+                    root.method  = (root.method + 1) % 3
+                    root.update_slider()
+                    root.update_text()
+        BoxLayout:
+            orientation: 'horizontal'
+            size_hint: 1, 0.5
+            Label:
+                text: (_('Target') if root.method > 0 else _('Fee')) + ':'
+            Label:
+                id: fee_target
                 text: ''
         Slider:
             id: slider
             range: 0, 4
             step: 1
             on_value: root.on_slider(self.value)
+        Widget:
+            size_hint: 1, 0.5
         BoxLayout:
             orientation: 'horizontal'
             size_hint: 1, 0.5
-            Label:
-                text: _('Dynamic Fees')
-            CheckBox:
-                id: dynfees
-                on_active: root.on_checkbox(self.active)
+            TopLabel:
+                id: fee_estimate
+                text: ''
+                font_size: '14dp'
         Widget:
-            size_hint: 1, 1
+            size_hint: 1, 0.5
         BoxLayout:
             orientation: 'horizontal'
             size_hint: 1, 0.5
@@ -60,53 +76,57 @@ class FeeDialog(Factory.Popup):
         self.config = config
         self.fee_rate = self.config.fee_per_kb()
         self.callback = callback
-        self.dynfees = self.config.get('dynamic_fees', True)
-        self.ids.dynfees.active = self.dynfees
+        mempool = self.config.use_mempool_fees()
+        dynfees = self.config.is_dynfee()
+        self.method = (2 if mempool else 1) if dynfees else 0
         self.update_slider()
         self.update_text()
 
     def update_text(self):
-        value = int(self.ids.slider.value)
-        self.ids.fee_per_kb.text = self.get_fee_text(value)
+        pos = int(self.ids.slider.value)
+        dynfees, mempool = self.get_method()
+        if self.method == 2:
+            fee_rate = self.config.depth_to_fee(pos)
+            target, estimate = self.config.get_fee_text(pos, dynfees, mempool, fee_rate)
+            msg = 'In the current network conditions, in order to be positioned %s, a transaction will require a fee of %s.' % (target, estimate)
+        elif self.method == 1:
+            fee_rate = self.config.eta_to_fee(pos)
+            target, estimate = self.config.get_fee_text(pos, dynfees, mempool, fee_rate)
+            msg = 'In the last few days, transactions that confirmed %s usually paid a fee of at least %s.' % (target.lower(), estimate)
+        else:
+            fee_rate = self.config.static_fee(pos)
+            target, estimate = self.config.get_fee_text(pos, dynfees, True, fee_rate)
+            msg = 'In the current network conditions, a transaction paying %s would be positioned %s.' % (target, estimate)
+
+        self.ids.fee_target.text = target
+        self.ids.fee_estimate.text = msg
+
+    def get_method(self):
+        dynfees = self.method > 0
+        mempool = self.method == 2
+        return dynfees, mempool
 
     def update_slider(self):
         slider = self.ids.slider
-        if self.dynfees:
-            slider.range = (0, 4)
-            slider.step = 1
-            slider.value = self.config.get('fee_level', 2)
-        else:
-            slider.range = (0, 9)
-            slider.step = 1
-            slider.value = self.config.static_fee_index(self.fee_rate)
-
-    def get_fee_text(self, value):
-        if self.ids.dynfees.active:
-            tooltip = fee_levels[value]
-            if self.config.has_fee_estimates():
-                dynfee = self.config.dynfee(value)
-                tooltip += '\n' + (self.app.format_amount_and_units(dynfee)) + '/kB'
-        else:
-            fee_rate = self.config.static_fee(value)
-            tooltip = self.app.format_amount_and_units(fee_rate) + '/kB'
-            if self.config.has_fee_estimates():
-                i = self.config.reverse_dynfee(fee_rate)
-                tooltip += '\n' + (_('low fee') if i < 0 else 'Within %d blocks'%i)
-        return tooltip
+        dynfees, mempool = self.get_method()
+        maxp, pos, fee_rate = self.config.get_fee_slider(dynfees, mempool)
+        slider.range = (0, maxp)
+        slider.step = 1
+        slider.value = pos
 
     def on_ok(self):
         value = int(self.ids.slider.value)
-        self.config.set_key('dynamic_fees', self.dynfees, False)
-        if self.dynfees:
-            self.config.set_key('fee_level', value, True)
+        dynfees, mempool = self.get_method()
+        self.config.set_key('dynamic_fees', dynfees, False)
+        self.config.set_key('mempool_fees', mempool, False)
+        if dynfees:
+            if mempool:
+                self.config.set_key('depth_level', value, True)
+            else:
+                self.config.set_key('fee_level', value, True)
         else:
             self.config.set_key('fee_per_kb', self.config.static_fee(value), True)
         self.callback()
 
     def on_slider(self, value):
         self.update_text()
-
-    def on_checkbox(self, b):
-        self.dynfees = b
-        self.update_slider()
-        self.update_text()

gui/kivy/uix/dialogs/fx_dialog.py

@@ -106,4 +106,6 @@ class FxDialog(Factory.Popup):
             if ccy != self.fx.get_currency():
                 self.fx.set_currency(ccy)
             self.app.fiat_unit = ccy
+        else:
+            self.app.is_fiat = False
         Clock.schedule_once(lambda dt: self.add_exchanges())

gui/kivy/uix/dialogs/installwizard.py

@@ -135,7 +135,7 @@ Builder.load_string('''
         height: self.minimum_height
         Label:
             color: root.text_color
-            text: _('From %d cosigners')%n.value
+            text: _('From {} cosigners').format(n.value)
         Slider:
             id: n
             range: 2, 5
@@ -143,7 +143,7 @@ Builder.load_string('''
             value: 2
         Label:
             color: root.text_color
-            text: _('Require %d signatures')%m.value
+            text: _('Require {} signatures').format(m.value)
         Slider:
             id: m
             range: 1, n.value
@@ -613,7 +613,7 @@ class RestoreSeedDialog(WizardDialog):
             for c in line.children:
                 if isinstance(c, Button):
                     if c.text in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ':
-                        c.disabled = (c.text.lower() not in p) and last_word
+                        c.disabled = (c.text.lower() not in p) and bool(last_word)
                     elif c.text == ' ':
                         c.disabled = not enable_space
 
@@ -807,7 +807,7 @@ class InstallWizard(BaseWizard, Widget):
         popup.init(message, callback)
         popup.open()
 
-    def request_password(self, run_next):
+    def request_password(self, run_next, force_disable_encrypt_cb=False):
         def callback(pin):
             if pin:
                 self.run('confirm_password', pin, run_next)

gui/kivy/uix/dialogs/settings.py

@@ -8,7 +8,6 @@ from electrum.i18n import languages
 from electrum_gui.kivy.i18n import _
 from electrum.plugins import run_hook
 from electrum import coinchooser
-from electrum.util import fee_levels
 
 from .choice_dialog import ChoiceDialog
 
@@ -49,12 +48,6 @@ Builder.load_string('''
                     description: _("Base unit for Zclassic amounts.")
                     action: partial(root.unit_dialog, self)
                 CardSeparator
-                SettingsItem:
-                    status: root.fee_status()
-                    title: _('Fees') + ': ' + self.status
-                    description: _("Fees paid to the Zclassic miners.")
-                    action: partial(root.fee_dialog, self)
-                CardSeparator
                 SettingsItem:
                     status: root.fx_status()
                     title: _('Fiat Currency') + ': ' + self.status
@@ -113,7 +106,6 @@ class SettingsDialog(Factory.Popup):
         layout.bind(minimum_height=layout.setter('height'))
         # cached dialogs
         self._fx_dialog = None
-        self._fee_dialog = None
         self._proxy_dialog = None
         self._language_dialog = None
         self._unit_dialog = None
@@ -204,18 +196,7 @@ class SettingsDialog(Factory.Popup):
         d.open()
 
     def fee_status(self):
-        if self.config.get('dynamic_fees', True):
-            return fee_levels[self.config.get('fee_level', 2)]
-        else:
-            return self.app.format_amount_and_units(self.config.fee_per_kb()) + '/kB'
-
-    def fee_dialog(self, label, dt):
-        if self._fee_dialog is None:
-            from .fee_dialog import FeeDialog
-            def cb():
-                label.status = self.fee_status()
-            self._fee_dialog = FeeDialog(self.app, self.config, cb)
-        self._fee_dialog.open()
+        return self.config.get_fee_status()
 
     def boolean_dialog(self, name, title, message, dt):
         from .checkbox_dialog import CheckBoxDialog

gui/kivy/uix/dialogs/tx_dialog.py

@@ -20,6 +20,7 @@ Builder.load_string('''
     can_rbf: False
     fee_str: ''
     date_str: ''
+    date_label:''
     amount_str: ''
     tx_hash: ''
     status_str: ''
@@ -46,7 +47,7 @@ Builder.load_string('''
                         text: _('Description') if root.description else ''
                         value: root.description
                     BoxLabel:
-                        text: _('Date') if root.date_str else ''
+                        text: root.date_label
                         value: root.date_str
                     BoxLabel:
                         text: _('Amount sent') if root.is_mine else _('Amount received')
@@ -110,10 +111,13 @@ class TxDialog(Factory.Popup):
         tx_hash, self.status_str, self.description, self.can_broadcast, self.can_rbf, amount, fee, height, conf, timestamp, exp_n = self.wallet.get_tx_info(self.tx)
         self.tx_hash = tx_hash or ''
         if timestamp:
+            self.date_label = _('Date')
             self.date_str = datetime.fromtimestamp(timestamp).isoformat(' ')[:-3]
         elif exp_n:
-            self.date_str = _('Within %d blocks') % exp_n if exp_n > 0 else _('unknown (low fee)')
+            self.date_label = _('Mempool depth')
+            self.date_str = _('{} from tip').format('%.2f MB'%(exp_n/1000000))
         else:
+            self.date_label = ''
             self.date_str = ''
 
         if amount is None:

gui/kivy/uix/screens.py

@@ -87,9 +87,9 @@ class CScreen(Factory.Screen):
         self.add_widget(self.context_menu)
 
 
+# note: this list needs to be kept in sync with another in qt
 TX_ICONS = [
-    "close",
-    "close",
+    "unconfirmed",
     "close",
     "unconfirmed",
     "close",
@@ -143,14 +143,13 @@ class HistoryScreen(CScreen):
         ri.icon = icon
         ri.date = status_str
         ri.message = label
-        ri.value = value or 0
-        ri.amount = self.app.format_amount(value, True) if value is not None else '--'
         ri.confirmations = conf
-        if self.app.fiat_unit and date:
-            rate = self.app.fx.history_rate(date)
-            if rate:
-                s = self.app.fx.value_str(value, rate)
-                ri.quote_text = '' if s is None else s + ' ' + self.app.fiat_unit
+        if value is not None:
+            ri.is_mine = value < 0
+            if value < 0: value = - value
+            ri.amount = self.app.format_amount_and_units(value)
+            if self.app.fiat_unit and date:
+                ri.quote_text = self.app.fx.historical_value_str(value, date) + ' ' + self.app.fx.ccy
         return ri
 
     def update(self, see_all=False):
@@ -162,13 +161,8 @@ class HistoryScreen(CScreen):
         count = 0
         for item in history:
             ri = self.get_card(*item)
-            count += 1
             history_card.add_widget(ri)
 
-        if count == 0:
-            msg = _('This screen shows your list of transactions. It is currently empty.')
-            history_card.add_widget(EmptyLabel(text=msg))
-
 
 class SendScreen(CScreen):
 
@@ -379,6 +373,8 @@ class ReceiveScreen(CScreen):
 
     def save_request(self):
         addr = self.screen.address
+        if not addr:
+            return
         amount = self.screen.amount
         message = self.screen.message
         amount = self.app.get_amount(amount) if amount else 0
@@ -521,7 +517,12 @@ class AddressScreen(CScreen):
     def update(self):
         self.menu_actions = [('Receive', self.do_show), ('Details', self.do_view)]
         wallet = self.app.wallet
-        _list = wallet.get_change_addresses() if self.screen.show_change else wallet.get_receiving_addresses()
+        if self.screen.show_change == 0:
+            _list = wallet.get_receiving_addresses()
+        elif self.screen.show_change == 1:
+            _list = wallet.get_change_addresses()
+        else:
+            _list = wallet.get_addresses()
         search = self.screen.message
         container = self.screen.ids.search_container
         container.clear_widgets()

gui/kivy/uix/ui_screens/address.kv

@@ -24,33 +24,18 @@
             shorten: True
         Widget
         AddressLabel:
-            text: root.memo
+            text: (root.amount if root.status == 'Funded' else root.status) + '     ' + root.memo
             color: .699, .699, .699, 1
             font_size: '13sp'
             shorten: True
         Widget
-    BoxLayout:
-        spacing: '8dp'
-        height: '32dp'
-        orientation: 'vertical'
-        Widget
-        AddressLabel:
-            text: root.amount
-            halign: 'right'
-            font_size: '15sp'
-        Widget
-        AddressLabel:
-            text: root.status
-            halign: 'right'
-            font_size: '13sp'
-            color: .699, .699, .699, 1 
 
 AddressScreen:
     id: addr_screen
     name: 'address'
     message: ''
     pr_status: 'Pending'
-    show_change: False
+    show_change: 0
     show_used: 0
     on_message:
         self.parent.update()
@@ -70,9 +55,9 @@ AddressScreen:
                 spacing: '5dp'
                 AddressButton:
                     id: search
-                    text: _('Change') if root.show_change else _('Receiving')
+                    text: {0:_('Receiving'), 1:_('Change'), 2:_('All')}[root.show_change]
                     on_release:
-                        root.show_change = not root.show_change
+                        root.show_change = (root.show_change + 1) % 3
                         Clock.schedule_once(lambda dt: app.address_screen.update())
             AddressFilter:
                 opacity: 1
@@ -103,4 +88,3 @@ AddressScreen:
                 id: search_container
                 size_hint_y: None
                 height: self.minimum_height
-                spacing: '2dp'

gui/kivy/uix/ui_screens/history.kv

@@ -19,56 +19,57 @@
 <HistoryItem@CardItem>
     icon: 'atlas://gui/kivy/theming/light/important'
     message: ''
-    value: 0
+    is_mine: True
     amount: '--'
-    amount_color: '#FF6657' if self.value < 0 else '#2EA442'
+    action: _('Sent') if self.is_mine else _('Received')
+    amount_color: '#FF6657' if self.is_mine else '#2EA442'
     confirmations: 0
     date: ''
     quote_text: ''
-    spacing: '9dp'
     Image:
         id: icon
         source: root.icon
         size_hint: None, 1
-        width: self.height *.54
+        allow_stretch: True
+        width: self.height*1.5
         mipmap: True
     BoxLayout:
         orientation: 'vertical'
         Widget
         CardLabel:
-            text: root.date
-            font_size: '14sp'
+            text:
+                u'[color={color}]{s}[/color]'.format(s='<<' if root.is_mine else '>>', color=root.amount_color)\
+                + ' ' + root.action + ' ' + (root.quote_text if app.is_fiat else root.amount)
+            font_size: '15sp'
         CardLabel:
             color: .699, .699, .699, 1
-            font_size: '13sp'
+            font_size: '14sp'
             shorten: True
-            text: root.message
+            text: root.date + '   ' + root.message
         Widget
-    CardLabel:
-        halign: 'right'
-        font_size: '15sp'
-        size_hint: None, 1
-        width: '110sp'
-        markup: True
-        font_name: font_light
-        text:
-            u'[color={amount_color}]{sign}{amount} {unit}[/color]\n'\
-            u'[color=#B2B3B3][size=13sp]{qt}[/size]'\
-            u'[/color]'.format(amount_color=root.amount_color,\
-            amount=root.amount[1:], qt=root.quote_text, sign=root.amount[0],\
-            unit=app.base_unit)
+
 
 
 HistoryScreen:
     name: 'history'
     content: content
-    ScrollView:
-        id: content
-        do_scroll_x: False
-        GridLayout
-            id: history_container
-            cols: 1
-            size_hint: 1, None
-            height: self.minimum_height
-            padding: '12dp'
-            spacing: '2dp'
+    BoxLayout:
+        orientation: 'vertical'
+        Button:
+            background_color: 0, 0, 0, 0
+            text: app.fiat_balance if app.is_fiat else app.balance
+            markup: True
+            color: .9, .9, .9, 1
+            font_size: '30dp'
+            bold: True
+            size_hint: 1, 0.25
+            on_release: app.is_fiat = not app.is_fiat if app.fx.is_enabled() else False
+        ScrollView:
+            id: content
+            do_scroll_x: False
+            size_hint: 1, 0.75
+            GridLayout
+                id: history_container
+                cols: 1
+                size_hint: 1, None
+                height: self.minimum_height

gui/kivy/uix/ui_screens/network.kv

@@ -11,7 +11,7 @@ Popup:
                 height: self.minimum_height
                 padding: '10dp'
                 SettingsItem:
-                    value: _("%d connections.")% app.num_nodes if app.num_nodes else _("Not connected")
+                    value: _("{} connections.").format(app.num_nodes) if app.num_nodes else _("Not connected")
                     title: _("Status") + ': ' + self.value
                     description: _("Connections with Electrum servers")
                     action: lambda x: None
@@ -46,7 +46,7 @@ Popup:
 
                 CardSeparator
                 SettingsItem:
-                    title: _('Fork detected at block %d')%app.blockchain_checkpoint if app.num_chains>1 else _('No fork detected')
+                    title: _('Fork detected at block {}').format(app.blockchain_checkpoint) if app.num_chains>1 else _('No fork detected')
                     fork_description: (_('You are following branch') if app.auto_connect else _("Your server is on branch")) + ' ' + app.blockchain_name
                     description: self.fork_description if app.num_chains>1 else _('Connected nodes are on the same chain')
                     action: app.choose_blockchain_dialog

gui/kivy/uix/ui_screens/send.kv

@@ -71,6 +71,24 @@ SendScreen:
                     text: s.message if s.message else (_('No Description') if root.is_pr else _('Description'))
                     disabled: root.is_pr
                     on_release: Clock.schedule_once(lambda dt: app.description_dialog(s))
+            CardSeparator:
+                opacity: int(not root.is_pr)
+                color: blue_bottom.foreground_color
+            BoxLayout:
+                size_hint: 1, None
+                height: blue_bottom.item_height
+                spacing: '5dp'
+                Image:
+                    source: 'atlas://gui/kivy/theming/light/star_big_inactive'
+                    opacity: 0.7
+                    size_hint: None, None
+                    size: '22dp', '22dp'
+                    pos_hint: {'center_y': .5}
+                BlueButton:
+                    id: fee_e
+                    default_text: _('Fee')
+                    text: app.fee_status
+                    on_release: Clock.schedule_once(lambda dt: app.fee_dialog(s, True))
         BoxLayout:
             size_hint: 1, None
             height: '48dp'

gui/qt/__init__.py

@@ -25,6 +25,7 @@
 
 import signal
 import sys
+import traceback
 
 
 try:
@@ -94,6 +95,8 @@ class ElectrumGui:
         QtCore.QCoreApplication.setAttribute(QtCore.Qt.AA_X11InitThreads)
         if hasattr(QtCore.Qt, "AA_ShareOpenGLContexts"):
             QtCore.QCoreApplication.setAttribute(QtCore.Qt.AA_ShareOpenGLContexts)
+        if hasattr(QGuiApplication, 'setDesktopFileName'):
+            QGuiApplication.setDesktopFileName('electrum.desktop')
         self.config = config
         self.daemon = daemon
         self.plugins = plugins
@@ -190,8 +193,10 @@ class ElectrumGui:
         else:
             try:
                 wallet = self.daemon.load_wallet(path, None)
-            except  BaseException as e:
-                d = QMessageBox(QMessageBox.Warning, _('Error'), 'Cannot load wallet:\n' + str(e))
+            except BaseException as e:
+                traceback.print_exc(file=sys.stdout)
+                d = QMessageBox(QMessageBox.Warning, _('Error'),
+                                _('Cannot load wallet:') + '\n' + str(e))
                 d.exec_()
                 return
             if not wallet:
@@ -208,7 +213,14 @@ class ElectrumGui:
                     return
                 wallet.start_threads(self.daemon.network)
                 self.daemon.add_wallet(wallet)
-            w = self.create_window_for_wallet(wallet)
+            try:
+                w = self.create_window_for_wallet(wallet)
+            except BaseException as e:
+                traceback.print_exc(file=sys.stdout)
+                d = QMessageBox(QMessageBox.Warning, _('Error'),
+                                _('Cannot create window for wallet:') + '\n' + str(e))
+                d.exec_()
+                return
         if uri:
             w.pay_to_URI(uri)
         w.bring_to_top()
@@ -241,8 +253,7 @@ class ElectrumGui:
             return
         except GoBack:
             return
-        except:
-            import traceback
+        except BaseException as e:
             traceback.print_exc(file=sys.stdout)
             return
         self.timer.start()

gui/qt/address_list.py

@@ -24,47 +24,56 @@
 # SOFTWARE.
 import webbrowser
 
-from .util import *
 from electrum.i18n import _
 from electrum.util import block_explorer_URL
 from electrum.plugins import run_hook
 from electrum.bitcoin import is_address
 
+from .util import *
+
 
 class AddressList(MyTreeWidget):
-    filter_columns = [0, 1, 2]  # Address, Label, Balance
+    filter_columns = [0, 1, 2, 3]  # Type, Address, Label, Balance
 
     def __init__(self, parent=None):
-        MyTreeWidget.__init__(self, parent, self.create_menu, [], 1)
+        MyTreeWidget.__init__(self, parent, self.create_menu, [], 2)
         self.refresh_headers()
         self.setSelectionMode(QAbstractItemView.ExtendedSelection)
-        self.show_change = False
+        self.setSortingEnabled(True)
+        self.show_change = 0
         self.show_used = 0
         self.change_button = QComboBox(self)
         self.change_button.currentIndexChanged.connect(self.toggle_change)
-        for t in [_('Receiving'), _('Change')]:
+        for t in [_('All'), _('Receiving'), _('Change')]:
             self.change_button.addItem(t)
         self.used_button = QComboBox(self)
         self.used_button.currentIndexChanged.connect(self.toggle_used)
         for t in [_('All'), _('Unused'), _('Funded'), _('Used')]:
             self.used_button.addItem(t)
 
-    def get_list_header(self):
-        return QLabel(_("Filter ")), self.change_button, self.used_button
+    def get_toolbar_buttons(self):
+        return QLabel(_("Filter:")), self.change_button, self.used_button
+
+    def on_hide_toolbar(self):
+        self.show_change = 0
+        self.show_used = 0
+        self.update()
+
+    def save_toolbar_state(self, state, config):
+        config.set_key('show_toolbar_addresses', state)
 
     def refresh_headers(self):
-        headers = [ _('Address'), _('Label'), _('Balance')]
+        headers = [_('Type'), _('Address'), _('Label'), _('Balance')]
         fx = self.parent.fx
         if fx and fx.get_fiat_address_config():
             headers.extend([_(fx.get_currency()+' Balance')])
         headers.extend([_('Tx')])
         self.update_headers(headers)
 
-    def toggle_change(self, show):
-        show = bool(show)
-        if show == self.show_change:
+    def toggle_change(self, state):
+        if state == self.show_change:
             return
-        self.show_change = show
+        self.show_change = state
         self.update()
 
     def toggle_used(self, state):
@@ -77,10 +86,15 @@ class AddressList(MyTreeWidget):
         self.wallet = self.parent.wallet
         item = self.currentItem()
         current_address = item.data(0, Qt.UserRole) if item else None
-        addr_list = self.wallet.get_change_addresses() if self.show_change else self.wallet.get_receiving_addresses()
+        if self.show_change == 1:
+            addr_list = self.wallet.get_receiving_addresses()
+        elif self.show_change == 2:
+            addr_list = self.wallet.get_change_addresses()
+        else:
+            addr_list = self.wallet.get_addresses()
         self.clear()
         for address in addr_list:
-            num = len(self.wallet.history.get(address,[]))
+            num = len(self.wallet.get_address_history(address))
             is_used = self.wallet.is_used(address)
             label = self.wallet.labels.get(address, '')
             c, u, x = self.wallet.get_addr_balance(address)
@@ -91,23 +105,29 @@ class AddressList(MyTreeWidget):
                 continue
             if self.show_used == 3 and not is_used:
                 continue
-            balance_text = self.parent.format_amount(balance)
+            balance_text = self.parent.format_amount(balance, whitespaces=True)
             fx = self.parent.fx
             if fx and fx.get_fiat_address_config():
                 rate = fx.exchange_rate()
                 fiat_balance = fx.value_str(balance, rate)
-                address_item = QTreeWidgetItem([address, label, balance_text, fiat_balance, "%d"%num])
-                address_item.setTextAlignment(3, Qt.AlignRight)
+                address_item = SortableTreeWidgetItem(['', address, label, balance_text, fiat_balance, "%d"%num])
+                address_item.setTextAlignment(4, Qt.AlignRight)
+                address_item.setFont(4, QFont(MONOSPACE_FONT))
             else:
-                address_item = QTreeWidgetItem([address, label, balance_text, "%d"%num])
-                address_item.setTextAlignment(2, Qt.AlignRight)
-            address_item.setFont(0, QFont(MONOSPACE_FONT))
-            address_item.setData(0, Qt.UserRole, address)
-            address_item.setData(0, Qt.UserRole+1, True) # label can be edited
+                address_item = SortableTreeWidgetItem(['', address, label, balance_text, "%d"%num])
+            address_item.setFont(3, QFont(MONOSPACE_FONT))
+            if self.wallet.is_change(address):
+                address_item.setText(0, _('change'))
+                address_item.setBackground(0, ColorScheme.YELLOW.as_color(True))
+            else:
+                address_item.setText(0, _('receiving'))
+                address_item.setBackground(0, ColorScheme.GREEN.as_color(True))
+            address_item.setFont(1, QFont(MONOSPACE_FONT))
+            address_item.setData(1, Qt.UserRole, address)
             if self.wallet.is_frozen(address):
-                address_item.setBackground(0, ColorScheme.BLUE.as_color(True))
-            if self.wallet.is_beyond_limit(address, self.show_change):
-                address_item.setBackground(0, ColorScheme.RED.as_color(True))
+                address_item.setBackground(1, ColorScheme.BLUE.as_color(True))
+            if self.wallet.is_beyond_limit(address):
+                address_item.setBackground(1, ColorScheme.RED.as_color(True))
             self.addChild(address_item)
             if address == current_address:
                 self.setCurrentItem(address_item)
@@ -118,7 +138,7 @@ class AddressList(MyTreeWidget):
         can_delete = self.wallet.can_delete_address()
         selected = self.selectedItems()
         multi_select = len(selected) > 1
-        addrs = [item.text(0) for item in selected]
+        addrs = [item.text(1) for item in selected]
         if not addrs:
             return
         if not multi_select:
@@ -135,10 +155,10 @@ class AddressList(MyTreeWidget):
         if not multi_select:
             column_title = self.headerItem().text(col)
             copy_text = item.text(col)
-            menu.addAction(_("Copy %s")%column_title, lambda: self.parent.app.clipboard().setText(copy_text))
+            menu.addAction(_("Copy {}").format(column_title), lambda: self.parent.app.clipboard().setText(copy_text))
             menu.addAction(_('Details'), lambda: self.parent.show_address(addr))
             if col in self.editable_columns:
-                menu.addAction(_("Edit %s")%column_title, lambda: self.editItem(item, col))
+                menu.addAction(_("Edit {}").format(column_title), lambda: self.editItem(item, col))
             menu.addAction(_("Request payment"), lambda: self.parent.receive_at(addr))
             if self.wallet.can_export():
                 menu.addAction(_("Private key"), lambda: self.parent.show_private_key(addr))

gui/qt/amountedit.py

@@ -106,12 +106,7 @@ class BTCAmountEdit(AmountEdit):
 
 class FeerateEdit(BTCAmountEdit):
     def _base_unit(self):
-        p = self.decimal_point()
-        if p == 2:
-            return 'mZCL/kB'
-        if p == 0:
-            return 'zat/byte'
-        raise Exception('Unknown base unit')
+        return 'zat/byte'
 
     def get_amount(self):
         sat_per_byte_amount = BTCAmountEdit.get_amount(self)

gui/qt/console.py

@@ -203,7 +203,8 @@ class Console(QtWidgets.QPlainTextEdit):
                     self.skip = not self.skip
 
             if type(self.namespace.get(command)) == type(lambda:None):
-                self.appendPlainText("'%s' is a function. Type '%s()' to use it in the Python console."%(command, command))
+                self.appendPlainText("'{}' is a function. Type '{}()' to use it in the Python console."
+                                     .format(command, command))
                 self.newPrompt()
                 return
 

gui/qt/contact_list.py

@@ -32,7 +32,7 @@ from PyQt5.QtGui import *
 from PyQt5.QtCore import *
 from PyQt5.QtWidgets import (
     QAbstractItemView, QFileDialog, QMenu, QTreeWidgetItem)
-from .util import MyTreeWidget
+from .util import MyTreeWidget, import_meta_gui, export_meta_gui
 
 
 class ContactList(MyTreeWidget):
@@ -53,12 +53,10 @@ class ContactList(MyTreeWidget):
         self.parent.set_contact(item.text(0), item.text(1))
 
     def import_contacts(self):
-        wallet_folder = self.parent.get_wallet_folder()
-        filename, __ = QFileDialog.getOpenFileName(self.parent, "Select your wallet file", wallet_folder)
-        if not filename:
-            return
-        self.parent.contacts.import_file(filename)
-        self.on_update()
+        import_meta_gui(self.parent, _('contacts'), self.parent.contacts.import_file, self.on_update)
+
+    def export_contacts(self):
+        export_meta_gui(self.parent, _('contacts'), self.parent.contacts.export_file)
 
     def create_menu(self, position):
         menu = QMenu()
@@ -66,16 +64,17 @@ class ContactList(MyTreeWidget):
         if not selected:
             menu.addAction(_("New contact"), lambda: self.parent.new_contact_dialog())
             menu.addAction(_("Import file"), lambda: self.import_contacts())
+            menu.addAction(_("Export file"), lambda: self.export_contacts())
         else:
             names = [item.text(0) for item in selected]
             keys = [item.text(1) for item in selected]
             column = self.currentColumn()
             column_title = self.headerItem().text(column)
             column_data = '\n'.join([item.text(column) for item in selected])
-            menu.addAction(_("Copy %s")%column_title, lambda: self.parent.app.clipboard().setText(column_data))
+            menu.addAction(_("Copy {}").format(column_title), lambda: self.parent.app.clipboard().setText(column_data))
             if column in self.editable_columns:
                 item = self.currentItem()
-                menu.addAction(_("Edit %s")%column_title, lambda: self.editItem(item, column))
+                menu.addAction(_("Edit {}").format(column_title), lambda: self.editItem(item, column))
             menu.addAction(_("Pay to"), lambda: self.parent.payto_contacts(keys))
             menu.addAction(_("Delete"), lambda: self.parent.delete_contacts(keys))
             URLs = [block_explorer_URL(self.config, 'addr', key) for key in filter(is_address, keys)]

gui/qt/exception_window.py

@@ -0,0 +1,206 @@
+#!/usr/bin/env python
+#
+# Electrum - lightweight Bitcoin client
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation files
+# (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge,
+# publish, distribute, sublicense, and/or sell copies of the Software,
+# and to permit persons to whom the Software is furnished to do so,
+# subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+import json
+import locale
+import platform
+import traceback
+import os
+import sys
+import subprocess
+
+import requests
+from PyQt5.QtCore import QObject
+import PyQt5.QtCore as QtCore
+from PyQt5.QtGui import QIcon
+from PyQt5.QtWidgets import *
+
+from electrum.i18n import _
+from electrum import ELECTRUM_VERSION, bitcoin, constants
+
+issue_template = """<h2>Traceback</h2>
+<pre>
+{traceback}
+</pre>
+
+<h2>Additional information</h2>
+<ul>
+  <li>Electrum version: {app_version}</li>
+  <li>Operating system: {os}</li>
+  <li>Wallet type: {wallet_type}</li>
+  <li>Locale: {locale}</li>
+</ul>
+"""
+report_server = "https://crashhub.electrum.org/crash"
+
+
+class Exception_Window(QWidget):
+    _active_window = None
+
+    def __init__(self, main_window, exctype, value, tb):
+        self.exc_args = (exctype, value, tb)
+        self.main_window = main_window
+        QWidget.__init__(self)
+        self.setWindowTitle('Electrum - ' + _('An Error Occured'))
+        self.setMinimumSize(600, 300)
+
+        main_box = QVBoxLayout()
+
+        heading = QLabel('<h2>' + _('Sorry!') + '</h2>')
+        main_box.addWidget(heading)
+        main_box.addWidget(QLabel(_('Something went wrong while executing Electrum.')))
+
+        main_box.addWidget(QLabel(
+            _('To help us diagnose and fix the problem, you can send us a bug report that contains useful debug '
+              'information:')))
+
+        collapse_info = QPushButton(_("Show report contents"))
+        collapse_info.clicked.connect(lambda: QMessageBox.about(self, "Report contents", self.get_report_string()))
+        main_box.addWidget(collapse_info)
+
+        main_box.addWidget(QLabel(_("Please briefly describe what led to the error (optional):")))
+
+        self.description_textfield = QTextEdit()
+        self.description_textfield.setFixedHeight(50)
+        main_box.addWidget(self.description_textfield)
+
+        main_box.addWidget(QLabel(_("Do you want to send this report?")))
+
+        buttons = QHBoxLayout()
+
+        report_button = QPushButton(_('Send Bug Report'))
+        report_button.clicked.connect(self.send_report)
+        report_button.setIcon(QIcon(":icons/tab_send.png"))
+        buttons.addWidget(report_button)
+
+        never_button = QPushButton(_('Never'))
+        never_button.clicked.connect(self.show_never)
+        buttons.addWidget(never_button)
+
+        close_button = QPushButton(_('Not Now'))
+        close_button.clicked.connect(self.close)
+        buttons.addWidget(close_button)
+
+        main_box.addLayout(buttons)
+
+        self.setLayout(main_box)
+        self.show()
+
+    def send_report(self):
+        if constants.net.GENESIS[-4:] not in ["4943", "e26f"] and ".electrum.org" in report_server:
+            # Gah! Some kind of altcoin wants to send us crash reports.
+            self.main_window.show_critical(_("Please report this issue manually."))
+            return
+        report = self.get_traceback_info()
+        report.update(self.get_additional_info())
+        report = json.dumps(report)
+        try:
+            response = requests.post(report_server, data=report, timeout=20)
+        except BaseException as e:
+            traceback.print_exc(file=sys.stderr)
+            self.main_window.show_critical(_('There was a problem with the automatic reporting:') + '\n' +
+                                           str(e) + '\n' +
+                                           _("Please report this issue manually."))
+            return
+        else:
+            QMessageBox.about(self, "Crash report", response.text)
+            self.close()
+
+    def on_close(self):
+        Exception_Window._active_window = None
+        sys.__excepthook__(*self.exc_args)
+        self.close()
+
+    def show_never(self):
+        self.main_window.config.set_key("show_crash_reporter", False)
+        self.close()
+
+    def closeEvent(self, event):
+        self.on_close()
+        event.accept()
+
+    def get_traceback_info(self):
+        exc_string = str(self.exc_args[1])
+        stack = traceback.extract_tb(self.exc_args[2])
+        readable_trace = "".join(traceback.format_list(stack))
+        id = {
+            "file": stack[-1].filename,
+            "name": stack[-1].name,
+            "type": self.exc_args[0].__name__
+        }
+        return {
+            "exc_string": exc_string,
+            "stack": readable_trace,
+            "id": id
+        }
+
+    def get_additional_info(self):
+        args = {
+            "app_version": ELECTRUM_VERSION,
+            "os": platform.platform(),
+            "wallet_type": "unknown",
+            "locale": locale.getdefaultlocale()[0],
+            "description": self.description_textfield.toPlainText()
+        }
+        try:
+            args["wallet_type"] = self.main_window.wallet.wallet_type
+        except:
+            # Maybe the wallet isn't loaded yet
+            pass
+        try:
+            args["app_version"] = self.get_git_version()
+        except:
+            # This is probably not running from source
+            pass
+        return args
+
+    def get_report_string(self):
+        info = self.get_additional_info()
+        info["traceback"] = "".join(traceback.format_exception(*self.exc_args))
+        return issue_template.format(**info)
+
+    @staticmethod
+    def get_git_version():
+        dir = os.path.dirname(os.path.realpath(sys.argv[0]))
+        version = subprocess.check_output(['git', 'describe', '--always'], cwd=dir)
+        return str(version, "utf8").strip()
+
+
+def _show_window(*args):
+    if not Exception_Window._active_window:
+        Exception_Window._active_window = Exception_Window(*args)
+
+
+class Exception_Hook(QObject):
+    _report_exception = QtCore.pyqtSignal(object, object, object, object)
+
+    def __init__(self, main_window, *args, **kwargs):
+        super(Exception_Hook, self).__init__(*args, **kwargs)
+        if not main_window.config.get("show_crash_reporter", default=True):
+            return
+        self.main_window = main_window
+        sys.excepthook = self.handler
+        self._report_exception.connect(_show_window)
+
+    def handler(self, *args):
+        self._report_exception.emit(self.main_window, *args)

gui/qt/fee_slider.py

@@ -1,6 +1,4 @@
-
 from electrum.i18n import _
-
 from PyQt5.QtGui import *
 from PyQt5.QtCore import *
 from PyQt5.QtWidgets import QSlider, QToolTip
@@ -18,47 +16,43 @@ class FeeSlider(QSlider):
         self.lock = threading.RLock()
         self.update()
         self.valueChanged.connect(self.moved)
+        self._active = True
 
     def moved(self, pos):
         with self.lock:
-            fee_rate = self.config.dynfee(pos) if self.dyn else self.config.static_fee(pos)
+            if self.dyn:
+                fee_rate = self.config.depth_to_fee(pos) if self.config.use_mempool_fees() else self.config.eta_to_fee(pos)
+            else:
+                fee_rate = self.config.static_fee(pos)
             tooltip = self.get_tooltip(pos, fee_rate)
             QToolTip.showText(QCursor.pos(), tooltip, self)
             self.setToolTip(tooltip)
             self.callback(self.dyn, pos, fee_rate)
 
     def get_tooltip(self, pos, fee_rate):
-        from electrum.util import fee_levels
-        rate_str = self.window.format_fee_rate(fee_rate) if fee_rate else _('unknown')
+        mempool = self.config.use_mempool_fees()
+        target, estimate = self.config.get_fee_text(pos, self.dyn, mempool, fee_rate)
         if self.dyn:
-            tooltip = fee_levels[pos] + '\n' + rate_str
+            return _('Target') + ': ' + target + '\n' + _('Current rate') + ': ' + estimate
         else:
-            tooltip = 'Fixed rate: ' + rate_str
-            if self.config.has_fee_estimates():
-                i = self.config.reverse_dynfee(fee_rate)
-                tooltip += '\n' + (_('Low fee') if i < 0 else 'Within %d blocks'%i)
-        return tooltip
+            return _('Fixed rate') + ': ' + target + '\n' + _('Estimate') + ': ' + estimate
 
     def update(self):
         with self.lock:
             self.dyn = self.config.is_dynfee()
-            if self.dyn:
-                pos = self.config.get('fee_level', 2)
-                fee_rate = self.config.dynfee(pos)
-                self.setRange(0, 4)
-                self.setValue(pos)
-            else:
-                fee_rate = self.config.fee_per_kb()
-                pos = self.config.static_fee_index(fee_rate)
-                self.setRange(0, 9)
-                self.setValue(pos)
+            mempool = self.config.use_mempool_fees()
+            maxp, pos, fee_rate = self.config.get_fee_slider(self.dyn, mempool)
+            self.setRange(0, maxp)
+            self.setValue(pos)
             tooltip = self.get_tooltip(pos, fee_rate)
             self.setToolTip(tooltip)
 
     def activate(self):
+        self._active = True
         self.setStyleSheet('')
 
     def deactivate(self):
+        self._active = False
         # TODO it would be nice to find a platform-independent solution
         # that makes the slider look as if it was disabled
         self.setStyleSheet(
@@ -79,3 +73,6 @@ class FeeSlider(QSlider):
             }
             """
         )
+
+    def is_active(self):
+        return self._active

gui/qt/history_list.py

@@ -24,19 +24,24 @@
 # SOFTWARE.
 
 import webbrowser
+import datetime
 
+from electrum.wallet import AddTransactionException, TX_HEIGHT_LOCAL
 from .util import *
 from electrum.i18n import _
-from electrum.util import block_explorer_URL
-from electrum.util import timestamp_to_datetime, profiler
+from electrum.util import block_explorer_URL, profiler
 
+try:
+    from electrum.plot import plot_history, NothingToPlotException
+except:
+    plot_history = None
 
+# note: this list needs to be kept in sync with another in kivy
 TX_ICONS = [
-    "warning.png",
-    "warning.png",
+    "unconfirmed.png",
     "warning.png",
     "unconfirmed.png",
-    "unconfirmed.png",
+    "offline_tx.png",
     "clock1.png",
     "clock2.png",
     "clock3.png",
@@ -46,51 +51,204 @@ TX_ICONS = [
 ]
 
 
-class HistoryList(MyTreeWidget):
+class HistoryList(MyTreeWidget, AcceptFileDragDrop):
     filter_columns = [2, 3, 4]  # Date, Description, Amount
 
     def __init__(self, parent=None):
         MyTreeWidget.__init__(self, parent, self.create_menu, [], 3)
+        AcceptFileDragDrop.__init__(self, ".txn")
         self.refresh_headers()
         self.setColumnHidden(1, True)
+        self.setSortingEnabled(True)
+        self.sortByColumn(0, Qt.AscendingOrder)
+        self.start_timestamp = None
+        self.end_timestamp = None
+        self.years = []
+        self.create_toolbar_buttons()
+
+    def format_date(self, d):
+        return str(datetime.date(d.year, d.month, d.day)) if d else _('None')
 
     def refresh_headers(self):
-        headers = ['', '', _('Date'), _('Description') , _('Amount'), _('Balance')]
+        headers = ['', '', _('Date'), _('Description'), _('Amount'), _('Balance')]
         fx = self.parent.fx
         if fx and fx.show_history():
-            headers.extend(['%s '%fx.ccy + _('Amount'), '%s '%fx.ccy + _('Balance')])
+            headers.extend(['%s '%fx.ccy + _('Value')])
+            self.editable_columns |= {6}
+            if fx.get_history_capital_gains_config():
+                headers.extend(['%s '%fx.ccy + _('Acquisition price')])
+                headers.extend(['%s '%fx.ccy + _('Capital Gains')])
+        else:
+            self.editable_columns -= {6}
         self.update_headers(headers)
 
     def get_domain(self):
         '''Replaced in address_dialog.py'''
         return self.wallet.get_addresses()
 
+    def on_combo(self, x):
+        s = self.period_combo.itemText(x)
+        x = s == _('Custom')
+        self.start_button.setEnabled(x)
+        self.end_button.setEnabled(x)
+        if s == _('All'):
+            self.start_timestamp = None
+            self.end_timestamp = None
+            self.start_button.setText("-")
+            self.end_button.setText("-")
+        else:
+            try:
+                year = int(s)
+            except:
+                return
+            start_date = datetime.datetime(year, 1, 1)
+            end_date = datetime.datetime(year+1, 1, 1)
+            self.start_timestamp = time.mktime(start_date.timetuple())
+            self.end_timestamp = time.mktime(end_date.timetuple())
+            self.start_button.setText(_('From') + ' ' + self.format_date(start_date))
+            self.end_button.setText(_('To') + ' ' + self.format_date(end_date))
+        self.update()
+
+    def create_toolbar_buttons(self):
+        self.period_combo = QComboBox()
+        self.start_button = QPushButton('-')
+        self.start_button.pressed.connect(self.select_start_date)
+        self.start_button.setEnabled(False)
+        self.end_button = QPushButton('-')
+        self.end_button.pressed.connect(self.select_end_date)
+        self.end_button.setEnabled(False)
+        self.period_combo.addItems([_('All'), _('Custom')])
+        self.period_combo.activated.connect(self.on_combo)
+
+    def get_toolbar_buttons(self):
+        return self.period_combo, self.start_button, self.end_button
+
+    def on_hide_toolbar(self):
+        self.start_timestamp = None
+        self.end_timestamp = None
+        self.update()
+
+    def save_toolbar_state(self, state, config):
+        config.set_key('show_toolbar_history', state)
+
+    def select_start_date(self):
+        self.start_timestamp = self.select_date(self.start_button)
+        self.update()
+
+    def select_end_date(self):
+        self.end_timestamp = self.select_date(self.end_button)
+        self.update()
+
+    def select_date(self, button):
+        d = WindowModalDialog(self, _("Select date"))
+        d.setMinimumSize(600, 150)
+        d.date = None
+        vbox = QVBoxLayout()
+        def on_date(date):
+            d.date = date
+        cal = QCalendarWidget()
+        cal.setGridVisible(True)
+        cal.clicked[QDate].connect(on_date)
+        vbox.addWidget(cal)
+        vbox.addLayout(Buttons(OkButton(d), CancelButton(d)))
+        d.setLayout(vbox)
+        if d.exec_():
+            if d.date is None:
+                return None
+            date = d.date.toPyDate()
+            button.setText(self.format_date(date))
+            return time.mktime(date.timetuple())
+
+    def show_summary(self):
+        h = self.summary
+        start_date = h.get('start_date')
+        end_date = h.get('end_date')
+        format_amount = lambda x: self.parent.format_amount(x.value) + ' ' + self.parent.base_unit()
+        d = WindowModalDialog(self, _("Summary"))
+        d.setMinimumSize(600, 150)
+        vbox = QVBoxLayout()
+        grid = QGridLayout()
+        grid.addWidget(QLabel(_("Start")), 0, 0)
+        grid.addWidget(QLabel(self.format_date(start_date)), 0, 1)
+        grid.addWidget(QLabel(_("End")), 1, 0)
+        grid.addWidget(QLabel(self.format_date(end_date)), 1, 1)
+        grid.addWidget(QLabel(_("Initial balance")), 2, 0)
+        grid.addWidget(QLabel(format_amount(h['start_balance'])), 2, 1)
+        grid.addWidget(QLabel(str(h.get('start_fiat_balance'))), 2, 2)
+        grid.addWidget(QLabel(_("Final balance")), 4, 0)
+        grid.addWidget(QLabel(format_amount(h['end_balance'])), 4, 1)
+        grid.addWidget(QLabel(str(h.get('end_fiat_balance'))), 4, 2)
+        grid.addWidget(QLabel(_("Income")), 5, 0)
+        grid.addWidget(QLabel(format_amount(h.get('income'))), 5, 1)
+        grid.addWidget(QLabel(str(h.get('fiat_income'))), 5, 2)
+        grid.addWidget(QLabel(_("Expenditures")), 6, 0)
+        grid.addWidget(QLabel(format_amount(h.get('expenditures'))), 6, 1)
+        grid.addWidget(QLabel(str(h.get('fiat_expenditures'))), 6, 2)
+        grid.addWidget(QLabel(_("Capital gains")), 7, 0)
+        grid.addWidget(QLabel(str(h.get('capital_gains'))), 7, 2)
+        grid.addWidget(QLabel(_("Unrealized gains")), 8, 0)
+        grid.addWidget(QLabel(str(h.get('unrealized_gains', ''))), 8, 2)
+        vbox.addLayout(grid)
+        vbox.addLayout(Buttons(CloseButton(d)))
+        d.setLayout(vbox)
+        d.exec_()
+
+    def plot_history_dialog(self):
+        if plot_history is None:
+            self.parent.show_message(
+                _("Can't plot history.") + '\n' +
+                _("Perhaps some dependencies are missing...") + " (matplotlib?)")
+            return
+        try:
+            plt = plot_history(self.transactions)
+            plt.show()
+        except NothingToPlotException as e:
+            self.parent.show_message(str(e))
+
     @profiler
     def on_update(self):
         self.wallet = self.parent.wallet
-        h = self.wallet.get_history(self.get_domain())
+        fx = self.parent.fx
+        r = self.wallet.get_full_history(domain=self.get_domain(), from_timestamp=self.start_timestamp, to_timestamp=self.end_timestamp, fx=fx)
+        self.transactions = r['transactions']
+        self.summary = r['summary']
+        if not self.years and self.transactions:
+            from datetime import date
+            start_date = self.transactions[0].get('date') or date.today()
+            end_date = self.transactions[-1].get('date') or date.today()
+            self.years = [str(i) for i in range(start_date.year, end_date.year + 1)]
+            self.period_combo.insertItems(1, self.years)
         item = self.currentItem()
         current_tx = item.data(0, Qt.UserRole) if item else None
         self.clear()
-        fx = self.parent.fx
         if fx: fx.history_used_spot = False
-        for h_item in h:
-            tx_hash, height, conf, timestamp, value, balance = h_item
+        for tx_item in self.transactions:
+            tx_hash = tx_item['txid']
+            height = tx_item['height']
+            conf = tx_item['confirmations']
+            timestamp = tx_item['timestamp']
+            value = tx_item['value'].value
+            balance = tx_item['balance'].value
+            label = tx_item['label']
             status, status_str = self.wallet.get_tx_status(tx_hash, height, conf, timestamp)
             has_invoice = self.wallet.invoices.paid.get(tx_hash)
             icon = QIcon(":icons/" + TX_ICONS[status])
             v_str = self.parent.format_amount(value, True, whitespaces=True)
             balance_str = self.parent.format_amount(balance, whitespaces=True)
-            label = self.wallet.get_label(tx_hash)
             entry = ['', tx_hash, status_str, label, v_str, balance_str]
-            if fx and fx.show_history():
-                date = timestamp_to_datetime(time.time() if conf <= 0 else timestamp)
-                for amount in [value, balance]:
-                    text = fx.historical_value_str(amount, date)
-                    entry.append(text)
-            item = QTreeWidgetItem(entry)
+            fiat_value = None
+            if value is not None and fx and fx.show_history():
+                fiat_value = tx_item['fiat_value'].value
+                value_str = fx.format_fiat(fiat_value)
+                entry.append(value_str)
+                # fixme: should use is_mine
+                if value < 0:
+                    entry.append(fx.format_fiat(tx_item['acquisition_price'].value))
+                    entry.append(fx.format_fiat(tx_item['capital_gain'].value))
+            item = SortableTreeWidgetItem(entry)
             item.setIcon(0, icon)
             item.setToolTip(0, str(conf) + " confirmation" + ("s" if conf != 1 else ""))
+            item.setData(0, SortableTreeWidgetItem.DataRole, (status, conf))
             if has_invoice:
                 item.setIcon(3, QIcon(":icons/seal"))
             for i in range(len(entry)):
@@ -101,12 +259,27 @@ class HistoryList(MyTreeWidget):
             if value and value < 0:
                 item.setForeground(3, QBrush(QColor("#BC1E1E")))
                 item.setForeground(4, QBrush(QColor("#BC1E1E")))
+            if fiat_value and not tx_item['fiat_default']:
+                item.setForeground(6, QBrush(QColor("#1E1EFF")))
             if tx_hash:
                 item.setData(0, Qt.UserRole, tx_hash)
             self.insertTopLevelItem(0, item)
             if current_tx == tx_hash:
                 self.setCurrentItem(item)
 
+    def on_edited(self, item, column, prior):
+        '''Called only when the text actually changes'''
+        key = item.data(0, Qt.UserRole)
+        text = item.text(column)
+        # fixme
+        if column == 3:
+            self.parent.wallet.set_label(key, text)
+            self.update_labels()
+            self.parent.update_completions()
+        elif column == 6:
+            self.parent.wallet.set_fiat_value(key, self.parent.fx.ccy, text)
+            self.on_update()
+
     def on_doubleclick(self, item, column):
         if self.permit_edit(item, column):
             super(HistoryList, self).on_doubleclick(item, column)
@@ -131,6 +304,7 @@ class HistoryList(MyTreeWidget):
         if items:
             item = items[0]
             item.setIcon(0, icon)
+            item.setData(0, SortableTreeWidgetItem.DataRole, (status, conf))
             item.setText(2, status_str)
 
     def create_menu(self, position):
@@ -148,20 +322,18 @@ class HistoryList(MyTreeWidget):
         else:
             column_title = self.headerItem().text(column)
             column_data = item.text(column)
-
         tx_URL = block_explorer_URL(self.config, 'tx', tx_hash)
         height, conf, timestamp = self.wallet.get_tx_height(tx_hash)
         tx = self.wallet.transactions.get(tx_hash)
         is_relevant, is_mine, v, fee = self.wallet.get_wallet_delta(tx)
         is_unconfirmed = height <= 0
         pr_key = self.wallet.invoices.paid.get(tx_hash)
-
         menu = QMenu()
-
-        menu.addAction(_("Copy %s")%column_title, lambda: self.parent.app.clipboard().setText(column_data))
-        if column in self.editable_columns:
-            menu.addAction(_("Edit %s")%column_title, lambda: self.editItem(item, column))
-
+        if height == TX_HEIGHT_LOCAL:
+            menu.addAction(_("Remove"), lambda: self.remove_local_tx(tx_hash))
+        menu.addAction(_("Copy {}").format(column_title), lambda: self.parent.app.clipboard().setText(column_data))
+        for c in self.editable_columns:
+            menu.addAction(_("Edit {}").format(self.headerItem().text(c)), lambda: self.editItem(item, c))
         menu.addAction(_("Details"), lambda: self.parent.show_transaction(tx))
         if is_unconfirmed and tx:
             rbf = is_mine and not tx.is_final()
@@ -176,3 +348,73 @@ class HistoryList(MyTreeWidget):
         if tx_URL:
             menu.addAction(_("View on block explorer"), lambda: webbrowser.open(tx_URL))
         menu.exec_(self.viewport().mapToGlobal(position))
+
+    def remove_local_tx(self, delete_tx):
+        to_delete = {delete_tx}
+        to_delete |= self.wallet.get_depending_transactions(delete_tx)
+        question = _("Are you sure you want to remove this transaction?")
+        if len(to_delete) > 1:
+            question = _(
+                "Are you sure you want to remove this transaction and {} child transactions?".format(len(to_delete) - 1)
+            )
+        answer = QMessageBox.question(self.parent, _("Please confirm"), question, QMessageBox.Yes, QMessageBox.No)
+        if answer == QMessageBox.No:
+            return
+        for tx in to_delete:
+            self.wallet.remove_transaction(tx)
+        self.wallet.save_transactions(write=True)
+        # need to update at least: history_list, utxo_list, address_list
+        self.parent.need_update.set()
+
+    def onFileAdded(self, fn):
+        try:
+            with open(fn) as f:
+                tx = self.parent.tx_from_text(f.read())
+                self.parent.save_transaction_into_wallet(tx)
+        except IOError as e:
+            self.parent.show_error(e)
+
+    def export_history_dialog(self):
+        d = WindowModalDialog(self, _('Export History'))
+        d.setMinimumSize(400, 200)
+        vbox = QVBoxLayout(d)
+        defaultname = os.path.expanduser('~/electrum-history.csv')
+        select_msg = _('Select file to export your wallet transactions to')
+        hbox, filename_e, csv_button = filename_field(self, self.config, defaultname, select_msg)
+        vbox.addLayout(hbox)
+        vbox.addStretch(1)
+        hbox = Buttons(CancelButton(d), OkButton(d, _('Export')))
+        vbox.addLayout(hbox)
+        #run_hook('export_history_dialog', self, hbox)
+        self.update()
+        if not d.exec_():
+            return
+        filename = filename_e.text()
+        if not filename:
+            return
+        try:
+            self.do_export_history(self.wallet, filename, csv_button.isChecked())
+        except (IOError, os.error) as reason:
+            export_error_label = _("Electrum was unable to produce a transaction export.")
+            self.parent.show_critical(export_error_label + "\n" + str(reason), title=_("Unable to export history"))
+            return
+        self.parent.show_message(_("Your wallet history has been successfully exported."))
+
+    def do_export_history(self, wallet, fileName, is_csv):
+        history = self.transactions
+        lines = []
+        for item in history:
+            if is_csv:
+                lines.append([item['txid'], item.get('label', ''), item['confirmations'], item['value'], item['date']])
+            else:
+                lines.append(item)
+        with open(fileName, "w+") as f:
+            if is_csv:
+                import csv
+                transaction = csv.writer(f, lineterminator='\n')
+                transaction.writerow(["transaction_hash","label", "confirmations", "value", "timestamp"])
+                for line in lines:
+                    transaction.writerow(line)
+            else:
+                from electrum.util import json_encode
+                f.write(json_encode(history))

gui/qt/installwizard.py

@@ -10,29 +10,25 @@ from PyQt5.QtWidgets import *
 
 from electrum import Wallet, WalletStorage
 from electrum.util import UserCancelled, InvalidPassword
-from electrum.base_wizard import BaseWizard
+from electrum.base_wizard import BaseWizard, HWD_SETUP_DECRYPT_WALLET
 from electrum.i18n import _
 
 from .seed_dialog import SeedLayout, KeysLayout
 from .network_dialog import NetworkChoiceLayout
 from .util import *
-from .password_dialog import PasswordLayout, PW_NEW
+from .password_dialog import PasswordLayout, PasswordLayoutForHW, PW_NEW
 
 
 class GoBack(Exception):
     pass
 
-MSG_GENERATING_WAIT = _("Generating your addresses, please wait...")
-MSG_ENTER_ANYTHING = _("Please enter a seed phrase, a master key, a list of "
-                       "Zclassic addresses, or a list of private keys")
-MSG_ENTER_SEED_OR_MPK = _("Please enter a seed phrase or a master key (xpub or xprv):")
-MSG_COSIGNER = _("Please enter the master public key of cosigner #%d:")
+
 MSG_ENTER_PASSWORD = _("Choose a password to encrypt your wallet keys.") + '\n'\
                      + _("Leave this field empty if you want to disable encryption.")
-MSG_RESTORE_PASSPHRASE = \
-    _("Please enter your seed derivation passphrase. "
-      "Note: this is NOT your encryption password. "
-      "Leave this field empty if you did not use one or are unsure.")
+MSG_HW_STORAGE_ENCRYPTION = _("Set wallet file encryption.") + '\n'\
+                          + _("Your wallet file does not contain secrets, mostly just metadata. ") \
+                          + _("It also contains your master public key that allows watching your addresses.") + '\n\n'\
+                          + _("Note: If you enable this setting, you will need your hardware device to open your wallet.")
 
 
 class CosignWidget(QWidget):
@@ -196,12 +192,18 @@ class InstallWizard(QDialog, MessageBoxMixin, BaseWizard):
                     msg =_("This file does not exist.") + '\n' \
                           + _("Press 'Next' to create this wallet, or choose another file.")
                     pw = False
-                elif self.storage.file_exists() and self.storage.is_encrypted():
-                    msg = _("This file is encrypted.") + '\n' + _('Enter your password or choose another file.')
-                    pw = True
                 else:
-                    msg = _("Press 'Next' to open this wallet.")
-                    pw = False
+                    if self.storage.is_encrypted_with_user_pw():
+                        msg = _("This file is encrypted with a password.") + '\n' \
+                              + _('Enter your password or choose another file.')
+                        pw = True
+                    elif self.storage.is_encrypted_with_hw_device():
+                        msg = _("This file is encrypted using a hardware device.") + '\n' \
+                              + _("Press 'Next' to choose device to decrypt.")
+                        pw = False
+                    else:
+                        msg = _("Press 'Next' to open this wallet.")
+                        pw = False
             else:
                 msg = _('Cannot read file')
                 pw = False
@@ -227,23 +229,46 @@ class InstallWizard(QDialog, MessageBoxMixin, BaseWizard):
             if not self.storage.file_exists():
                 break
             if self.storage.file_exists() and self.storage.is_encrypted():
-                password = self.pw_e.text()
-                try:
-                    self.storage.decrypt(password)
-                    break
-                except InvalidPassword as e:
-                    QMessageBox.information(None, _('Error'), str(e))
-                    continue
-                except BaseException as e:
-                    traceback.print_exc(file=sys.stdout)
-                    QMessageBox.information(None, _('Error'), str(e))
-                    return
+                if self.storage.is_encrypted_with_user_pw():
+                    password = self.pw_e.text()
+                    try:
+                        self.storage.decrypt(password)
+                        break
+                    except InvalidPassword as e:
+                        QMessageBox.information(None, _('Error'), str(e))
+                        continue
+                    except BaseException as e:
+                        traceback.print_exc(file=sys.stdout)
+                        QMessageBox.information(None, _('Error'), str(e))
+                        return
+                elif self.storage.is_encrypted_with_hw_device():
+                    try:
+                        self.run('choose_hw_device', HWD_SETUP_DECRYPT_WALLET)
+                    except InvalidPassword as e:
+                        # FIXME if we get here because of mistyped passphrase
+                        # then that passphrase gets "cached"
+                        QMessageBox.information(
+                            None, _('Error'),
+                            _('Failed to decrypt using this hardware device.') + '\n' +
+                            _('If you use a passphrase, make sure it is correct.'))
+                        self.stack = []
+                        return self.run_and_get_wallet()
+                    except BaseException as e:
+                        traceback.print_exc(file=sys.stdout)
+                        QMessageBox.information(None, _('Error'), str(e))
+                        return
+                    if self.storage.is_past_initial_decryption():
+                        break
+                    else:
+                        return
+                else:
+                    raise Exception('Unexpected encryption version')
 
         path = self.storage.path
         if self.storage.requires_split():
             self.hide()
-            msg = _("The wallet '%s' contains multiple accounts, which are no longer supported since Electrum 2.7.\n\n"
-                    "Do you want to split your wallet into multiple files?"%path)
+            msg = _("The wallet '{}' contains multiple accounts, which are no longer supported since Electrum 2.7.\n\n"
+                    "Do you want to split your wallet into multiple files?").format(path)
             if not self.question(msg):
                 return
             file_list = '\n'.join(self.storage.split_accounts())
@@ -261,10 +286,10 @@ class InstallWizard(QDialog, MessageBoxMixin, BaseWizard):
         action = self.storage.get_action()
         if action and action != 'new':
             self.hide()
-            msg = _("The file '%s' contains an incompletely created wallet.\n"
-                    "Do you want to complete its creation now?") % path
+            msg = _("The file '{}' contains an incompletely created wallet.\n"
+                    "Do you want to complete its creation now?").format(path)
             if not self.question(msg):
-                if self.question(_("Do you want to delete '%s'?") % path):
+                if self.question(_("Do you want to delete '{}'?").format(path)):
                     os.remove(path)
                     self.show_warning(_('The file was removed'))
                 return
@@ -386,17 +411,25 @@ class InstallWizard(QDialog, MessageBoxMixin, BaseWizard):
         self.exec_layout(slayout)
         return slayout.is_ext
 
-    def pw_layout(self, msg, kind):
-        playout = PasswordLayout(None, msg, kind, self.next_button)
+    def pw_layout(self, msg, kind, force_disable_encrypt_cb):
+        playout = PasswordLayout(None, msg, kind, self.next_button,
+                                 force_disable_encrypt_cb=force_disable_encrypt_cb)
         playout.encrypt_cb.setChecked(True)
         self.exec_layout(playout.layout())
         return playout.new_password(), playout.encrypt_cb.isChecked()
 
     @wizard_dialog
-    def request_password(self, run_next):
+    def request_password(self, run_next, force_disable_encrypt_cb=False):
         """Request the user enter a new password and confirm it.  Return
         the password or None for no password."""
-        return self.pw_layout(MSG_ENTER_PASSWORD, PW_NEW)
+        return self.pw_layout(MSG_ENTER_PASSWORD, PW_NEW, force_disable_encrypt_cb)
+
+    @wizard_dialog
+    def request_storage_encryption(self, run_next):
+        playout = PasswordLayoutForHW(None, MSG_HW_STORAGE_ENCRYPTION, PW_NEW, self.next_button)
+        playout.encrypt_cb.setChecked(True)
+        self.exec_layout(playout.layout())
+        return playout.encrypt_cb.isChecked()
 
     def show_restore(self, wallet, network):
         # FIXME: these messages are shown after the install wizard is
@@ -437,7 +470,7 @@ class InstallWizard(QDialog, MessageBoxMixin, BaseWizard):
         self.accept_signal.emit()
 
     def waiting_dialog(self, task, msg):
-        self.please_wait.setText(MSG_GENERATING_WAIT)
+        self.please_wait.setText(msg)
         self.refresh_gui()
         t = threading.Thread(target = task)
         t.start()

gui/qt/invoice_list.py

@@ -23,10 +23,11 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .util import *
 from electrum.i18n import _
 from electrum.util import format_time
 
+from .util import *
+
 
 class InvoiceList(MyTreeWidget):
     filter_columns = [0, 1, 2, 3]  # Date, Requested By, Description, Amount
@@ -57,12 +58,10 @@ class InvoiceList(MyTreeWidget):
         self.parent.invoices_label.setVisible(len(inv_list))
 
     def import_invoices(self):
-        wallet_folder = self.parent.get_wallet_folder()
-        filename, __ = QFileDialog.getOpenFileName(self.parent, "Select your wallet file", wallet_folder)
-        if not filename:
-            return
-        self.parent.invoices.import_file(filename)
-        self.on_update()
+        import_meta_gui(self.parent, _('invoices'), self.parent.invoices.import_file, self.on_update)
+
+    def export_invoices(self):
+        export_meta_gui(self.parent, _('invoices'), self.parent.invoices.export_file)
 
     def create_menu(self, position):
         menu = QMenu()
@@ -76,7 +75,7 @@ class InvoiceList(MyTreeWidget):
         pr = self.parent.invoices.get(key)
         status = self.parent.invoices.get_status(key)
         if column_data:
-            menu.addAction(_("Copy %s")%column_title, lambda: self.parent.app.clipboard().setText(column_data))
+            menu.addAction(_("Copy {}").format(column_title), lambda: self.parent.app.clipboard().setText(column_data))
         menu.addAction(_("Details"), lambda: self.parent.show_invoice(key))
         if status == PR_UNPAID:
             menu.addAction(_("Pay Now"), lambda: self.parent.do_pay_invoice(key))

gui/qt/main_window.py

@@ -32,34 +32,32 @@ from decimal import Decimal
 import base64
 from functools import partial
 
-from PyQt5.QtCore import Qt
 from PyQt5.QtGui import *
+from PyQt5.QtCore import *
+import PyQt5.QtCore as QtCore
+
+from .exception_window import Exception_Hook
 from PyQt5.QtWidgets import *
 
-from electrum.util import bh2u, bfh
-
 from electrum import keystore, simple_config
-from electrum.bitcoin import COIN, is_address, TYPE_ADDRESS, NetworkConstants
+from electrum.bitcoin import COIN, is_address, TYPE_ADDRESS
+from electrum import constants
 from electrum.plugins import run_hook
 from electrum.i18n import _
 from electrum.util import (format_time, format_satoshis, PrintError,
                            format_satoshis_plain, NotEnoughFunds,
-                           UserCancelled, NoDynamicFeeEstimates)
+                           UserCancelled, NoDynamicFeeEstimates, profiler,
+                           export_meta, import_meta, bh2u, bfh)
 from electrum import Transaction
 from electrum import util, bitcoin, commands, coinchooser
 from electrum import paymentrequest
-from electrum.wallet import Multisig_Wallet
-try:
-    from electrum.plot import plot_history
-except:
-    plot_history = None
+from electrum.wallet import Multisig_Wallet, AddTransactionException
 
 from .amountedit import AmountEdit, BTCAmountEdit, MyLineEdit, FeerateEdit
 from .qrcodewidget import QRCodeWidget, QRDialog
 from .qrtextedit import ShowQRTextEdit, ScanQRTextEdit
 from .transaction_dialog import show_transaction
 from .fee_slider import FeeSlider
-
 from .util import *
 
 
@@ -102,6 +100,9 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
         self.gui_object = gui_object
         self.config = config = gui_object.config
+
+        self.setup_exception_hook()
+
         self.network = gui_object.daemon.network
         self.fx = gui_object.daemon.fx
         self.invoices = wallet.invoices
@@ -123,9 +124,8 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         self.create_status_bar()
         self.need_update = threading.Event()
 
-        self.decimal_point = config.get('decimal_point', 8)
-        self.fee_unit = config.get('fee_unit', 0)
-        self.num_zeros = int(config.get('num_zeros', 0))
+        self.decimal_point = config.get('decimal_point', 5)
+        self.num_zeros     = int(config.get('num_zeros',0))
 
         self.completions = QStringListModel()
 
@@ -203,6 +203,9 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
     def on_history(self, b):
         self.new_fx_history_signal.emit()
 
+    def setup_exception_hook(self):
+        Exception_Hook(self)
+
     def on_fx_history(self):
         self.history_list.refresh_headers()
         self.history_list.update()
@@ -283,7 +286,6 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             self.need_update.set()
             self.gui_object.network_updated_signal_obj.network_updated_signal \
                 .emit(event, args)
-
         elif event == 'new_transaction':
             self.tx_notifications.append(args[0])
             self.notify_transactions_signal.emit()
@@ -305,6 +307,12 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             if self.config.is_dynfee():
                 self.fee_slider.update()
                 self.do_update_fee()
+        elif event == 'fee_histogram':
+            if self.config.is_dynfee():
+                self.fee_slider.update()
+                self.do_update_fee()
+            # todo: update only unconfirmed tx
+            self.history_list.update()
         else:
             self.print_error("unexpected network_qt signal:", event, args)
 
@@ -325,6 +333,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             self.print_error('close_wallet', self.wallet.storage.path)
         run_hook('close_wallet', self.wallet)
 
+    @profiler
     def load_wallet(self, wallet):
         wallet.thread = TaskThread(self, self.on_error)
         self.wallet = wallet
@@ -363,7 +372,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             self.setGeometry(100, 100, 840, 400)
 
     def watching_only_changed(self):
-        name = "[TESTNET] Zclassic Electrum" if NetworkConstants.TESTNET else "Zclassic Electrum"
+        name = "Zclassic Electrum TESTNET" if constants.net.TESTNET else "Zclassic Electrum"
         title = '%s %s  -  %s' % (name, self.wallet.electrum_version,
                                         self.wallet.basename())
         extra = [self.wallet.storage.get('wallet_type', '?')]
@@ -372,7 +381,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             extra.append(_('watching only'))
         title += '  [%s]'% ', '.join(extra)
         self.setWindowTitle(title)
-        self.password_menu.setEnabled(self.wallet.can_change_password())
+        self.password_menu.setEnabled(self.wallet.may_have_password())
         self.import_privkey_menu.setVisible(self.wallet.can_import_privkey())
         self.import_address_menu.setVisible(self.wallet.can_import_address())
         self.export_menu.setEnabled(self.wallet.can_export())
@@ -467,17 +476,23 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         self.import_address_menu = wallet_menu.addAction(_("Import Addresses"), self.import_addresses)
         wallet_menu.addSeparator()
 
+        addresses_menu = wallet_menu.addMenu(_("&Addresses"))
+        addresses_menu.addAction(_("&Filter"), lambda: self.address_list.toggle_toolbar(self.config))
         labels_menu = wallet_menu.addMenu(_("&Labels"))
         labels_menu.addAction(_("&Import"), self.do_import_labels)
         labels_menu.addAction(_("&Export"), self.do_export_labels)
+        history_menu = wallet_menu.addMenu(_("&History"))
+        history_menu.addAction(_("&Filter"), lambda: self.history_list.toggle_toolbar(self.config))
+        history_menu.addAction(_("&Summary"), self.history_list.show_summary)
+        history_menu.addAction(_("&Plot"), self.history_list.plot_history_dialog)
+        history_menu.addAction(_("&Export"), self.history_list.export_history_dialog)
         contacts_menu = wallet_menu.addMenu(_("Contacts"))
         contacts_menu.addAction(_("&New"), self.new_contact_dialog)
         contacts_menu.addAction(_("Import"), lambda: self.contact_list.import_contacts())
+        contacts_menu.addAction(_("Export"), lambda: self.contact_list.export_contacts())
         invoices_menu = wallet_menu.addMenu(_("Invoices"))
         invoices_menu.addAction(_("Import"), lambda: self.invoice_list.import_invoices())
-        hist_menu = wallet_menu.addMenu(_("&History"))
-        hist_menu.addAction("Plot", self.plot_history_dialog).setEnabled(plot_history is not None)
-        hist_menu.addAction("Export", self.export_history_dialog)
+        invoices_menu.addAction(_("Export"), lambda: self.invoice_list.export_invoices())
 
         wallet_menu.addSeparator()
         wallet_menu.addAction(_("Find"), self.toggle_search).setShortcut(QKeySequence("Ctrl+F"))
@@ -559,24 +574,24 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             return
         self.print_error("Notifying GUI")
         if len(self.tx_notifications) > 0:
-            # Combine the transactions if there are more then three
-            tx_amount = len(self.tx_notifications)
-            if(tx_amount >= 3):
+            # Combine the transactions if there are at least three
+            num_txns = len(self.tx_notifications)
+            if num_txns >= 3:
                 total_amount = 0
                 for tx in self.tx_notifications:
                     is_relevant, is_mine, v, fee = self.wallet.get_wallet_delta(tx)
-                    if(v > 0):
+                    if v > 0:
                         total_amount += v
-                self.notify(_("%(txs)s new transactions received: Total amount received in the new transactions %(amount)s") \
-                            % { 'txs' : tx_amount, 'amount' : self.format_amount_and_units(total_amount)})
+                self.notify(_("{} new transactions received: Total amount received in the new transactions {}")
+                            .format(num_txns, self.format_amount_and_units(total_amount)))
                 self.tx_notifications = []
             else:
-              for tx in self.tx_notifications:
-                  if tx:
-                      self.tx_notifications.remove(tx)
-                      is_relevant, is_mine, v, fee = self.wallet.get_wallet_delta(tx)
-                      if(v > 0):
-                          self.notify(_("Inbound Transaction - %(amount)s") % { 'amount' : self.format_amount_and_units(v)})
+                for tx in self.tx_notifications:
+                    if tx:
+                        self.tx_notifications.remove(tx)
+                        is_relevant, is_mine, v, fee = self.wallet.get_wallet_delta(tx)
+                        if v > 0:
+                            self.notify(_("New Transaction: {}").format(self.format_amount_and_units(v)))
 
     def notify(self, message):
         if self.tray:
@@ -625,16 +640,13 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
     def format_amount_and_units(self, amount):
         text = self.format_amount(amount) + ' '+ self.base_unit()
-        x = self.fx.format_amount_and_units(amount)
+        x = self.fx.format_amount_and_units(amount) if self.fx else None
         if text and x:
             text += ' (%s)'%x
         return text
 
     def format_fee_rate(self, fee_rate):
-        if self.fee_unit == 0:
-            return format_satoshis(fee_rate/1000, False, self.num_zeros, 0, False)  + ' zat/byte'
-        else:
-            return self.format_amount(fee_rate) + ' ' + self.base_unit() + '/kB'
+        return format_satoshis(fee_rate/1000, False, self.num_zeros, 0, False)  + ' zat/byte'
 
     def get_decimal_point(self):
         return self.decimal_point
@@ -704,7 +716,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
                 text = _("Synchronizing...")
                 icon = QIcon(":icons/status_waiting.png")
             elif server_lag > 1:
-                text = _("Server is lagging (%d blocks)"%server_lag)
+                text = _("Server is lagging ({} blocks)").format(server_lag)
                 icon = QIcon(":icons/status_lagging.png")
             else:
                 c, u, x = self.wallet.get_balance()
@@ -749,7 +761,10 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         from .history_list import HistoryList
         self.history_list = l = HistoryList(self)
         l.searchable_list = l
-        return l
+        toolbar = l.create_toolbar(self.config)
+        toolbar_shown = self.config.get('show_toolbar_history', False)
+        l.show_toolbar(toolbar_shown)
+        return self.create_list_tab(l, toolbar)
 
     def show_address(self, addr):
         from . import address_dialog
@@ -883,14 +898,15 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             if alias_addr:
                 if self.wallet.is_mine(alias_addr):
                     msg = _('This payment request will be signed.') + '\n' + _('Please enter your password')
-                    password = self.password_dialog(msg)
-                    if password:
-                        try:
-                            self.wallet.sign_payment_request(addr, alias, alias_addr, password)
-                        except Exception as e:
-                            self.show_error(str(e))
+                    password = None
+                    if self.wallet.has_keystore_encryption():
+                        password = self.password_dialog(msg)
+                        if not password:
                             return
-                    else:
+                    try:
+                        self.wallet.sign_payment_request(addr, alias, alias_addr, password)
+                    except Exception as e:
+                        self.show_error(str(e))
                         return
                 else:
                     return
@@ -905,11 +921,17 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         i = self.expires_combo.currentIndex()
         expiration = list(map(lambda x: x[1], expiration_values))[i]
         req = self.wallet.make_payment_request(addr, amount, message, expiration)
-        self.wallet.add_payment_request(req, self.config)
-        self.sign_payment_request(addr)
-        self.request_list.update()
-        self.address_list.update()
-        self.save_request_button.setEnabled(False)
+        try:
+            self.wallet.add_payment_request(req, self.config)
+        except Exception as e:
+            traceback.print_exc(file=sys.stderr)
+            self.show_error(_('Error adding payment request') + ':\n' + str(e))
+        else:
+            self.sign_payment_request(addr)
+            self.save_request_button.setEnabled(False)
+        finally:
+            self.request_list.update()
+            self.address_list.update()
 
     def view_and_paste(self, title, msg, data):
         dialog = WindowModalDialog(self, title)
@@ -1005,6 +1027,10 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         if self.qr_window and self.qr_window.isVisible():
             self.qr_window.set_content(addr, amount, message, uri)
 
+    def set_feerounding_text(self, num_satoshis_added):
+        self.feerounding_text = (_('Additional {} satoshis are going to be added.')
+                                 .format(num_satoshis_added))
+
     def create_send_tab(self):
         # A 4-column grid layout.  All the stretch is in the last column.
         # The exchange rate plugin adds a fiat widget in column 2
@@ -1070,12 +1096,17 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
         def fee_cb(dyn, pos, fee_rate):
             if dyn:
-                self.config.set_key('fee_level', pos, False)
+                if self.config.use_mempool_fees():
+                    self.config.set_key('depth_level', pos, False)
+                else:
+                    self.config.set_key('fee_level', pos, False)
             else:
                 self.config.set_key('fee_per_kb', fee_rate, False)
 
             if fee_rate:
                 self.feerate_e.setAmount(fee_rate // 1000)
+            else:
+                self.feerate_e.setAmount(None)
             self.fee_e.setModified(False)
 
             self.fee_slider.activate()
@@ -1108,7 +1139,8 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         self.size_e.setFixedWidth(140)
         self.size_e.setStyleSheet(ColorScheme.DEFAULT.as_stylesheet())
 
-        self.feerate_e = FeerateEdit(lambda: 2 if self.fee_unit else 0)
+        self.feerate_e = FeerateEdit(lambda: 0)
+        self.feerate_e.setAmount(self.config.fee_per_byte())
         self.feerate_e.textEdited.connect(partial(on_fee_or_feerate, self.feerate_e, False))
         self.feerate_e.editingFinished.connect(partial(on_fee_or_feerate, self.feerate_e, True))
 
@@ -1116,6 +1148,20 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         self.fee_e.textEdited.connect(partial(on_fee_or_feerate, self.fee_e, False))
         self.fee_e.editingFinished.connect(partial(on_fee_or_feerate, self.fee_e, True))
 
+        def feerounding_onclick():
+            text = (self.feerounding_text + '\n\n' +
+                    _('To somewhat protect your privacy, Electrum tries to create change with similar precision to other outputs.') + ' ' +
+                    _('At most 100 satoshis might be lost due to this rounding.') + ' ' +
+                    _("You can disable this setting in '{}'.").format(_('Preferences')) + '\n' +
+                    _('Also, dust is not kept as change, but added to the fee.'))
+            QMessageBox.information(self, 'Fee rounding', text)
+
+        self.feerounding_icon = QPushButton(QIcon(':icons/info.png'), '')
+        self.feerounding_icon.setFixedWidth(20)
+        self.feerounding_icon.setFlat(True)
+        self.feerounding_icon.clicked.connect(feerounding_onclick)
+        self.feerounding_icon.setVisible(False)
+
         self.connect_fields(self, self.amount_e, self.fiat_send_e, self.fee_e)
 
         vbox_feelabel = QVBoxLayout()
@@ -1129,14 +1175,16 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         hbox.addWidget(self.feerate_e)
         hbox.addWidget(self.size_e)
         hbox.addWidget(self.fee_e)
+        hbox.addWidget(self.feerounding_icon, Qt.AlignLeft)
+        hbox.addStretch(1)
 
         vbox_feecontrol = QVBoxLayout()
         vbox_feecontrol.addWidget(self.fee_adv_controls)
         vbox_feecontrol.addWidget(self.fee_slider)
 
-        grid.addLayout(vbox_feecontrol, 5, 1, 1, 3)
+        grid.addLayout(vbox_feecontrol, 5, 1, 1, -1)
 
-        if not self.config.get('show_fee', True):
+        if not self.config.get('show_fee', False):
             self.fee_adv_controls.setVisible(False)
 
         self.preview_button = EnterButton(_("Preview"), self.do_preview)
@@ -1233,9 +1281,6 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         '''Recalculate the fee.  If the fee was manually input, retain it, but
         still build the TX to see if there are enough funds.
         '''
-        if not self.config.get('offline') and self.config.is_dynfee() and not self.config.has_fee_estimates():
-            self.statusBar().showMessage(_('Waiting for fee estimates...'))
-            return False
         freeze_fee = self.is_send_fee_frozen()
         freeze_feerate = self.is_send_feerate_frozen()
         amount = '!' if self.is_max else self.amount_e.get_amount()
@@ -1258,15 +1303,22 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             try:
                 tx = make_tx(fee_estimator)
                 self.not_enough_funds = False
-            except NotEnoughFunds:
-                self.not_enough_funds = True
+            except (NotEnoughFunds, NoDynamicFeeEstimates) as e:
                 if not freeze_fee:
                     self.fee_e.setAmount(None)
-                return
-            except NoDynamicFeeEstimates:
-                tx = make_tx(0)
-                size = tx.estimated_size()
-                self.size_e.setAmount(size)
+                if not freeze_feerate:
+                    self.feerate_e.setAmount(None)
+                self.feerounding_icon.setVisible(False)
+
+                if isinstance(e, NotEnoughFunds):
+                    self.not_enough_funds = True
+                elif isinstance(e, NoDynamicFeeEstimates):
+                    try:
+                        tx = make_tx(0)
+                        size = tx.estimated_size()
+                        self.size_e.setAmount(size)
+                    except BaseException:
+                        pass
                 return
             except BaseException:
                 traceback.print_exc(file=sys.stderr)
@@ -1276,12 +1328,32 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             self.size_e.setAmount(size)
 
             fee = tx.get_fee()
-            if not freeze_fee:
-                fee = None if self.not_enough_funds else fee
-                self.fee_e.setAmount(fee)
-            if not freeze_feerate:
-                fee_rate = fee // size if fee is not None else None
-                self.feerate_e.setAmount(fee_rate)
+            fee = None if self.not_enough_funds else fee
+
+            # Displayed fee/fee_rate values are set according to user input.
+            # Due to rounding or dropping dust in CoinChooser,
+            # actual fees often differ somewhat.
+            if freeze_feerate or self.fee_slider.is_active():
+                displayed_feerate = self.feerate_e.get_amount()
+                displayed_feerate = displayed_feerate // 1000 if displayed_feerate else 0
+                displayed_fee = displayed_feerate * size
+                self.fee_e.setAmount(displayed_fee)
+            else:
+                if freeze_fee:
+                    displayed_fee = self.fee_e.get_amount()
+                else:
+                    # fallback to actual fee if nothing is frozen
+                    displayed_fee = fee
+                    self.fee_e.setAmount(displayed_fee)
+                displayed_fee = displayed_fee if displayed_fee else 0
+                displayed_feerate = displayed_fee // size if displayed_fee is not None else None
+                self.feerate_e.setAmount(displayed_feerate)
+
+            # show/hide fee rounding icon
+            feerounding = (fee - displayed_fee) if fee else 0
+            self.set_feerounding_text(feerounding)
+            self.feerounding_icon.setToolTip(self.feerounding_text)
+            self.feerounding_icon.setVisible(bool(feerounding))
 
             if self.is_max:
                 amount = tx.output_value()
@@ -1331,7 +1403,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         def request_password(self, *args, **kwargs):
             parent = self.top_level_window()
             password = None
-            while self.wallet.has_password():
+            while self.wallet.has_keystore_encryption():
                 password = self.password_dialog(parent=parent)
                 if password is None:
                     # User cancelled password input
@@ -1360,7 +1432,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             fee_estimator = self.fee_e.get_amount()
         elif self.is_send_feerate_frozen():
             amount = self.feerate_e.get_amount()
-            amount = 0 if amount is None else float(amount)
+            amount = 0 if amount is None else amount
             fee_estimator = partial(
                 simple_config.SimpleConfig.estimate_fee_for_feerate, amount)
         else:
@@ -1369,7 +1441,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
     def read_send_tab(self):
         if self.payment_request and self.payment_request.has_expired():
-            self.show_error(_('Payment request has expired.'))
+            self.show_error(_('Payment request has expired'))
             return
         label = self.message_e.text()
 
@@ -1384,7 +1456,8 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
             if self.payto_e.is_alias and self.payto_e.validated is False:
                 alias = self.payto_e.toPlainText()
-                msg = _('WARNING: the alias "%s" could not be validated via an additional security check, DNSSEC, and thus may not be correct.'%alias) + '\n'
+                msg = _('WARNING: the alias "{}" could not be validated via an additional '
+                        'security check, DNSSEC, and thus may not be correct.').format(alias) + '\n'
                 msg += _('Do you wish to continue?')
                 if not self.question(msg):
                     return
@@ -1439,13 +1512,20 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             tx.set_rbf(True)
 
         if fee < self.wallet.relayfee() * tx.estimated_size() / 1000:
-            self.show_error(_("This transaction requires a higher fee, or it will not be propagated by the network"))
+            self.show_error('\n'.join([
+                _("This transaction requires a higher fee, or it will not be propagated by your current server"),
+                _("Try to raise your transaction fee, or use a server with a lower relay fee.")
+            ]))
             return
 
         if preview:
             self.show_transaction(tx, tx_desc)
             return
 
+        if not self.network:
+            self.show_error(_("You can't broadcast a transaction without a live network connection."))
+            return
+
         # confirmation dialog
         msg = [
             _("Amount to be sent") + ": " + self.format_amount_and_units(amount),
@@ -1457,11 +1537,11 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             x_fee_address, x_fee_amount = x_fee
             msg.append( _("Additional fees") + ": " + self.format_amount_and_units(x_fee_amount) )
 
-        confirm_rate = 2 * self.config.max_fee_rate()
+        confirm_rate = simple_config.FEERATE_WARNING_HIGH_FEE
         if fee > confirm_rate * tx.estimated_size() / 1000:
             msg.append(_('Warning') + ': ' + _("The fee for this transaction seems unusually high."))
 
-        if self.wallet.has_password():
+        if self.wallet.has_keystore_encryption():
             msg.append("")
             msg.append(_("Enter your password to proceed"))
             password = self.password_dialog('\n'.join(msg))
@@ -1514,7 +1594,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             pr = self.payment_request
             if pr and pr.has_expired():
                 self.payment_request = None
-                return False, _("Payment request has expired.")
+                return False, _("Payment request has expired")
             status, msg =  self.network.broadcast(tx)
             if pr and status is True:
                 self.invoices.set_paid(pr, tx.txid())
@@ -1606,7 +1686,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             self.payment_request_error_signal.emit()
 
     def pay_to_URI(self, URI):
-        if not URI or not isinstance(URI, str):
+        if not URI:
             return
         try:
             out = util.parse_URI(URI, self.on_pr)
@@ -1646,7 +1726,9 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             e.setText('')
             e.setFrozen(False)
         self.fee_slider.activate()
+        self.feerate_e.setAmount(self.config.fee_per_byte())
         self.size_e.setAmount(0)
+        self.feerounding_icon.setVisible(False)
         self.set_pay_from([])
         self.tx_external_keypairs = {}
         self.update_status()
@@ -1658,26 +1740,25 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         self.utxo_list.update()
         self.update_fee()
 
-    def create_list_tab(self, l, list_header=None):
+    def create_list_tab(self, l, toolbar=None):
         w = QWidget()
         w.searchable_list = l
         vbox = QVBoxLayout()
         w.setLayout(vbox)
         vbox.setContentsMargins(0, 0, 0, 0)
         vbox.setSpacing(0)
-        if list_header:
-            hbox = QHBoxLayout()
-            for b in list_header:
-                hbox.addWidget(b)
-            hbox.addStretch()
-            vbox.addLayout(hbox)
+        if toolbar:
+            vbox.addLayout(toolbar)
         vbox.addWidget(l)
         return w
 
     def create_addresses_tab(self):
         from .address_list import AddressList
         self.address_list = l = AddressList(self)
-        return self.create_list_tab(l, l.get_list_header())
+        toolbar = l.create_toolbar(self.config)
+        toolbar_shown = self.config.get('show_toolbar_addresses', False)
+        l.show_toolbar(toolbar_shown)
+        return self.create_list_tab(l, toolbar)
 
     def create_utxo_tab(self):
         from .utxo_list import UTXOList
@@ -1741,8 +1822,8 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         return True
 
     def delete_contacts(self, labels):
-        if not self.question(_("Remove %s from your list of contacts?")
-                             % " + ".join(labels)):
+        if not self.question(_("Remove {} from your list of contacts?")
+                             .format(" + ".join(labels))):
             return
         for label in labels:
             self.contacts.pop(label)
@@ -1862,17 +1943,37 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
     def update_buttons_on_seed(self):
         self.seed_button.setVisible(self.wallet.has_seed())
-        self.password_button.setVisible(self.wallet.can_change_password())
+        self.password_button.setVisible(self.wallet.may_have_password())
         self.send_button.setVisible(not self.wallet.is_watching_only())
 
     def change_password_dialog(self):
-        from .password_dialog import ChangePasswordDialog
-        d = ChangePasswordDialog(self, self.wallet)
-        ok, password, new_password, encrypt_file = d.run()
+        from electrum.storage import STO_EV_XPUB_PW
+        if self.wallet.get_available_storage_encryption_version() == STO_EV_XPUB_PW:
+            from .password_dialog import ChangePasswordDialogForHW
+            d = ChangePasswordDialogForHW(self, self.wallet)
+            ok, encrypt_file = d.run()
+            if not ok:
+                return
+
+            try:
+                hw_dev_pw = self.wallet.keystore.get_password_for_storage_encryption()
+            except UserCancelled:
+                return
+            except BaseException as e:
+                traceback.print_exc(file=sys.stderr)
+                self.show_error(str(e))
+                return
+            old_password = hw_dev_pw if self.wallet.has_password() else None
+            new_password = hw_dev_pw if encrypt_file else None
+        else:
+            from .password_dialog import ChangePasswordDialogForSW
+            d = ChangePasswordDialogForSW(self, self.wallet)
+            ok, old_password, new_password, encrypt_file = d.run()
+
         if not ok:
             return
         try:
-            self.wallet.update_password(password, new_password, encrypt_file)
+            self.wallet.update_password(old_password, new_password, encrypt_file)
         except BaseException as e:
             self.show_error(str(e))
             return
@@ -1880,11 +1981,15 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             traceback.print_exc(file=sys.stdout)
             self.show_error(_('Failed to update password'))
             return
-        msg = _('Password was updated successfully') if new_password else _('Password is disabled, this wallet is not protected')
+        msg = _('Password was updated successfully') if self.wallet.has_password() else _('Password is disabled, this wallet is not protected')
         self.show_message(msg, title=_("Success"))
         self.update_lock_icon()
 
     def toggle_search(self):
+        tab = self.tabs.currentWidget()
+        #if hasattr(tab, 'searchable_list'):
+        #    tab.searchable_list.toggle_toolbar()
+        #return
         self.search_box.setHidden(not self.search_box.isHidden())
         if not self.search_box.isHidden():
             self.search_box.setFocus(1)
@@ -2017,8 +2122,6 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             rds_e = ShowQRTextEdit(text=redeem_script)
             rds_e.addCopyButton(self.app)
             vbox.addWidget(rds_e)
-        if xtype in ['p2wpkh', 'p2wsh', 'p2wpkh-p2sh', 'p2wsh-p2sh']:
-            vbox.addWidget(WWLabel(_("Warning: the format of private keys associated to segwit addresses may not be compatible with other wallets")))
         vbox.addLayout(Buttons(CloseButton(d)))
         d.setLayout(vbox)
         d.exec_()
@@ -2036,6 +2139,9 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         if not bitcoin.is_address(address):
             self.show_message(_('Invalid Zclassic address.'))
             return
+        if self.wallet.is_watching_only():
+            self.show_message(_('This is a watching-only wallet.'))
+            return
         if not self.wallet.is_mine(address):
             self.show_message(_('Address not in wallet.'))
             return
@@ -2047,7 +2153,12 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         task = partial(self.wallet.sign_message, address, message, password)
 
         def show_signed_message(sig):
-            signature.setText(base64.b64encode(sig).decode('ascii'))
+            try:
+                signature.setText(base64.b64encode(sig).decode('ascii'))
+            except RuntimeError:
+                # (signature) wrapped C/C++ object has been deleted
+                pass
+
         self.wallet.thread.add(task, on_success=show_signed_message)
 
     def do_verify(self, address, message, signature):
@@ -2106,9 +2217,20 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
     @protected
     def do_decrypt(self, message_e, pubkey_e, encrypted_e, password):
+        if self.wallet.is_watching_only():
+            self.show_message(_('This is a watching-only wallet.'))
+            return
         cyphertext = encrypted_e.toPlainText()
         task = partial(self.wallet.decrypt_message, pubkey_e.text(), cyphertext, password)
-        self.wallet.thread.add(task, on_success=lambda text: message_e.setText(text.decode('utf-8')))
+
+        def setText(text):
+            try:
+                message_e.setText(text.decode('utf-8'))
+            except RuntimeError:
+                # (message_e) wrapped C/C++ object has been deleted
+                pass
+
+        self.wallet.thread.add(task, on_success=setText)
 
     def do_encrypt(self, message_e, pubkey_e, encrypted_e):
         message = message_e.toPlainText()
@@ -2207,25 +2329,17 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         return self.tx_from_text(file_content)
 
     def do_process_from_text(self):
-        from electrum.transaction import SerializationError
         text = text_dialog(self, _('Input raw transaction'), _("Transaction:"), _("Load transaction"))
         if not text:
             return
-        try:
-            tx = self.tx_from_text(text)
-            if tx:
-                self.show_transaction(tx)
-        except SerializationError as e:
-            self.show_critical(_("Electrum was unable to deserialize the transaction:") + "\n" + str(e))
+        tx = self.tx_from_text(text)
+        if tx:
+            self.show_transaction(tx)
 
     def do_process_from_file(self):
-        from electrum.transaction import SerializationError
-        try:
-            tx = self.read_tx_from_file()
-            if tx:
-                self.show_transaction(tx)
-        except SerializationError as e:
-            self.show_critical(_("Electrum was unable to deserialize the transaction:") + "\n" + str(e))
+        tx = self.read_tx_from_file()
+        if tx:
+            self.show_transaction(tx)
 
     def do_process_from_txid(self):
         from electrum import transaction
@@ -2251,7 +2365,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
                               _('It can not be "backed up" by simply exporting these private keys.'))
 
         d = WindowModalDialog(self, _('Private keys'))
-        d.setMinimumSize(850, 300)
+        d.setMinimumSize(980, 300)
         vbox = QVBoxLayout(d)
 
         msg = "%s\n%s\n%s" % (_("WARNING: ALL your private keys are secret."),
@@ -2344,102 +2458,23 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
                 f.write(json.dumps(pklist, indent = 4))
 
     def do_import_labels(self):
-        labelsFile = self.getOpenFileName(_("Open Labels File"), "*.json")
-        if not labelsFile: return
-        try:
-            with open(labelsFile, 'r') as f:
-                data = f.read()
-            for key, value in json.loads(data).items():
-                self.wallet.set_label(key, value)
-            self.show_message(_("Your labels were imported from") + " '%s'" % str(labelsFile))
-        except (IOError, os.error) as reason:
-            self.show_critical(_("Electrum was unable to import your labels.") + "\n" + str(reason))
-        self.address_list.update()
-        self.history_list.update()
+        def import_labels(path):
+            def _validate(data):
+                return data  # TODO
+
+            def import_labels_assign(data):
+                for key, value in data.items():
+                    self.wallet.set_label(key, value)
+            import_meta(path, _validate, import_labels_assign)
+
+        def on_import():
+            self.need_update.set()
+        import_meta_gui(self, _('labels'), import_labels, on_import)
 
     def do_export_labels(self):
-        labels = self.wallet.labels
-        try:
-            fileName = self.getSaveFileName(_("Select destination file for your labels:"), 'electrum_labels.json', "*.json")
-            if fileName:
-                with open(fileName, 'w+') as f:
-                    json.dump(labels, f, indent=4, sort_keys=True)
-                self.show_message(_("Your labels were exported to") + " '%s'" % str(fileName))
-        except (IOError, os.error) as reason:
-            self.show_critical(_("Electrum was unable to export your labels.") + "\n" + str(reason))
-
-    def export_history_dialog(self):
-        d = WindowModalDialog(self, _('Export History'))
-        d.setMinimumSize(400, 200)
-        vbox = QVBoxLayout(d)
-        defaultname = os.path.expanduser('~/electrum-history.csv')
-        select_msg = _('Select destination file for your wallet transaction history:')
-        hbox, filename_e, csv_button = filename_field(self, self.config, defaultname, select_msg)
-        vbox.addLayout(hbox)
-        vbox.addStretch(1)
-        hbox = Buttons(CancelButton(d), OkButton(d, _('Export')))
-        vbox.addLayout(hbox)
-        run_hook('export_history_dialog', self, hbox)
-        self.update()
-        if not d.exec_():
-            return
-        filename = filename_e.text()
-        if not filename:
-            return
-        try:
-            self.do_export_history(self.wallet, filename, csv_button.isChecked())
-        except (IOError, os.error) as reason:
-            export_error_label = _("Electrum was unable to produce a transaction export.")
-            self.show_critical(export_error_label + "\n" + str(reason), title=_("Unable to export history"))
-            return
-        self.show_message(_("Your wallet history has been successfully exported."))
-
-    def plot_history_dialog(self):
-        if plot_history is None:
-            return
-        wallet = self.wallet
-        history = wallet.get_history()
-        if len(history) > 0:
-            plt = plot_history(self.wallet, history)
-            plt.show()
-
-    def do_export_history(self, wallet, fileName, is_csv):
-        history = wallet.get_history()
-        lines = []
-        for item in history:
-            tx_hash, height, confirmations, timestamp, value, balance = item
-            if height>0:
-                if timestamp is not None:
-                    time_string = format_time(timestamp)
-                else:
-                    time_string = _("unverified")
-            else:
-                time_string = _("unconfirmed")
-
-            if value is not None:
-                value_string = format_satoshis(value, True)
-            else:
-                value_string = '--'
-
-            if tx_hash:
-                label = wallet.get_label(tx_hash)
-            else:
-                label = ""
-
-            if is_csv:
-                lines.append([tx_hash, label, confirmations, value_string, time_string])
-            else:
-                lines.append({'txid':tx_hash, 'date':"%16s"%time_string, 'label':label, 'value':value_string})
-
-        with open(fileName, "w+") as f:
-            if is_csv:
-                transaction = csv.writer(f, lineterminator='\n')
-                transaction.writerow(["transaction_hash","label", "confirmations", "value", "timestamp"])
-                for line in lines:
-                    transaction.writerow(line)
-            else:
-                import json
-                f.write(json.dumps(lines, indent = 4))
+        def export_labels(filename):
+            export_meta(self.wallet.labels, filename)
+        export_meta_gui(self, _('labels'), export_labels)
 
     def sweep_key_dialog(self):
         d = WindowModalDialog(self, title=_('Sweep private keys'))
@@ -2590,6 +2625,21 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         nz.valueChanged.connect(on_nz)
         gui_widgets.append((nz_label, nz))
 
+        msg = '\n'.join([
+            _('Time based: fee rate is based on average confirmation time estimates'),
+            _('Mempool based: fee rate is targetting a depth in the memory pool')
+            ]
+        )
+        fee_type_label = HelpLabel(_('Fee estimation') + ':', msg)
+        fee_type_combo = QComboBox()
+        fee_type_combo.addItems([_('Static'), _('ETA'), _('Mempool')])
+        fee_type_combo.setCurrentIndex(1 if self.config.use_mempool_fees() else 0)
+        def on_fee_type(x):
+            self.config.set_key('mempool_fees', x==2)
+            self.config.set_key('dynamic_fees', x>0)
+            self.fee_slider.update()
+        fee_type_combo.currentIndexChanged.connect(on_fee_type)
+        fee_widgets.append((fee_type_label, fee_type_combo))
 
         use_rbf_cb = QCheckBox(_('Use Replace-By-Fee'))
         use_rbf_cb.setChecked(self.config.get('use_rbf', True))
@@ -2602,18 +2652,6 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         use_rbf_cb.stateChanged.connect(on_use_rbf)
         fee_widgets.append((use_rbf_cb, None))
 
-        self.fee_unit = self.config.get('fee_unit', 0)
-        fee_unit_label = HelpLabel(_('Fee Unit') + ':', '')
-        fee_unit_combo = QComboBox()
-        fee_unit_combo.addItems([_('zat/byte'), _('mZCL/kB')])
-        fee_unit_combo.setCurrentIndex(self.fee_unit)
-        def on_fee_unit(x):
-            self.fee_unit = x
-            self.config.set_key('fee_unit', x)
-            self.fee_slider.update()
-        fee_unit_combo.currentIndexChanged.connect(on_fee_unit)
-        fee_widgets.append((fee_unit_label, fee_unit_combo))
-
         msg = _('OpenAlias record, used to receive coins and to sign payment requests.') + '\n\n'\
               + _('The following alias providers are available:') + '\n'\
               + '\n'.join(['https://cryptoname.co/', 'http://xmr.link']) + '\n\n'\
@@ -2672,7 +2710,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         unit_combo = QComboBox()
         unit_combo.addItems(units)
         unit_combo.setCurrentIndex(units.index(self.base_unit()))
-        def on_unit(x):
+        def on_unit(x, nz):
             unit_result = units[unit_combo.currentIndex()]
             if self.base_unit() == unit_result:
                 return
@@ -2687,13 +2725,14 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             else:
                 raise Exception('Unknown base unit')
             self.config.set_key('decimal_point', self.decimal_point, True)
+            nz.setMaximum(self.decimal_point)
             self.history_list.update()
             self.request_list.update()
             self.address_list.update()
             for edit, amount in zip(edits, amounts):
                 edit.setAmount(amount)
             self.update_status()
-        unit_combo.currentIndexChanged.connect(on_unit)
+        unit_combo.currentIndexChanged.connect(lambda x: on_unit(x, nz))
         gui_widgets.append((unit_label, unit_combo))
 
         block_explorers = sorted(util.block_explorer_info().keys())
@@ -2783,8 +2822,21 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         unconf_cb.stateChanged.connect(on_unconf)
         tx_widgets.append((unconf_cb, None))
 
+        def on_outrounding(x):
+            self.config.set_key('coin_chooser_output_rounding', bool(x))
+        enable_outrounding = self.config.get('coin_chooser_output_rounding', False)
+        outrounding_cb = QCheckBox(_('Enable output value rounding'))
+        outrounding_cb.setToolTip(
+            _('Set the value of the change output so that it has similar precision to the other outputs.') + '\n' +
+            _('This might improve your privacy somewhat.') + '\n' +
+            _('If enabled, at most 100 zatoshis might be lost due to this, per transaction.'))
+        outrounding_cb.setChecked(enable_outrounding)
+        outrounding_cb.stateChanged.connect(on_outrounding)
+        tx_widgets.append((outrounding_cb, None))
+
         # Fiat Currency
         hist_checkbox = QCheckBox()
+        hist_capgains_checkbox = QCheckBox()
         fiat_address_checkbox = QCheckBox()
         ccy_combo = QComboBox()
         ex_combo = QComboBox()
@@ -2806,6 +2858,11 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             if not self.fx: return
             fiat_address_checkbox.setChecked(self.fx.get_fiat_address_config())
 
+        def update_history_capgains_cb():
+            if not self.fx: return
+            hist_capgains_checkbox.setChecked(self.fx.get_history_capital_gains_config())
+            hist_capgains_checkbox.setEnabled(hist_checkbox.isChecked())
+
         def update_exchanges():
             if not self.fx: return
             b = self.fx.is_enabled()
@@ -2844,6 +2901,12 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
             if self.fx.is_enabled() and checked:
                 # reset timeout to get historical rates
                 self.fx.timeout = 0
+            update_history_capgains_cb()
+
+        def on_history_capgains(checked):
+            if not self.fx: return
+            self.fx.set_history_capital_gains_config(checked)
+            self.history_list.refresh_headers()
 
         def on_fiat_address(checked):
             if not self.fx: return
@@ -2853,16 +2916,19 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
 
         update_currencies()
         update_history_cb()
+        update_history_capgains_cb()
         update_fiat_address_cb()
         update_exchanges()
         ccy_combo.currentIndexChanged.connect(on_currency)
         hist_checkbox.stateChanged.connect(on_history)
+        hist_capgains_checkbox.stateChanged.connect(on_history_capgains)
         fiat_address_checkbox.stateChanged.connect(on_fiat_address)
         ex_combo.currentIndexChanged.connect(on_exchange)
 
         fiat_widgets = []
         fiat_widgets.append((QLabel(_('Fiat currency')), ccy_combo))
         fiat_widgets.append((QLabel(_('Show history rates')), hist_checkbox))
+        fiat_widgets.append((QLabel(_('Show capital gains in history')), hist_capgains_checkbox))
         fiat_widgets.append((QLabel(_('Show Fiat balance for addresses')), fiat_address_checkbox))
         fiat_widgets.append((QLabel(_('Source')), ex_combo))
 
@@ -3017,6 +3083,7 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         grid.addWidget(QLabel(_('Output amount') + ':'), 2, 0)
         grid.addWidget(output_amount, 2, 1)
         fee_e = BTCAmountEdit(self.get_decimal_point)
+        # FIXME with dyn fees, without estimates, there are all kinds of crashes here
         def f(x):
             a = max_fee - fee_e.get_amount()
             output_amount.setText((self.format_amount(a) + ' ' + self.base_unit()) if a else '')
@@ -3081,3 +3148,21 @@ class ElectrumWindow(QMainWindow, MessageBoxMixin, PrintError):
         if is_final:
             new_tx.set_rbf(False)
         self.show_transaction(new_tx, tx_label)
+
+    def save_transaction_into_wallet(self, tx):
+        try:
+            if not self.wallet.add_transaction(tx.txid(), tx):
+                self.show_error(_("Transaction could not be saved.") + "\n" +
+                                       _("It conflicts with current history."))
+                return False
+        except AddTransactionException as e:
+            self.show_error(e)
+            return False
+        else:
+            self.wallet.save_transactions(write=True)
+            # need to update at least: history_list, utxo_list, address_list
+            self.need_update.set()
+            self.msg_box(QPixmap(":icons/offline_tx.png"), None, _('Success'), _("Transaction added to wallet history"))
+            return True
+
+

gui/qt/network_dialog.py

@@ -31,8 +31,9 @@ from PyQt5.QtWidgets import *
 import PyQt5.QtCore as QtCore
 
 from electrum.i18n import _
-from electrum.bitcoin import NetworkConstants
+from electrum import constants
 from electrum.util import print_error
+from electrum.network import serialize_server, deserialize_server
 
 from .util import *
 
@@ -145,7 +146,7 @@ class ServerListWidget(QTreeWidget):
         menu.exec_(self.viewport().mapToGlobal(position))
 
     def set_server(self, s):
-        host, port, protocol = s.split(':')
+        host, port, protocol = deserialize_server(s)
         self.parent.server_host.setText(host)
         self.parent.server_port.setText(port)
         self.parent.set_server()
@@ -170,7 +171,7 @@ class ServerListWidget(QTreeWidget):
             port = d.get(protocol)
             if port:
                 x = QTreeWidgetItem([_host, port])
-                server = _host+':'+port+':'+protocol
+                server = serialize_server(_host, port, protocol)
                 x.setData(1, Qt.UserRole, server)
                 self.addTopLevelItem(x)
 
@@ -395,7 +396,7 @@ class NetworkChoiceLayout(object):
     def change_protocol(self, use_ssl):
         p = 's' if use_ssl else 't'
         host = self.server_host.text()
-        pp = self.servers.get(host, NetworkConstants.DEFAULT_PORTS)
+        pp = self.servers.get(host, constants.net.DEFAULT_PORTS)
         if p not in pp.keys():
             p = list(pp.keys())[0]
         port = pp[p]
@@ -411,7 +412,7 @@ class NetworkChoiceLayout(object):
     def follow_server(self, server):
         self.network.switch_to_interface(server)
         host, port, protocol, proxy, auto_connect = self.network.get_parameters()
-        host, port, protocol = server.split(':')
+        host, port, protocol = deserialize_server(server)
         self.network.set_parameters(host, port, protocol, proxy, auto_connect)
         self.update()
 
@@ -420,7 +421,7 @@ class NetworkChoiceLayout(object):
             self.change_server(str(x.text(0)), self.protocol)
 
     def change_server(self, host, protocol):
-        pp = self.servers.get(host, NetworkConstants.DEFAULT_PORTS)
+        pp = self.servers.get(host, constants.net.DEFAULT_PORTS)
         if protocol and protocol not in protocol_letters:
             protocol = None
         if protocol:
@@ -444,7 +445,6 @@ class NetworkChoiceLayout(object):
         host, port, protocol, proxy, auto_connect = self.network.get_parameters()
         host = str(self.server_host.text())
         port = str(self.server_port.text())
-        protocol = 't' if self.config.get('nossl') else 's'
         auto_connect = self.autoconnect_cb.isChecked()
         self.network.set_parameters(host, port, protocol, proxy, auto_connect)
 

gui/qt/password_dialog.py

@@ -57,7 +57,7 @@ class PasswordLayout(object):
 
     titles = [_("Enter Password"), _("Change Password"), _("Enter Passphrase")]
 
-    def __init__(self, wallet, msg, kind, OK_button):
+    def __init__(self, wallet, msg, kind, OK_button, force_disable_encrypt_cb=False):
         self.wallet = wallet
 
         self.pw = QLineEdit()
@@ -126,7 +126,8 @@ class PasswordLayout(object):
         def enable_OK():
             ok = self.new_pw.text() == self.conf_pw.text()
             OK_button.setEnabled(ok)
-            self.encrypt_cb.setEnabled(ok and bool(self.new_pw.text()))
+            self.encrypt_cb.setEnabled(ok and bool(self.new_pw.text())
+                                       and not force_disable_encrypt_cb)
         self.new_pw.textChanged.connect(enable_OK)
         self.conf_pw.textChanged.connect(enable_OK)
 
@@ -163,11 +164,84 @@ class PasswordLayout(object):
         return pw
 
 
-class ChangePasswordDialog(WindowModalDialog):
+class PasswordLayoutForHW(object):
+
+    def __init__(self, wallet, msg, kind, OK_button):
+        self.wallet = wallet
+
+        self.kind = kind
+        self.OK_button = OK_button
+
+        vbox = QVBoxLayout()
+        label = QLabel(msg + "\n")
+        label.setWordWrap(True)
+
+        grid = QGridLayout()
+        grid.setSpacing(8)
+        grid.setColumnMinimumWidth(0, 150)
+        grid.setColumnMinimumWidth(1, 100)
+        grid.setColumnStretch(1,1)
+
+        logo_grid = QGridLayout()
+        logo_grid.setSpacing(8)
+        logo_grid.setColumnMinimumWidth(0, 70)
+        logo_grid.setColumnStretch(1,1)
+
+        logo = QLabel()
+        logo.setAlignment(Qt.AlignCenter)
+
+        logo_grid.addWidget(logo,  0, 0)
+        logo_grid.addWidget(label, 0, 1, 1, 2)
+        vbox.addLayout(logo_grid)
+
+        if wallet and wallet.has_storage_encryption():
+            lockfile = ":icons/lock.png"
+        else:
+            lockfile = ":icons/unlock.png"
+        logo.setPixmap(QPixmap(lockfile).scaledToWidth(36))
+
+        vbox.addLayout(grid)
+
+        self.encrypt_cb = QCheckBox(_('Encrypt wallet file'))
+        grid.addWidget(self.encrypt_cb, 1, 0, 1, 2)
+
+        self.vbox = vbox
+
+    def title(self):
+        return _("Toggle Encryption")
+
+    def layout(self):
+        return self.vbox
+
+
+class ChangePasswordDialogBase(WindowModalDialog):
 
     def __init__(self, parent, wallet):
         WindowModalDialog.__init__(self, parent)
-        is_encrypted = wallet.storage.is_encrypted()
+        is_encrypted = wallet.has_storage_encryption()
+        OK_button = OkButton(self)
+
+        self.create_password_layout(wallet, is_encrypted, OK_button)
+
+        self.setWindowTitle(self.playout.title())
+        vbox = QVBoxLayout(self)
+        vbox.addLayout(self.playout.layout())
+        vbox.addStretch(1)
+        vbox.addLayout(Buttons(CancelButton(self), OK_button))
+        self.playout.encrypt_cb.setChecked(is_encrypted)
+
+    def create_password_layout(self, wallet, is_encrypted, OK_button):
+        raise NotImplementedError()
+
+
+class ChangePasswordDialogForSW(ChangePasswordDialogBase):
+
+    def __init__(self, parent, wallet):
+        ChangePasswordDialogBase.__init__(self, parent, wallet)
+        if not wallet.has_password():
+            self.playout.encrypt_cb.setChecked(True)
+
+    def create_password_layout(self, wallet, is_encrypted, OK_button):
         if not wallet.has_password():
             msg = _('Your wallet is not protected.')
             msg += ' ' + _('Use this dialog to add a password to your wallet.')
@@ -177,14 +251,9 @@ class ChangePasswordDialog(WindowModalDialog):
             else:
                 msg = _('Your wallet is password protected and encrypted.')
             msg += ' ' + _('Use this dialog to change your password.')
-        OK_button = OkButton(self)
-        self.playout = PasswordLayout(wallet, msg, PW_CHANGE, OK_button)
-        self.setWindowTitle(self.playout.title())
-        vbox = QVBoxLayout(self)
-        vbox.addLayout(self.playout.layout())
-        vbox.addStretch(1)
-        vbox.addLayout(Buttons(CancelButton(self), OK_button))
-        self.playout.encrypt_cb.setChecked(is_encrypted or not wallet.has_password())
+        self.playout = PasswordLayout(
+            wallet, msg, PW_CHANGE, OK_button,
+            force_disable_encrypt_cb=not wallet.can_have_keystore_encryption())
 
     def run(self):
         if not self.exec_():
@@ -192,6 +261,26 @@ class ChangePasswordDialog(WindowModalDialog):
         return True, self.playout.old_password(), self.playout.new_password(), self.playout.encrypt_cb.isChecked()
 
 
+class ChangePasswordDialogForHW(ChangePasswordDialogBase):
+
+    def __init__(self, parent, wallet):
+        ChangePasswordDialogBase.__init__(self, parent, wallet)
+
+    def create_password_layout(self, wallet, is_encrypted, OK_button):
+        if not is_encrypted:
+            msg = _('Your wallet file is NOT encrypted.')
+        else:
+            msg = _('Your wallet file is encrypted.')
+        msg += '\n' + _('Note: If you enable this setting, you will need your hardware device to open your wallet.')
+        msg += '\n' + _('Use this dialog to toggle encryption.')
+        self.playout = PasswordLayoutForHW(wallet, msg, PW_CHANGE, OK_button)
+
+    def run(self):
+        if not self.exec_():
+            return False, None
+        return True, self.playout.encrypt_cb.isChecked()
+
+
 class PasswordDialog(WindowModalDialog):
 
     def __init__(self, parent=None, msg=None):

gui/qt/request_list.py

@@ -115,7 +115,7 @@ class RequestList(MyTreeWidget):
         column_title = self.headerItem().text(column)
         column_data = item.text(column)
         menu = QMenu(self)
-        menu.addAction(_("Copy %s")%column_title, lambda: self.parent.app.clipboard().setText(column_data))
+        menu.addAction(_("Copy {}").format(column_title), lambda: self.parent.app.clipboard().setText(column_data))
         menu.addAction(_("Copy URI"), lambda: self.parent.view_and_paste('URI', '', self.parent.get_request_URI(addr)))
         menu.addAction(_("Save as BIP70 file"), lambda: self.parent.export_payment_request(addr))
         menu.addAction(_("Delete"), lambda: self.parent.delete_payment_request(addr))

gui/qt/transaction_dialog.py

@@ -25,6 +25,7 @@
 import copy
 import datetime
 import json
+import traceback
 
 from PyQt5.QtCore import *
 from PyQt5.QtGui import *
@@ -33,16 +34,27 @@ from PyQt5.QtWidgets import *
 from electrum.bitcoin import base_encode
 from electrum.i18n import _
 from electrum.plugins import run_hook
+from electrum import simple_config
 
 from electrum.util import bfh
+from electrum.wallet import AddTransactionException
+from electrum.transaction import SerializationError
+
 from .util import *
 
 dialogs = []  # Otherwise python randomly garbage collects the dialogs...
 
+
 def show_transaction(tx, parent, desc=None, prompt_if_unsaved=False):
-    d = TxDialog(tx, parent, desc, prompt_if_unsaved)
-    dialogs.append(d)
-    d.show()
+    try:
+        d = TxDialog(tx, parent, desc, prompt_if_unsaved)
+    except SerializationError as e:
+        traceback.print_exc(file=sys.stderr)
+        parent.show_critical(_("Electrum was unable to deserialize the transaction:") + "\n" + str(e))
+    else:
+        dialogs.append(d)
+        d.show()
+
 
 class TxDialog(QDialog, MessageBoxMixin):
 
@@ -56,7 +68,10 @@ class TxDialog(QDialog, MessageBoxMixin):
         # e.g. the FX plugin.  If this happens during or after a long
         # sign operation the signatures are lost.
         self.tx = copy.deepcopy(tx)
-        self.tx.deserialize()
+        try:
+            self.tx.deserialize()
+        except BaseException as e:
+            raise SerializationError(e)
         self.main_window = parent
         self.wallet = parent.wallet
         self.prompt_if_unsaved = prompt_if_unsaved
@@ -98,8 +113,17 @@ class TxDialog(QDialog, MessageBoxMixin):
         self.broadcast_button = b = QPushButton(_("Broadcast"))
         b.clicked.connect(self.do_broadcast)
 
-        self.save_button = b = QPushButton(_("Save"))
-        b.clicked.connect(self.save)
+        self.save_button = QPushButton(_("Save"))
+        save_button_disabled = not tx.is_complete()
+        self.save_button.setDisabled(save_button_disabled)
+        if save_button_disabled:
+            self.save_button.setToolTip(_("Please sign this transaction in order to save it"))
+        else:
+            self.save_button.setToolTip("")
+        self.save_button.clicked.connect(self.save)
+
+        self.export_button = b = QPushButton(_("Export"))
+        b.clicked.connect(self.export)
 
         self.cancel_button = b = QPushButton(_("Close"))
         b.clicked.connect(self.close)
@@ -112,9 +136,9 @@ class TxDialog(QDialog, MessageBoxMixin):
         self.copy_button = CopyButton(lambda: str(self.tx), parent.app)
 
         # Action buttons
-        self.buttons = [self.sign_button, self.broadcast_button, self.cancel_button]
+        self.buttons = [self.sign_button, self.broadcast_button, self.save_button, self.cancel_button]
         # Transaction sharing buttons
-        self.sharing_buttons = [self.copy_button, self.qr_button, self.save_button]
+        self.sharing_buttons = [self.copy_button, self.qr_button, self.export_button]
 
         run_hook('transaction_dialog', self)
 
@@ -136,11 +160,14 @@ class TxDialog(QDialog, MessageBoxMixin):
 
     def closeEvent(self, event):
         if (self.prompt_if_unsaved and not self.saved
-            and not self.question(_('This transaction is not saved. Close anyway?'), title=_("Warning"))):
+                and not self.question(_('This transaction is not saved. Close anyway?'), title=_("Warning"))):
             event.ignore()
         else:
             event.accept()
-            dialogs.remove(self)
+            try:
+                dialogs.remove(self)
+            except ValueError:
+                pass  # was not in list already
 
     def show_qr(self):
         text = bfh(str(self.tx))
@@ -155,6 +182,8 @@ class TxDialog(QDialog, MessageBoxMixin):
             if success:
                 self.prompt_if_unsaved = True
                 self.saved = False
+            self.save_button.setDisabled(False)
+            self.save_button.setToolTip("")
             self.update()
             self.main_window.pop_top_level_window(self)
 
@@ -163,12 +192,18 @@ class TxDialog(QDialog, MessageBoxMixin):
         self.main_window.sign_tx(self.tx, sign_done)
 
     def save(self):
+        if self.main_window.save_transaction_into_wallet(self.tx):
+            self.save_button.setDisabled(True)
+            self.saved = True
+
+
+    def export(self):
         name = 'signed_%s.txn' % (self.tx.txid()[0:8]) if self.tx.is_complete() else 'unsigned.txn'
         fileName = self.main_window.getSaveFileName(_("Select where to save your signed transaction"), name, "*.txn")
         if fileName:
             with open(fileName, "w+") as f:
                 f.write(json.dumps(self.tx.as_dict(), indent=4) + '\n')
-            self.show_message(_("Transaction saved successfully"))
+            self.show_message(_("Transaction exported successfully"))
             self.saved = True
 
     def update(self):
@@ -191,11 +226,11 @@ class TxDialog(QDialog, MessageBoxMixin):
 
         if timestamp:
             time_str = datetime.datetime.fromtimestamp(timestamp).isoformat(' ')[:-3]
-            self.date_label.setText(_("Date: %s")%time_str)
+            self.date_label.setText(_("Date: {}").format(time_str))
             self.date_label.show()
         elif exp_n:
-            text = '%d blocks'%(exp_n) if exp_n > 0 else _('unknown (low fee)')
-            self.date_label.setText(_('Expected confirmation time') + ': ' + text)
+            text = '%.2f MB'%(exp_n/1000000)
+            self.date_label.setText(_('Position in mempool') + ': ' + text + ' ' + _('from tip'))
             self.date_label.show()
         else:
             self.date_label.hide()
@@ -206,9 +241,13 @@ class TxDialog(QDialog, MessageBoxMixin):
         else:
             amount_str = _("Amount sent:") + ' %s'% format_amount(-amount) + ' ' + base_unit
         size_str = _("Size:") + ' %d bytes'% size
-        fee_str = _("Fee") + ': %s'% (format_amount(fee) + ' ' + base_unit if fee is not None else _('Unknown'))
+        fee_str = _("Fee") + ': %s' % (format_amount(fee) + ' ' + base_unit if fee is not None else _('unknown'))
         if fee is not None:
-            fee_str += '  ( %s ) '%  self.main_window.format_fee_rate(fee/size*1000)
+            fee_rate = fee/size*1000
+            fee_str += '  ( %s ) ' % self.main_window.format_fee_rate(fee_rate)
+            confirm_rate = simple_config.FEERATE_WARNING_HIGH_FEE
+            if fee_rate > confirm_rate:
+                fee_str += ' - ' + _('Warning') + ': ' + _("high fee") + '!'
         self.amount_label.setText(amount_str)
         self.fee_label.setText(fee_str)
         self.size_label.setText(size_str)
@@ -224,7 +263,7 @@ class TxDialog(QDialog, MessageBoxMixin):
         rec.setBackground(QBrush(ColorScheme.GREEN.as_color(background=True)))
         rec.setToolTip(_("Wallet receive address"))
         chg = QTextCharFormat()
-        chg.setBackground(QBrush(QColor("yellow")))
+        chg.setBackground(QBrush(ColorScheme.YELLOW.as_color(background=True)))
         chg.setToolTip(_("Wallet change address"))
 
         def text_format(addr):
@@ -254,7 +293,7 @@ class TxDialog(QDialog, MessageBoxMixin):
                     if _addr:
                         addr = _addr
                 if addr is None:
-                    addr = _('Unknown')
+                    addr = _('unknown')
                 cursor.insertText(addr, text_format(addr))
                 if x.get('value'):
                     cursor.insertText(format_amount(x['value']), ext)

gui/qt/util.py

@@ -6,11 +6,15 @@ import queue
 from collections import namedtuple
 from functools import partial
 
-from electrum.i18n import _
 from PyQt5.QtGui import *
 from PyQt5.QtCore import *
 from PyQt5.QtWidgets import *
 
+from electrum.i18n import _
+from electrum.util import FileImportFailed, FileExportFailed
+from electrum.paymentrequest import PR_UNPAID, PR_PAID, PR_EXPIRED
+
+
 if platform.system() == 'Windows':
     MONOSPACE_FONT = 'Lucida Console'
 elif platform.system() == 'Darwin':
@@ -21,8 +25,6 @@ else:
 
 dialogs = []
 
-from electrum.paymentrequest import PR_UNPAID, PR_PAID, PR_EXPIRED
-
 pr_icons = {
     PR_UNPAID:":icons/unpaid.png",
     PR_PAID:":icons/confirmed.png",
@@ -200,9 +202,14 @@ class MessageBoxMixin(object):
     def msg_box(self, icon, parent, title, text, buttons=QMessageBox.Ok,
                 defaultButton=QMessageBox.NoButton):
         parent = parent or self.top_level_window()
-        d = QMessageBox(icon, title, str(text), buttons, parent)
+        if type(icon) is QPixmap:
+            d = QMessageBox(QMessageBox.Information, title, str(text), buttons, parent)
+            d.setIconPixmap(icon)
+        else:
+            d = QMessageBox(icon, title, str(text), buttons, parent)
         d.setWindowModality(Qt.WindowModal)
         d.setDefaultButton(defaultButton)
+        d.setTextInteractionFlags(Qt.TextSelectableByMouse)
         return d.exec_()
 
 class WindowModalDialog(QDialog, MessageBoxMixin):
@@ -216,7 +223,7 @@ class WindowModalDialog(QDialog, MessageBoxMixin):
 
 
 class WaitingDialog(WindowModalDialog):
-    '''Shows a please wait dialog whilst runnning a task.  It is not
+    '''Shows a please wait dialog whilst running a task.  It is not
     necessary to maintain a reference to this dialog.'''
     def __init__(self, parent, message, task, on_success=None, on_error=None):
         assert parent
@@ -228,6 +235,7 @@ class WaitingDialog(WindowModalDialog):
         self.accepted.connect(self.on_accepted)
         self.show()
         self.thread = TaskThread(self)
+        self.thread.finished.connect(self.deleteLater)  # see #3956
         self.thread.add(task, on_success, self.accept, on_error)
 
     def wait(self):
@@ -254,7 +262,7 @@ def line_dialog(parent, title, label, ok_label, default=None):
 def text_dialog(parent, title, label, ok_label, default=None, allow_multi=False):
     from .qrtextedit import ScanQRTextEdit
     dialog = WindowModalDialog(parent, title)
-    dialog.setMinimumWidth(500)
+    dialog.setMinimumWidth(600)
     l = QVBoxLayout()
     dialog.setLayout(l)
     l.addWidget(QLabel(label))
@@ -389,13 +397,18 @@ class MyTreeWidget(QTreeWidget):
         self.editor = None
         self.pending_update = False
         if editable_columns is None:
-            editable_columns = [stretch_column]
+            editable_columns = {stretch_column}
+        else:
+            editable_columns = set(editable_columns)
         self.editable_columns = editable_columns
         self.setItemDelegate(ElectrumItemDelegate(self))
         self.itemDoubleClicked.connect(self.on_doubleclick)
         self.update_headers(headers)
         self.current_filter = ""
 
+        self.setRootIsDecorated(False)  # remove left margin
+        self.toolbar_shown = False
+
     def update_headers(self, headers):
         self.setColumnCount(len(headers))
         self.setHeaderLabels(headers)
@@ -406,11 +419,15 @@ class MyTreeWidget(QTreeWidget):
 
     def editItem(self, item, column):
         if column in self.editable_columns:
-            self.editing_itemcol = (item, column, item.text(column))
-            # Calling setFlags causes on_changed events for some reason
-            item.setFlags(item.flags() | Qt.ItemIsEditable)
-            QTreeWidget.editItem(self, item, column)
-            item.setFlags(item.flags() & ~Qt.ItemIsEditable)
+            try:
+                self.editing_itemcol = (item, column, item.text(column))
+                # Calling setFlags causes on_changed events for some reason
+                item.setFlags(item.flags() | Qt.ItemIsEditable)
+                QTreeWidget.editItem(self, item, column)
+                item.setFlags(item.flags() & ~Qt.ItemIsEditable)
+            except RuntimeError:
+                # (item) wrapped C/C++ object has been deleted
+                pass
 
     def keyPressEvent(self, event):
         if event.key() in [ Qt.Key_F2, Qt.Key_Return ] and self.editor is None:
@@ -478,8 +495,12 @@ class MyTreeWidget(QTreeWidget):
             self.pending_update = True
         else:
             self.setUpdatesEnabled(False)
+            scroll_pos = self.verticalScrollBar().value()
             self.on_update()
             self.setUpdatesEnabled(True)
+            # To paint the list before resetting the scroll position
+            self.parent.app.processEvents()
+            self.verticalScrollBar().setValue(scroll_pos)
         if self.current_filter:
             self.filter(self.current_filter)
 
@@ -503,6 +524,37 @@ class MyTreeWidget(QTreeWidget):
             item.setHidden(all([item.text(column).lower().find(p) == -1
                                 for column in columns]))
 
+    def create_toolbar(self, config=None):
+        hbox = QHBoxLayout()
+        buttons = self.get_toolbar_buttons()
+        for b in buttons:
+            b.setVisible(False)
+            hbox.addWidget(b)
+        hide_button = QPushButton('x')
+        hide_button.setVisible(False)
+        hide_button.pressed.connect(lambda: self.show_toolbar(False, config))
+        self.toolbar_buttons = buttons + (hide_button,)
+        hbox.addStretch()
+        hbox.addWidget(hide_button)
+        return hbox
+
+    def save_toolbar_state(self, state, config):
+        pass  # implemented in subclasses
+
+    def show_toolbar(self, state, config=None):
+        if state == self.toolbar_shown:
+            return
+        self.toolbar_shown = state
+        if config:
+            self.save_toolbar_state(state, config)
+        for b in self.toolbar_buttons:
+            b.setVisible(state)
+        if not state:
+            self.on_hide_toolbar()
+
+    def toggle_toolbar(self, config=None):
+        self.show_toolbar(not self.toolbar_shown, config)
+
 
 class ButtonsWidget(QWidget):
 
@@ -589,12 +641,12 @@ class TaskThread(QThread):
             except BaseException:
                 self.doneSig.emit(sys.exc_info(), task.cb_done, task.cb_error)
 
-    def on_done(self, result, cb_done, cb):
+    def on_done(self, result, cb_done, cb_result):
         # This runs in the parent's thread.
         if cb_done:
             cb_done()
-        if cb:
-            cb(result)
+        if cb_result:
+            cb_result(result)
 
     def stop(self):
         self.tasks.put(None)
@@ -621,6 +673,7 @@ class ColorScheme:
     dark_scheme = False
 
     GREEN = ColorSchemeItem("#117c11", "#8af296")
+    YELLOW = ColorSchemeItem("#ffff00", "#ffff00")
     RED = ColorSchemeItem("#7c1111", "#f18c8c")
     BLUE = ColorSchemeItem("#123b7c", "#8cb3f2")
     DEFAULT = ColorSchemeItem("black", "white")
@@ -635,6 +688,97 @@ class ColorScheme:
         if ColorScheme.has_dark_background(widget):
             ColorScheme.dark_scheme = True
 
+
+class AcceptFileDragDrop:
+    def __init__(self, file_type=""):
+        assert isinstance(self, QWidget)
+        self.setAcceptDrops(True)
+        self.file_type = file_type
+
+    def validateEvent(self, event):
+        if not event.mimeData().hasUrls():
+            event.ignore()
+            return False
+        for url in event.mimeData().urls():
+            if not url.toLocalFile().endswith(self.file_type):
+                event.ignore()
+                return False
+        event.accept()
+        return True
+
+    def dragEnterEvent(self, event):
+        self.validateEvent(event)
+
+    def dragMoveEvent(self, event):
+        if self.validateEvent(event):
+            event.setDropAction(Qt.CopyAction)
+
+    def dropEvent(self, event):
+        if self.validateEvent(event):
+            for url in event.mimeData().urls():
+                self.onFileAdded(url.toLocalFile())
+
+    def onFileAdded(self, fn):
+        raise NotImplementedError()
+
+
+def import_meta_gui(electrum_window, title, importer, on_success):
+    filter_ = "JSON (*.json);;All files (*)"
+    filename = electrum_window.getOpenFileName(_("Open {} file").format(title), filter_)
+    if not filename:
+        return
+    try:
+        importer(filename)
+    except FileImportFailed as e:
+        electrum_window.show_critical(str(e))
+    else:
+        electrum_window.show_message(_("Your {} were successfully imported").format(title))
+        on_success()
+
+
+def export_meta_gui(electrum_window, title, exporter):
+    filter_ = "JSON (*.json);;All files (*)"
+    filename = electrum_window.getSaveFileName(_("Select file to save your {}").format(title),
+                                               'electrum_{}.json'.format(title), filter_)
+    if not filename:
+        return
+    try:
+        exporter(filename)
+    except FileExportFailed as e:
+        electrum_window.show_critical(str(e))
+    else:
+        electrum_window.show_message(_("Your {0} were exported to '{1}'")
+                                     .format(title, str(filename)))
+
+
+def get_parent_main_window(widget):
+    """Returns a reference to the ElectrumWindow this widget belongs to."""
+    from .main_window import ElectrumWindow
+    for _ in range(100):
+        if widget is None:
+            return None
+        if not isinstance(widget, ElectrumWindow):
+            widget = widget.parentWidget()
+        else:
+            return widget
+    return None
+
+class SortableTreeWidgetItem(QTreeWidgetItem):
+    DataRole = Qt.UserRole + 1
+
+    def __lt__(self, other):
+        column = self.treeWidget().sortColumn()
+        if None not in [x.data(column, self.DataRole) for x in [self, other]]:
+            # We have set custom data to sort by
+            return self.data(column, self.DataRole) < other.data(column, self.DataRole)
+        try:
+            # Is the value something numeric?
+            return float(self.text(column)) < float(other.text(column))
+        except ValueError:
+            # If not, we will just do string comparison
+            return self.text(column) < other.text(column)
+
+
 if __name__ == "__main__":
     app = QApplication([])
     t = WaitingDialog(None, 'testing ...', lambda: [time.sleep(1)], lambda x: QMessageBox.information(None, 'done', "done"))

gui/qt/utxo_list.py

@@ -32,6 +32,7 @@ class UTXOList(MyTreeWidget):
     def __init__(self, parent=None):
         MyTreeWidget.__init__(self, parent, self.create_menu, [ _('Address'), _('Label'), _('Amount'), _('Height'), _('Output point')], 1)
         self.setSelectionMode(QAbstractItemView.ExtendedSelection)
+        self.setSortingEnabled(True)
 
     def get_name(self, x):
         return x.get('prevout_hash') + ":%d"%x.get('prevout_n')
@@ -46,9 +47,10 @@ class UTXOList(MyTreeWidget):
             height = x.get('height')
             name = self.get_name(x)
             label = self.wallet.get_label(x.get('prevout_hash'))
-            amount = self.parent.format_amount(x['value'])
-            utxo_item = QTreeWidgetItem([address, label, amount, '%d'%height, name[0:10] + '...' + name[-2:]])
+            amount = self.parent.format_amount(x['value'], whitespaces=True)
+            utxo_item = SortableTreeWidgetItem([address, label, amount, '%d'%height, name[0:10] + '...' + name[-2:]])
             utxo_item.setFont(0, QFont(MONOSPACE_FONT))
+            utxo_item.setFont(2, QFont(MONOSPACE_FONT))
             utxo_item.setFont(4, QFont(MONOSPACE_FONT))
             utxo_item.setData(0, Qt.UserRole, name)
             if self.wallet.is_frozen(address):

icons.qrc

@@ -14,6 +14,7 @@
     <file>icons/electrum_light_icon.png</file>
     <file>icons/electrum_dark_icon.png</file>
     <file>icons/file.png</file>
+    <file>icons/info.png</file>
     <file>icons/keepkey.png</file>
     <file>icons/keepkey_unpaired.png</file>
     <file>icons/key.png</file>
@@ -22,6 +23,7 @@
     <file>icons/lock.png</file>
     <file>icons/microphone.png</file>
     <file>icons/network.png</file>
+    <file>icons/offline_tx.png</file>
     <file>icons/qrcode.png</file>
     <file>icons/qrcode_white.png</file>
     <file>icons/preferences.png</file>

lib/base_wizard.py

@@ -24,12 +24,19 @@
 # SOFTWARE.
 
 import os
+import sys
+import traceback
+
 from . import bitcoin
 from . import keystore
 from .keystore import bip44_derivation
 from .wallet import Imported_Wallet, Standard_Wallet, Multisig_Wallet, wallet_types
+from .storage import STO_EV_USER_PW, STO_EV_XPUB_PW, get_derivation_used_for_hw_device_encryption
 from .i18n import _
+from .util import UserCancelled
 
+# hardware device setup purpose
+HWD_SETUP_NEW_WALLET, HWD_SETUP_DECRYPT_WALLET = range(0, 2)
 
 class ScriptTypeNotSupported(Exception): pass
 
@@ -123,7 +130,7 @@ class BaseWizard(object):
             choices = [
                 ('choose_seed_type', _('Create a new seed')),
                 ('restore_from_seed', _('I already have a seed')),
-                ('restore_from_key', _('Use public or private keys')),
+                ('restore_from_key', _('Use a master key')),
             ]
             if not self.is_kivy:
                 choices.append(('choose_hw_device',  _('Use a hardware device')))
@@ -146,17 +153,22 @@ class BaseWizard(object):
                              is_valid=v, allow_multi=True)
 
     def on_import(self, text):
+        # create a temporary wallet and exploit that modifications
+        # will be reflected on self.storage
         if keystore.is_address_list(text):
-            self.wallet = Imported_Wallet(self.storage)
+            w = Imported_Wallet(self.storage)
             for x in text.split():
-                self.wallet.import_address(x)
+                w.import_address(x)
         elif keystore.is_private_key_list(text):
             k = keystore.Imported_KeyStore({})
             self.storage.put('keystore', k.dump())
-            self.wallet = Imported_Wallet(self.storage)
+            w = Imported_Wallet(self.storage)
             for x in text.split():
-                self.wallet.import_private_key(x, None)
-        self.terminate()
+                w.import_private_key(x, None)
+            self.keystores.append(w.keystore)
+        else:
+            return self.terminate()
+        return self.run('create_wallet')
 
     def restore_from_key(self):
         if self.wallet_type == 'standard':
@@ -175,7 +187,7 @@ class BaseWizard(object):
         k = keystore.from_master_key(text)
         self.on_keystore(k)
 
-    def choose_hw_device(self):
+    def choose_hw_device(self, purpose=HWD_SETUP_NEW_WALLET):
         title = _('Hardware Keystore')
         # check available plugins
         support = self.plugins.get_hardware_support()
@@ -184,7 +196,7 @@ class BaseWizard(object):
                 _('No hardware wallet support found on your system.'),
                 _('Please install the relevant libraries (eg python-trezor for Trezor).'),
             ])
-            self.confirm_dialog(title=title, message=msg, run_next= lambda x: self.choose_hw_device())
+            self.confirm_dialog(title=title, message=msg, run_next= lambda x: self.choose_hw_device(purpose))
             return
         # scan devices
         devices = []
@@ -204,34 +216,51 @@ class BaseWizard(object):
                 _('If your device is not detected on Windows, go to "Settings", "Devices", "Connected devices", and do "Remove device". Then, plug your device again.') + ' ',
                 _('On Linux, you might have to add a new permission to your udev rules.'),
             ])
-            self.confirm_dialog(title=title, message=msg, run_next= lambda x: self.choose_hw_device())
+            self.confirm_dialog(title=title, message=msg, run_next= lambda x: self.choose_hw_device(purpose))
             return
         # select device
         self.devices = devices
         choices = []
         for name, info in devices:
             state = _("initialized") if info.initialized else _("wiped")
-            label = info.label or _("An unnamed %s")%name
+            label = info.label or _("An unnamed {}").format(name)
             descr = "%s [%s, %s]" % (label, name, state)
             choices.append(((name, info), descr))
         msg = _('Select a device') + ':'
-        self.choice_dialog(title=title, message=msg, choices=choices, run_next=self.on_device)
+        self.choice_dialog(title=title, message=msg, choices=choices, run_next= lambda *args: self.on_device(*args, purpose=purpose))
 
-    def on_device(self, name, device_info):
+    def on_device(self, name, device_info, *, purpose):
         self.plugin = self.plugins.get_plugin(name)
         try:
-            self.plugin.setup_device(device_info, self)
+            self.plugin.setup_device(device_info, self, purpose)
+        except OSError as e:
+            self.show_error(_('We encountered an error while connecting to your device:')
+                            + '\n' + str(e) + '\n'
+                            + _('To try to fix this, we will now re-pair with your device.') + '\n'
+                            + _('Please try again.'))
+            devmgr = self.plugins.device_manager
+            devmgr.unpair_id(device_info.device.id_)
+            self.choose_hw_device(purpose)
+            return
         except BaseException as e:
             self.show_error(str(e))
-            self.choose_hw_device()
+            self.choose_hw_device(purpose)
             return
-        if self.wallet_type=='multisig':
-            # There is no general standard for HD multisig.
-            # This is partially compatible with BIP45; assumes index=0
-            self.on_hw_derivation(name, device_info, "m/45'/0")
+        if purpose == HWD_SETUP_NEW_WALLET:
+            if self.wallet_type=='multisig':
+                # There is no general standard for HD multisig.
+                # This is partially compatible with BIP45; assumes index=0
+                self.on_hw_derivation(name, device_info, "m/45'/0")
+            else:
+                f = lambda x: self.run('on_hw_derivation', name, device_info, str(x))
+                self.derivation_dialog(f)
+        elif purpose == HWD_SETUP_DECRYPT_WALLET:
+            derivation = get_derivation_used_for_hw_device_encryption()
+            xpub = self.plugin.get_xpub(device_info.device.id_, derivation, 'standard', self)
+            password = keystore.Xpub.get_pubkey_from_xpub(xpub, ())
+            self.storage.decrypt(password)
         else:
-            f = lambda x: self.run('on_hw_derivation', name, device_info, str(x))
-            self.derivation_dialog(f)
+            raise Exception('unknown purpose: %s' % purpose)
 
     def derivation_dialog(self, f):
         default = bip44_derivation(0, bip43_purpose=44)
@@ -364,13 +393,45 @@ class BaseWizard(object):
                 self.run('create_wallet')
 
     def create_wallet(self):
-        if any(k.may_have_password() for k in self.keystores):
-            self.request_password(run_next=self.on_password)
+        encrypt_keystore = any(k.may_have_password() for k in self.keystores)
+        # note: the following condition ("if") is duplicated logic from
+        # wallet.get_available_storage_encryption_version()
+        if self.wallet_type == 'standard' and isinstance(self.keystores[0], keystore.Hardware_KeyStore):
+            # offer encrypting with a pw derived from the hw device
+            k = self.keystores[0]
+            try:
+                k.handler = self.plugin.create_handler(self)
+                password = k.get_password_for_storage_encryption()
+            except UserCancelled:
+                devmgr = self.plugins.device_manager
+                devmgr.unpair_xpub(k.xpub)
+                self.choose_hw_device()
+                return
+            except BaseException as e:
+                traceback.print_exc(file=sys.stderr)
+                self.show_error(str(e))
+                return
+            self.request_storage_encryption(
+                run_next=lambda encrypt_storage: self.on_password(
+                    password,
+                    encrypt_storage=encrypt_storage,
+                    storage_enc_version=STO_EV_XPUB_PW,
+                    encrypt_keystore=False))
         else:
-            self.on_password(None, False)
+            # prompt the user to set an arbitrary password
+            self.request_password(
+                run_next=lambda password, encrypt_storage: self.on_password(
+                    password,
+                    encrypt_storage=encrypt_storage,
+                    storage_enc_version=STO_EV_USER_PW,
+                    encrypt_keystore=encrypt_keystore),
+                force_disable_encrypt_cb=not encrypt_keystore)
 
-    def on_password(self, password, encrypt):
-        self.storage.set_password(password, encrypt)
+    def on_password(self, password, *, encrypt_storage,
+                    storage_enc_version=STO_EV_USER_PW, encrypt_keystore):
+        self.storage.set_keystore_encryption(bool(password) and encrypt_keystore)
+        if encrypt_storage:
+            self.storage.set_password(password, enc_version=storage_enc_version)
         for k in self.keystores:
             if k.may_have_password():
                 k.update_password(None, password)
@@ -386,6 +447,13 @@ class BaseWizard(object):
             self.storage.write()
             self.wallet = Multisig_Wallet(self.storage)
             self.run('create_addresses')
+        elif self.wallet_type == 'imported':
+            if len(self.keystores) > 0:
+                keys = self.keystores[0].dump()
+                self.storage.put('keystore', keys)
+            self.wallet = Imported_Wallet(self.storage)
+            self.wallet.storage.write()
+            self.terminate()
 
     def show_xpub_and_add_cosigners(self, xpub):
         self.show_xpub_dialog(xpub=xpub, run_next=lambda x: self.run('choose_keystore'))
@@ -444,5 +512,5 @@ class BaseWizard(object):
             self.wallet.synchronize()
             self.wallet.storage.write()
             self.terminate()
-        msg = _("Electrum is generating your addresses, please wait.")
+        msg = _("Electrum is generating your addresses, please wait...")
         self.waiting_dialog(task, msg)

lib/bitcoin.py

@@ -37,89 +37,11 @@ from .util import bfh, bh2u, to_string
 from . import version
 from .util import print_error, InvalidPassword, assert_bytes, to_bytes, inv_dict
 from . import segwit_addr
+from . import constants
 
-def read_json(filename, default):
-    path = os.path.join(os.path.dirname(__file__), filename)
-    try:
-        with open(path, 'r') as f:
-            r = json.loads(f.read())
-    except:
-        r = default
-    return r
-
-
-
-
-# Version numbers for BIP32 extended keys
-# standard: xprv, xpub
-# segwit in p2sh: yprv, ypub
-# native segwit: zprv, zpub
-XPRV_HEADERS = {
-    'standard': 0x0488ade4,
-    'p2wpkh-p2sh': 0x049d7878,
-    'p2wsh-p2sh': 0x295b005,
-    'p2wpkh': 0x4b2430c,
-    'p2wsh': 0x2aa7a99
-}
-XPUB_HEADERS = {
-    'standard': 0x0488b21e,
-    'p2wpkh-p2sh': 0x049d7cb2,
-    'p2wsh-p2sh': 0x295b43f,
-    'p2wpkh': 0x4b24746,
-    'p2wsh': 0x2aa7ed3
-}
-
-
-class NetworkConstants:
-
-    # https://github.com/z-classic/zclassic/blob/master/src/chainparams.cpp#L103
-    @classmethod
-    def set_mainnet(cls):
-        cls.TESTNET = False
-        cls.WIF_PREFIX = 0x80
-        cls.ADDRTYPE_P2PKH = [0x1C, 0xB8]
-        cls.ADDRTYPE_P2SH = [0x1C, 0xBD]
-        cls.ADDRTYPE_SHIELDED = [0x16, 0x9A]
-        cls.SEGWIT_HRP = "bc" #TODO zcl has no segwit
-        cls.GENESIS = "0007104ccda289427919efc39dc9e4d499804b7bebc22df55f8b834301260602"
-        cls.DEFAULT_PORTS = {'t': '50001', 's': '50002'}
-        cls.DEFAULT_SERVERS = read_json('servers.json', {})
-        cls.CHECKPOINTS = read_json('checkpoints.json', [])
-        cls.EQUIHASH_N = 200
-        cls.EQUIHASH_K = 9
-        cls.HEADERS_URL = "http://headers.zcl-electrum.com/blockchain_headers"
-
-        cls.CHUNK_SIZE = 200
-
-    # https://github.com/z-classic/zclassic/blob/master/src/chainparams.cpp#L234
-    @classmethod
-    def set_testnet(cls):
-        cls.TESTNET = True
-        cls.WIF_PREFIX = 0xef
-        cls.ADDRTYPE_P2PKH = [0x1D, 0x25]
-        cls.ADDRTYPE_P2SH = [0x1C, 0xBA]
-        cls.ADDRTYPE_SHIELDED = [0x16, 0xB6]
-        cls.SEGWIT_HRP = "tb" #TODO zcl has no segwit
-        cls.GENESIS = "03e1c4bb705c871bf9bfda3e74b7f8f86bff267993c215a89d5795e3708e5e1f"
-        cls.DEFAULT_PORTS = {'t': '51001', 's': '51002'}
-        cls.DEFAULT_SERVERS = read_json('servers_testnet.json', {})
-        cls.CHECKPOINTS = read_json('checkpoints_testnet.json', [])
-        cls.EQUIHASH_N = 200
-        cls.EQUIHASH_K = 9
-
-        #cls.HEADERS_URL = "http://35.224.186.7/blockchain_headers"
-
-        cls.CHUNK_SIZE = 200
-
-NetworkConstants.set_mainnet()
 
 ################################## transactions
 
-FEE_STEP = 10000
-DEFAULT_FEE_RATE = 10000
-MAX_FEE_RATE = 300000
-FEE_TARGETS = [25, 10, 5, 2]
-
 COINBASE_MATURITY = 100
 COIN = 100000000
 
@@ -332,11 +254,11 @@ def ser_uint256(u):
     return rs
 
 def sha256(x):
-    if isinstance(x, str):
-        x = x.encode('utf8')
+    x = to_bytes(x, 'utf8')
     return bytes(hashlib.sha256(x).digest())
 
 def Hash(x):
+    x = to_bytes(x, 'utf8')
     out = bytes(sha256(sha256(x)))
     return out
 
@@ -429,16 +351,16 @@ def b58_address_to_hash160(addr):
 
 
 def hash160_to_p2pkh(h160):
-    return hash160_to_b58_address(h160, NetworkConstants.ADDRTYPE_P2PKH)
+    return hash160_to_b58_address(h160, constants.net.ADDRTYPE_P2PKH)
 
 def hash160_to_p2sh(h160):
-    return hash160_to_b58_address(h160, NetworkConstants.ADDRTYPE_P2SH)
+    return hash160_to_b58_address(h160, constants.net.ADDRTYPE_P2SH)
 
 def public_key_to_p2pkh(public_key):
     return hash160_to_p2pkh(hash_160(public_key))
 
 def hash_to_segwit_addr(h):
-    return segwit_addr.encode(NetworkConstants.SEGWIT_HRP, 0, h)
+    return segwit_addr.encode(constants.net.SEGWIT_HRP, 0, h)
 
 def public_key_to_p2wpkh(public_key):
     return hash_to_segwit_addr(hash_160(public_key))
@@ -484,7 +406,7 @@ def script_to_address(script):
     return addr
 
 def address_to_script(addr):
-    witver, witprog = segwit_addr.decode(NetworkConstants.SEGWIT_HRP, addr)
+    witver, witprog = segwit_addr.decode(constants.net.SEGWIT_HRP, addr)
     if witprog is not None:
         assert (0 <= witver <= 16)
         OP_n = witver + 0x50 if witver > 0 else 0
@@ -492,11 +414,11 @@ def address_to_script(addr):
         script += push_script(bh2u(bytes(witprog)))
         return script
     addrtype, hash_160 = b58_address_to_hash160(addr)
-    if addrtype == NetworkConstants.ADDRTYPE_P2PKH:
+    if addrtype == constants.net.ADDRTYPE_P2PKH:
         script = '76a9'                                      # op_dup, op_hash_160
         script += push_script(bh2u(hash_160))
         script += '88ac'                                     # op_equalverify, op_checksig
-    elif addrtype == NetworkConstants.ADDRTYPE_P2SH:
+    elif addrtype == constants.net.ADDRTYPE_P2SH:
         script = 'a9'                                        # op_hash_160
         script += push_script(bh2u(hash_160))
         script += '87'                                       # op_equal
@@ -600,9 +522,8 @@ def DecodeBase58Check(psz):
         return key
 
 
-
-# extended key export format for segwit
-
+# backwards compat
+# extended WIF for segwit (used in 3.0.x; but still used internally)
 SCRIPT_TYPES = {
     'p2pkh':0,
     'p2wpkh':1,
@@ -613,26 +534,43 @@ SCRIPT_TYPES = {
 }
 
 
-def serialize_privkey(secret, compressed, txin_type):
-    prefix = bytes([(SCRIPT_TYPES[txin_type]+NetworkConstants.WIF_PREFIX)&255])
+def serialize_privkey(secret, compressed, txin_type, internal_use=False):
+    if internal_use:
+        prefix = bytes([(SCRIPT_TYPES[txin_type] + constants.net.WIF_PREFIX) & 255])
+    else:
+        prefix = bytes([constants.net.WIF_PREFIX])
     suffix = b'\01' if compressed else b''
     vchIn = prefix + secret + suffix
-    return EncodeBase58Check(vchIn)
+    base58_wif = EncodeBase58Check(vchIn)
+    if internal_use:
+        return base58_wif
+    else:
+        return '{}:{}'.format(txin_type, base58_wif)
 
 
 def deserialize_privkey(key):
-    # whether the pubkey is compressed should be visible from the keystore
-    vch = DecodeBase58Check(key)
     if is_minikey(key):
         return 'p2pkh', minikey_to_private_key(key), True
-    elif vch:
-        txin_type = inv_dict(SCRIPT_TYPES)[vch[0] - NetworkConstants.WIF_PREFIX]
-        assert len(vch) in [33, 34]
-        compressed = len(vch) == 34
-        return txin_type, vch[1:33], compressed
-    else:
+
+    txin_type = None
+    if ':' in key:
+        txin_type, key = key.split(sep=':', maxsplit=1)
+        assert txin_type in SCRIPT_TYPES
+    vch = DecodeBase58Check(key)
+    if not vch:
         raise BaseException("cannot deserialize", key)
 
+    if txin_type is None:
+        # keys exported in version 3.0.x encoded script type in first byte
+        txin_type = inv_dict(SCRIPT_TYPES)[vch[0] - constants.net.WIF_PREFIX]
+    else:
+        assert vch[0] == constants.net.WIF_PREFIX
+
+    assert len(vch) in [33, 34]
+    compressed = len(vch) == 34
+    return txin_type, vch[1:33], compressed
+
+
 def regenerate_key(pk):
     assert len(pk) == 32
     return EC_KEY(pk)
@@ -662,7 +600,7 @@ def address_from_private_key(sec):
 
 def is_segwit_address(addr):
     try:
-        witver, witprog = segwit_addr.decode(NetworkConstants.SEGWIT_HRP, addr)
+        witver, witprog = segwit_addr.decode(constants.net.SEGWIT_HRP, addr)
     except Exception as e:
         return False
     return witprog is not None
@@ -672,7 +610,7 @@ def is_b58_address(addr):
         addrtype, h = b58_address_to_hash160(addr)
     except Exception as e:
         return False
-    if addrtype not in [NetworkConstants.ADDRTYPE_P2PKH, NetworkConstants.ADDRTYPE_P2SH]:
+    if addrtype not in [constants.net.ADDRTYPE_P2PKH, constants.net.ADDRTYPE_P2SH]:
         return False
     return addr == hash160_to_b58_address(h, addrtype)
 
@@ -735,8 +673,8 @@ def verify_message(address, sig, message):
         return False
 
 
-def encrypt_message(message, pubkey):
-    return EC_KEY.encrypt_message(message, bfh(pubkey))
+def encrypt_message(message, pubkey, magic=b'BIE1'):
+    return EC_KEY.encrypt_message(message, bfh(pubkey), magic)
 
 
 def chunks(l, n):
@@ -881,7 +819,7 @@ class EC_KEY(object):
     # ECIES encryption/decryption methods; AES-128-CBC with PKCS7 is used as the cipher; hmac-sha256 is used as the mac
 
     @classmethod
-    def encrypt_message(self, message, pubkey):
+    def encrypt_message(self, message, pubkey, magic=b'BIE1'):
         assert_bytes(message)
 
         pk = ser_to_point(pubkey)
@@ -895,20 +833,20 @@ class EC_KEY(object):
         iv, key_e, key_m = key[0:16], key[16:32], key[32:]
         ciphertext = aes_encrypt_with_iv(key_e, iv, message)
         ephemeral_pubkey = bfh(ephemeral.get_public_key(compressed=True))
-        encrypted = b'BIE1' + ephemeral_pubkey + ciphertext
+        encrypted = magic + ephemeral_pubkey + ciphertext
         mac = hmac.new(key_m, encrypted, hashlib.sha256).digest()
 
         return base64.b64encode(encrypted + mac)
 
-    def decrypt_message(self, encrypted):
+    def decrypt_message(self, encrypted, magic=b'BIE1'):
         encrypted = base64.b64decode(encrypted)
         if len(encrypted) < 85:
             raise Exception('invalid ciphertext: length')
-        magic = encrypted[:4]
+        magic_found = encrypted[:4]
         ephemeral_pubkey = encrypted[4:37]
         ciphertext = encrypted[37:-32]
         mac = encrypted[-32:]
-        if magic != b'BIE1':
+        if magic_found != magic:
             raise Exception('invalid ciphertext: invalid magic bytes')
         try:
             ephemeral_pubkey = ser_to_point(ephemeral_pubkey)
@@ -984,25 +922,35 @@ def _CKD_pub(cK, c, s):
     return cK_n, c_n
 
 
-def xprv_header(xtype):
-    return bfh("%08x" % XPRV_HEADERS[xtype])
+def xprv_header(xtype, *, net=None):
+    if net is None:
+        net = constants.net
+    return bfh("%08x" % net.XPRV_HEADERS[xtype])
 
 
-def xpub_header(xtype):
-    return bfh("%08x" % XPUB_HEADERS[xtype])
+def xpub_header(xtype, *, net=None):
+    if net is None:
+        net = constants.net
+    return bfh("%08x" % net.XPUB_HEADERS[xtype])
 
 
-def serialize_xprv(xtype, c, k, depth=0, fingerprint=b'\x00'*4, child_number=b'\x00'*4):
-    xprv = xprv_header(xtype) + bytes([depth]) + fingerprint + child_number + c + bytes([0]) + k
+def serialize_xprv(xtype, c, k, depth=0, fingerprint=b'\x00'*4,
+                   child_number=b'\x00'*4, *, net=None):
+    xprv = xprv_header(xtype, net=net) \
+           + bytes([depth]) + fingerprint + child_number + c + bytes([0]) + k
     return EncodeBase58Check(xprv)
 
 
-def serialize_xpub(xtype, c, cK, depth=0, fingerprint=b'\x00'*4, child_number=b'\x00'*4):
-    xpub = xpub_header(xtype) + bytes([depth]) + fingerprint + child_number + c + cK
+def serialize_xpub(xtype, c, cK, depth=0, fingerprint=b'\x00'*4,
+                   child_number=b'\x00'*4, *, net=None):
+    xpub = xpub_header(xtype, net=net) \
+           + bytes([depth]) + fingerprint + child_number + c + cK
     return EncodeBase58Check(xpub)
 
 
-def deserialize_xkey(xkey, prv):
+def deserialize_xkey(xkey, prv, *, net=None):
+    if net is None:
+        net = constants.net
     xkey = DecodeBase58Check(xkey)
     if len(xkey) != 78:
         raise BaseException('Invalid length')
@@ -1011,7 +959,7 @@ def deserialize_xkey(xkey, prv):
     child_number = xkey[9:13]
     c = xkey[13:13+32]
     header = int('0x' + bh2u(xkey[0:4]), 16)
-    headers = XPRV_HEADERS if prv else XPUB_HEADERS
+    headers = net.XPRV_HEADERS if prv else net.XPUB_HEADERS
     if header not in headers.values():
         raise BaseException('Invalid xpub format', hex(header))
     xtype = list(headers.keys())[list(headers.values()).index(header)]
@@ -1020,11 +968,11 @@ def deserialize_xkey(xkey, prv):
     return xtype, depth, fingerprint, child_number, c, K_or_k
 
 
-def deserialize_xpub(xkey):
-    return deserialize_xkey(xkey, False)
+def deserialize_xpub(xkey, *, net=None):
+    return deserialize_xkey(xkey, False, net=net)
 
-def deserialize_xprv(xkey):
-    return deserialize_xkey(xkey, True)
+def deserialize_xprv(xkey, *, net=None):
+    return deserialize_xkey(xkey, True, net=net)
 
 def xpub_type(x):
     return deserialize_xpub(x)[0]

lib/blockchain.py

@@ -27,6 +27,7 @@ from io import BytesIO
 
 from . import util
 from . import bitcoin
+from . import constants
 from .bitcoin import *
 import base64
 
@@ -84,7 +85,10 @@ def hash_header(header):
         return '0' * 64
     if header.get('prev_block_hash') is None:
         header['prev_block_hash'] = '00'*64
-    return hash_to_str(Hash(serialize_header(header)))
+        '''
+        TODO 32?
+        '''
+    return hash_encode(Hash(bfh(serialize_header(header))))
 
 
 blockchains = {}
@@ -131,7 +135,7 @@ class Blockchain(util.PrintError):
         self.config = config
         self.catch_up = None # interface catching up
         self.checkpoint = checkpoint
-        self.checkpoints = NetworkConstants.CHECKPOINTS
+        self.checkpoints = constants.net.CHECKPOINTS
         self.parent_id = parent_id
         self.lock = threading.Lock()
         with self.lock:
@@ -175,34 +179,34 @@ class Blockchain(util.PrintError):
 
     def update_size(self):
         p = self.path()
-        self._size = os.path.getsize(p)//bitcoin.HEADER_SIZE if os.path.exists(p) else 0
+        self._size = os.path.getsize(p)//80 if os.path.exists(p) else 0
 
-    def verify_header(self, header, prev_header):
-        if prev_header:
-            prev_hash = hash_header(prev_header)
-            if prev_hash != header.get('prev_block_hash'):
-                raise BaseException("prev hash mismatch: %s vs %s" % (prev_hash, header.get('prev_block_hash')))
-        _powhash = sha256_header(header)
-        target = self.bits_to_target(header['bits'])
-        if _powhash > target:
-            raise BaseException("insufficient proof of work: %s vs target %s" % (int('0x' + _powhash, 16), target))
+    def verify_header(self, header, prev_hash, target):
+        _hash = hash_header(header)
+        if prev_hash != header.get('prev_block_hash'):
+            raise BaseException("prev hash mismatch: %s vs %s" % (prev_hash, header.get('prev_block_hash')))
+        if constants.net.TESTNET:
+            return
+        bits = self.target_to_bits(target)
+        if bits != header.get('bits'):
+            raise BaseException("bits mismatch: %s vs %s" % (bits, header.get('bits')))
+        if int('0x' + _hash, 16) > target:
+            raise BaseException("insufficient proof of work: %s vs target %s" % (int('0x' + _hash, 16), target))
         nonce = uint256_from_bytes(str_to_hash(header.get('nonce')))
         n_solution = vector_from_bytes(base64.b64decode(header.get('n_solution').encode('utf8')))
         if not is_gbp_valid(serialize_header(header), nonce, n_solution,
-            NetworkConstants.EQUIHASH_N, NetworkConstants.EQUIHASH_K):
+            constants.net.EQUIHASH_N, constants.net.EQUIHASH_K):
             raise BaseException("Equihash invalid")
 
     def verify_chunk(self, index, data):
         num = len(data) // bitcoin.HEADER_SIZE
-        prev_header = None
-        if index != 0:
-            prev_header = self.read_header(index * NetworkConstants.CHUNK_SIZE - 1)
-
+        prev_hash = self.get_hash(index * constants.net.CHUNK_SIZE - 1)
+        target = self.get_target(index-1)
         for i in range(num):
             raw_header = data[i*bitcoin.HEADER_SIZE:(i+1) * bitcoin.HEADER_SIZE]
-            header = deserialize_header(raw_header, index*NetworkConstants.CHUNK_SIZE + i)
-            self.verify_header(header, prev_header)
-            prev_header = header
+            header = deserialize_header(raw_header, index*constants.net.CHUNK_SIZE + i)
+            self.verify_header(header, prev_hash, target)
+            prev_hash = hash_header(header)
 
     def path(self):
         d = util.get_headers_dir(self.config)
@@ -211,11 +215,12 @@ class Blockchain(util.PrintError):
 
     def save_chunk(self, index, chunk):
         filename = self.path()
-        d = (index * NetworkConstants.CHUNK_SIZE - self.checkpoint) * bitcoin.HEADER_SIZE
+        d = (index * constants.net.CHUNK_SIZE - self.checkpoint) * bitcoin.HEADER_SIZE
         if d < 0:
             chunk = chunk[-d:]
             d = 0
-        self.write(chunk, d)
+        truncate = index >= len(self.checkpoints)
+        self.write(chunk, d, truncate)
         self.swap_with_parent()
 
     def swap_with_parent(self):
@@ -253,11 +258,11 @@ class Blockchain(util.PrintError):
         blockchains[self.checkpoint] = self
         blockchains[parent.checkpoint] = parent
 
-    def write(self, data, offset):
+    def write(self, data, offset, truncate=True):
         filename = self.path()
         with self.lock:
             with open(filename, 'rb+') as f:
-                if offset != self._size*bitcoin.HEADER_SIZE:
+                if truncate and offset != self._size*bitcoin.HEADER_SIZE:
                     f.seek(offset)
                     f.truncate()
                 f.seek(offset)
@@ -268,7 +273,7 @@ class Blockchain(util.PrintError):
 
     def save_header(self, header):
         delta = header.get('block_height') - self.checkpoint
-        data = serialize_header(header)
+        data = bfh(serialize_header(header))
         assert delta == self.size()
         assert len(data) == bitcoin.HEADER_SIZE
         self.write(data, delta*bitcoin.HEADER_SIZE)
@@ -289,25 +294,52 @@ class Blockchain(util.PrintError):
             with open(name, 'rb') as f:
                 f.seek(delta * bitcoin.HEADER_SIZE)
                 h = f.read(bitcoin.HEADER_SIZE)
+        if h == bytes([0])*bitcoin.HEADER_SIZE:
+            return None
         return deserialize_header(h, height)
 
     def get_hash(self, height):
-        return self.hash_header(self.read_header(height))
+        if height == -1:
+            return '0000000000000000000000000000000000000000000000000000000000000000'
+        elif height == 0:
+            return constants.net.GENESIS
+        elif height < len(self.checkpoints) * constants.net.CHUNK_SIZE:
+            assert (height+1) % constants.net.CHUNK_SIZE == 0, height
+            index = height // constants.net.CHUNK_SIZE
+            h, t = self.checkpoints[index]
+            return h
+        else:
+            return hash_header(self.read_header(height))
 
-    def hash_header(self, header):
-        return hash_header(header)
+    def get_target(self, index):
+        # compute target from chunk x, used in chunk x+1
+        if constants.net.TESTNET:
+            return 0
+        if index == -1:
+            return MAX_TARGET
+        if index < len(self.checkpoints):
+            h, t = self.checkpoints[index]
+            return t
+        # new target
+        first = self.read_header(index * constants.net.CHUNK_SIZE)
+        last = self.read_header(index * constants.net.CHUNK_SIZE + (constants.net.CHUNK_SIZE - 1))
+        bits = last.get('bits')
+        target = self.bits_to_target(bits)
+        nActualTimespan = last.get('timestamp') - first.get('timestamp')
+        nTargetTimespan = 14 * 24 * 60 * 60
+        nActualTimespan = max(nActualTimespan, nTargetTimespan // 4)
+        nActualTimespan = min(nActualTimespan, nTargetTimespan * 4)
+        new_target = min(MAX_TARGET, (target * nActualTimespan) // nTargetTimespan)
+        return new_target
 
     def bits_to_target(self, bits):
         bitsN = (bits >> 24) & 0xff
-        # if not (bitsN >= 0x03 and bitsN <= 0x1d):
-        #     raise BaseException("First part of bits should be in [0x03, 0x1d]")
+        if not (bitsN >= 0x03 and bitsN <= 0x1d):
+            raise BaseException("First part of bits should be in [0x03, 0x1d]")
         bitsBase = bits & 0xffffff
-        # if not (bitsBase >= 0x8000 and bitsBase <= 0x7fffff):
-        #     raise BaseException("Second part of bits should be in [0x8000, 0x7fffff]")
-        if bitsN <= 3:
-            return bitsBase >> (8 * (3 - bitsN))
-        else:
-            return bitsBase << (8 * (bitsN - 3))
+        if not (bitsBase >= 0x8000 and bitsBase <= 0x7fffff):
+            raise BaseException("Second part of bits should be in [0x8000, 0x7fffff]")
+        return bitsBase << (8 * (bitsN-3))
 
     def target_to_bits(self, target):
         c = ("%064x" % target)[2:]
@@ -320,38 +352,42 @@ class Blockchain(util.PrintError):
         return bitsN << 24 | bitsBase
 
     def can_connect(self, header, check_height=True):
-        # import pdb; pdb.set_trace()
         height = header['block_height']
         if check_height and self.height() != height - 1:
             self.print_error("cannot connect at height", height)
             return False
         if height == 0:
-            return hash_header(header) == NetworkConstants.GENESIS
+            return hash_header(header) == constants.net.GENESIS
         try:
-            prev_header = self.read_header(height - 1)
-            prev_hash = self.hash_header(prev_header)
+            prev_hash = self.get_hash(height - 1)
         except:
             return False
         if prev_hash != header.get('prev_block_hash'):
             return False
+        target = self.get_target(height // constants.net.CHUNK_SIZE - 1)
         try:
-            self.verify_header(header, prev_header)
+            self.verify_header(header, prev_hash, target)
         except BaseException as e:
-            import traceback
-            traceback.print_exc()
-            self.print_error('verify_header failed', str(e))
             return False
         return True
 
     def connect_chunk(self, idx, hexdata):
         try:
-            data = bytes.fromhex(hexdata)
+            data = bfh(hexdata)
             self.verify_chunk(idx, data)
-            self.print_error("validated chunk %d" % idx)
+            #self.print_error("validated chunk %d" % idx)
             self.save_chunk(idx, data)
             return True
         except BaseException as e:
-            import traceback
-            traceback.print_exc()
-            self.print_error('verify_chunk failed', str(e))
+            self.print_error('verify_chunk %d failed'%idx, str(e))
             return False
+
+    def get_checkpoints(self):
+        # for each chunk, store the hash of the last block and the target after the chunk
+        cp = []
+        n = self.height() // 2016
+        for index in range(n):
+            h = self.get_hash((index+1) * 2016 -1)
+            target = self.get_target(index)
+            cp.append((h, target))
+        return cp

lib/coinchooser.py

@@ -25,7 +25,7 @@
 from collections import defaultdict, namedtuple
 from math import floor, log10
 
-from .bitcoin import sha256, COIN, TYPE_ADDRESS
+from .bitcoin import sha256, COIN, TYPE_ADDRESS, is_address
 from .transaction import Transaction
 from .util import NotEnoughFunds, PrintError
 
@@ -87,6 +87,8 @@ def strip_unneeded(bkts, sufficient_funds):
 
 class CoinChooserBase(PrintError):
 
+    enable_output_value_rounding = False
+
     def keys(self, coins):
         raise NotImplementedError
 
@@ -135,7 +137,13 @@ class CoinChooserBase(PrintError):
         zeroes = [trailing_zeroes(i) for i in output_amounts]
         min_zeroes = min(zeroes)
         max_zeroes = max(zeroes)
-        zeroes = range(max(0, min_zeroes - 1), (max_zeroes + 1) + 1)
+
+        if n > 1:
+            zeroes = range(max(0, min_zeroes - 1), (max_zeroes + 1) + 1)
+        else:
+            # if there is only one change output, this will ensure that we aim
+            # to have one that is exactly as precise as the most precise output
+            zeroes = [min_zeroes]
 
         # Calculate change; randomize it a bit if using more than 1 output
         remaining = change_amount
@@ -150,8 +158,10 @@ class CoinChooserBase(PrintError):
             n -= 1
 
         # Last change output.  Round down to maximum precision but lose
-        # no more than 100 satoshis to fees (2dp)
-        N = pow(10, min(2, zeroes[0]))
+        # no more than 10**max_dp_to_round_for_privacy
+        # e.g. a max of 2 decimal places means losing 100 satoshis to fees
+        max_dp_to_round_for_privacy = 2 if self.enable_output_value_rounding else 0
+        N = pow(10, min(max_dp_to_round_for_privacy, zeroes[0]))
         amount = (remaining // N) * N
         amounts.append(amount)
 
@@ -230,6 +240,13 @@ class CoinChooserBase(PrintError):
         tx.add_inputs([coin for b in buckets for coin in b.coins])
         tx_weight = get_tx_weight(buckets)
 
+        # change is sent back to sending address unless specified
+        if not change_addrs:
+            change_addrs = [tx.inputs()[0]['address']]
+            # note: this is not necessarily the final "first input address"
+            # because the inputs had not been sorted at this point
+            assert is_address(change_addrs[0])
+
         # This takes a count of change outputs and returns a tx fee
         output_weight = 4 * Transaction.estimated_output_size(change_addrs[0])
         fee = lambda count: fee_estimator_w(tx_weight + count * output_weight)
@@ -273,6 +290,8 @@ class CoinChooserRandom(CoinChooserBase):
                     candidates.add(tuple(sorted(permutation[:count + 1])))
                     break
             else:
+                # FIXME this assumes that the effective value of any bkt is >= 0
+                # we should make sure not to choose buckets with <= 0 eff. val.
                 raise NotEnoughFunds()
 
         candidates = [[buckets[n] for n in c] for c in candidates]
@@ -284,7 +303,7 @@ class CoinChooserRandom(CoinChooserBase):
         Any bucket can be:
         1. "confirmed" if it only contains confirmed coins; else
         2. "unconfirmed" if it does not contain coins with unconfirmed parents
-        3. "unconfirmed parent" otherwise
+        3. other: e.g. "unconfirmed parent" or "local"
 
         This method tries to only use buckets of type 1, and if the coins there
         are not enough, tries to use the next type but while also selecting
@@ -292,9 +311,9 @@ class CoinChooserRandom(CoinChooserBase):
         """
         conf_buckets = [bkt for bkt in buckets if bkt.min_height > 0]
         unconf_buckets = [bkt for bkt in buckets if bkt.min_height == 0]
-        unconf_par_buckets = [bkt for bkt in buckets if bkt.min_height == -1]
+        other_buckets = [bkt for bkt in buckets if bkt.min_height < 0]
 
-        bucket_sets = [conf_buckets, unconf_buckets, unconf_par_buckets]
+        bucket_sets = [conf_buckets, unconf_buckets, other_buckets]
         already_selected_buckets = []
 
         for bkts_choose_from in bucket_sets:
@@ -368,4 +387,6 @@ def get_name(config):
 
 def get_coin_chooser(config):
     klass = COIN_CHOOSERS[get_name(config)]
-    return klass()
+    coinchooser = klass()
+    coinchooser.enable_output_value_rounding = config.get('coin_chooser_output_rounding', False)
+    return coinchooser

lib/commands.py

@@ -34,7 +34,7 @@ from functools import wraps
 from decimal import Decimal
 
 from .import util
-from .util import bfh, bh2u, format_satoshis
+from .util import bfh, bh2u, format_satoshis, json_decode, print_error
 from .import bitcoin
 from .bitcoin import is_address,  hash_160, COIN, TYPE_ADDRESS
 from .i18n import _
@@ -82,7 +82,7 @@ def command(s):
             password = kwargs.get('password')
             if c.requires_wallet and wallet is None:
                 raise BaseException("wallet not loaded. Use 'electrum daemon load_wallet'")
-            if c.requires_password and password is None and wallet.storage.get('use_encryption'):
+            if c.requires_password and password is None and wallet.has_password():
                 return {'error': 'Password required' }
             return func(*args, **kwargs)
         return func_wrapper
@@ -138,6 +138,8 @@ class Commands:
     @command('wp')
     def password(self, password=None, new_password=None):
         """Change wallet password. """
+        if self.wallet.storage.is_encrypted_with_hw_device() and new_password:
+            raise Exception("Can't change the password of a wallet encrypted with a hw device.")
         b = self.wallet.storage.is_encrypted()
         self.wallet.update_password(password, new_password, b)
         self.wallet.storage.write()
@@ -151,10 +153,8 @@ class Commands:
     @command('')
     def setconfig(self, key, value):
         """Set a configuration variable. 'value' may be a string or a Python expression."""
-        try:
-            value = ast.literal_eval(value)
-        except:
-            pass
+        if key not in ('rpcuser', 'rpcpassword'):
+            value = json_decode(value)
         self.config.set_key(key, value)
         return True
 
@@ -177,7 +177,8 @@ class Commands:
         """Return the transaction history of any address. Note: This is a
         walletless server query, results are not checked by SPV.
         """
-        return self.network.synchronous_get(('blockchain.address.get_history', [address]))
+        sh = bitcoin.address_to_scripthash(address)
+        return self.network.synchronous_get(('blockchain.scripthash.get_history', [sh]))
 
     @command('w')
     def listunspent(self):
@@ -194,7 +195,8 @@ class Commands:
         """Returns the UTXO list of any address. Note: This
         is a walletless server query, results are not checked by SPV.
         """
-        return self.network.synchronous_get(('blockchain.address.listunspent', [address]))
+        sh = bitcoin.address_to_scripthash(address)
+        return self.network.synchronous_get(('blockchain.scripthash.listunspent', [sh]))
 
     @command('')
     def serialize(self, jsontx):
@@ -316,20 +318,12 @@ class Commands:
         """Return the balance of any address. Note: This is a walletless
         server query, results are not checked by SPV.
         """
-        out = self.network.synchronous_get(('blockchain.address.get_balance', [address]))
+        sh = bitcoin.address_to_scripthash(address)
+        out = self.network.synchronous_get(('blockchain.scripthash.get_balance', [sh]))
         out["confirmed"] =  str(Decimal(out["confirmed"])/COIN)
         out["unconfirmed"] =  str(Decimal(out["unconfirmed"])/COIN)
         return out
 
-    @command('n')
-    def getproof(self, address):
-        """Get Merkle branch of an address in the UTXO set"""
-        p = self.network.synchronous_get(('blockchain.address.get_proof', [address]))
-        out = []
-        for i,s in p:
-            out.append(i)
-        return out
-
     @command('n')
     def getmerkle(self, txid, height):
         """Get Merkle branch of a transaction included in a block. Electrum
@@ -448,46 +442,20 @@ class Commands:
         return tx.as_dict()
 
     @command('w')
-    def history(self):
+    def history(self, year=None, show_addresses=False, show_fiat=False):
         """Wallet history. Returns the transaction history of your wallet."""
-        balance = 0
-        out = []
-        for item in self.wallet.get_history():
-            tx_hash, height, conf, timestamp, value, balance = item
-            if timestamp:
-                date = datetime.datetime.fromtimestamp(timestamp).isoformat(' ')[:-3]
-            else:
-                date = "----"
-            label = self.wallet.get_label(tx_hash)
-            tx = self.wallet.transactions.get(tx_hash)
-            tx.deserialize()
-            input_addresses = []
-            output_addresses = []
-            for x in tx.inputs():
-                if x['type'] == 'coinbase': continue
-                addr = x.get('address')
-                if addr == None: continue
-                if addr == "(pubkey)":
-                    prevout_hash = x.get('prevout_hash')
-                    prevout_n = x.get('prevout_n')
-                    _addr = self.wallet.find_pay_to_pubkey_address(prevout_hash, prevout_n)
-                    if _addr:
-                        addr = _addr
-                input_addresses.append(addr)
-            for addr, v in tx.get_outputs():
-                output_addresses.append(addr)
-            out.append({
-                'txid': tx_hash,
-                'timestamp': timestamp,
-                'date': date,
-                'input_addresses': input_addresses,
-                'output_addresses': output_addresses,
-                'label': label,
-                'value': str(Decimal(value)/COIN) if value is not None else None,
-                'height': height,
-                'confirmations': conf
-            })
-        return out
+        kwargs = {'show_addresses': show_addresses}
+        if year:
+            import time
+            start_date = datetime.datetime(year, 1, 1)
+            end_date = datetime.datetime(year+1, 1, 1)
+            kwargs['from_timestamp'] = time.mktime(start_date.timetuple())
+            kwargs['to_timestamp'] = time.mktime(end_date.timetuple())
+        if show_fiat:
+            from .exchange_rate import FxThread
+            fx = FxThread(self.config, None)
+            kwargs['fx'] = fx
+        return self.wallet.get_full_history(**kwargs)
 
     @command('w')
     def setlabel(self, key, label):
@@ -631,6 +599,15 @@ class Commands:
         out = self.wallet.get_payment_request(addr, self.config)
         return self._format_request(out)
 
+    @command('w')
+    def addtransaction(self, tx):
+        """ Add a transaction to the wallet history """
+        tx = Transaction(tx)
+        if not self.wallet.add_transaction(tx.txid(), tx):
+            return False
+        self.wallet.save_transactions()
+        return tx.txid()
+
     @command('wp')
     def signrequest(self, address, password=None):
         "Sign payment request with an OpenAlias"
@@ -658,13 +635,15 @@ class Commands:
             import urllib.request
             headers = {'content-type':'application/json'}
             data = {'address':address, 'status':x.get('result')}
+            serialized_data = util.to_bytes(json.dumps(data))
             try:
-                req = urllib.request.Request(URL, json.dumps(data), headers)
+                req = urllib.request.Request(URL, serialized_data, headers)
                 response_stream = urllib.request.urlopen(req, timeout=5)
                 util.print_error('Got Response for %s' % address)
             except BaseException as e:
                 util.print_error(str(e))
-        self.network.send([('blockchain.address.subscribe', [address])], callback)
+        h = self.network.addr_to_scripthash(address)
+        self.network.send([('blockchain.scripthash.subscribe', [h])], callback)
         return True
 
     @command('wn')
@@ -672,6 +651,12 @@ class Commands:
         """ return wallet synchronization status """
         return self.wallet.is_up_to_date()
 
+    @command('n')
+    def getfeerate(self):
+        """Return current optimal fee rate per kilobyte, according
+        to config settings (static/dynamic)"""
+        return self.config.fee_per_kb()
+
     @command('')
     def help(self):
         # for the python console
@@ -727,6 +712,9 @@ command_options = {
     'pending':     (None, "Show only pending requests."),
     'expired':     (None, "Show only expired requests."),
     'paid':        (None, "Show only paid requests."),
+    'show_addresses': (None, "Show input and output addresses"),
+    'show_fiat':   (None, "Show fiat value of transactions"),
+    'year':        (None, "Show history for a given year"),
 }
 
 
@@ -737,6 +725,7 @@ arg_types = {
     'num': int,
     'nbits': int,
     'imax': int,
+    'year': int,
     'entropy': int,
     'tx': tx_from_str,
     'pubkeys': json_loads,
@@ -800,7 +789,7 @@ def subparser_call(self, parser, namespace, values, option_string=None):
         parser = self._name_parser_map[parser_name]
     except KeyError:
         tup = parser_name, ', '.join(self._name_parser_map)
-        msg = _('unknown parser %r (choices: %s)') % tup
+        msg = _('unknown parser {!r} (choices: {})').format(*tup)
         raise ArgumentError(self, msg)
     # parse all the remaining options into the namespace
     # store any unrecognized options on the object, so that the top
@@ -825,7 +814,6 @@ def add_global_options(parser):
     group.add_argument("-P", "--portable", action="store_true", dest="portable", default=False, help="Use local 'electrum_data' directory")
     group.add_argument("-w", "--wallet", dest="wallet_path", help="wallet path")
     group.add_argument("--testnet", action="store_true", dest="testnet", default=False, help="Use Testnet")
-    group.add_argument("--nossl", action="store_true", dest="nossl", default=False, help="Disable SSL")
 
 def get_parser():
     # create main parser

lib/constants.py

@@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Electrum - lightweight Bitcoin client
+# Copyright (C) 2018 The Electrum developers
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation files
+# (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge,
+# publish, distribute, sublicense, and/or sell copies of the Software,
+# and to permit persons to whom the Software is furnished to do so,
+# subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import os
+import json
+
+
+def read_json(filename, default):
+    path = os.path.join(os.path.dirname(__file__), filename)
+    try:
+        with open(path, 'r') as f:
+            r = json.loads(f.read())
+    except:
+        r = default
+    return r
+
+
+class BitcoinMainnet:
+
+    TESTNET = False
+    WIF_PREFIX = 0x80
+    ADDRTYPE_P2PKH = [0x1C, 0xB8] 
+    ADDRTYPE_P2SH = [0x1C, 0xBD]
+    ADDRTYPE_SHIELDED = [0x16, 0x9A]
+    SEGWIT_HRP = "bc" # (No ZCL Segwit)
+    GENESIS = "0007104ccda289427919efc39dc9e4d499804b7bebc22df55f8b834301260602"
+    DEFAULT_PORTS = {'t': '50001', 's': '50002'}
+    DEFAULT_SERVERS = read_json('servers.json', {})
+    CHECKPOINTS = read_json('checkpoints.json', [])
+
+    EQUIHASH_N = 200
+    EQUIHASH_K = 9
+
+    CHUNK_SIZE = 200
+
+    HEADERS_URL = "http://headers.zcl-electrum.com/blockchain_headers"
+
+    XPRV_HEADERS = {
+        'standard':    0x0488ade4,  # xprv
+        'p2wpkh-p2sh': 0x049d7878,  # yprv
+        'p2wsh-p2sh':  0x0295b005,  # Yprv
+        'p2wpkh':      0x04b2430c,  # zprv
+        'p2wsh':       0x02aa7a99,  # Zprv
+    }
+    XPUB_HEADERS = {
+        'standard':    0x0488b21e,  # xpub
+        'p2wpkh-p2sh': 0x049d7cb2,  # ypub
+        'p2wsh-p2sh':  0x0295b43f,  # Ypub
+        'p2wpkh':      0x04b24746,  # zpub
+        'p2wsh':       0x02aa7ed3,  # Zpub
+    }
+
+
+class BitcoinTestnet:
+
+    TESTNET = True
+    WIF_PREFIX = 0xef
+    ADDRTYPE_P2PKH = [0x1D, 0x25] 
+    ADDRTYPE_P2SH = [0x1C, 0xBA] 
+    ADDTYPE_SHIELDED = [0x16, 0xB6]
+    SEGWIT_HRP = "tb" # (ZCL has no Segwit)
+    GENESIS = "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943"
+    DEFAULT_PORTS = {'t': '51001', 's': '51002'}
+    DEFAULT_SERVERS = read_json('servers_testnet.json', {})
+    CHECKPOINTS = read_json('checkpoints_testnet.json', [])
+
+    EQUIHASH_N = 200
+    EQUIHASH_K = 9
+
+    CHUNK_SIZE = 200
+
+    XPRV_HEADERS = {
+        'standard':    0x04358394,  # tprv
+        'p2wpkh-p2sh': 0x044a4e28,  # uprv
+        'p2wsh-p2sh':  0x024285b5,  # Uprv
+        'p2wpkh':      0x045f18bc,  # vprv
+        'p2wsh':       0x02575048,  # Vprv
+    }
+    XPUB_HEADERS = {
+        'standard':    0x043587cf,  # tpub
+        'p2wpkh-p2sh': 0x044a5262,  # upub
+        'p2wsh-p2sh':  0x024285ef,  # Upub
+        'p2wpkh':      0x045f1cf6,  # vpub
+        'p2wsh':       0x02575483,  # Vpub
+    }
+
+
+# don't import net directly, import the module instead (so that net is singleton)
+net = BitcoinMainnet
+
+
+def set_mainnet():
+    global net
+    net = BitcoinMainnet
+
+
+def set_testnet():
+    global net
+    net = BitcoinTestnet

lib/contacts.py

@@ -23,9 +23,12 @@
 import re
 import dns
 import json
+import traceback
+import sys
 
 from . import bitcoin
 from . import dnssec
+from .util import export_meta, import_meta
 
 
 class Contacts(dict):
@@ -48,14 +51,15 @@ class Contacts(dict):
         self.storage.put('contacts', dict(self))
 
     def import_file(self, path):
-        try:
-            with open(path, 'r') as f:
-                d = self._validate(json.loads(f.read()))
-        except:
-            return
-        self.update(d)
+        import_meta(path, self._validate, self.load_meta)
+
+    def load_meta(self, data):
+        self.update(data)
         self.save()
 
+    def export_file(self, filename):
+        export_meta(self, filename)
+
     def __setitem__(self, key, value):
         dict.__setitem__(self, key, value)
         self.save()
@@ -113,13 +117,13 @@ class Contacts(dict):
             return None
             
     def _validate(self, data):
-        for k,v in list(data.items()):
+        for k, v in list(data.items()):
             if k == 'contacts':
                 return self._validate(v)
             if not bitcoin.is_address(k):
                 data.pop(k)
             else:
-                _type,_ = v
+                _type, _ = v
                 if _type != 'address':
                     data.pop(k)
         return data

lib/currencies.json

@@ -2,6 +2,7 @@
     "CoinMarketCap": [
         "USD",
         "EUR",
-        "CNY"
+        "RUB",
+        "USD"
     ]
-}
+ }

lib/daemon.py

@@ -25,6 +25,8 @@
 import ast
 import os
 import time
+import traceback
+import sys
 
 # from jsonrpc import JSONRPCResponseManager
 import jsonrpclib
@@ -121,13 +123,12 @@ class Daemon(DaemonThread):
         self.config = config
         if config.get('offline'):
             self.network = None
-            self.fx = None
         else:
             self.network = Network(config)
             self.network.start()
-            self.fx = FxThread(config, self.network)
+        self.fx = FxThread(config, self.network)
+        if self.network:
             self.network.add_jobs([self.fx])
-
         self.gui = None
         self.wallets = {}
         # Setup JSONRPC server
@@ -173,7 +174,7 @@ class Daemon(DaemonThread):
             path = config.get_wallet_path()
             wallet = self.load_wallet(path, config.get('password'))
             self.cmd_runner.wallet = wallet
-            response = True
+            response = wallet is not None
         elif sub == 'close_wallet':
             path = config.get_wallet_path()
             if path in self.wallets:
@@ -301,4 +302,8 @@ class Daemon(DaemonThread):
             gui_name = 'qt'
         gui = __import__('electrum_gui.' + gui_name, fromlist=['electrum_gui'])
         self.gui = gui.ElectrumGui(config, self, plugins)
-        self.gui.main()
+        try:
+            self.gui.main()
+        except BaseException as e:
+            traceback.print_exc(file=sys.stdout)
+            # app will exit now

lib/exchange_rate.py

@@ -2,9 +2,12 @@ from datetime import datetime
 import inspect
 import requests
 import sys
+import os
+import json
 from threading import Thread
 import time
 import csv
+import decimal
 from decimal import Decimal
 
 from .bitcoin import COIN
@@ -13,15 +16,12 @@ from .util import PrintError, ThreadJob
 
 
 # See https://en.wikipedia.org/wiki/ISO_4217
-CCY_PRECISIONS = {}
-'''
-{'BHD': 3, 'BIF': 0, 'BYR': 0, 'CLF': 4, 'CLP': 0,
-'CVE': 0, 'DJF': 0, 'GNF': 0, 'IQD': 3, 'ISK': 0,
-'JOD': 3, 'JPY': 0, 'KMF': 0, 'KRW': 0, 'KWD': 3,
-'LYD': 3, 'MGA': 1, 'MRO': 1, 'OMR': 3, 'PYG': 0,
-'RWF': 0, 'TND': 3, 'UGX': 0, 'UYI': 0, 'VND': 0,
-'VUV': 0, 'XAF': 0, 'XAU': 4, 'XOF': 0, 'XPF': 0}
-'''
+CCY_PRECISIONS = {'BHD': 3, 'BIF': 0, 'BYR': 0, 'CLF': 4, 'CLP': 0,
+                  'CVE': 0, 'DJF': 0, 'GNF': 0, 'IQD': 3, 'ISK': 0,
+                  'JOD': 3, 'JPY': 0, 'KMF': 0, 'KRW': 0, 'KWD': 3,
+                  'LYD': 3, 'MGA': 1, 'MRO': 1, 'OMR': 3, 'PYG': 0,
+                  'RWF': 0, 'TND': 3, 'UGX': 0, 'UYI': 0, 'VND': 0,
+                  'VUV': 0, 'XAF': 0, 'XAU': 4, 'XOF': 0, 'XPF': 0}
 
 class ExchangeBase(PrintError):
 
@@ -34,7 +34,7 @@ class ExchangeBase(PrintError):
     def get_json(self, site, get_string):
         # APIs must have https
         url = ''.join(['https://', site, get_string])
-        response = requests.request('GET', url, headers={'User-Agent' : 'Electrum'})
+        response = requests.request('GET', url, headers={'User-Agent' : 'Electrum'}, timeout=10)
         return response.json()
 
     def get_csv(self, site, get_string):
@@ -60,28 +60,54 @@ class ExchangeBase(PrintError):
         t.setDaemon(True)
         t.start()
 
-    def get_historical_rates_safe(self, ccy):
+    def read_historical_rates(self, ccy, cache_dir):
+        filename = os.path.join(cache_dir, self.name() + '_'+ ccy)
+        if os.path.exists(filename):
+            timestamp = os.stat(filename).st_mtime
+            try:
+                with open(filename, 'r') as f:
+                    h = json.loads(f.read())
+                h['timestamp'] = timestamp
+            except:
+                h = None
+        else:
+            h = None
+        if h:
+            self.history[ccy] = h
+            self.on_history()
+        return h
+
+    def get_historical_rates_safe(self, ccy, cache_dir):
         try:
             self.print_error("requesting fx history for", ccy)
-            self.history[ccy] = self.historical_rates(ccy)
+            h = self.request_history(ccy)
             self.print_error("received fx history for", ccy)
-            self.on_history()
         except BaseException as e:
             self.print_error("failed fx history:", e)
+            return
+        filename = os.path.join(cache_dir, self.name() + '_' + ccy)
+        with open(filename, 'w') as f:
+            f.write(json.dumps(h))
+        h['timestamp'] = time.time()
+        self.history[ccy] = h
+        self.on_history()
 
-    def get_historical_rates(self, ccy):
-        result = self.history.get(ccy)
-        if not result and ccy in self.history_ccys():
-            t = Thread(target=self.get_historical_rates_safe, args=(ccy,))
+    def get_historical_rates(self, ccy, cache_dir):
+        if ccy not in self.history_ccys():
+            return
+        h = self.history.get(ccy)
+        if h is None:
+            h = self.read_historical_rates(ccy, cache_dir)
+        if h is None or h['timestamp'] < time.time() - 24*3600:
+            t = Thread(target=self.get_historical_rates_safe, args=(ccy, cache_dir))
             t.setDaemon(True)
             t.start()
-        return result
 
     def history_ccys(self):
         return []
 
     def historical_rate(self, ccy, d_t):
-        return self.history.get(ccy, {}).get(d_t.strftime('%Y-%m-%d'))
+        return self.history.get(ccy, {}).get(d_t.strftime('%Y-%m-%d'), 'NaN')
 
     def get_currencies(self):
         rates = self.get_rates('')
@@ -126,7 +152,9 @@ def get_exchanges_and_currencies():
         exchange = klass(None, None)
         try:
             d[name] = exchange.get_currencies()
+            print(name, "ok")
         except:
+            print(name, "error")
             continue
     with open(path, 'w') as f:
         f.write(json.dumps(d, indent=4, sort_keys=True))
@@ -157,7 +185,10 @@ class FxThread(ThreadJob):
         self.history_used_spot = False
         self.ccy_combo = None
         self.hist_checkbox = None
+        self.cache_dir = os.path.join(config.path, 'cache')
         self.set_exchange(self.config_exchange())
+        if not os.path.exists(self.cache_dir):
+            os.mkdir(self.cache_dir)
 
     def get_currencies(self, h):
         d = get_exchanges_by_ccy(h)
@@ -170,13 +201,17 @@ class FxThread(ThreadJob):
     def ccy_amount_str(self, amount, commas):
         prec = CCY_PRECISIONS.get(self.ccy, 2)
         fmt_str = "{:%s.%df}" % ("," if commas else "", max(0, prec))
-        return fmt_str.format(round(amount, prec))
+        try:
+            rounded_amount = round(amount, prec)
+        except decimal.InvalidOperation:
+            rounded_amount = amount
+        return fmt_str.format(rounded_amount)
 
     def run(self):
         # This runs from the plugins thread which catches exceptions
         if self.is_enabled():
             if self.timeout ==0 and self.show_history():
-                self.exchange.get_historical_rates(self.ccy)
+                self.exchange.get_historical_rates(self.ccy, self.cache_dir)
             if self.timeout <= time.time():
                 self.timeout = time.time() + 150
                 self.exchange.update(self.ccy)
@@ -193,6 +228,12 @@ class FxThread(ThreadJob):
     def set_history_config(self, b):
         self.config.set_key('history_rates', bool(b))
 
+    def get_history_capital_gains_config(self):
+        return bool(self.config.get('history_rates_capital_gains', False))
+
+    def set_history_capital_gains_config(self, b):
+        self.config.set_key('history_rates_capital_gains', bool(b))
+
     def get_fiat_address_config(self):
         return bool(self.config.get('fiat_address'))
 
@@ -224,45 +265,65 @@ class FxThread(ThreadJob):
         # A new exchange means new fx quotes, initially empty.  Force
         # a quote refresh
         self.timeout = 0
+        self.exchange.read_historical_rates(self.ccy, self.cache_dir)
 
     def on_quotes(self):
-        self.network.trigger_callback('on_quotes')
+        if self.network:
+            self.network.trigger_callback('on_quotes')
 
     def on_history(self):
-        self.network.trigger_callback('on_history')
+        if self.network:
+            self.network.trigger_callback('on_history')
 
     def exchange_rate(self):
         '''Returns None, or the exchange rate as a Decimal'''
         rate = self.exchange.quotes.get(self.ccy)
-        if rate:
-            return Decimal(rate)
+        if rate is None:
+            return Decimal('NaN')
+        return Decimal(rate)
+
+    def format_amount(self, btc_balance):
+        rate = self.exchange_rate()
+        return '' if rate.is_nan() else "%s" % self.value_str(btc_balance, rate)
 
     def format_amount_and_units(self, btc_balance):
         rate = self.exchange_rate()
-        return '' if rate is None else "%s %s" % (self.value_str(btc_balance, rate), self.ccy)
+        return '' if rate.is_nan() else "%s %s" % (self.value_str(btc_balance, rate), self.ccy)
 
     def get_fiat_status_text(self, btc_balance, base_unit, decimal_point):
         rate = self.exchange_rate()
-        return _("  (No exchange rate available)") if rate is None else " 1 %s=%s %s" % (base_unit,
+        return _("  (No FX rate available)") if rate.is_nan() else " 1 %s~%s %s" % (base_unit,
             self.value_str(COIN / (10**(8 - decimal_point)), rate), self.ccy)
 
+    def fiat_value(self, satoshis, rate):
+        return Decimal('NaN') if satoshis is None else Decimal(satoshis) / COIN * Decimal(rate)
+
     def value_str(self, satoshis, rate):
-        if satoshis is None:  # Can happen with incomplete history
-            return _("Unknown")
-        if rate:
-            value = Decimal(satoshis) / COIN * Decimal(rate)
-            return "%s" % (self.ccy_amount_str(value, True))
-        return _("No data")
+        return self.format_fiat(self.fiat_value(satoshis, rate))
+
+    def format_fiat(self, value):
+        if value.is_nan():
+            return _("No data")
+        return "%s" % (self.ccy_amount_str(value, True))
 
     def history_rate(self, d_t):
+        if d_t is None:
+            return Decimal('NaN')
         rate = self.exchange.historical_rate(self.ccy, d_t)
         # Frequently there is no rate for today, until tomorrow :)
         # Use spot quotes in that case
-        if rate is None and (datetime.today().date() - d_t.date()).days <= 2:
-            rate = self.exchange.quotes.get(self.ccy)
+        if rate == 'NaN' and (datetime.today().date() - d_t.date()).days <= 2:
+            rate = self.exchange.quotes.get(self.ccy, 'NaN')
             self.history_used_spot = True
-        return rate
+        return Decimal(rate)
 
     def historical_value_str(self, satoshis, d_t):
-        rate = self.history_rate(d_t)
-        return self.value_str(satoshis, rate)
+        return self.format_fiat(self.historical_value(satoshis, d_t))
+
+    def historical_value(self, satoshis, d_t):
+        return self.fiat_value(satoshis, self.history_rate(d_t))
+
+    def timestamp_rate(self, timestamp):
+        from electrum.util import timestamp_to_datetime
+        date = timestamp_to_datetime(timestamp)
+        return self.history_rate(date)

lib/interface.py

@@ -31,9 +31,7 @@ import threading
 import time
 import traceback
 
-from .util import print_error, get_cert_path
-
-ca_path = get_cert_path()
+from .util import print_error
 
 from . import util
 from . import x509

lib/keystore.py

@@ -28,7 +28,7 @@ from unicodedata import normalize
 
 from . import bitcoin
 from .bitcoin import *
-
+from . import constants
 from .util import PrintError, InvalidPassword, hfu
 from .mnemonic import Mnemonic, load_wordlist
 from .plugins import run_hook
@@ -45,6 +45,10 @@ class KeyStore(PrintError):
     def can_import(self):
         return False
 
+    def may_have_password(self):
+        """Returns whether the keystore can be encrypted with a password."""
+        raise NotImplementedError()
+
     def get_tx_derivations(self, tx):
         keypairs = {}
         for txin in tx.inputs():
@@ -116,9 +120,6 @@ class Imported_KeyStore(Software_KeyStore):
     def is_deterministic(self):
         return False
 
-    def can_change_password(self):
-        return True
-
     def get_master_public_key(self):
         return None
 
@@ -138,7 +139,10 @@ class Imported_KeyStore(Software_KeyStore):
     def import_privkey(self, sec, password):
         txin_type, privkey, compressed = deserialize_privkey(sec)
         pubkey = public_key_from_private_key(privkey, compressed)
-        self.keypairs[pubkey] = pw_encode(sec, password)
+        # re-serialize the key so the internal storage format is consistent
+        serialized_privkey = serialize_privkey(
+            privkey, compressed, txin_type, internal_use=True)
+        self.keypairs[pubkey] = pw_encode(serialized_privkey, password)
         return txin_type, pubkey
 
     def delete_imported_key(self, key):
@@ -196,9 +200,6 @@ class Deterministic_KeyStore(Software_KeyStore):
     def is_watching_only(self):
         return not self.has_seed()
 
-    def can_change_password(self):
-        return not self.is_watching_only()
-
     def add_seed(self, seed):
         if self.seed:
             raise Exception("a seed exists")
@@ -522,9 +523,13 @@ class Hardware_KeyStore(KeyStore, Xpub):
         assert not self.has_seed()
         return False
 
-    def can_change_password(self):
-        return False
-
+    def get_password_for_storage_encryption(self):
+        from .storage import get_derivation_used_for_hw_device_encryption
+        client = self.plugin.get_client(self)
+        derivation = get_derivation_used_for_hw_device_encryption()
+        xpub = client.get_xpub(derivation, "standard")
+        password = self.get_pubkey_from_xpub(xpub, ())
+        return password
 
 
 def bip39_normalize_passphrase(passphrase):
@@ -683,7 +688,7 @@ is_bip32_key = lambda x: is_xprv(x) or is_xpub(x)
 
 
 def bip44_derivation(account_id, bip43_purpose=44):
-    coin = 1 if bitcoin.NetworkConstants.TESTNET else 147 # ZCL - SLIP0044
+    coin = 1 if constants.net.TESTNET else 147 # ZCL - SLIP0044
     return "m/%d'/%d'/%d'" % (bip43_purpose, coin, int(account_id))
 
 def from_seed(seed, passphrase, is_p2sh):

lib/network.py

@@ -37,9 +37,11 @@ import socks
 from . import util
 from . import bitcoin
 from .bitcoin import *
+from . import constants
 from .interface import Connection, Interface
 from . import blockchain
 from .version import ELECTRUM_VERSION, PROTOCOL_VERSION
+from .i18n import _
 
 
 NODES_RETRY_INTERVAL = 60
@@ -59,7 +61,7 @@ def parse_servers(result):
             for v in item[2]:
                 if re.match("[st]\d*", v):
                     protocol, port = v[0], v[1:]
-                    if port == '': port = NetworkConstants.DEFAULT_PORTS[protocol]
+                    if port == '': port = constants.net.DEFAULT_PORTS[protocol]
                     out[protocol] = port
                 elif re.match("v(.?)+", v):
                     version = v[1:]
@@ -93,7 +95,7 @@ def filter_protocol(hostmap, protocol = 's'):
 
 def pick_random_server(hostmap = None, protocol = 's', exclude_set = set()):
     if hostmap is None:
-        hostmap = NetworkConstants.DEFAULT_SERVERS
+        hostmap = constants.net.DEFAULT_SERVERS
     eligible = list(set(filter_protocol(hostmap, protocol)) - exclude_set)
     return random.choice(eligible) if eligible else None
 
@@ -137,7 +139,7 @@ def deserialize_proxy(s):
 
 
 def deserialize_server(server_str):
-    host, port, protocol = str(server_str).split(':')
+    host, port, protocol = str(server_str).rsplit(':', 2)
     assert protocol in 'st'
     int(port)    # Throw if cannot be converted to int
     return host, port, protocol
@@ -171,17 +173,17 @@ class Network(util.DaemonThread):
         self.blockchain_index = config.get('blockchain_index', 0)
         if self.blockchain_index not in self.blockchains.keys():
             self.blockchain_index = 0
-        self.protocol = 't' if self.config.get('nossl') else 's'
         # Server for addresses and transactions
-        self.default_server = self.config.get('server')
+        self.default_server = self.config.get('server', None)
         # Sanitize default server
-        try:
-            host, port, protocol = deserialize_server(self.default_server)
-            assert protocol == self.protocol
-        except:
-            self.default_server = None
+        if self.default_server:
+            try:
+                deserialize_server(self.default_server)
+            except:
+                self.print_error('Warning: failed to parse server-string; falling back to random.')
+                self.default_server = None
         if not self.default_server:
-            self.default_server = pick_random_server(protocol=self.protocol)
+            self.default_server = pick_random_server()
         self.lock = threading.Lock()
         self.pending_sends = []
         self.message_id = 0
@@ -220,7 +222,8 @@ class Network(util.DaemonThread):
         self.connecting = set()
         self.requested_chunks = set()
         self.socket_queue = queue.Queue()
-        self.start_network(self.protocol, deserialize_proxy(self.config.get('proxy')))
+        self.start_network(deserialize_server(self.default_server)[2],
+                           deserialize_proxy(self.config.get('proxy')))
 
     def register_callback(self, callback, events):
         with self.lock:
@@ -305,6 +308,9 @@ class Network(util.DaemonThread):
         # Resend unanswered requests
         requests = self.unanswered_requests.values()
         self.unanswered_requests = {}
+        if self.interface.ping_required():
+            params = [ELECTRUM_VERSION, PROTOCOL_VERSION]
+            self.queue_request('server.version', params, self.interface)
         for request in requests:
             message_id = self.queue_request(request[0], request[1])
             self.unanswered_requests[message_id] = request
@@ -313,15 +319,14 @@ class Network(util.DaemonThread):
         self.queue_request('server.peers.subscribe', [])
         self.request_fee_estimates()
         self.queue_request('blockchain.relayfee', [])
-        if self.interface.ping_required():
-            params = [ELECTRUM_VERSION, PROTOCOL_VERSION]
-            self.queue_request('server.version', params, self.interface)
         for h in self.subscribed_addresses:
             self.queue_request('blockchain.scripthash.subscribe', [h])
 
     def request_fee_estimates(self):
+        from .simple_config import FEE_ETA_TARGETS
         self.config.requested_fee_estimates()
-        for i in bitcoin.FEE_TARGETS:
+        self.queue_request('mempool.get_fee_histogram', [])
+        for i in FEE_ETA_TARGETS:
             self.queue_request('blockchain.estimatefee', [i])
 
     def get_status_value(self, key):
@@ -331,6 +336,8 @@ class Network(util.DaemonThread):
             value = self.banner
         elif key == 'fee':
             value = self.config.fee_estimates
+        elif key == 'fee_histogram':
+            value = self.config.mempool_fees
         elif key == 'updated':
             value = (self.get_local_height(), self.get_server_height())
         elif key == 'servers':
@@ -358,7 +365,7 @@ class Network(util.DaemonThread):
         return list(self.interfaces.keys())
 
     def get_servers(self):
-        out = NetworkConstants.DEFAULT_SERVERS
+        out = constants.net.DEFAULT_SERVERS
         if self.irc_servers:
             out.update(filter_version(self.irc_servers.copy()))
         else:
@@ -542,6 +549,11 @@ class Network(util.DaemonThread):
         elif method == 'server.donation_address':
             if error is None:
                 self.donation_address = result
+        elif method == 'mempool.get_fee_histogram':
+            if error is None:
+                self.print_error('fee_histogram', result)
+                self.config.mempool_fees = result
+                self.notify('fee_histogram')
         elif method == 'blockchain.estimatefee':
             if error is None and result > 0:
                 i = params[0]
@@ -767,29 +779,33 @@ class Network(util.DaemonThread):
         error = response.get('error')
         result = response.get('result')
         params = response.get('params')
+        blockchain = interface.blockchain
         if result is None or params is None or error is not None:
             interface.print_error(error or 'bad response')
             return
         index = params[0]
         # Ignore unsolicited chunks
         if index not in self.requested_chunks:
+            interface.print_error("received chunk %d (unsolicited)" % index)
             return
+        else:
+            interface.print_error("received chunk %d" % index)
         self.requested_chunks.remove(index)
-        connect = interface.blockchain.connect_chunk(index, result)
+        connect = blockchain.connect_chunk(index, result)
         if not connect:
             self.connection_down(interface.server)
             return
         # If not finished, get the next chunk
-        if interface.blockchain.height() < interface.tip:
+        if index >= len(blockchain.checkpoints) and blockchain.height() < interface.tip:
             self.request_chunk(interface, index+1)
         else:
             interface.mode = 'default'
-            interface.print_error('catch up done', interface.blockchain.height())
-            interface.blockchain.catch_up = None
+            interface.print_error('catch up done', blockchain.height())
+            blockchain.catch_up = None
         self.notify('updated')
 
     def request_header(self, interface, height):
-        interface.print_error("requesting header %d" % height)
+        #interface.print_error("requesting header %d" % height)
         self.queue_request('blockchain.block.get_header', [height], interface)
         interface.request = height
         interface.req_time = time.time()
@@ -802,11 +818,10 @@ class Network(util.DaemonThread):
             self.connection_down(interface.server)
             return
         height = header.get('block_height')
-        if int(interface.request) != height:
+        if interface.request != height:
             interface.print_error("unsolicited header",interface.request, height)
             self.connection_down(interface.server)
             return
-        interface.print_error("interface.mode %s" % interface.mode)
         chain = blockchain.check_header(header)
         if interface.mode == 'backward':
             can_connect = blockchain.can_connect(header)
@@ -822,6 +837,7 @@ class Network(util.DaemonThread):
                 interface.blockchain = chain
                 interface.good = height
                 next_height = (interface.bad + interface.good) // 2
+                assert next_height >= self.max_checkpoint(), (interface.bad, interface.good)
             else:
                 if height == 0:
                     self.connection_down(interface.server)
@@ -830,7 +846,8 @@ class Network(util.DaemonThread):
                     interface.bad = height
                     interface.bad_header = header
                     delta = interface.tip - height
-                    next_height = max(0, interface.tip - 2 * delta)
+                    next_height = max(self.max_checkpoint(), interface.tip - 2 * delta)
+
         elif interface.mode == 'binary':
             if chain:
                 interface.good = height
@@ -840,6 +857,7 @@ class Network(util.DaemonThread):
                 interface.bad_header = header
             if interface.bad != interface.good + 1:
                 next_height = (interface.bad + interface.good) // 2
+                assert next_height >= self.max_checkpoint()
             elif not interface.blockchain.can_connect(interface.bad_header, check_height=False):
                 self.connection_down(interface.server)
                 next_height = None
@@ -908,7 +926,7 @@ class Network(util.DaemonThread):
         # If not finished, get the next header
         if next_height:
             if interface.mode == 'catch_up' and interface.tip > next_height + 50:
-                self.request_chunk(interface, next_height // NetworkConstants.CHUNK_SIZE)
+                self.request_chunk(interface, next_height // constants.net.CHUNK_SIZE)
             else:
                 self.request_header(interface, next_height)
         else:
@@ -949,36 +967,18 @@ class Network(util.DaemonThread):
 
     def init_headers_file(self):
         b = self.blockchains[0]
-        print(b.get_hash(0), NetworkConstants.GENESIS)
-        if b.get_hash(0) == NetworkConstants.GENESIS:
-            self.downloading_headers = False
-            return
         filename = b.path()
-        def download_thread():
-            try:
-                import urllib, socket
-                socket.setdefaulttimeout(30)
-                self.print_error("downloading ", NetworkConstants.HEADERS_URL)
-                urllib.request.urlretrieve(NetworkConstants.HEADERS_URL, filename)
-                self.print_error("done.")
-            except Exception:
-                import traceback
-                traceback.print_exc()
-                self.print_error("download failed. creating file", filename)
-                open(filename, 'wb+').close()
-            b = self.blockchains[0]
-            with b.lock: b.update_size()
-            self.downloading_headers = False
-
-        self.downloading_headers = True
-        t = threading.Thread(target = download_thread)
-        t.daemon = True
-        t.start()
+        length = bitcoin.HEADER_SIZE * len(constants.net.CHECKPOINTS) * constants.net.CHUNK_SIZE
+        if not os.path.exists(filename) or os.path.getsize(filename) < length:
+            with open(filename, 'wb') as f:
+                if length>0:
+                    f.seek(length-1)
+                    f.write(b'\x00')
+        with b.lock:
+            b.update_size()
 
     def run(self):
         self.init_headers_file()
-        while self.is_running() and self.downloading_headers:
-            time.sleep(1)
         while self.is_running():
             self.maintain_sockets()
             self.wait_on_sockets()
@@ -990,10 +990,11 @@ class Network(util.DaemonThread):
 
     def on_notify_header(self, interface, header):
         height = header.get('block_height')
-
         if not height:
             return
-
+        if height < self.max_checkpoint():
+            self.connection_down(interface.server)
+            return
         interface.tip_header = header
         interface.tip = height
 
@@ -1078,7 +1079,7 @@ class Network(util.DaemonThread):
         try:
             r = q.get(True, timeout)
         except queue.Empty:
-            raise BaseException('Server did not answer')
+            raise util.TimeoutException(_('Server did not answer'))
         if r.get('error'):
             raise BaseException(r.get('error'))
         return r.get('result')
@@ -1092,3 +1093,12 @@ class Network(util.DaemonThread):
         if out != tx_hash:
             return False, "error: " + out
         return True, out
+
+    def export_checkpoints(self, path):
+        # run manually from the console to generate checkpoints
+        cp = self.blockchain().get_checkpoints()
+        with open(path, 'w') as f:
+            f.write(json.dumps(cp, indent=4))
+
+    def max_checkpoint(self):
+        return max(0, len(constants.net.CHECKPOINTS) * 2016 - 1)

lib/paymentrequest.py

@@ -39,7 +39,8 @@ except ImportError:
 
 from . import bitcoin
 from . import util
-from .util import print_error, bh2u, bfh, get_cert_path
+from .util import print_error, bh2u, bfh
+from .util import export_meta, import_meta
 from . import transaction
 from . import x509
 from . import rsakey
@@ -49,7 +50,7 @@ from .bitcoin import TYPE_ADDRESS
 REQUEST_HEADERS = {'Accept': 'application/bitcoin-paymentrequest', 'User-Agent': 'Electrum'}
 ACK_HEADERS = {'Content-Type':'application/bitcoin-payment','Accept':'application/bitcoin-paymentack','User-Agent':'Electrum'}
 
-ca_path = get_cert_path()
+ca_path = requests.certs.where()
 ca_list = None
 ca_keyID = None
 
@@ -467,24 +468,29 @@ class InvoiceStore(object):
                 continue
 
     def import_file(self, path):
-        try:
-            with open(path, 'r') as f:
-                d = json.loads(f.read())
-                self.load(d)
-        except:
-            traceback.print_exc(file=sys.stderr)
-            return
+        def validate(data):
+            return data  # TODO
+        import_meta(path, validate, self.on_import)
+
+    def on_import(self, data):
+        self.load(data)
         self.save()
 
-    def save(self):
-        l = {}
+    def export_file(self, filename):
+        export_meta(self.dump(), filename)
+
+    def dump(self):
+        d = {}
         for k, pr in self.invoices.items():
-            l[k] = {
+            d[k] = {
                 'hex': bh2u(pr.raw),
                 'requestor': pr.requestor,
                 'txid': pr.tx
             }
-        self.storage.put('invoices', l)
+        return d
+
+    def save(self):
+        self.storage.put('invoices', self.dump())
 
     def get_status(self, key):
         pr = self.get(key)

lib/plot.py

@@ -14,17 +14,23 @@ from matplotlib.patches import Ellipse
 from matplotlib.offsetbox import AnchoredOffsetbox, TextArea, DrawingArea, HPacker
 
 
-def plot_history(wallet, history):
+class NothingToPlotException(Exception):
+    def __str__(self):
+        return _("Nothing to plot.")
+
+
+def plot_history(history):
+    if len(history) == 0:
+        raise NothingToPlotException()
     hist_in = defaultdict(int)
     hist_out = defaultdict(int)
     for item in history:
-        tx_hash, height, confirmations, timestamp, value, balance = item
-        if not confirmations:
+        if not item['confirmations']:
             continue
-        if timestamp is None:
+        if item['timestamp'] is None:
             continue
-        value = value*1./COIN
-        date = datetime.datetime.fromtimestamp(timestamp)
+        value = item['value'].value/COIN
+        date = item['date']
         datenum = int(md.date2num(datetime.date(date.year, date.month, 1)))
         if value > 0:
             hist_in[datenum] += value
@@ -43,12 +49,19 @@ def plot_history(wallet, history):
     xfmt = md.DateFormatter('%Y-%m')
     ax.xaxis.set_major_formatter(xfmt)
     width = 20
-    dates, values = zip(*sorted(hist_in.items()))
-    r1 = axarr[0].bar(dates, values, width, label='incoming')
-    axarr[0].legend(loc='upper left')
+
+    r1 = None
+    r2 = None
+    dates_values = list(zip(*sorted(hist_in.items())))
+    if dates_values and len(dates_values) == 2:
+        dates, values = dates_values
+        r1 = axarr[0].bar(dates, values, width, label='incoming')
+        axarr[0].legend(loc='upper left')
     dates_values = list(zip(*sorted(hist_out.items())))
     if dates_values and len(dates_values) == 2:
         dates, values = dates_values
         r2 = axarr[1].bar(dates, values, width, color='r', label='outgoing')
         axarr[1].legend(loc='upper left')
+    if r1 is None and r2 is None:
+        raise NothingToPlotException()
     return plt

lib/plugins.py

@@ -312,6 +312,8 @@ class DeviceMgr(ThreadJob, PrintError):
         # What we recognise.  Each entry is a (vendor_id, product_id)
         # pair.
         self.recognised_hardware = set()
+        # Custom enumerate functions for devices we don't know about.
+        self.enumerate_func = set()
         # For synchronization
         self.lock = threading.RLock()
         self.hid_lock = threading.RLock()
@@ -334,6 +336,9 @@ class DeviceMgr(ThreadJob, PrintError):
         for pair in device_pairs:
             self.recognised_hardware.add(pair)
 
+    def register_enumerate_func(self, func):
+        self.enumerate_func.add(func)
+
     def create_client(self, device, handler, plugin):
         # Get from cache first
         client = self.client_lookup(device.id_)
@@ -362,15 +367,20 @@ class DeviceMgr(ThreadJob, PrintError):
             if not xpub in self.xpub_ids:
                 return
             _id = self.xpub_ids.pop(xpub)
-        client = self.client_lookup(_id)
-        self.clients.pop(client, None)
-        if client:
-            client.close()
+            self._close_client(_id)
 
     def unpair_id(self, id_):
         xpub = self.xpub_by_id(id_)
         if xpub:
             self.unpair_xpub(xpub)
+        else:
+            self._close_client(id_)
+
+    def _close_client(self, id_):
+        client = self.client_lookup(id_)
+        self.clients.pop(client, None)
+        if client:
+            client.close()
 
     def pair_xpub(self, xpub, id_):
         with self.lock:
@@ -442,11 +452,11 @@ class DeviceMgr(ThreadJob, PrintError):
         # The user input has wrong PIN or passphrase, or cancelled input,
         # or it is not pairable
         raise DeviceUnpairableError(
-            _('Electrum cannot pair with your %s.\n\n'
+            _('Electrum cannot pair with your {}.\n\n'
               'Before you request bitcoins to be sent to addresses in this '
               'wallet, ensure you can pair with your device, or that you have '
               'its seed (and passphrase, if any).  Otherwise all bitcoins you '
-              'receive will be unspendable.') % plugin.device)
+              'receive will be unspendable.').format(plugin.device))
 
     def unpaired_device_infos(self, handler, plugin, devices=None):
         '''Returns a list of DeviceInfo objects: one for each connected,
@@ -472,9 +482,9 @@ class DeviceMgr(ThreadJob, PrintError):
             infos = self.unpaired_device_infos(handler, plugin, devices)
             if infos:
                 break
-            msg = _('Please insert your %s.  Verify the cable is '
+            msg = _('Please insert your {}.  Verify the cable is '
                     'connected and that no other application is using it.\n\n'
-                    'Try to connect again?') % plugin.device
+                    'Try to connect again?').format(plugin.device)
             if not handler.yes_no_question(msg):
                 raise UserCancelled()
             devices = None
@@ -484,7 +494,7 @@ class DeviceMgr(ThreadJob, PrintError):
         for info in infos:
             if info.label == keystore.label:
                 return info
-        msg = _("Please select which %s device to use:") % plugin.device
+        msg = _("Please select which {} device to use:").format(plugin.device)
         descriptions = [info.label + ' (%s)'%(_("initialized") if info.initialized else _("wiped")) for info in infos]
         c = handler.query_choice(msg, descriptions)
         if c is None:
@@ -492,7 +502,8 @@ class DeviceMgr(ThreadJob, PrintError):
         info = infos[c]
         # save new label
         keystore.set_label(info.label)
-        handler.win.wallet.save_keystore()
+        if handler.win.wallet is not None:
+            handler.win.wallet.save_keystore()
         return info
 
     def scan_devices(self):
@@ -504,6 +515,7 @@ class DeviceMgr(ThreadJob, PrintError):
         self.print_error("scanning devices...")
         with self.hid_lock:
             hid_list = hid.enumerate(0, 0)
+
         # First see what's connected that we know about
         devices = []
         for d in hid_list:
@@ -519,6 +531,10 @@ class DeviceMgr(ThreadJob, PrintError):
                 devices.append(Device(d['path'], interface_number,
                                       id_, product_key, usage_page))
 
+        # Let plugin handlers enumerate devices we don't know about
+        for f in self.enumerate_func:
+            devices.extend(f())
+
         # Now find out what was disconnected
         pairs = [(dev.path, dev.id_) for dev in devices]
         disconnected_ids = []

lib/qrscanner.py

@@ -29,8 +29,8 @@ import ctypes
 
 if sys.platform == 'darwin':
     name = 'libzbar.dylib'
-elif sys.platform == 'windows':
-    name = 'libzbar.dll'
+elif sys.platform in ('windows', 'win32'):
+    name = 'libzbar-0.dll'
 else:
     name = 'libzbar.so.0'
 
@@ -40,7 +40,7 @@ except OSError:
     libzbar = None
 
 
-def scan_barcode(device='', timeout=-1, display=True, threaded=False):
+def scan_barcode(device='', timeout=-1, display=True, threaded=False, try_again=True):
     if libzbar is None:
         raise RuntimeError("Cannot start QR scanner; zbar not available.")
     libzbar.zbar_symbol_get_data.restype = ctypes.c_char_p
@@ -50,6 +50,10 @@ def scan_barcode(device='', timeout=-1, display=True, threaded=False):
     proc = libzbar.zbar_processor_create(threaded)
     libzbar.zbar_processor_request_size(proc, 640, 480)
     if libzbar.zbar_processor_init(proc, device.encode('utf-8'), display) != 0:
+        if try_again:
+            # workaround for a bug in "ZBar for Windows"
+            # libzbar.zbar_processor_init always seem to fail the first time around
+            return scan_barcode(device, timeout, display, threaded, try_again=False)
         raise RuntimeError("Can not start QR scanner; initialization failed.")
     libzbar.zbar_processor_set_visible(proc)
     if libzbar.zbar_process_one(proc, timeout):

lib/servers-orig.json

@@ -1,231 +0,0 @@
-{
-    "E-X.not.fyi": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "ELECTRUMX.not.fyi": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "ELEX01.blackpole.online": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "VPS.hsmiths.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "bitcoin.freedomnode.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "btc.smsys.me": {
-        "pruning": "-",
-        "s": "995",
-        "version": "1.1"
-    },
-    "currentlane.lovebitco.in": {
-        "pruning": "-",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "daedalus.bauerj.eu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "de01.hamster.science": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "ecdsa.net": {
-        "pruning": "-",
-        "s": "110",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "elec.luggs.co": {
-        "pruning": "-",
-        "s": "443",
-        "version": "1.1"
-    },
-    "electrum.akinbo.org": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.antumbra.se": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.be": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.coinucopia.io": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.cutie.ga": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.festivaldelhumor.org": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.hsmiths.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.qtornado.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum.vom-stausee.de": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrum3.hachre.de": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrumx.bot.nu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "electrumx.westeurope.cloudapp.azure.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "elx01.knas.systems": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "ex-btc.server-on.net": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "helicarrier.bauerj.eu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "mooo.not.fyi": {
-        "pruning": "-",
-        "s": "50012",
-        "t": "50011",
-        "version": "1.1"
-    },
-    "ndnd.selfhost.eu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "node.arihanc.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "node.xbt.eu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "node1.volatilevictory.com": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "noserver4u.de": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "qmebr.spdns.org": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "raspi.hsmiths.com": {
-        "pruning": "-",
-        "s": "51002",
-        "t": "51001",
-        "version": "1.1"
-    },
-    "s2.noip.pl": {
-        "pruning": "-",
-        "s": "50102",
-        "version": "1.1"
-    },
-    "s5.noip.pl": {
-        "pruning": "-",
-        "s": "50105",
-        "version": "1.1"
-    },
-    "songbird.bauerj.eu": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "us.electrum.be": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    },
-    "us01.hamster.science": {
-        "pruning": "-",
-        "s": "50002",
-        "t": "50001",
-        "version": "1.1"
-    }
-}

lib/servers.json

@@ -1,4 +1,4 @@
 {
-    "electrum.zclassic.community": {"s":"50002"},
-    "zclele.duckdns.org": {"s":"50002"}
+    "electrum.zclassic.community": {"s":"50002", "pruning": "-", "version": "1.1"},
+    "zclele.duckdns.org": {"s":"50002", "pruning": "-", "version": "1.1"}
 }

lib/servers_testnet.json

@@ -1,3 +1,3 @@
 {
-    "localhost": {"t":"51001", "s":"51002"}
+    "localhost": {"t":"51001", "s":"51002", "pruning": "-", "version": "1.1"}
 }

lib/simple_config.py

@@ -5,11 +5,21 @@ import os
 import stat
 
 from copy import deepcopy
-from .util import user_dir, print_error, print_stderr, PrintError
 
-from .bitcoin import DEFAULT_FEE_RATE, MAX_FEE_RATE, FEE_TARGETS
+from .util import (user_dir, print_error, PrintError,
+                   NoDynamicFeeEstimates, format_satoshis)
+from .i18n import _
+
+FEE_ETA_TARGETS = [25, 10, 5, 2]
+FEE_DEPTH_TARGETS = [10000000, 5000000, 2000000, 1000000, 500000, 200000, 100000]
+
+# satoshi per kbyte
+FEERATE_MAX_DYNAMIC = 1500000
+FEERATE_WARNING_HIGH_FEE = 600000
+FEERATE_FALLBACK_STATIC_FEE = 150000
+FEERATE_DEFAULT_RELAY = 1000
+FEERATE_STATIC_VALUES = [5000, 10000, 20000, 30000, 50000, 70000, 100000, 150000, 200000, 300000]
 
-SYSTEM_CONFIG_PATH = "/etc/electrum.conf"
 
 config = None
 
@@ -24,35 +34,37 @@ def set_config(c):
     config = c
 
 
+FINAL_CONFIG_VERSION = 2
+
+
 class SimpleConfig(PrintError):
     """
     The SimpleConfig class is responsible for handling operations involving
     configuration files.
 
-    There are 3 different sources of possible configuration values:
+    There are two different sources of possible configuration values:
         1. Command line options.
         2. User configuration (in the user's config directory)
-        3. System configuration (in /etc/)
-    They are taken in order (1. overrides config options set in 2., that
-    override config set in 3.)
+    They are taken in order (1. overrides config options set in 2.)
     """
-    fee_rates = [5000, 10000, 20000, 30000, 50000, 70000, 100000, 150000, 200000, 300000]
 
-    def __init__(self, options={}, read_system_config_function=None,
-                 read_user_config_function=None, read_user_dir_function=None):
+    def __init__(self, options=None, read_user_config_function=None,
+                 read_user_dir_function=None):
+
+        if options is None:
+            options = {}
 
         # This lock needs to be acquired for updating and reading the config in
         # a thread-safe way.
         self.lock = threading.RLock()
 
+        self.mempool_fees = {}
         self.fee_estimates = {}
         self.fee_estimates_last_updated = {}
         self.last_time_fee_estimates_requested = 0  # zero ensures immediate fees
 
         # The following two functions are there for dependency injection when
         # testing.
-        if read_system_config_function is None:
-            read_system_config_function = read_system_config
         if read_user_config_function is None:
             read_user_config_function = read_user_config
         if read_user_dir_function is None:
@@ -62,24 +74,30 @@ class SimpleConfig(PrintError):
 
         # The command line options
         self.cmdline_options = deepcopy(options)
-
-        # Portable wallets don't use a system config
-        if self.cmdline_options.get('portable', False):
-            self.system_config = {}
-        else:
-            self.system_config = read_system_config_function()
+        # don't allow to be set on CLI:
+        self.cmdline_options.pop('config_version', None)
 
         # Set self.path and read the user config
         self.user_config = {}  # for self.get in electrum_path()
         self.path = self.electrum_path()
         self.user_config = read_user_config_function(self.path)
-        # Upgrade obsolete keys
-        self.fixup_keys({'auto_cycle': 'auto_connect'})
+        if not self.user_config:
+            # avoid new config getting upgraded
+            self.user_config = {'config_version': FINAL_CONFIG_VERSION}
+
+        # config "upgrade" - CLI options
+        self.rename_config_keys(
+            self.cmdline_options, {'auto_cycle': 'auto_connect'}, True)
+
+        # config upgrade - user config
+        if self.requires_upgrade():
+            self.upgrade()
+
         # Make a singleton instance of 'self'
         set_config(self)
 
     def electrum_path(self):
-        # Read electrum_path from command line / system configuration
+        # Read electrum_path from command line
         # Otherwise use the user's default data directory.
         path = self.get('electrum_path')
         if path is None:
@@ -101,45 +119,92 @@ class SimpleConfig(PrintError):
         self.print_error("electrum directory", path)
         return path
 
-    def fixup_config_keys(self, config, keypairs):
+    def rename_config_keys(self, config, keypairs, deprecation_warning=False):
+        """Migrate old key names to new ones"""
         updated = False
         for old_key, new_key in keypairs.items():
             if old_key in config:
-                if not new_key in config:
+                if new_key not in config:
                     config[new_key] = config[old_key]
+                    if deprecation_warning:
+                        self.print_stderr('Note that the {} variable has been deprecated. '
+                                     'You should use {} instead.'.format(old_key, new_key))
                 del config[old_key]
                 updated = True
         return updated
 
-    def fixup_keys(self, keypairs):
-        '''Migrate old key names to new ones'''
-        self.fixup_config_keys(self.cmdline_options, keypairs)
-        self.fixup_config_keys(self.system_config, keypairs)
-        if self.fixup_config_keys(self.user_config, keypairs):
-            self.save_user_config()
-
-    def set_key(self, key, value, save = True):
+    def set_key(self, key, value, save=True):
         if not self.is_modifiable(key):
-            print_stderr("Warning: not changing config key '%s' set on the command line" % key)
+            self.print_stderr("Warning: not changing config key '%s' set on the command line" % key)
             return
+        self._set_key_in_user_config(key, value, save)
 
+    def _set_key_in_user_config(self, key, value, save=True):
         with self.lock:
-            self.user_config[key] = value
+            if value is not None:
+                self.user_config[key] = value
+            else:
+                self.user_config.pop(key, None)
             if save:
                 self.save_user_config()
-        return
 
     def get(self, key, default=None):
         with self.lock:
             out = self.cmdline_options.get(key)
             if out is None:
-                out = self.user_config.get(key)
-                if out is None:
-                    out = self.system_config.get(key, default)
+                out = self.user_config.get(key, default)
         return out
 
+    def requires_upgrade(self):
+        return self.get_config_version() < FINAL_CONFIG_VERSION
+
+    def upgrade(self):
+        with self.lock:
+            self.print_error('upgrading config')
+
+            self.convert_version_2()
+
+            self.set_key('config_version', FINAL_CONFIG_VERSION, save=True)
+
+    def convert_version_2(self):
+        if not self._is_upgrade_method_needed(1, 1):
+            return
+
+        self.rename_config_keys(self.user_config, {'auto_cycle': 'auto_connect'})
+
+        try:
+            # change server string FROM host:port:proto TO host:port:s
+            server_str = self.user_config.get('server')
+            host, port, protocol = str(server_str).rsplit(':', 2)
+            assert protocol in ('s', 't')
+            int(port)  # Throw if cannot be converted to int
+            server_str = '{}:{}:s'.format(host, port)
+            self._set_key_in_user_config('server', server_str)
+        except BaseException:
+            self._set_key_in_user_config('server', None)
+
+        self.set_key('config_version', 2)
+
+    def _is_upgrade_method_needed(self, min_version, max_version):
+        cur_version = self.get_config_version()
+        if cur_version > max_version:
+            return False
+        elif cur_version < min_version:
+            raise BaseException(
+                ('config upgrade: unexpected version %d (should be %d-%d)'
+                 % (cur_version, min_version, max_version)))
+        else:
+            return True
+
+    def get_config_version(self):
+        config_version = self.get('config_version', 1)
+        if config_version > FINAL_CONFIG_VERSION:
+            self.print_stderr('WARNING: config version ({}) is higher than ours ({})'
+                             .format(config_version, FINAL_CONFIG_VERSION))
+        return config_version
+
     def is_modifiable(self, key):
-        return not key in self.cmdline_options
+        return key not in self.cmdline_options
 
     def save_user_config(self):
         if not self.path:
@@ -203,68 +268,197 @@ class SimpleConfig(PrintError):
             path = wallet.storage.path
             self.set_key('gui_last_wallet', path)
 
-    def default_fee_rate(self):
-        f = self.get('default_fee_rate', DEFAULT_FEE_RATE)
-        if f==0:
-            f = DEFAULT_FEE_RATE
-        return f
+    def impose_hard_limits_on_fee(func):
+        def get_fee_within_limits(self, *args, **kwargs):
+            fee = func(self, *args, **kwargs)
+            if fee is None:
+                return fee
+            fee = min(FEERATE_MAX_DYNAMIC, fee)
+            fee = max(FEERATE_DEFAULT_RELAY, fee)
+            return fee
+        return get_fee_within_limits
 
-    def max_fee_rate(self):
-        f = self.get('max_fee_rate', MAX_FEE_RATE)
-        if f==0:
-            f = MAX_FEE_RATE
-        return f
-
-    def dynfee(self, i):
+    @impose_hard_limits_on_fee
+    def eta_to_fee(self, i):
+        """Returns fee in sat/kbyte."""
         if i < 4:
-            j = FEE_TARGETS[i]
+            j = FEE_ETA_TARGETS[i]
             fee = self.fee_estimates.get(j)
         else:
             assert i == 4
             fee = self.fee_estimates.get(2)
             if fee is not None:
                 fee += fee/2
-        if fee is not None:
-            fee = min(5*MAX_FEE_RATE, fee)
         return fee
 
-    def reverse_dynfee(self, fee_per_kb):
+    def fee_to_depth(self, target_fee):
+        depth = 0
+        for fee, s in self.mempool_fees:
+            depth += s
+            if fee <= target_fee:
+                break
+        else:
+            return 0
+        return depth
+
+    @impose_hard_limits_on_fee
+    def depth_to_fee(self, i):
+        """Returns fee in zat/kbyte."""
+        target = self.depth_target(i)
+        depth = 0
+        for fee, s in self.mempool_fees:
+            depth += s
+            if depth > target:
+                break
+        else:
+            return 0
+        return fee * 1000
+
+    def depth_target(self, i):
+        return FEE_DEPTH_TARGETS[i]
+
+    def eta_target(self, i):
+        if i == len(FEE_ETA_TARGETS):
+            return 1
+        return FEE_ETA_TARGETS[i]
+
+    def fee_to_eta(self, fee_per_kb):
         import operator
-        l = list(self.fee_estimates.items()) + [(1, self.dynfee(4))]
+        l = list(self.fee_estimates.items()) + [(1, self.eta_to_fee(4))]
         dist = map(lambda x: (x[0], abs(x[1] - fee_per_kb)), l)
         min_target, min_value = min(dist, key=operator.itemgetter(1))
         if fee_per_kb < self.fee_estimates.get(25)/2:
             min_target = -1
         return min_target
 
+    def depth_tooltip(self, depth):
+        return "%.1f MB from tip"%(depth/1000000)
+
+    def eta_tooltip(self, x):
+        if x < 0:
+            return _('Low fee')
+        elif x == 1:
+            return _('In the next block')
+        else:
+            return _('Within {} blocks').format(x)
+
+    def get_fee_status(self):
+        dyn = self.is_dynfee()
+        mempool = self.use_mempool_fees()
+        pos = self.get_depth_level() if mempool else self.get_fee_level()
+        fee_rate = self.fee_per_kb()
+        target, tooltip = self.get_fee_text(pos, dyn, mempool, fee_rate)
+        return tooltip + '  [%s]'%target if dyn else target + '  [Static]'
+
+    def get_fee_text(self, pos, dyn, mempool, fee_rate):
+        """Returns (text, tooltip) where
+        text is what we target: static fee / num blocks to confirm in / mempool depth
+        tooltip is the corresponding estimate (e.g. num blocks for a static fee)
+        """
+        rate_str = (format_satoshis(fee_rate/1000, False, 0, 0, False)  + ' sat/byte') if fee_rate is not None else 'unknown'
+        if dyn:
+            if mempool:
+                depth = self.depth_target(pos)
+                text = self.depth_tooltip(depth)
+            else:
+                eta = self.eta_target(pos)
+                text = self.eta_tooltip(eta)
+            tooltip = rate_str
+        else:
+            text = rate_str
+            if mempool and self.has_fee_mempool():
+                depth = self.fee_to_depth(fee_rate)
+                tooltip = self.depth_tooltip(depth)
+            elif not mempool and self.has_fee_etas():
+                eta = self.fee_to_eta(fee_rate)
+                tooltip = self.eta_tooltip(eta)
+            else:
+                tooltip = ''
+        return text, tooltip
+
+    def get_depth_level(self):
+        maxp = len(FEE_DEPTH_TARGETS) - 1
+        return min(maxp, self.get('depth_level', 2))
+
+    def get_fee_level(self):
+        maxp = len(FEE_ETA_TARGETS)  # not (-1) to have "next block"
+        return min(maxp, self.get('fee_level', 2))
+
+    def get_fee_slider(self, dyn, mempool):
+        if dyn:
+            if mempool:
+                pos = self.get_depth_level()
+                maxp = len(FEE_DEPTH_TARGETS) - 1
+                fee_rate = self.depth_to_fee(pos)
+            else:
+                pos = self.get_fee_level()
+                maxp = len(FEE_ETA_TARGETS)  # not (-1) to have "next block"
+                fee_rate = self.eta_to_fee(pos)
+        else:
+            fee_rate = self.fee_per_kb()
+            pos = self.static_fee_index(fee_rate)
+            maxp = 9
+        return maxp, pos, fee_rate
+
     def static_fee(self, i):
-        return self.fee_rates[i]
+        return FEERATE_STATIC_VALUES[i]
 
     def static_fee_index(self, value):
-        dist = list(map(lambda x: abs(x - value), self.fee_rates))
+        dist = list(map(lambda x: abs(x - value), FEERATE_STATIC_VALUES))
         return min(range(len(dist)), key=dist.__getitem__)
 
-    def has_fee_estimates(self):
-        return len(self.fee_estimates)==4
+    def has_fee_etas(self):
+        return len(self.fee_estimates) == 4
+
+    def has_fee_mempool(self):
+        return bool(self.mempool_fees)
+
+    def has_dynamic_fees_ready(self):
+        if self.use_mempool_fees():
+            return self.has_fee_mempool()
+        else:
+            return self.has_fee_etas()
 
-    # 'dynamic' fees are disabled - we use 'static' (but adjustable) fees
     def is_dynfee(self):
-        return self.get('dynamic_fees', False)
+        return bool(self.get('dynamic_fees', True))
+
+    def use_mempool_fees(self):
+        return bool(self.get('mempool_fees', False))
 
     def fee_per_kb(self):
-        dyn = self.is_dynfee()
-        if dyn:
-            fee_rate = self.dynfee(self.get('fee_level', 2))
+        """Returns sat/kvB fee to pay for a txn.
+        Note: might return None.
+        """
+        if self.is_dynfee():
+            if self.use_mempool_fees():
+                fee_rate = self.depth_to_fee(self.get_depth_level())
+            else:
+                fee_rate = self.eta_to_fee(self.get_fee_level())
         else:
-            fee_rate = self.get('fee_per_kb', self.max_fee_rate()/2)
+            fee_rate = self.get('fee_per_kb', FEERATE_FALLBACK_STATIC_FEE)
         return fee_rate
 
+    def fee_per_byte(self):
+        """Returns sat/vB fee to pay for a txn.
+        Note: might return None.
+        """
+        fee_per_kb = self.fee_per_kb()
+        return fee_per_kb / 1000 if fee_per_kb is not None else None
+
     def estimate_fee(self, size):
-        return self.estimate_fee_for_feerate(self.fee_per_kb(), size)
+        fee_per_kb = self.fee_per_kb()
+        if fee_per_kb is None:
+            raise NoDynamicFeeEstimates()
+        return self.estimate_fee_for_feerate(fee_per_kb, size)
 
     @classmethod
     def estimate_fee_for_feerate(cls, fee_per_kb, size):
-        return int(fee_per_kb * size / 1000.)
+        # note: We only allow integer sat/byte values atm.
+        # The GUI for simplicity reasons only displays integer sat/byte,
+        # and for the sake of consistency, we thus only use integer sat/byte in
+        # the backend too.
+        fee_per_byte = int(fee_per_kb / 1000)
+        return int(fee_per_byte * size)
 
     def update_fee_estimates(self, key, value):
         self.fee_estimates[key] = value
@@ -275,11 +469,7 @@ class SimpleConfig(PrintError):
         Returns True if an update should be requested.
         """
         now = time.time()
-        prev_updates = self.fee_estimates_last_updated.values()
-        oldest_fee_time = min(prev_updates) if prev_updates else 0
-        stale_fees = now - oldest_fee_time > 7200
-        old_request = now - self.last_time_fee_estimates_requested > 60
-        return stale_fees and old_request
+        return now - self.last_time_fee_estimates_requested > 60
 
     def requested_fee_estimates(self):
         self.last_time_fee_estimates_requested = time.time()
@@ -291,21 +481,6 @@ class SimpleConfig(PrintError):
         return device
 
 
-def read_system_config(path=SYSTEM_CONFIG_PATH):
-    """Parse and return the system config settings in /etc/electrum.conf."""
-    result = {}
-    if os.path.exists(path):
-        import configparser
-        p = configparser.ConfigParser()
-        try:
-            p.read(path)
-            for k, v in p.items('client'):
-                result[k] = v
-        except (configparser.NoSectionError, configparser.MissingSectionHeaderError):
-            pass
-
-    return result
-
 def read_user_config(path):
     """Parse and store the user config settings in electrum.conf into user_config[]."""
     if not path:

lib/storage.py

@@ -33,7 +33,7 @@ import pbkdf2, hmac, hashlib
 import base64
 import zlib
 
-from .util import PrintError, profiler
+from .util import PrintError, profiler, InvalidPassword
 from .plugins import run_hook, plugin_loaders
 from .keystore import bip44_derivation
 from . import bitcoin
@@ -56,6 +56,13 @@ def multisig_type(wallet_type):
         match = [int(x) for x in match.group(1, 2)]
     return match
 
+def get_derivation_used_for_hw_device_encryption():
+    return ("m"
+            "/4541509'"      # ascii 'ELE'  as decimal ("BIP43 purpose")
+            "/1112098098'")  # ascii 'BIE2' as decimal
+
+# storage encryption version
+STO_EV_PLAINTEXT, STO_EV_USER_PW, STO_EV_XPUB_PW = range(0, 3)
 
 class WalletStorage(PrintError):
 
@@ -70,9 +77,11 @@ class WalletStorage(PrintError):
         if self.file_exists():
             with open(self.path, "r") as f:
                 self.raw = f.read()
+            self._encryption_version = self._init_encryption_version()
             if not self.is_encrypted():
                 self.load_data(self.raw)
         else:
+            self._encryption_version = STO_EV_PLAINTEXT
             # avoid new wallets getting 'upgraded'
             self.put('seed_version', FINAL_SEED_VERSION)
 
@@ -106,11 +115,47 @@ class WalletStorage(PrintError):
             if self.requires_upgrade():
                 self.upgrade()
 
+    def is_past_initial_decryption(self):
+        """Return if storage is in a usable state for normal operations.
+
+        The value is True exactly
+            if encryption is disabled completely (self.is_encrypted() == False),
+            or if encryption is enabled but the contents have already been decrypted.
+        """
+        return bool(self.data)
+
     def is_encrypted(self):
+        """Return if storage encryption is currently enabled."""
+        return self.get_encryption_version() != STO_EV_PLAINTEXT
+
+    def is_encrypted_with_user_pw(self):
+        return self.get_encryption_version() == STO_EV_USER_PW
+
+    def is_encrypted_with_hw_device(self):
+        return self.get_encryption_version() == STO_EV_XPUB_PW
+
+    def get_encryption_version(self):
+        """Return the version of encryption used for this storage.
+
+        0: plaintext / no encryption
+
+        ECIES, private key derived from a password,
+        1: password is provided by user
+        2: password is derived from an xpub; used with hw wallets
+        """
+        return self._encryption_version
+
+    def _init_encryption_version(self):
         try:
-            return base64.b64decode(self.raw)[0:4] == b'BIE1'
+            magic = base64.b64decode(self.raw)[0:4]
+            if magic == b'BIE1':
+                return STO_EV_USER_PW
+            elif magic == b'BIE2':
+                return STO_EV_XPUB_PW
+            else:
+                return STO_EV_PLAINTEXT
         except:
-            return False
+            return STO_EV_PLAINTEXT
 
     def file_exists(self):
         return self.path and os.path.exists(self.path)
@@ -120,20 +165,50 @@ class WalletStorage(PrintError):
         ec_key = bitcoin.EC_KEY(secret)
         return ec_key
 
+    def _get_encryption_magic(self):
+        v = self._encryption_version
+        if v == STO_EV_USER_PW:
+            return b'BIE1'
+        elif v == STO_EV_XPUB_PW:
+            return b'BIE2'
+        else:
+            raise Exception('no encryption magic for version: %s' % v)
+
     def decrypt(self, password):
         ec_key = self.get_key(password)
-        s = zlib.decompress(ec_key.decrypt_message(self.raw)) if self.raw else None
+        if self.raw:
+            enc_magic = self._get_encryption_magic()
+            s = zlib.decompress(ec_key.decrypt_message(self.raw, enc_magic))
+        else:
+            s = None
         self.pubkey = ec_key.get_public_key()
         s = s.decode('utf8')
         self.load_data(s)
 
-    def set_password(self, password, encrypt):
-        self.put('use_encryption', bool(password))
-        if encrypt and password:
+    def check_password(self, password):
+        """Raises an InvalidPassword exception on invalid password"""
+        if not self.is_encrypted():
+            return
+        if self.pubkey and self.pubkey != self.get_key(password).get_public_key():
+            raise InvalidPassword()
+
+    def set_keystore_encryption(self, enable):
+        self.put('use_encryption', enable)
+
+    def set_password(self, password, enc_version=None):
+        """Set a password to be used for encrypting this storage."""
+        if enc_version is None:
+            enc_version = self._encryption_version
+        if password and enc_version != STO_EV_PLAINTEXT:
             ec_key = self.get_key(password)
             self.pubkey = ec_key.get_public_key()
+            self._encryption_version = enc_version
         else:
             self.pubkey = None
+            self._encryption_version = STO_EV_PLAINTEXT
+        # make sure next storage.write() saves changes
+        with self.lock:
+            self.modified = True
 
     def get(self, key, default=None):
         with self.lock:
@@ -175,7 +250,8 @@ class WalletStorage(PrintError):
         if self.pubkey:
             s = bytes(s, 'utf8')
             c = zlib.compress(s)
-            s = bitcoin.encrypt_message(c, self.pubkey)
+            enc_magic = self._get_encryption_magic()
+            s = bitcoin.encrypt_message(c, self.pubkey, enc_magic)
             s = s.decode('utf8')
 
         temp_path = "%s.tmp.%s" % (self.path, os.getpid())
@@ -263,6 +339,9 @@ class WalletStorage(PrintError):
         self.write()
 
     def convert_wallet_type(self):
+        if not self._is_upgrade_method_needed(0, 13):
+            return
+
         wallet_type = self.get('wallet_type')
         if wallet_type == 'btchip': wallet_type = 'ledger'
         if self.get('keystore') or self.get('x1/') or wallet_type=='imported':
@@ -446,6 +525,9 @@ class WalletStorage(PrintError):
         self.put('seed_version', 16)
 
     def convert_imported(self):
+        if not self._is_upgrade_method_needed(0, 13):
+            return
+
         # '/x' is the internal ID for imported accounts
         d = self.get('accounts', {}).get('/x', {}).get('imported',{})
         if not d:
@@ -472,6 +554,9 @@ class WalletStorage(PrintError):
             raise BaseException('no addresses or privkeys')
 
     def convert_account(self):
+        if not self._is_upgrade_method_needed(0, 13):
+            return
+
         self.put('accounts', None)
 
     def _is_upgrade_method_needed(self, min_version, max_version):

lib/synchronizer.py

@@ -84,11 +84,13 @@ class Synchronizer(ThreadJob):
         return bh2u(hashlib.sha256(status.encode('ascii')).digest())
 
     def on_address_status(self, response):
+        if self.wallet.synchronizer is None:
+            return  # we have been killed, this was just an orphan callback
         params, result = self.parse_response(response)
         if not params:
             return
         addr = params[0]
-        history = self.wallet.get_address_history(addr)
+        history = self.wallet.history.get(addr, [])
         if self.get_status(history) != result:
             if self.requested_histories.get(addr) is None:
                 self.requested_histories[addr] = result
@@ -98,6 +100,8 @@ class Synchronizer(ThreadJob):
             self.requested_addrs.remove(addr)
 
     def on_address_history(self, response):
+        if self.wallet.synchronizer is None:
+            return  # we have been killed, this was just an orphan callback
         params, result = self.parse_response(response)
         if not params:
             return
@@ -127,6 +131,8 @@ class Synchronizer(ThreadJob):
         self.requested_histories.pop(addr)
 
     def tx_response(self, response):
+        if self.wallet.synchronizer is None:
+            return  # we have been killed, this was just an orphan callback
         params, result = self.parse_response(response)
         if not params:
             return

lib/tests/test_bitcoin.py

@@ -11,8 +11,9 @@ from lib.bitcoin import (
     var_int, op_push, address_to_script, regenerate_key,
     verify_message, deserialize_privkey, serialize_privkey, is_segwit_address,
     is_b58_address, address_to_scripthash, is_minikey, is_compressed, is_xpub,
-    xpub_type, is_xprv, is_bip32_derivation, seed_type, NetworkConstants)
+    xpub_type, is_xprv, is_bip32_derivation, seed_type)
 from lib.util import bfh
+from lib import constants
 
 try:
     import ecdsa
@@ -168,12 +169,12 @@ class Test_bitcoin_testnet(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
         super().setUpClass()
-        NetworkConstants.set_testnet()
+        constants.set_testnet()
 
     @classmethod
     def tearDownClass(cls):
         super().tearDownClass()
-        NetworkConstants.set_mainnet()
+        constants.set_mainnet()
 
     def test_address_to_script(self):
         # bech32 native segwit
@@ -271,6 +272,7 @@ class Test_keyImport(unittest.TestCase):
 
     priv_pub_addr = (
            {'priv': 'KzMFjMC2MPadjvX5Cd7b8AKKjjpBSoRKUTpoAtN6B3J9ezWYyXS6',
+            'exported_privkey': 'p2pkh:KzMFjMC2MPadjvX5Cd7b8AKKjjpBSoRKUTpoAtN6B3J9ezWYyXS6',
             'pub': '02c6467b7e621144105ed3e4835b0b4ab7e35266a2ae1c4f8baa19e9ca93452997',
             'address': '17azqT8T16coRmWKYFj3UjzJuxiYrYFRBR',
             'minikey' : False,
@@ -278,7 +280,17 @@ class Test_keyImport(unittest.TestCase):
             'compressed': True,
             'addr_encoding': 'base58',
             'scripthash': 'c9aecd1fef8d661a42c560bf75c8163e337099800b8face5ca3d1393a30508a7'},
+           {'priv': 'p2pkh:Kzj8VjwpZ99bQqVeUiRXrKuX9mLr1o6sWxFMCBJn1umC38BMiQTD',
+            'exported_privkey': 'p2pkh:Kzj8VjwpZ99bQqVeUiRXrKuX9mLr1o6sWxFMCBJn1umC38BMiQTD',
+            'pub': '0352d78b4b37e0f6d4e164423436f2925fa57817467178eca550a88f2821973c41',
+            'address': '1GXgZ5Qi6gmXTHVSpUPZLy4Ci2nbfb3ZNb',
+            'minikey': False,
+            'txin_type': 'p2pkh',
+            'compressed': True,
+            'addr_encoding': 'base58',
+            'scripthash': 'a9b2a76fc196c553b352186dfcca81fcf323a721cd8431328f8e9d54216818c1'},
            {'priv': '5Hxn5C4SQuiV6e62A1MtZmbSeQyrLFhu5uYks62pU5VBUygK2KD',
+            'exported_privkey': 'p2pkh:5Hxn5C4SQuiV6e62A1MtZmbSeQyrLFhu5uYks62pU5VBUygK2KD',
             'pub': '04e5fe91a20fac945845a5518450d23405ff3e3e1ce39827b47ee6d5db020a9075422d56a59195ada0035e4a52a238849f68e7a325ba5b2247013e0481c5c7cb3f',
             'address': '1GPHVTY8UD9my6jyP4tb2TYJwUbDetyNC6',
             'minikey': False,
@@ -286,7 +298,17 @@ class Test_keyImport(unittest.TestCase):
             'compressed': False,
             'addr_encoding': 'base58',
             'scripthash': 'f5914651408417e1166f725a5829ff9576d0dbf05237055bf13abd2af7f79473'},
+           {'priv': 'p2pkh:5KhYQCe1xd5g2tqpmmGpUWDpDuTbA8vnpbiCNDwMPAx29WNQYfN',
+            'exported_privkey': 'p2pkh:5KhYQCe1xd5g2tqpmmGpUWDpDuTbA8vnpbiCNDwMPAx29WNQYfN',
+            'pub': '048f0431b0776e8210376c81280011c2b68be43194cb00bd47b7e9aa66284b713ce09556cde3fee606051a07613f3c159ef3953b8927c96ae3dae94a6ba4182e0e',
+            'address': '147kiRHHm9fqeMQSgqf4k35XzuWLP9fmmS',
+            'minikey': False,
+            'txin_type': 'p2pkh',
+            'compressed': False,
+            'addr_encoding': 'base58',
+            'scripthash': '6dd2e07ad2de9ba8eec4bbe8467eb53f8845acff0d9e6f5627391acc22ff62df'},
            {'priv': 'LHJnnvRzsdrTX2j5QeWVsaBkabK7gfMNqNNqxnbBVRaJYfk24iJz',
+            'exported_privkey': 'p2wpkh-p2sh:Kz9XebiCXL2BZzhYJViiHDzn5iup1povWV8aqstzWU4sz1K5nVva',
             'pub': '0279ad237ca0d812fb503ab86f25e15ebd5fa5dd95c193639a8a738dcd1acbad81',
             'address': '3GeVJB3oKr7psgKR6BTXSxKtWUkfsHHhk7',
             'minikey': False,
@@ -294,7 +316,17 @@ class Test_keyImport(unittest.TestCase):
             'compressed': True,
             'addr_encoding': 'base58',
             'scripthash': 'd7b04e882fa6b13246829ac552a2b21461d9152eb00f0a6adb58457a3e63d7c5'},
+           {'priv': 'p2wpkh-p2sh:L3CZH1pm87X4bbE6mSGvZnAZ1KcFDRomBudUkrkBG7EZhDtBVXMW',
+            'exported_privkey': 'p2wpkh-p2sh:L3CZH1pm87X4bbE6mSGvZnAZ1KcFDRomBudUkrkBG7EZhDtBVXMW',
+            'pub': '0229da20a15b3363b2c28e3c5093c180b56c439df0b968a970366bb1f38435361e',
+            'address': '3C79goMwT7zSTjXnPoCg6VFGAnUpZAkyus',
+            'minikey': False,
+            'txin_type': 'p2wpkh-p2sh',
+            'compressed': True,
+            'addr_encoding': 'base58',
+            'scripthash': '714bf6bfe1083e69539f40d4c7a7dca85d187471b35642e55f20d7e866494cf7'},
            {'priv': 'L8g5V8kFFeg2WbecahRSdobARbHz2w2STH9S8ePHVSY4fmia7Rsj',
+            'exported_privkey': 'p2wpkh:Kz6SuyPM5VktY5dr2d2YqdVgBA6LCWkiHqXJaC3BzxnMPSUuYzmF',
             'pub': '03e9f948421aaa89415dc5f281a61b60dde12aae3181b3a76cd2d849b164fc6d0b',
             'address': 'bc1qqmpt7u5e9hfznljta5gnvhyvfd2kdd0r90hwue',
             'minikey': False,
@@ -302,8 +334,18 @@ class Test_keyImport(unittest.TestCase):
             'compressed': True,
             'addr_encoding': 'bech32',
             'scripthash': '1929acaaef3a208c715228e9f1ca0318e3a6b9394ab53c8d026137f847ecf97b'},
+           {'priv': 'p2wpkh:KyDWy5WbjLA58Zesh1o8m3pADGdJ3v33DKk4m7h8BD5zDKDmDFwo',
+            'exported_privkey': 'p2wpkh:KyDWy5WbjLA58Zesh1o8m3pADGdJ3v33DKk4m7h8BD5zDKDmDFwo',
+            'pub': '038c57657171c1f73e34d5b3971d05867d50221ad94980f7e87cbc2344425e6a1e',
+            'address': 'bc1qpakeeg4d9ydyjxd8paqrw4xy9htsg532xzxn50',
+            'minikey': False,
+            'txin_type': 'p2wpkh',
+            'compressed': True,
+            'addr_encoding': 'bech32',
+            'scripthash': '242f02adde84ebb2a7dd778b2f3a81b3826f111da4d8960d826d7a4b816cb261'},
            # from http://bitscan.com/articles/security/spotlight-on-mini-private-keys
            {'priv': 'SzavMBLoXU6kDrqtUVmffv',
+            'exported_privkey': 'p2pkh:L53fCHmQhbNp1B4JipfBtfeHZH7cAibzG9oK19XfiFzxHgAkz6JK',
             'pub': '02588d202afcc1ee4ab5254c7847ec25b9a135bbda0f2bc69ee1a714749fd77dc9',
             'address': '19GuvDvMMUZ8vq84wT79fvnvhMd5MnfTkR',
             'minikey': True,
@@ -344,6 +386,7 @@ class Test_keyImport(unittest.TestCase):
     def test_is_private_key(self):
         for priv_details in self.priv_pub_addr:
             self.assertTrue(is_private_key(priv_details['priv']))
+            self.assertTrue(is_private_key(priv_details['exported_privkey']))
             self.assertFalse(is_private_key(priv_details['pub']))
             self.assertFalse(is_private_key(priv_details['address']))
         self.assertFalse(is_private_key("not a privkey"))
@@ -352,8 +395,7 @@ class Test_keyImport(unittest.TestCase):
         for priv_details in self.priv_pub_addr:
             txin_type, privkey, compressed = deserialize_privkey(priv_details['priv'])
             priv2 = serialize_privkey(privkey, compressed, txin_type)
-            if not priv_details['minikey']:
-                self.assertEqual(priv_details['priv'], priv2)
+            self.assertEqual(priv_details['exported_privkey'], priv2)
 
     def test_address_to_scripthash(self):
         for priv_details in self.priv_pub_addr:

lib/tests/test_simple_config.py

@@ -6,8 +6,7 @@ import tempfile
 import shutil
 
 from io import StringIO
-from lib.simple_config import (SimpleConfig, read_system_config,
-                               read_user_config)
+from lib.simple_config import (SimpleConfig, read_user_config)
 
 
 class Test_SimpleConfig(unittest.TestCase):
@@ -37,18 +36,15 @@ class Test_SimpleConfig(unittest.TestCase):
 
     def test_simple_config_key_rename(self):
         """auto_cycle was renamed auto_connect"""
-        fake_read_system = lambda : {}
         fake_read_user = lambda _: {"auto_cycle": True}
         read_user_dir = lambda : self.user_dir
         config = SimpleConfig(options=self.options,
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         self.assertEqual(config.get("auto_connect"), True)
         self.assertEqual(config.get("auto_cycle"), None)
         fake_read_user = lambda _: {"auto_connect": False, "auto_cycle": True}
         config = SimpleConfig(options=self.options,
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         self.assertEqual(config.get("auto_connect"), False)
@@ -57,110 +53,51 @@ class Test_SimpleConfig(unittest.TestCase):
     def test_simple_config_command_line_overrides_everything(self):
         """Options passed by command line override all other configuration
         sources"""
-        fake_read_system = lambda : {"electrum_path": "a"}
         fake_read_user = lambda _: {"electrum_path": "b"}
         read_user_dir = lambda : self.user_dir
         config = SimpleConfig(options=self.options,
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         self.assertEqual(self.options.get("electrum_path"),
                          config.get("electrum_path"))
 
-    def test_simple_config_user_config_overrides_system_config(self):
-        """Options passed in user config override system config."""
-        fake_read_system = lambda : {"electrum_path": self.electrum_dir}
-        fake_read_user = lambda _: {"electrum_path": "b"}
-        read_user_dir = lambda : self.user_dir
-        config = SimpleConfig(options={},
-                              read_system_config_function=fake_read_system,
-                              read_user_config_function=fake_read_user,
-                              read_user_dir_function=read_user_dir)
-        self.assertEqual("b", config.get("electrum_path"))
-
-    def test_simple_config_system_config_ignored_if_portable(self):
-        """If electrum is started with the "portable" flag, system
-        configuration is completely ignored."""
-        fake_read_system = lambda : {"some_key": "some_value"}
-        fake_read_user = lambda _: {}
-        read_user_dir = lambda : self.user_dir
-        config = SimpleConfig(options={"portable": True},
-                              read_system_config_function=fake_read_system,
-                              read_user_config_function=fake_read_user,
-                              read_user_dir_function=read_user_dir)
-        self.assertEqual(config.get("some_key"), None)
-
     def test_simple_config_user_config_is_used_if_others_arent_specified(self):
         """If no system-wide configuration and no command-line options are
         specified, the user configuration is used instead."""
-        fake_read_system = lambda : {}
         fake_read_user = lambda _: {"electrum_path": self.electrum_dir}
         read_user_dir = lambda : self.user_dir
         config = SimpleConfig(options={},
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         self.assertEqual(self.options.get("electrum_path"),
                          config.get("electrum_path"))
 
     def test_cannot_set_options_passed_by_command_line(self):
-        fake_read_system = lambda : {}
         fake_read_user = lambda _: {"electrum_path": "b"}
         read_user_dir = lambda : self.user_dir
         config = SimpleConfig(options=self.options,
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         config.set_key("electrum_path", "c")
         self.assertEqual(self.options.get("electrum_path"),
                          config.get("electrum_path"))
 
-    def test_can_set_options_from_system_config(self):
-        fake_read_system = lambda : {"electrum_path": self.electrum_dir}
-        fake_read_user = lambda _: {}
-        read_user_dir = lambda : self.user_dir
-        config = SimpleConfig(options={},
-                              read_system_config_function=fake_read_system,
-                              read_user_config_function=fake_read_user,
-                              read_user_dir_function=read_user_dir)
-        config.set_key("electrum_path", "c")
-        self.assertEqual("c", config.get("electrum_path"))
-
     def test_can_set_options_set_in_user_config(self):
         another_path = tempfile.mkdtemp()
-        fake_read_system = lambda : {}
         fake_read_user = lambda _: {"electrum_path": self.electrum_dir}
         read_user_dir = lambda : self.user_dir
         config = SimpleConfig(options={},
-                              read_system_config_function=fake_read_system,
-                              read_user_config_function=fake_read_user,
-                              read_user_dir_function=read_user_dir)
-        config.set_key("electrum_path", another_path)
-        self.assertEqual(another_path, config.get("electrum_path"))
-
-    def test_can_set_options_from_system_config_if_portable(self):
-        """If the "portable" flag is set, the user can overwrite system
-        configuration options."""
-        another_path = tempfile.mkdtemp()
-        fake_read_system = lambda : {"electrum_path": self.electrum_dir}
-        fake_read_user = lambda _: {}
-        read_user_dir = lambda : self.user_dir
-        config = SimpleConfig(options={"portable": True},
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         config.set_key("electrum_path", another_path)
         self.assertEqual(another_path, config.get("electrum_path"))
 
     def test_user_config_is_not_written_with_read_only_config(self):
-        """The user config does not contain command-line options or system
-        options when saved."""
-        fake_read_system = lambda : {"something": "b"}
+        """The user config does not contain command-line options when saved."""
         fake_read_user = lambda _: {"something": "a"}
         read_user_dir = lambda : self.user_dir
         self.options.update({"something": "c"})
         config = SimpleConfig(options=self.options,
-                              read_system_config_function=fake_read_system,
                               read_user_config_function=fake_read_user,
                               read_user_dir_function=read_user_dir)
         config.save_user_config()
@@ -168,48 +105,10 @@ class Test_SimpleConfig(unittest.TestCase):
         with open(os.path.join(self.electrum_dir, "config"), "r") as f:
             contents = f.read()
         result = ast.literal_eval(contents)
+        result.pop('config_version', None)
         self.assertEqual({"something": "a"}, result)
 
 
-class TestSystemConfig(unittest.TestCase):
-
-    sample_conf = """
-[client]
-gap_limit = 5
-
-[something_else]
-everything = 42
-"""
-
-    def setUp(self):
-        super(TestSystemConfig, self).setUp()
-        self.thefile = tempfile.mkstemp(suffix=".electrum.test.conf")[1]
-
-    def tearDown(self):
-        super(TestSystemConfig, self).tearDown()
-        os.remove(self.thefile)
-
-    def test_read_system_config_file_does_not_exist(self):
-        somefile = "/foo/I/do/not/exist/electrum.conf"
-        result = read_system_config(somefile)
-        self.assertEqual({}, result)
-
-    def test_read_system_config_file_returns_file_options(self):
-        with open(self.thefile, "w") as f:
-            f.write(self.sample_conf)
-
-        result = read_system_config(self.thefile)
-        self.assertEqual({"gap_limit": "5"}, result)
-
-    def test_read_system_config_file_no_sections(self):
-
-        with open(self.thefile, "w") as f:
-            f.write("gap_limit = 5")  # The file has no sections at all
-
-        result = read_system_config(self.thefile)
-        self.assertEqual({}, result)
-
-
 class TestUserConfig(unittest.TestCase):
 
     def setUp(self):

lib/tests/test_transaction.py

@@ -231,6 +231,389 @@ class TestTransaction(unittest.TestCase):
         tx = transaction.Transaction('010000000001010d350cefa29138de18a2d63a93cffda63721b07a6ecfa80a902f9514104b55ca0000000000fdffffff012a4a824a00000000160014b869999d342a5d42d6dc7af1efc28456da40297a024730440220475bb55814a52ea1036919e4408218c693b8bf93637b9f54c821b5baa3b846e102207276ed7a79493142c11fb01808a4142bbdd525ae7bdccdf8ecb7b8e3c856b4d90121024cdeaca7a53a7e23a1edbe9260794eaa83063534b5f111ee3c67d8b0cb88f0eec8010000')
         self.assertEqual('51087ece75c697cc872d2e643d646b0f3e1f2666fa1820b7bff4343d50dd680e', tx.txid())
 
+    def test_txid_input_p2wsh_p2sh_not_multisig(self):
+        tx = transaction.Transaction('0100000000010160f84fdcda039c3ca1b20038adea2d49a53db92f7c467e8def13734232bb610804000000232200202814720f16329ab81cb8867c4d447bd13255931f23e6655944c9ada1797fcf88ffffffff0ba3dcfc04000000001976a91488124a57c548c9e7b1dd687455af803bd5765dea88acc9f44900000000001976a914da55045a0ccd40a56ce861946d13eb861eb5f2d788ac49825e000000000017a914ca34d4b190e36479aa6e0023cfe0a8537c6aa8dd87680c0d00000000001976a914651102524c424b2e7c44787c4f21e4c54dffafc088acf02fa9000000000017a914ee6c596e6f7066466d778d4f9ba633a564a6e95d874d250900000000001976a9146ca7976b48c04fd23867748382ee8401b1d27c2988acf5119600000000001976a914cf47d5dcdba02fd547c600697097252d38c3214a88ace08a12000000000017a914017bef79d92d5ec08c051786bad317e5dd3befcf87e3d76201000000001976a9148ec1b88b66d142bcbdb42797a0fd402c23e0eec288ac718f6900000000001976a914e66344472a224ce6f843f2989accf435ae6a808988ac65e51300000000001976a914cad6717c13a2079066f876933834210ebbe68c3f88ac0347304402201a4907c4706104320313e182ecbb1b265b2d023a79586671386de86bb47461590220472c3db9fc99a728ebb9b555a72e3481d20b181bd059a9c1acadfb853d90c96c01210338a46f2a54112fef8803c8478bc17e5f8fc6a5ec276903a946c1fafb2e3a8b181976a914eda8660085bf607b82bd18560ca8f3a9ec49178588ac00000000')
+        self.assertEqual('e9933221a150f78f9f224899f8568ff6422ffcc28ca3d53d87936368ff7c4b1d', tx.txid())
+
+    # input: p2sh, not multisig
+    def test_txid_regression_issue_3899(self):
+        tx = transaction.Transaction('0100000004328685b0352c981d3d451b471ae3bfc78b82565dc2a54049a81af273f0a9fd9c010000000b0009630330472d5fae685bffffffff328685b0352c981d3d451b471ae3bfc78b82565dc2a54049a81af273f0a9fd9c020000000b0009630359646d5fae6858ffffffff328685b0352c981d3d451b471ae3bfc78b82565dc2a54049a81af273f0a9fd9c030000000b000963034bd4715fae6854ffffffff328685b0352c981d3d451b471ae3bfc78b82565dc2a54049a81af273f0a9fd9c040000000b000963036de8705fae6860ffffffff0130750000000000001976a914b5abca61d20f9062fb1fdbb880d9d93bac36675188ac00000000')
+        self.assertEqual('f570d5d1e965ee61bcc7005f8fefb1d3abbed9d7ddbe035e2a68fa07e5fc4a0d', tx.txid())
+
+
+# these transactions are from Bitcoin Core unit tests --->
+# https://github.com/bitcoin/bitcoin/blob/11376b5583a283772c82f6d32d0007cdbf5b8ef0/src/test/data/tx_valid.json
+
+    def test_txid_bitcoin_core_0001(self):
+        tx = transaction.Transaction('0100000001b14bdcbc3e01bdaad36cc08e81e69c82e1060bc14e518db2b49aa43ad90ba26000000000490047304402203f16c6f40162ab686621ef3000b04e75418a0c0cb2d8aebeac894ae360ac1e780220ddc15ecdfc3507ac48e1681a33eb60996631bf6bf5bc0a0682c4db743ce7ca2b01ffffffff0140420f00000000001976a914660d4ef3a743e3e696ad990364e555c271ad504b88ac00000000')
+        self.assertEqual('23b397edccd3740a74adb603c9756370fafcde9bcc4483eb271ecad09a94dd63', tx.txid())
+
+    def test_txid_bitcoin_core_0002(self):
+        tx = transaction.Transaction('0100000001b14bdcbc3e01bdaad36cc08e81e69c82e1060bc14e518db2b49aa43ad90ba260000000004a0048304402203f16c6f40162ab686621ef3000b04e75418a0c0cb2d8aebeac894ae360ac1e780220ddc15ecdfc3507ac48e1681a33eb60996631bf6bf5bc0a0682c4db743ce7ca2bab01ffffffff0140420f00000000001976a914660d4ef3a743e3e696ad990364e555c271ad504b88ac00000000')
+        self.assertEqual('fcabc409d8e685da28536e1e5ccc91264d755cd4c57ed4cae3dbaa4d3b93e8ed', tx.txid())
+
+    def test_txid_bitcoin_core_0003(self):
+        tx = transaction.Transaction('0100000001b14bdcbc3e01bdaad36cc08e81e69c82e1060bc14e518db2b49aa43ad90ba260000000004a01ff47304402203f16c6f40162ab686621ef3000b04e75418a0c0cb2d8aebeac894ae360ac1e780220ddc15ecdfc3507ac48e1681a33eb60996631bf6bf5bc0a0682c4db743ce7ca2b01ffffffff0140420f00000000001976a914660d4ef3a743e3e696ad990364e555c271ad504b88ac00000000')
+        self.assertEqual('c9aa95f2c48175fdb70b34c23f1c3fc44f869b073a6f79b1343fbce30c3cb575', tx.txid())
+
+    def test_txid_bitcoin_core_0004(self):
+        tx = transaction.Transaction('0100000001b14bdcbc3e01bdaad36cc08e81e69c82e1060bc14e518db2b49aa43ad90ba26000000000495147304402203f16c6f40162ab686621ef3000b04e75418a0c0cb2d8aebeac894ae360ac1e780220ddc15ecdfc3507ac48e1681a33eb60996631bf6bf5bc0a0682c4db743ce7ca2b01ffffffff0140420f00000000001976a914660d4ef3a743e3e696ad990364e555c271ad504b88ac00000000')
+        self.assertEqual('da94fda32b55deb40c3ed92e135d69df7efc4ee6665e0beb07ef500f407c9fd2', tx.txid())
+
+    def test_txid_bitcoin_core_0005(self):
+        tx = transaction.Transaction('0100000001b14bdcbc3e01bdaad36cc08e81e69c82e1060bc14e518db2b49aa43ad90ba26000000000494f47304402203f16c6f40162ab686621ef3000b04e75418a0c0cb2d8aebeac894ae360ac1e780220ddc15ecdfc3507ac48e1681a33eb60996631bf6bf5bc0a0682c4db743ce7ca2b01ffffffff0140420f00000000001976a914660d4ef3a743e3e696ad990364e555c271ad504b88ac00000000')
+        self.assertEqual('f76f897b206e4f78d60fe40f2ccb542184cfadc34354d3bb9bdc30cc2f432b86', tx.txid())
+
+    def test_txid_bitcoin_core_0006(self):
+        tx = transaction.Transaction('01000000010276b76b07f4935c70acf54fbf1f438a4c397a9fb7e633873c4dd3bc062b6b40000000008c493046022100d23459d03ed7e9511a47d13292d3430a04627de6235b6e51a40f9cd386f2abe3022100e7d25b080f0bb8d8d5f878bba7d54ad2fda650ea8d158a33ee3cbd11768191fd004104b0e2c879e4daf7b9ab68350228c159766676a14f5815084ba166432aab46198d4cca98fa3e9981d0a90b2effc514b76279476550ba3663fdcaff94c38420e9d5000000000100093d00000000001976a9149a7b0f3b80c6baaeedce0a0842553800f832ba1f88ac00000000')
+        self.assertEqual('c99c49da4c38af669dea436d3e73780dfdb6c1ecf9958baa52960e8baee30e73', tx.txid())
+
+    def test_txid_bitcoin_core_0007(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000006a473044022067288ea50aa799543a536ff9306f8e1cba05b9c6b10951175b924f96732555ed022026d7b5265f38d21541519e4a1e55044d5b9e17e15cdbaf29ae3792e99e883e7a012103ba8c8b86dea131c22ab967e6dd99bdae8eff7a1f75a2c35f1f944109e3fe5e22ffffffff010000000000000000015100000000')
+        self.assertEqual('e41ffe19dff3cbedb413a2ca3fbbcd05cb7fd7397ffa65052f8928aa9c700092', tx.txid())
+
+    def test_txid_bitcoin_core_0008(self):
+        tx = transaction.Transaction('01000000023d6cf972d4dff9c519eff407ea800361dd0a121de1da8b6f4138a2f25de864b4000000008a4730440220ffda47bfc776bcd269da4832626ac332adfca6dd835e8ecd83cd1ebe7d709b0e022049cffa1cdc102a0b56e0e04913606c70af702a1149dc3b305ab9439288fee090014104266abb36d66eb4218a6dd31f09bb92cf3cfa803c7ea72c1fc80a50f919273e613f895b855fb7465ccbc8919ad1bd4a306c783f22cd3227327694c4fa4c1c439affffffff21ebc9ba20594737864352e95b727f1a565756f9d365083eb1a8596ec98c97b7010000008a4730440220503ff10e9f1e0de731407a4a245531c9ff17676eda461f8ceeb8c06049fa2c810220c008ac34694510298fa60b3f000df01caa244f165b727d4896eb84f81e46bcc4014104266abb36d66eb4218a6dd31f09bb92cf3cfa803c7ea72c1fc80a50f919273e613f895b855fb7465ccbc8919ad1bd4a306c783f22cd3227327694c4fa4c1c439affffffff01f0da5200000000001976a914857ccd42dded6df32949d4646dfa10a92458cfaa88ac00000000')
+        self.assertEqual('f7fdd091fa6d8f5e7a8c2458f5c38faffff2d3f1406b6e4fe2c99dcc0d2d1cbb', tx.txid())
+
+    def test_txid_bitcoin_core_0009(self):
+        tx = transaction.Transaction('01000000020002000000000000000000000000000000000000000000000000000000000000000000000151ffffffff0001000000000000000000000000000000000000000000000000000000000000000000006b483045022100c9cdd08798a28af9d1baf44a6c77bcc7e279f47dc487c8c899911bc48feaffcc0220503c5c50ae3998a733263c5c0f7061b483e2b56c4c41b456e7d2f5a78a74c077032102d5c25adb51b61339d2b05315791e21bbe80ea470a49db0135720983c905aace0ffffffff010000000000000000015100000000')
+        self.assertEqual('b56471690c3ff4f7946174e51df68b47455a0d29344c351377d712e6d00eabe5', tx.txid())
+
+    def test_txid_bitcoin_core_0010(self):
+        tx = transaction.Transaction('010000000100010000000000000000000000000000000000000000000000000000000000000000000009085768617420697320ffffffff010000000000000000015100000000')
+        self.assertEqual('99517e5b47533453cc7daa332180f578be68b80370ecfe84dbfff7f19d791da4', tx.txid())
+
+    def test_txid_bitcoin_core_0011(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000006e493046022100c66c9cdf4c43609586d15424c54707156e316d88b0a1534c9e6b0d4f311406310221009c0fe51dbc9c4ab7cc25d3fdbeccf6679fe6827f08edf2b4a9f16ee3eb0e438a0123210338e8034509af564c62644c07691942e0c056752008a173c89f60ab2a88ac2ebfacffffffff010000000000000000015100000000')
+        self.assertEqual('ab097537b528871b9b64cb79a769ae13c3c3cd477cc9dddeebe657eabd7fdcea', tx.txid())
+
+    def test_txid_bitcoin_core_0012(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000006e493046022100e1eadba00d9296c743cb6ecc703fd9ddc9b3cd12906176a226ae4c18d6b00796022100a71aef7d2874deff681ba6080f1b278bac7bb99c61b08a85f4311970ffe7f63f012321030c0588dc44d92bdcbf8e72093466766fdc265ead8db64517b0c542275b70fffbacffffffff010040075af0750700015100000000')
+        self.assertEqual('4d163e00f1966e9a1eab8f9374c3e37f4deb4857c247270e25f7d79a999d2dc9', tx.txid())
+
+    def test_txid_bitcoin_core_0013(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000006d483045022027deccc14aa6668e78a8c9da3484fbcd4f9dcc9bb7d1b85146314b21b9ae4d86022100d0b43dece8cfb07348de0ca8bc5b86276fa88f7f2138381128b7c36ab2e42264012321029bb13463ddd5d2cc05da6e84e37536cb9525703cfd8f43afdb414988987a92f6acffffffff020040075af075070001510000000000000000015100000000')
+        self.assertEqual('9fe2ef9dde70e15d78894a4800b7df3bbfb1addb9a6f7d7c204492fdb6ee6cc4', tx.txid())
+
+    def test_txid_bitcoin_core_0014(self):
+        tx = transaction.Transaction('01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff025151ffffffff010000000000000000015100000000')
+        self.assertEqual('99d3825137602e577aeaf6a2e3c9620fd0e605323dc5265da4a570593be791d4', tx.txid())
+
+    def test_txid_bitcoin_core_0015(self):
+        tx = transaction.Transaction('01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff6451515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151515151ffffffff010000000000000000015100000000')
+        self.assertEqual('c0d67409923040cc766bbea12e4c9154393abef706db065ac2e07d91a9ba4f84', tx.txid())
+
+    def test_txid_bitcoin_core_0016(self):
+        tx = transaction.Transaction('010000000200010000000000000000000000000000000000000000000000000000000000000000000049483045022100d180fd2eb9140aeb4210c9204d3f358766eb53842b2a9473db687fa24b12a3cc022079781799cd4f038b85135bbe49ec2b57f306b2bb17101b17f71f000fcab2b6fb01ffffffff0002000000000000000000000000000000000000000000000000000000000000000000004847304402205f7530653eea9b38699e476320ab135b74771e1c48b81a5d041e2ca84b9be7a802200ac8d1f40fb026674fe5a5edd3dea715c27baa9baca51ed45ea750ac9dc0a55e81ffffffff010100000000000000015100000000')
+        self.assertEqual('c610d85d3d5fdf5046be7f123db8a0890cee846ee58de8a44667cfd1ab6b8666', tx.txid())
+
+    def test_txid_bitcoin_core_0017(self):
+        tx = transaction.Transaction('01000000020001000000000000000000000000000000000000000000000000000000000000000000004948304502203a0f5f0e1f2bdbcd04db3061d18f3af70e07f4f467cbc1b8116f267025f5360b022100c792b6e215afc5afc721a351ec413e714305cb749aae3d7fee76621313418df101010000000002000000000000000000000000000000000000000000000000000000000000000000004847304402205f7530653eea9b38699e476320ab135b74771e1c48b81a5d041e2ca84b9be7a802200ac8d1f40fb026674fe5a5edd3dea715c27baa9baca51ed45ea750ac9dc0a55e81ffffffff010100000000000000015100000000')
+        self.assertEqual('a647a7b3328d2c698bfa1ee2dd4e5e05a6cea972e764ccb9bd29ea43817ca64f', tx.txid())
+
+    def test_txid_bitcoin_core_0018(self):
+        tx = transaction.Transaction('010000000370ac0a1ae588aaf284c308d67ca92c69a39e2db81337e563bf40c59da0a5cf63000000006a4730440220360d20baff382059040ba9be98947fd678fb08aab2bb0c172efa996fd8ece9b702201b4fb0de67f015c90e7ac8a193aeab486a1f587e0f54d0fb9552ef7f5ce6caec032103579ca2e6d107522f012cd00b52b9a65fb46f0c57b9b8b6e377c48f526a44741affffffff7d815b6447e35fbea097e00e028fb7dfbad4f3f0987b4734676c84f3fcd0e804010000006b483045022100c714310be1e3a9ff1c5f7cacc65c2d8e781fc3a88ceb063c6153bf950650802102200b2d0979c76e12bb480da635f192cc8dc6f905380dd4ac1ff35a4f68f462fffd032103579ca2e6d107522f012cd00b52b9a65fb46f0c57b9b8b6e377c48f526a44741affffffff3f1f097333e4d46d51f5e77b53264db8f7f5d2e18217e1099957d0f5af7713ee010000006c493046022100b663499ef73273a3788dea342717c2640ac43c5a1cf862c9e09b206fcb3f6bb8022100b09972e75972d9148f2bdd462e5cb69b57c1214b88fc55ca638676c07cfc10d8032103579ca2e6d107522f012cd00b52b9a65fb46f0c57b9b8b6e377c48f526a44741affffffff0380841e00000000001976a914bfb282c70c4191f45b5a6665cad1682f2c9cfdfb88ac80841e00000000001976a9149857cc07bed33a5cf12b9c5e0500b675d500c81188ace0fd1c00000000001976a91443c52850606c872403c0601e69fa34b26f62db4a88ac00000000')
+        self.assertEqual('afd9c17f8913577ec3509520bd6e5d63e9c0fd2a5f70c787993b097ba6ca9fae', tx.txid())
+
+    def test_txid_bitcoin_core_0019(self):
+        tx = transaction.Transaction('01000000012312503f2491a2a97fcd775f11e108a540a5528b5d4dee7a3c68ae4add01dab300000000fdfe0000483045022100f6649b0eddfdfd4ad55426663385090d51ee86c3481bdc6b0c18ea6c0ece2c0b0220561c315b07cffa6f7dd9df96dbae9200c2dee09bf93cc35ca05e6cdf613340aa0148304502207aacee820e08b0b174e248abd8d7a34ed63b5da3abedb99934df9fddd65c05c4022100dfe87896ab5ee3df476c2655f9fbe5bd089dccbef3e4ea05b5d121169fe7f5f4014c695221031d11db38972b712a9fe1fc023577c7ae3ddb4a3004187d41c45121eecfdbb5b7210207ec36911b6ad2382860d32989c7b8728e9489d7bbc94a6b5509ef0029be128821024ea9fac06f666a4adc3fc1357b7bec1fd0bdece2b9d08579226a8ebde53058e453aeffffffff0180380100000000001976a914c9b99cddf847d10685a4fabaa0baf505f7c3dfab88ac00000000')
+        self.assertEqual('f4b05f978689c89000f729cae187dcfbe64c9819af67a4f05c0b4d59e717d64d', tx.txid())
+
+    def test_txid_bitcoin_core_0020(self):
+        tx = transaction.Transaction('0100000001f709fa82596e4f908ee331cb5e0ed46ab331d7dcfaf697fe95891e73dac4ebcb000000008c20ca42095840735e89283fec298e62ac2ddea9b5f34a8cbb7097ad965b87568100201b1b01dc829177da4a14551d2fc96a9db00c6501edfa12f22cd9cefd335c227f483045022100a9df60536df5733dd0de6bc921fab0b3eee6426501b43a228afa2c90072eb5ca02201c78b74266fac7d1db5deff080d8a403743203f109fbcabf6d5a760bf87386d20100ffffffff01c075790000000000232103611f9a45c18f28f06f19076ad571c344c82ce8fcfe34464cf8085217a2d294a6ac00000000')
+        self.assertEqual('cc60b1f899ec0a69b7c3f25ddf32c4524096a9c5b01cbd84c6d0312a0c478984', tx.txid())
+
+    def test_txid_bitcoin_core_0021(self):
+        tx = transaction.Transaction('01000000012c651178faca83be0b81c8c1375c4b0ad38d53c8fe1b1c4255f5e795c25792220000000049483045022100d6044562284ac76c985018fc4a90127847708c9edb280996c507b28babdc4b2a02203d74eca3f1a4d1eea7ff77b528fde6d5dc324ec2dbfdb964ba885f643b9704cd01ffffffff010100000000000000232102c2410f8891ae918cab4ffc4bb4a3b0881be67c7a1e7faa8b5acf9ab8932ec30cac00000000')
+        self.assertEqual('1edc7f214659d52c731e2016d258701911bd62a0422f72f6c87a1bc8dd3f8667', tx.txid())
+
+    def test_txid_bitcoin_core_0022(self):
+        tx = transaction.Transaction('0100000001f725ea148d92096a79b1709611e06e94c63c4ef61cbae2d9b906388efd3ca99c000000000100ffffffff0101000000000000002321028a1d66975dbdf97897e3a4aef450ebeb5b5293e4a0b4a6d3a2daaa0b2b110e02ac00000000')
+        self.assertEqual('018adb7133fde63add9149a2161802a1bcf4bdf12c39334e880c073480eda2ff', tx.txid())
+
+    def test_txid_bitcoin_core_0023(self):
+        tx = transaction.Transaction('0100000001be599efaa4148474053c2fa031c7262398913f1dc1d9ec201fd44078ed004e44000000004900473044022022b29706cb2ed9ef0cb3c97b72677ca2dfd7b4160f7b4beb3ba806aa856c401502202d1e52582412eba2ed474f1f437a427640306fd3838725fab173ade7fe4eae4a01ffffffff010100000000000000232103ac4bba7e7ca3e873eea49e08132ad30c7f03640b6539e9b59903cf14fd016bbbac00000000')
+        self.assertEqual('1464caf48c708a6cc19a296944ded9bb7f719c9858986d2501cf35068b9ce5a2', tx.txid())
+
+    def test_txid_bitcoin_core_0024(self):
+        tx = transaction.Transaction('010000000112b66d5e8c7d224059e946749508efea9d66bf8d0c83630f080cf30be8bb6ae100000000490047304402206ffe3f14caf38ad5c1544428e99da76ffa5455675ec8d9780fac215ca17953520220779502985e194d84baa36b9bd40a0dbd981163fa191eb884ae83fc5bd1c86b1101ffffffff010100000000000000232103905380c7013e36e6e19d305311c1b81fce6581f5ee1c86ef0627c68c9362fc9fac00000000')
+        self.assertEqual('1fb73fbfc947d52f5d80ba23b67c06a232ad83fdd49d1c0a657602f03fbe8f7a', tx.txid())
+
+    def test_txid_bitcoin_core_0025(self):
+        tx = transaction.Transaction('0100000001b0ef70cc644e0d37407e387e73bfad598d852a5aa6d691d72b2913cebff4bceb000000004a00473044022068cd4851fc7f9a892ab910df7a24e616f293bcb5c5fbdfbc304a194b26b60fba022078e6da13d8cb881a22939b952c24f88b97afd06b4c47a47d7f804c9a352a6d6d0100ffffffff0101000000000000002321033bcaa0a602f0d44cc9d5637c6e515b0471db514c020883830b7cefd73af04194ac00000000')
+        self.assertEqual('24cecfce0fa880b09c9b4a66c5134499d1b09c01cc5728cd182638bea070e6ab', tx.txid())
+
+    def test_txid_bitcoin_core_0026(self):
+        tx = transaction.Transaction('0100000001c188aa82f268fcf08ba18950f263654a3ea6931dabc8bf3ed1d4d42aaed74cba000000004b0000483045022100940378576e069aca261a6b26fb38344e4497ca6751bb10905c76bb689f4222b002204833806b014c26fd801727b792b1260003c55710f87c5adbd7a9cb57446dbc9801ffffffff0101000000000000002321037c615d761e71d38903609bf4f46847266edc2fb37532047d747ba47eaae5ffe1ac00000000')
+        self.assertEqual('9eaa819e386d6a54256c9283da50c230f3d8cd5376d75c4dcc945afdeb157dd7', tx.txid())
+
+    def test_txid_bitcoin_core_0027(self):
+        tx = transaction.Transaction('01000000012432b60dc72cebc1a27ce0969c0989c895bdd9e62e8234839117f8fc32d17fbc000000004a493046022100a576b52051962c25e642c0fd3d77ee6c92487048e5d90818bcf5b51abaccd7900221008204f8fb121be4ec3b24483b1f92d89b1b0548513a134e345c5442e86e8617a501ffffffff010000000000000000016a00000000')
+        self.assertEqual('46224764c7870f95b58f155bce1e38d4da8e99d42dbb632d0dd7c07e092ee5aa', tx.txid())
+
+    def test_txid_bitcoin_core_0028(self):
+        tx = transaction.Transaction('01000000014710b0e7cf9f8930de259bdc4b84aa5dfb9437b665a3e3a21ff26e0bf994e183000000004a493046022100a166121a61b4eeb19d8f922b978ff6ab58ead8a5a5552bf9be73dc9c156873ea02210092ad9bc43ee647da4f6652c320800debcf08ec20a094a0aaf085f63ecb37a17201ffffffff010000000000000000016a00000000')
+        self.assertEqual('8d66836045db9f2d7b3a75212c5e6325f70603ee27c8333a3bce5bf670d9582e', tx.txid())
+
+    def test_txid_bitcoin_core_0029(self):
+        tx = transaction.Transaction('01000000015ebaa001d8e4ec7a88703a3bcf69d98c874bca6299cca0f191512bf2a7826832000000004948304502203bf754d1c6732fbf87c5dcd81258aefd30f2060d7bd8ac4a5696f7927091dad1022100f5bcb726c4cf5ed0ed34cc13dadeedf628ae1045b7cb34421bc60b89f4cecae701ffffffff010000000000000000016a00000000')
+        self.assertEqual('aab7ef280abbb9cc6fbaf524d2645c3daf4fcca2b3f53370e618d9cedf65f1f8', tx.txid())
+
+    def test_txid_bitcoin_core_0030(self):
+        tx = transaction.Transaction('010000000144490eda355be7480f2ec828dcc1b9903793a8008fad8cfe9b0c6b4d2f0355a900000000924830450221009c0a27f886a1d8cb87f6f595fbc3163d28f7a81ec3c4b252ee7f3ac77fd13ffa02203caa8dfa09713c8c4d7ef575c75ed97812072405d932bd11e6a1593a98b679370148304502201e3861ef39a526406bad1e20ecad06be7375ad40ddb582c9be42d26c3a0d7b240221009d0a3985e96522e59635d19cc4448547477396ce0ef17a58e7d74c3ef464292301ffffffff010000000000000000016a00000000')
+        self.assertEqual('6327783a064d4e350c454ad5cd90201aedf65b1fc524e73709c52f0163739190', tx.txid())
+
+    def test_txid_bitcoin_core_0031(self):
+        tx = transaction.Transaction('010000000144490eda355be7480f2ec828dcc1b9903793a8008fad8cfe9b0c6b4d2f0355a9000000004a48304502207a6974a77c591fa13dff60cabbb85a0de9e025c09c65a4b2285e47ce8e22f761022100f0efaac9ff8ac36b10721e0aae1fb975c90500b50c56e8a0cc52b0403f0425dd0100ffffffff010000000000000000016a00000000')
+        self.assertEqual('892464645599cc3c2d165adcc612e5f982a200dfaa3e11e9ce1d228027f46880', tx.txid())
+
+    def test_txid_bitcoin_core_0032(self):
+        tx = transaction.Transaction('010000000144490eda355be7480f2ec828dcc1b9903793a8008fad8cfe9b0c6b4d2f0355a9000000004a483045022100fa4a74ba9fd59c59f46c3960cf90cbe0d2b743c471d24a3d5d6db6002af5eebb02204d70ec490fd0f7055a7c45f86514336e3a7f03503dacecabb247fc23f15c83510151ffffffff010000000000000000016a00000000')
+        self.assertEqual('578db8c6c404fec22c4a8afeaf32df0e7b767c4dda3478e0471575846419e8fc', tx.txid())
+
+    def test_txid_bitcoin_core_0033(self):
+        tx = transaction.Transaction('0100000001e0be9e32f1f89c3d916c4f21e55cdcd096741b895cc76ac353e6023a05f4f7cc00000000d86149304602210086e5f736a2c3622ebb62bd9d93d8e5d76508b98be922b97160edc3dcca6d8c47022100b23c312ac232a4473f19d2aeb95ab7bdf2b65518911a0d72d50e38b5dd31dc820121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ac4730440220508fa761865c8abd81244a168392876ee1d94e8ed83897066b5e2df2400dad24022043f5ee7538e87e9c6aef7ef55133d3e51da7cc522830a9c4d736977a76ef755c0121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ffffffff010000000000000000016a00000000')
+        self.assertEqual('974f5148a0946f9985e75a240bb24c573adbbdc25d61e7b016cdbb0a5355049f', tx.txid())
+
+    def test_txid_bitcoin_core_0034(self):
+        tx = transaction.Transaction('01000000013c6f30f99a5161e75a2ce4bca488300ca0c6112bde67f0807fe983feeff0c91001000000e608646561646265656675ab61493046022100ce18d384221a731c993939015e3d1bcebafb16e8c0b5b5d14097ec8177ae6f28022100bcab227af90bab33c3fe0a9abfee03ba976ee25dc6ce542526e9b2e56e14b7f10121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ac493046022100c3b93edcc0fd6250eb32f2dd8a0bba1754b0f6c3be8ed4100ed582f3db73eba2022100bf75b5bd2eff4d6bf2bda2e34a40fcc07d4aa3cf862ceaa77b47b81eff829f9a01ab21038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ffffffff010000000000000000016a00000000')
+        self.assertEqual('b0097ec81df231893a212657bf5fe5a13b2bff8b28c0042aca6fc4159f79661b', tx.txid())
+
+    def test_txid_bitcoin_core_0035(self):
+        tx = transaction.Transaction('01000000016f3dbe2ca96fa217e94b1017860be49f20820dea5c91bdcb103b0049d5eb566000000000fd1d0147304402203989ac8f9ad36b5d0919d97fa0a7f70c5272abee3b14477dc646288a8b976df5022027d19da84a066af9053ad3d1d7459d171b7e3a80bc6c4ef7a330677a6be548140147304402203989ac8f9ad36b5d0919d97fa0a7f70c5272abee3b14477dc646288a8b976df5022027d19da84a066af9053ad3d1d7459d171b7e3a80bc6c4ef7a330677a6be548140121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ac47304402203757e937ba807e4a5da8534c17f9d121176056406a6465054bdd260457515c1a02200f02eccf1bec0f3a0d65df37889143c2e88ab7acec61a7b6f5aa264139141a2b0121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ffffffff010000000000000000016a00000000')
+        self.assertEqual('feeba255656c80c14db595736c1c7955c8c0a497622ec96e3f2238fbdd43a7c9', tx.txid())
+
+    def test_txid_bitcoin_core_0036(self):
+        tx = transaction.Transaction('01000000012139c555ccb81ee5b1e87477840991ef7b386bc3ab946b6b682a04a621006b5a01000000fdb40148304502201723e692e5f409a7151db386291b63524c5eb2030df652b1f53022fd8207349f022100b90d9bbf2f3366ce176e5e780a00433da67d9e5c79312c6388312a296a5800390148304502201723e692e5f409a7151db386291b63524c5eb2030df652b1f53022fd8207349f022100b90d9bbf2f3366ce176e5e780a00433da67d9e5c79312c6388312a296a5800390121038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f2204148304502201723e692e5f409a7151db386291b63524c5eb2030df652b1f53022fd8207349f022100b90d9bbf2f3366ce176e5e780a00433da67d9e5c79312c6388312a296a5800390175ac4830450220646b72c35beeec51f4d5bc1cbae01863825750d7f490864af354e6ea4f625e9c022100f04b98432df3a9641719dbced53393022e7249fb59db993af1118539830aab870148304502201723e692e5f409a7151db386291b63524c5eb2030df652b1f53022fd8207349f022100b90d9bbf2f3366ce176e5e780a00433da67d9e5c79312c6388312a296a580039017521038479a0fa998cd35259a2ef0a7a5c68662c1474f88ccb6d08a7677bbec7f22041ffffffff010000000000000000016a00000000')
+        self.assertEqual('a0c984fc820e57ddba97f8098fa640c8a7eb3fe2f583923da886b7660f505e1e', tx.txid())
+
+    def test_txid_bitcoin_core_0037(self):
+        tx = transaction.Transaction('0100000002f9cbafc519425637ba4227f8d0a0b7160b4e65168193d5af39747891de98b5b5000000006b4830450221008dd619c563e527c47d9bd53534a770b102e40faa87f61433580e04e271ef2f960220029886434e18122b53d5decd25f1f4acb2480659fea20aabd856987ba3c3907e0121022b78b756e2258af13779c1a1f37ea6800259716ca4b7f0b87610e0bf3ab52a01ffffffff42e7988254800876b69f24676b3e0205b77be476512ca4d970707dd5c60598ab00000000fd260100483045022015bd0139bcccf990a6af6ec5c1c52ed8222e03a0d51c334df139968525d2fcd20221009f9efe325476eb64c3958e4713e9eefe49bf1d820ed58d2112721b134e2a1a53034930460221008431bdfa72bc67f9d41fe72e94c88fb8f359ffa30b33c72c121c5a877d922e1002210089ef5fc22dd8bfc6bf9ffdb01a9862d27687d424d1fefbab9e9c7176844a187a014c9052483045022015bd0139bcccf990a6af6ec5c1c52ed8222e03a0d51c334df139968525d2fcd20221009f9efe325476eb64c3958e4713e9eefe49bf1d820ed58d2112721b134e2a1a5303210378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c71210378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c7153aeffffffff01a08601000000000017a914d8dacdadb7462ae15cd906f1878706d0da8660e68700000000')
+        self.assertEqual('5df1375ffe61ac35ca178ebb0cab9ea26dedbd0e96005dfcee7e379fa513232f', tx.txid())
+
+    def test_txid_bitcoin_core_0038(self):
+        tx = transaction.Transaction('0100000002dbb33bdf185b17f758af243c5d3c6e164cc873f6bb9f40c0677d6e0f8ee5afce000000006b4830450221009627444320dc5ef8d7f68f35010b4c050a6ed0d96b67a84db99fda9c9de58b1e02203e4b4aaa019e012e65d69b487fdf8719df72f488fa91506a80c49a33929f1fd50121022b78b756e2258af13779c1a1f37ea6800259716ca4b7f0b87610e0bf3ab52a01ffffffffdbb33bdf185b17f758af243c5d3c6e164cc873f6bb9f40c0677d6e0f8ee5afce010000009300483045022015bd0139bcccf990a6af6ec5c1c52ed8222e03a0d51c334df139968525d2fcd20221009f9efe325476eb64c3958e4713e9eefe49bf1d820ed58d2112721b134e2a1a5303483045022015bd0139bcccf990a6af6ec5c1c52ed8222e03a0d51c334df139968525d2fcd20221009f9efe325476eb64c3958e4713e9eefe49bf1d820ed58d2112721b134e2a1a5303ffffffff01a0860100000000001976a9149bc0bbdd3024da4d0c38ed1aecf5c68dd1d3fa1288ac00000000')
+        self.assertEqual('ded7ff51d89a4e1ec48162aee5a96447214d93dfb3837946af2301a28f65dbea', tx.txid())
+
+    def test_txid_bitcoin_core_0039(self):
+        tx = transaction.Transaction('010000000100010000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000')
+        self.assertEqual('3444be2e216abe77b46015e481d8cc21abd4c20446aabf49cd78141c9b9db87e', tx.txid())
+
+    def test_txid_bitcoin_core_0040(self):
+        tx = transaction.Transaction('0100000001000100000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000ff64cd1d')
+        self.assertEqual('abd62b4627d8d9b2d95fcfd8c87e37d2790637ce47d28018e3aece63c1d62649', tx.txid())
+
+    def test_txid_bitcoin_core_0041(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000065cd1d')
+        self.assertEqual('58b6de8413603b7f556270bf48caedcf17772e7105f5419f6a80be0df0b470da', tx.txid())
+
+    def test_txid_bitcoin_core_0042(self):
+        tx = transaction.Transaction('0100000001000100000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000ffffffff')
+        self.assertEqual('5f99c0abf511294d76cbe144d86b77238a03e086974bc7a8ea0bdb2c681a0324', tx.txid())
+
+    def test_txid_bitcoin_core_0043(self):
+        tx = transaction.Transaction('010000000100010000000000000000000000000000000000000000000000000000000000000000000000feffffff0100000000000000000000000000')
+        self.assertEqual('25d35877eaba19497710666473c50d5527d38503e3521107a3fc532b74cd7453', tx.txid())
+
+    def test_txid_bitcoin_core_0044(self):
+        tx = transaction.Transaction('0100000001000100000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000feffffff')
+        self.assertEqual('1b9aef851895b93c62c29fbd6ca4d45803f4007eff266e2f96ff11e9b6ef197b', tx.txid())
+
+    def test_txid_bitcoin_core_0045(self):
+        tx = transaction.Transaction('010000000100010000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000')
+        self.assertEqual('3444be2e216abe77b46015e481d8cc21abd4c20446aabf49cd78141c9b9db87e', tx.txid())
+
+    def test_txid_bitcoin_core_0046(self):
+        tx = transaction.Transaction('01000000010001000000000000000000000000000000000000000000000000000000000000000000000251b1000000000100000000000000000001000000')
+        self.assertEqual('f53761038a728b1f17272539380d96e93f999218f8dcb04a8469b523445cd0fd', tx.txid())
+
+    def test_txid_bitcoin_core_0047(self):
+        tx = transaction.Transaction('0100000001000100000000000000000000000000000000000000000000000000000000000000000000030251b1000000000100000000000000000001000000')
+        self.assertEqual('d193f0f32fceaf07bb25c897c8f99ca6f69a52f6274ca64efc2a2e180cb97fc1', tx.txid())
+
+    def test_txid_bitcoin_core_0048(self):
+        tx = transaction.Transaction('010000000132211bdd0d568506804eef0d8cc3db68c3d766ab9306cdfcc0a9c89616c8dbb1000000006c493045022100c7bb0faea0522e74ff220c20c022d2cb6033f8d167fb89e75a50e237a35fd6d202203064713491b1f8ad5f79e623d0219ad32510bfaa1009ab30cbee77b59317d6e30001210237af13eb2d84e4545af287b919c2282019c9691cc509e78e196a9d8274ed1be0ffffffff0100000000000000001976a914f1b3ed2eda9a2ebe5a9374f692877cdf87c0f95b88ac00000000')
+        self.assertEqual('50a1e0e6a134a564efa078e3bd088e7e8777c2c0aec10a752fd8706470103b89', tx.txid())
+
+    def test_txid_bitcoin_core_0049(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000')
+        self.assertEqual('e2207d1aaf6b74e5d98c2fa326d2dc803b56b30a3f90ce779fa5edb762f38755', tx.txid())
+
+    def test_txid_bitcoin_core_0050(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffff00000100000000000000000000000000')
+        self.assertEqual('f335864f7c12ec7946d2c123deb91eb978574b647af125a414262380c7fbd55c', tx.txid())
+
+    def test_txid_bitcoin_core_0051(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffbf7f0100000000000000000000000000')
+        self.assertEqual('d1edbcde44691e98a7b7f556bd04966091302e29ad9af3c2baac38233667e0d2', tx.txid())
+
+    def test_txid_bitcoin_core_0052(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000000040000100000000000000000000000000')
+        self.assertEqual('3a13e1b6371c545147173cc4055f0ed73686a9f73f092352fb4b39ca27d360e6', tx.txid())
+
+    def test_txid_bitcoin_core_0053(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffff40000100000000000000000000000000')
+        self.assertEqual('bffda23e40766d292b0510a1b556453c558980c70c94ab158d8286b3413e220d', tx.txid())
+
+    def test_txid_bitcoin_core_0054(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffff7f0100000000000000000000000000')
+        self.assertEqual('01a86c65460325dc6699714d26df512a62a854a669f6ed2e6f369a238e048cfd', tx.txid())
+
+    def test_txid_bitcoin_core_0055(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000000000800100000000000000000000000000')
+        self.assertEqual('f6d2359c5de2d904e10517d23e7c8210cca71076071bbf46de9fbd5f6233dbf1', tx.txid())
+
+    def test_txid_bitcoin_core_0056(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000feffffff0100000000000000000000000000')
+        self.assertEqual('19c2b7377229dae7aa3e50142a32fd37cef7171a01682f536e9ffa80c186f6c9', tx.txid())
+
+    def test_txid_bitcoin_core_0057(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffffff0100000000000000000000000000')
+        self.assertEqual('c9dda3a24cc8a5acb153d1085ecd2fecf6f87083122f8cdecc515b1148d4c40d', tx.txid())
+
+    def test_txid_bitcoin_core_0058(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffbf7f0100000000000000000000000000')
+        self.assertEqual('d1edbcde44691e98a7b7f556bd04966091302e29ad9af3c2baac38233667e0d2', tx.txid())
+
+    def test_txid_bitcoin_core_0059(self):
+        tx = transaction.Transaction('020000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffff7f0100000000000000000000000000')
+        self.assertEqual('01a86c65460325dc6699714d26df512a62a854a669f6ed2e6f369a238e048cfd', tx.txid())
+
+    def test_txid_bitcoin_core_0060(self):
+        tx = transaction.Transaction('02000000010001000000000000000000000000000000000000000000000000000000000000000000000251b2010000000100000000000000000000000000')
+        self.assertEqual('4b5e0aae1251a9dc66b4d5f483f1879bf518ea5e1765abc5a9f2084b43ed1ea7', tx.txid())
+
+    def test_txid_bitcoin_core_0061(self):
+        tx = transaction.Transaction('0200000001000100000000000000000000000000000000000000000000000000000000000000000000030251b2010000000100000000000000000000000000')
+        self.assertEqual('5f16eb3ca4581e2dfb46a28140a4ee15f85e4e1c032947da8b93549b53c105f5', tx.txid())
+
+    def test_txid_bitcoin_core_0062(self):
+        tx = transaction.Transaction('0100000000010100010000000000000000000000000000000000000000000000000000000000000000000000ffffffff01e8030000000000001976a9144c9c3dfac4207d5d8cb89df5722cb3d712385e3f88ac02483045022100cfb07164b36ba64c1b1e8c7720a56ad64d96f6ef332d3d37f9cb3c96477dc44502200a464cd7a9cf94cd70f66ce4f4f0625ef650052c7afcfe29d7d7e01830ff91ed012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7100000000')
+        self.assertEqual('b2ce556154e5ab22bec0a2f990b2b843f4f4085486c0d2cd82873685c0012004', tx.txid())
+
+    def test_txid_bitcoin_core_0063(self):
+        tx = transaction.Transaction('0100000000010100010000000000000000000000000000000000000000000000000000000000000000000000ffffffff01e8030000000000001976a9144c9c3dfac4207d5d8cb89df5722cb3d712385e3f88ac02483045022100aa5d8aa40a90f23ce2c3d11bc845ca4a12acd99cbea37de6b9f6d86edebba8cb022022dedc2aa0a255f74d04c0b76ece2d7c691f9dd11a64a8ac49f62a99c3a05f9d01232103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ac00000000')
+        self.assertEqual('b2ce556154e5ab22bec0a2f990b2b843f4f4085486c0d2cd82873685c0012004', tx.txid())
+
+    def test_txid_bitcoin_core_0064(self):
+        tx = transaction.Transaction('01000000000101000100000000000000000000000000000000000000000000000000000000000000000000171600144c9c3dfac4207d5d8cb89df5722cb3d712385e3fffffffff01e8030000000000001976a9144c9c3dfac4207d5d8cb89df5722cb3d712385e3f88ac02483045022100cfb07164b36ba64c1b1e8c7720a56ad64d96f6ef332d3d37f9cb3c96477dc44502200a464cd7a9cf94cd70f66ce4f4f0625ef650052c7afcfe29d7d7e01830ff91ed012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7100000000')
+        self.assertEqual('fee125c6cd142083fabd0187b1dd1f94c66c89ec6e6ef6da1374881c0c19aece', tx.txid())
+
+    def test_txid_bitcoin_core_0065(self):
+        tx = transaction.Transaction('0100000000010100010000000000000000000000000000000000000000000000000000000000000000000023220020ff25429251b5a84f452230a3c75fd886b7fc5a7865ce4a7bb7a9d7c5be6da3dbffffffff01e8030000000000001976a9144c9c3dfac4207d5d8cb89df5722cb3d712385e3f88ac02483045022100aa5d8aa40a90f23ce2c3d11bc845ca4a12acd99cbea37de6b9f6d86edebba8cb022022dedc2aa0a255f74d04c0b76ece2d7c691f9dd11a64a8ac49f62a99c3a05f9d01232103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ac00000000')
+        self.assertEqual('5f32557914351fee5f89ddee6c8983d476491d29e601d854e3927299e50450da', tx.txid())
+
+    def test_txid_bitcoin_core_0066(self):
+        tx = transaction.Transaction('0100000000010400010000000000000000000000000000000000000000000000000000000000000200000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000300000000ffffffff05540b0000000000000151d0070000000000000151840300000000000001513c0f00000000000001512c010000000000000151000248304502210092f4777a0f17bf5aeb8ae768dec5f2c14feabf9d1fe2c89c78dfed0f13fdb86902206da90a86042e252bcd1e80a168c719e4a1ddcc3cebea24b9812c5453c79107e9832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71000000000000')
+        self.assertEqual('07dfa2da3d67c8a2b9f7bd31862161f7b497829d5da90a88ba0f1a905e7a43f7', tx.txid())
+
+    def test_txid_bitcoin_core_0067(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b0000000000000151000248304502210092f4777a0f17bf5aeb8ae768dec5f2c14feabf9d1fe2c89c78dfed0f13fdb86902206da90a86042e252bcd1e80a168c719e4a1ddcc3cebea24b9812c5453c79107e9832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0068(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff0484030000000000000151d0070000000000000151540b0000000000000151c800000000000000015100024730440220699e6b0cfe015b64ca3283e6551440a34f901ba62dd4c72fe1cb815afb2e6761022021cc5e84db498b1479de14efda49093219441adc6c543e5534979605e273d80b032103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('f92bb6e4f3ff89172f23ef647f74c13951b665848009abb5862cdf7a0412415a', tx.txid())
+
+    def test_txid_bitcoin_core_0069(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b000000000000015100024730440220699e6b0cfe015b64ca3283e6551440a34f901ba62dd4c72fe1cb815afb2e6761022021cc5e84db498b1479de14efda49093219441adc6c543e5534979605e273d80b032103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0070(self):
+        tx = transaction.Transaction('0100000000010400010000000000000000000000000000000000000000000000000000000000000200000000ffffffff00010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000300000000ffffffff04b60300000000000001519e070000000000000151860b00000000000001009600000000000000015100000248304502210091b32274295c2a3fa02f5bce92fb2789e3fc6ea947fbe1a76e52ea3f4ef2381a022079ad72aefa3837a2e0c033a8652a59731da05fa4a813f4fc48e87c075037256b822103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('e657e25fc9f2b33842681613402759222a58cf7dd504d6cdc0b69a0b8c2e7dcb', tx.txid())
+
+    def test_txid_bitcoin_core_0071(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b0000000000000151000248304502210091b32274295c2a3fa02f5bce92fb2789e3fc6ea947fbe1a76e52ea3f4ef2381a022079ad72aefa3837a2e0c033a8652a59731da05fa4a813f4fc48e87c075037256b822103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0072(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff04b60300000000000001519e070000000000000151860b0000000000000100960000000000000001510002473044022022fceb54f62f8feea77faac7083c3b56c4676a78f93745adc8a35800bc36adfa022026927df9abcf0a8777829bcfcce3ff0a385fa54c3f9df577405e3ef24ee56479022103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('4ede5e22992d43d42ccdf6553fb46e448aa1065ba36423f979605c1e5ab496b8', tx.txid())
+
+    def test_txid_bitcoin_core_0073(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b00000000000001510002473044022022fceb54f62f8feea77faac7083c3b56c4676a78f93745adc8a35800bc36adfa022026927df9abcf0a8777829bcfcce3ff0a385fa54c3f9df577405e3ef24ee56479022103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0074(self):
+        tx = transaction.Transaction('01000000000103000100000000000000000000000000000000000000000000000000000000000000000000000200000000010000000000000000000000000000000000000000000000000000000000000100000000ffffffff000100000000000000000000000000000000000000000000000000000000000002000000000200000003e8030000000000000151d0070000000000000151b80b00000000000001510002473044022022fceb54f62f8feea77faac7083c3b56c4676a78f93745adc8a35800bc36adfa022026927df9abcf0a8777829bcfcce3ff0a385fa54c3f9df577405e3ef24ee56479022103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('cfe9f4b19f52b8366860aec0d2b5815e329299b2e9890d477edd7f1182be7ac8', tx.txid())
+
+    def test_txid_bitcoin_core_0075(self):
+        tx = transaction.Transaction('0100000000010400010000000000000000000000000000000000000000000000000000000000000200000000ffffffff00010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000300000000ffffffff03e8030000000000000151d0070000000000000151b80b0000000000000151000002483045022100a3cec69b52cba2d2de623eeef89e0ba1606184ea55476c0f8189fda231bc9cbb022003181ad597f7c380a7d1c740286b1d022b8b04ded028b833282e055e03b8efef812103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('aee8f4865ca40fa77ff2040c0d7de683bea048b103d42ca406dc07dd29d539cb', tx.txid())
+
+    def test_txid_bitcoin_core_0076(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b00000000000001510002483045022100a3cec69b52cba2d2de623eeef89e0ba1606184ea55476c0f8189fda231bc9cbb022003181ad597f7c380a7d1c740286b1d022b8b04ded028b833282e055e03b8efef812103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0077(self):
+        tx = transaction.Transaction('0100000000010300010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000200000000ffffffff03e8030000000000000151d0070000000000000151b80b00000000000001510002483045022100a3cec69b52cba2d2de623ffffffffff1606184ea55476c0f8189fda231bc9cbb022003181ad597f7c380a7d1c740286b1d022b8b04ded028b833282e055e03b8efef812103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('8a1bddf924d24570074b09d7967c145e54dc4cee7972a92fd975a2ad9e64b424', tx.txid())
+
+    def test_txid_bitcoin_core_0078(self):
+        tx = transaction.Transaction('0100000000010100010000000000000000000000000000000000000000000000000000000000000000000000ffffffff010000000000000000015102fd08020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002755100000000')
+        self.assertEqual('d93ab9e12d7c29d2adc13d5cdf619d53eec1f36eb6612f55af52be7ba0448e97', tx.txid())
+
+    def test_txid_bitcoin_core_0079(self):
+        tx = transaction.Transaction('0100000000010c00010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff0001000000000000000000000000000000000000000000000000000000000000020000006a473044022026c2e65b33fcd03b2a3b0f25030f0244bd23cc45ae4dec0f48ae62255b1998a00220463aa3982b718d593a6b9e0044513fd67a5009c2fdccc59992cffc2b167889f4012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff0001000000000000000000000000000000000000000000000000000000000000030000006a4730440220008bd8382911218dcb4c9f2e75bf5c5c3635f2f2df49b36994fde85b0be21a1a02205a539ef10fb4c778b522c1be852352ea06c67ab74200977c722b0bc68972575a012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff0001000000000000000000000000000000000000000000000000000000000000040000006b483045022100d9436c32ff065127d71e1a20e319e4fe0a103ba0272743dbd8580be4659ab5d302203fd62571ee1fe790b182d078ecfd092a509eac112bea558d122974ef9cc012c7012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff0001000000000000000000000000000000000000000000000000000000000000050000006a47304402200e2c149b114ec546015c13b2b464bbcb0cdc5872e6775787527af6cbc4830b6c02207e9396c6979fb15a9a2b96ca08a633866eaf20dc0ff3c03e512c1d5a1654f148012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff0001000000000000000000000000000000000000000000000000000000000000060000006b483045022100b20e70d897dc15420bccb5e0d3e208d27bdd676af109abbd3f88dbdb7721e6d6022005836e663173fbdfe069f54cde3c2decd3d0ea84378092a5d9d85ec8642e8a41012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff00010000000000000000000000000000000000000000000000000000000000000700000000ffffffff00010000000000000000000000000000000000000000000000000000000000000800000000ffffffff00010000000000000000000000000000000000000000000000000000000000000900000000ffffffff00010000000000000000000000000000000000000000000000000000000000000a00000000ffffffff00010000000000000000000000000000000000000000000000000000000000000b0000006a47304402206639c6e05e3b9d2675a7f3876286bdf7584fe2bbd15e0ce52dd4e02c0092cdc60220757d60b0a61fc95ada79d23746744c72bac1545a75ff6c2c7cdb6ae04e7e9592012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71ffffffff0ce8030000000000000151e9030000000000000151ea030000000000000151eb030000000000000151ec030000000000000151ed030000000000000151ee030000000000000151ef030000000000000151f0030000000000000151f1030000000000000151f2030000000000000151f30300000000000001510248304502210082219a54f61bf126bfc3fa068c6e33831222d1d7138c6faa9d33ca87fd4202d6022063f9902519624254d7c2c8ea7ba2d66ae975e4e229ae38043973ec707d5d4a83012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7102473044022017fb58502475848c1b09f162cb1688d0920ff7f142bed0ef904da2ccc88b168f02201798afa61850c65e77889cbcd648a5703b487895517c88f85cdd18b021ee246a012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7100000000000247304402202830b7926e488da75782c81a54cd281720890d1af064629ebf2e31bf9f5435f30220089afaa8b455bbeb7d9b9c3fe1ed37d07685ade8455c76472cda424d93e4074a012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7102473044022026326fcdae9207b596c2b05921dbac11d81040c4d40378513670f19d9f4af893022034ecd7a282c0163b89aaa62c22ec202cef4736c58cd251649bad0d8139bcbf55012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc71024730440220214978daeb2f38cd426ee6e2f44131a33d6b191af1c216247f1dd7d74c16d84a02205fdc05529b0bc0c430b4d5987264d9d075351c4f4484c16e91662e90a72aab24012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710247304402204a6e9f199dc9672cf2ff8094aaa784363be1eb62b679f7ff2df361124f1dca3302205eeb11f70fab5355c9c8ad1a0700ea355d315e334822fa182227e9815308ee8f012103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710000000000')
+        self.assertEqual('b83579db5246aa34255642768167132a0c3d2932b186cd8fb9f5490460a0bf91', tx.txid())
+
+    def test_txid_bitcoin_core_0080(self):
+        tx = transaction.Transaction('010000000100010000000000000000000000000000000000000000000000000000000000000000000000ffffffff01e803000000000000015100000000')
+        self.assertEqual('2b1e44fff489d09091e5e20f9a01bbc0e8d80f0662e629fd10709cdb4922a874', tx.txid())
+
+    def test_txid_bitcoin_core_0081(self):
+        tx = transaction.Transaction('0100000000010200010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff01d00700000000000001510003483045022100e078de4e96a0e05dcdc0a414124dd8475782b5f3f0ed3f607919e9a5eeeb22bf02201de309b3a3109adb3de8074b3610d4cf454c49b61247a2779a0bcbf31c889333032103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc711976a9144c9c3dfac4207d5d8cb89df5722cb3d712385e3f88ac00000000')
+        self.assertEqual('60ebb1dd0b598e20dd0dd462ef6723dd49f8f803b6a2492926012360119cfdd7', tx.txid())
+
+    def test_txid_bitcoin_core_0082(self):
+        tx = transaction.Transaction('0100000000010200010000000000000000000000000000000000000000000000000000000000000000000000ffffffff00010000000000000000000000000000000000000000000000000000000000000100000000ffffffff02e8030000000000000151e90300000000000001510247304402206d59682663faab5e4cb733c562e22cdae59294895929ec38d7c016621ff90da0022063ef0af5f970afe8a45ea836e3509b8847ed39463253106ac17d19c437d3d56b832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710248304502210085001a820bfcbc9f9de0298af714493f8a37b3b354bfd21a7097c3e009f2018c022050a8b4dbc8155d4d04da2f5cdd575dcf8dd0108de8bec759bd897ea01ecb3af7832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7100000000')
+        self.assertEqual('ed0c7f4163e275f3f77064f471eac861d01fdf55d03aa6858ebd3781f70bf003', tx.txid())
+
+    def test_txid_bitcoin_core_0083(self):
+        tx = transaction.Transaction('0100000000010200010000000000000000000000000000000000000000000000000000000000000100000000ffffffff00010000000000000000000000000000000000000000000000000000000000000000000000ffffffff02e9030000000000000151e80300000000000001510248304502210085001a820bfcbc9f9de0298af714493f8a37b3b354bfd21a7097c3e009f2018c022050a8b4dbc8155d4d04da2f5cdd575dcf8dd0108de8bec759bd897ea01ecb3af7832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc710247304402206d59682663faab5e4cb733c562e22cdae59294895929ec38d7c016621ff90da0022063ef0af5f970afe8a45ea836e3509b8847ed39463253106ac17d19c437d3d56b832103596d3451025c19dbbdeb932d6bf8bfb4ad499b95b6f88db8899efac102e5fc7100000000')
+        self.assertEqual('f531ddf5ce141e1c8a7fdfc85cc634e5ff686f446a5cf7483e9dbe076b844862', tx.txid())
+
+    def test_txid_bitcoin_core_0084(self):
+        tx = transaction.Transaction('01000000020001000000000000000000000000000000000000000000000000000000000000000000004847304402202a0b4b1294d70540235ae033d78e64b4897ec859c7b6f1b2b1d8a02e1d46006702201445e756d2254b0f1dfda9ab8e1e1bc26df9668077403204f32d16a49a36eb6983ffffffff00010000000000000000000000000000000000000000000000000000000000000100000049483045022100acb96cfdbda6dc94b489fd06f2d720983b5f350e31ba906cdbd800773e80b21c02200d74ea5bdf114212b4bbe9ed82c36d2e369e302dff57cb60d01c428f0bd3daab83ffffffff02e8030000000000000151e903000000000000015100000000')
+        self.assertEqual('98229b70948f1c17851a541f1fe532bf02c408267fecf6d7e174c359ae870654', tx.txid())
+
+    def test_txid_bitcoin_core_0085(self):
+        tx = transaction.Transaction('01000000000102fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e000000004847304402200af4e47c9b9629dbecc21f73af989bdaa911f7e6f6c2e9394588a3aa68f81e9902204f3fcf6ade7e5abb1295b6774c8e0abd94ae62217367096bc02ee5e435b67da201ffffffff0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f80000000000ffffffff0100f2052a010000001976a914a30741f8145e5acadf23f751864167f32e0963f788ac000347304402200de66acf4527789bfda55fc5459e214fa6083f936b430a762c629656216805ac0220396f550692cd347171cbc1ef1f51e15282e837bb2b30860dc77c8f78bc8501e503473044022027dc95ad6b740fe5129e7e62a75dd00f291a2aeb1200b84b09d9e3789406b6c002201a9ecd315dd6a0e632ab20bbb98948bc0c6fb204f2c286963bb48517a7058e27034721026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac00000000')
+        self.assertEqual('570e3730deeea7bd8bc92c836ccdeb4dd4556f2c33f2a1f7b889a4cb4e48d3ab', tx.txid())
+
+    def test_txid_bitcoin_core_0086(self):
+        tx = transaction.Transaction('01000000000102e9b542c5176808107ff1df906f46bb1f2583b16112b95ee5380665ba7fcfc0010000000000ffffffff80e68831516392fcd100d186b3c2c7b95c80b53c77e77c35ba03a66b429a2a1b0000000000ffffffff0280969800000000001976a914de4b231626ef508c9a74a8517e6783c0546d6b2888ac80969800000000001976a9146648a8cd4531e1ec47f35916de8e259237294d1e88ac02483045022100f6a10b8604e6dc910194b79ccfc93e1bc0ec7c03453caaa8987f7d6c3413566002206216229ede9b4d6ec2d325be245c5b508ff0339bf1794078e20bfe0babc7ffe683270063ab68210392972e2eb617b2388771abe27235fd5ac44af8e61693261550447a4c3e39da98ac024730440220032521802a76ad7bf74d0e2c218b72cf0cbc867066e2e53db905ba37f130397e02207709e2188ed7f08f4c952d9d13986da504502b8c3be59617e043552f506c46ff83275163ab68210392972e2eb617b2388771abe27235fd5ac44af8e61693261550447a4c3e39da98ac00000000')
+        self.assertEqual('e0b8142f587aaa322ca32abce469e90eda187f3851043cc4f2a0fff8c13fc84e', tx.txid())
+
+    def test_txid_bitcoin_core_0087(self):
+        tx = transaction.Transaction('0100000000010280e68831516392fcd100d186b3c2c7b95c80b53c77e77c35ba03a66b429a2a1b0000000000ffffffffe9b542c5176808107ff1df906f46bb1f2583b16112b95ee5380665ba7fcfc0010000000000ffffffff0280969800000000001976a9146648a8cd4531e1ec47f35916de8e259237294d1e88ac80969800000000001976a914de4b231626ef508c9a74a8517e6783c0546d6b2888ac024730440220032521802a76ad7bf74d0e2c218b72cf0cbc867066e2e53db905ba37f130397e02207709e2188ed7f08f4c952d9d13986da504502b8c3be59617e043552f506c46ff83275163ab68210392972e2eb617b2388771abe27235fd5ac44af8e61693261550447a4c3e39da98ac02483045022100f6a10b8604e6dc910194b79ccfc93e1bc0ec7c03453caaa8987f7d6c3413566002206216229ede9b4d6ec2d325be245c5b508ff0339bf1794078e20bfe0babc7ffe683270063ab68210392972e2eb617b2388771abe27235fd5ac44af8e61693261550447a4c3e39da98ac00000000')
+        self.assertEqual('b9ecf72df06b8f98f8b63748d1aded5ffc1a1186f8a302e63cf94f6250e29f4d', tx.txid())
+
+    def test_txid_bitcoin_core_0088(self):
+        tx = transaction.Transaction('0100000000010136641869ca081e70f394c6948e8af409e18b619df2ed74aa106c1ca29787b96e0100000023220020a16b5755f7f6f96dbd65f5f0d6ab9418b89af4b1f14a1bb8a09062c35f0dcb54ffffffff0200e9a435000000001976a914389ffce9cd9ae88dcc0631e88a821ffdbe9bfe2688acc0832f05000000001976a9147480a33f950689af511e6e84c138dbbd3c3ee41588ac080047304402206ac44d672dac41f9b00e28f4df20c52eeb087207e8d758d76d92c6fab3b73e2b0220367750dbbe19290069cba53d096f44530e4f98acaa594810388cf7409a1870ce01473044022068c7946a43232757cbdf9176f009a928e1cd9a1a8c212f15c1e11ac9f2925d9002205b75f937ff2f9f3c1246e547e54f62e027f64eefa2695578cc6432cdabce271502473044022059ebf56d98010a932cf8ecfec54c48e6139ed6adb0728c09cbe1e4fa0915302e022007cd986c8fa870ff5d2b3a89139c9fe7e499259875357e20fcbb15571c76795403483045022100fbefd94bd0a488d50b79102b5dad4ab6ced30c4069f1eaa69a4b5a763414067e02203156c6a5c9cf88f91265f5a942e96213afae16d83321c8b31bb342142a14d16381483045022100a5263ea0553ba89221984bd7f0b13613db16e7a70c549a86de0cc0444141a407022005c360ef0ae5a5d4f9f2f87a56c1546cc8268cab08c73501d6b3be2e1e1a8a08824730440220525406a1482936d5a21888260dc165497a90a15669636d8edca6b9fe490d309c022032af0c646a34a44d1f4576bf6a4a74b67940f8faa84c7df9abe12a01a11e2b4783cf56210307b8ae49ac90a048e9b53357a2354b3334e9c8bee813ecb98e99a7e07e8c3ba32103b28f0c28bfab54554ae8c658ac5c3e0ce6e79ad336331f78c428dd43eea8449b21034b8113d703413d57761b8b9781957b8c0ac1dfe69f492580ca4195f50376ba4a21033400f6afecb833092a9a21cfdf1ed1376e58c5d1f47de74683123987e967a8f42103a6d48b1131e94ba04d9737d61acdaa1322008af9602b3b14862c07a1789aac162102d8b661b0b3302ee2f162b09e07a55ad5dfbe673a9f01d9f0c19617681024306b56ae00000000')
+        self.assertEqual('27eae69aff1dd4388c0fa05cbbfe9a3983d1b0b5811ebcd4199b86f299370aac', tx.txid())
+
+    def test_txid_bitcoin_core_0089(self):
+        tx = transaction.Transaction('010000000169c12106097dc2e0526493ef67f21269fe888ef05c7a3a5dacab38e1ac8387f1581b0000b64830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0121037a3fb04bcdb09eba90f69961ba1692a3528e45e67c85b200df820212d7594d334aad4830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e01ffffffff0101000000000000000000000000')
+        self.assertEqual('22d020638e3b7e1f2f9a63124ac76f5e333c74387862e3675f64b25e960d3641', tx.txid())
+
+    def test_txid_bitcoin_core_0090(self):
+        tx = transaction.Transaction('0100000000010169c12106097dc2e0526493ef67f21269fe888ef05c7a3a5dacab38e1ac8387f14c1d000000ffffffff01010000000000000000034830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e012102a9781d66b61fb5a7ef00ac5ad5bc6ffc78be7b44a566e3c87870e1079368df4c4aad4830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0100000000')
+        self.assertEqual('2862bc0c69d2af55da7284d1b16a7cddc03971b77e5a97939cca7631add83bf5', tx.txid())
+
+    def test_txid_bitcoin_core_0091(self):
+        tx = transaction.Transaction('01000000019275cb8d4a485ce95741c013f7c0d28722160008021bb469a11982d47a662896581b0000fd6f01004830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0148304502205286f726690b2e9b0207f0345711e63fa7012045b9eb0f19c2458ce1db90cf43022100e89f17f86abc5b149eba4115d4f128bcf45d77fb3ecdd34f594091340c03959601522102cd74a2809ffeeed0092bc124fd79836706e41f048db3f6ae9df8708cefb83a1c2102e615999372426e46fd107b76eaf007156a507584aa2cc21de9eee3bdbd26d36c4c9552af4830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0148304502205286f726690b2e9b0207f0345711e63fa7012045b9eb0f19c2458ce1db90cf43022100e89f17f86abc5b149eba4115d4f128bcf45d77fb3ecdd34f594091340c0395960175ffffffff0101000000000000000000000000')
+        self.assertEqual('1aebf0c98f01381765a8c33d688f8903e4d01120589ac92b78f1185dc1f4119c', tx.txid())
+
+    def test_txid_bitcoin_core_0092(self):
+        tx = transaction.Transaction('010000000001019275cb8d4a485ce95741c013f7c0d28722160008021bb469a11982d47a6628964c1d000000ffffffff0101000000000000000007004830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0148304502205286f726690b2e9b0207f0345711e63fa7012045b9eb0f19c2458ce1db90cf43022100e89f17f86abc5b149eba4115d4f128bcf45d77fb3ecdd34f594091340c0395960101022102966f109c54e85d3aee8321301136cedeb9fc710fdef58a9de8a73942f8e567c021034ffc99dd9a79dd3cb31e2ab3e0b09e0e67db41ac068c625cd1f491576016c84e9552af4830450220487fb382c4974de3f7d834c1b617fe15860828c7f96454490edd6d891556dcc9022100baf95feb48f845d5bfc9882eb6aeefa1bc3790e39f59eaa46ff7f15ae626c53e0148304502205286f726690b2e9b0207f0345711e63fa7012045b9eb0f19c2458ce1db90cf43022100e89f17f86abc5b149eba4115d4f128bcf45d77fb3ecdd34f594091340c039596017500000000')
+        self.assertEqual('45d17fb7db86162b2b6ca29fa4e163acf0ef0b54110e49b819bda1f948d423a3', tx.txid())
+
+# txns from Bitcoin Core ends <---
+
 
 class NetworkMock(object):
 

lib/tests/test_wallet_vertical.py

@@ -128,7 +128,7 @@ class TestWalletKeystoreAddressIntegrity(unittest.TestCase):
         long_user_id, short_id = trustedcoin.get_user_id(
             {'x1/': {'xpub': xpub1},
              'x2/': {'xpub': xpub2}})
-        xpub3 = trustedcoin.make_xpub(trustedcoin.signing_xpub, long_user_id)
+        xpub3 = trustedcoin.make_xpub(trustedcoin.get_signing_xpub(), long_user_id)
         ks3 = keystore.from_xpub(xpub3)
         self._check_xpub_keystore_sanity(ks3)
         self.assertTrue(isinstance(ks3, keystore.BIP32_KeyStore))

lib/transaction.py

@@ -32,6 +32,8 @@ from .util import print_error, profiler
 from . import bitcoin
 from .bitcoin import *
 import struct
+import traceback
+import sys
 
 #
 # Workalike python implementation of Bitcoin's CDataStream class.
@@ -45,6 +47,14 @@ class SerializationError(Exception):
     """ Thrown when there's a problem deserializing or serializing """
 
 
+class UnknownTxinType(Exception):
+    pass
+
+
+class NotRecognizedRedeemScript(Exception):
+    pass
+
+
 class BCDataStream(object):
     def __init__(self):
         self.input = None
@@ -295,17 +305,30 @@ def parse_scriptSig(d, _bytes):
         decoded = [ x for x in script_GetOp(_bytes) ]
     except Exception as e:
         # coinbase transactions raise an exception
-        print_error("cannot find address in input script", bh2u(_bytes))
+        print_error("parse_scriptSig: cannot find address in input script (coinbase?)",
+                    bh2u(_bytes))
         return
 
     match = [ opcodes.OP_PUSHDATA4 ]
     if match_decoded(decoded, match):
         item = decoded[0][1]
         if item[0] == 0:
+            # segwit embedded into p2sh
+            # witness version 0
             d['address'] = bitcoin.hash160_to_p2sh(bitcoin.hash_160(item))
-            d['type'] = 'p2wpkh-p2sh' if len(item) == 22 else 'p2wsh-p2sh'
+            if len(item) == 22:
+                d['type'] = 'p2wpkh-p2sh'
+            elif len(item) == 34:
+                d['type'] = 'p2wsh-p2sh'
+            else:
+                print_error("unrecognized txin type", bh2u(item))
+        elif opcodes.OP_1 <= item[0] <= opcodes.OP_16:
+            # segwit embedded into p2sh
+            # witness version 1-16
+            pass
         else:
-            # payto_pubkey
+            # assert item[0] == 0x30
+            # pay-to-pubkey
             d['type'] = 'p2pk'
             d['address'] = "(pubkey)"
             d['signatures'] = [bh2u(item)]
@@ -314,9 +337,9 @@ def parse_scriptSig(d, _bytes):
             d['pubkeys'] = ["(pubkey)"]
         return
 
-    # non-generated TxIn transactions push a signature
-    # (seventy-something bytes) and then their public key
-    # (65 bytes) onto the stack:
+    # p2pkh TxIn transactions push a signature
+    # (71-73 bytes) and then their public key
+    # (33 or 65 bytes) onto the stack:
     match = [ opcodes.OP_PUSHDATA4, opcodes.OP_PUSHDATA4 ]
     if match_decoded(decoded, match):
         sig = bh2u(decoded[0][1])
@@ -325,7 +348,8 @@ def parse_scriptSig(d, _bytes):
             signatures = parse_sig([sig])
             pubkey, address = xpubkey_to_address(x_pubkey)
         except:
-            print_error("cannot find address in input script", bh2u(_bytes))
+            print_error("parse_scriptSig: cannot find address in input script (p2pkh?)",
+                        bh2u(_bytes))
             return
         d['type'] = 'p2pkh'
         d['signatures'] = signatures
@@ -337,31 +361,42 @@ def parse_scriptSig(d, _bytes):
 
     # p2sh transaction, m of n
     match = [ opcodes.OP_0 ] + [ opcodes.OP_PUSHDATA4 ] * (len(decoded) - 1)
-    if not match_decoded(decoded, match):
-        print_error("cannot find address in input script", bh2u(_bytes))
+    if match_decoded(decoded, match):
+        x_sig = [bh2u(x[1]) for x in decoded[1:-1]]
+        try:
+            m, n, x_pubkeys, pubkeys, redeemScript = parse_redeemScript(decoded[-1][1])
+        except NotRecognizedRedeemScript:
+            print_error("parse_scriptSig: cannot find address in input script (p2sh?)",
+                        bh2u(_bytes))
+            # we could still guess:
+            # d['address'] = hash160_to_p2sh(hash_160(decoded[-1][1]))
+            return
+        # write result in d
+        d['type'] = 'p2sh'
+        d['num_sig'] = m
+        d['signatures'] = parse_sig(x_sig)
+        d['x_pubkeys'] = x_pubkeys
+        d['pubkeys'] = pubkeys
+        d['redeemScript'] = redeemScript
+        d['address'] = hash160_to_p2sh(hash_160(bfh(redeemScript)))
         return
-    x_sig = [bh2u(x[1]) for x in decoded[1:-1]]
-    m, n, x_pubkeys, pubkeys, redeemScript = parse_redeemScript(decoded[-1][1])
-    # write result in d
-    d['type'] = 'p2sh'
-    d['num_sig'] = m
-    d['signatures'] = parse_sig(x_sig)
-    d['x_pubkeys'] = x_pubkeys
-    d['pubkeys'] = pubkeys
-    d['redeemScript'] = redeemScript
-    d['address'] = hash160_to_p2sh(hash_160(bfh(redeemScript)))
+
+    print_error("parse_scriptSig: cannot find address in input script (unknown)",
+                bh2u(_bytes))
 
 
 def parse_redeemScript(s):
     dec2 = [ x for x in script_GetOp(s) ]
-    m = dec2[0][0] - opcodes.OP_1 + 1
-    n = dec2[-2][0] - opcodes.OP_1 + 1
+    try:
+        m = dec2[0][0] - opcodes.OP_1 + 1
+        n = dec2[-2][0] - opcodes.OP_1 + 1
+    except IndexError:
+        raise NotRecognizedRedeemScript()
     op_m = opcodes.OP_1 + m - 1
     op_n = opcodes.OP_1 + n - 1
     match_multisig = [ op_m ] + [opcodes.OP_PUSHDATA4]*n + [ op_n, opcodes.OP_CHECKMULTISIG ]
     if not match_decoded(dec2, match_multisig):
-        print_error("cannot find address in input script", bh2u(s))
-        return
+        raise NotRecognizedRedeemScript()
     x_pubkeys = [bh2u(x[1]) for x in dec2[1:-2]]
     pubkeys = [safe_parse_pubkey(x) for x in x_pubkeys]
     redeemScript = multisig_script(pubkeys, m)
@@ -416,7 +451,11 @@ def parse_input(vds):
         d['num_sig'] = 0
         if scriptSig:
             d['scriptSig'] = bh2u(scriptSig)
-            parse_scriptSig(d, scriptSig)
+            try:
+                parse_scriptSig(d, scriptSig)
+            except BaseException:
+                traceback.print_exc(file=sys.stderr)
+                print_error('failed to parse scriptSig', bh2u(scriptSig))
         else:
             d['scriptSig'] = ''
 
@@ -430,21 +469,55 @@ def parse_witness(vds, txin):
     if n == 0xffffffff:
         txin['value'] = vds.read_uint64()
         n = vds.read_compact_size()
+    # now 'n' is the number of items in the witness
     w = list(bh2u(vds.read_bytes(vds.read_compact_size())) for i in range(n))
-    if txin['type'] == 'coinbase':
-        pass
-    elif n > 2:
-        txin['signatures'] = parse_sig(w[1:-1])
-        m, n, x_pubkeys, pubkeys, witnessScript = parse_redeemScript(bfh(w[-1]))
-        txin['num_sig'] = m
-        txin['x_pubkeys'] = x_pubkeys
-        txin['pubkeys'] = pubkeys
-        txin['witnessScript'] = witnessScript
-    else:
-        txin['num_sig'] = 1
-        txin['x_pubkeys'] = [w[1]]
-        txin['pubkeys'] = [safe_parse_pubkey(w[1])]
-        txin['signatures'] = parse_sig([w[0]])
+
+    add_w = lambda x: var_int(len(x) // 2) + x
+    txin['witness'] = var_int(n) + ''.join(add_w(i) for i in w)
+
+    # FIXME: witness version > 0 will probably fail here.
+    # For native segwit, we would need the scriptPubKey of the parent txn
+    # to determine witness program version, and properly parse the witness.
+    # In case of p2sh-segwit, we can tell based on the scriptSig in this txn.
+    # The code below assumes witness version 0.
+    # p2sh-segwit should work in that case; for native segwit we need to tell
+    # between p2wpkh and p2wsh; we do this based on number of witness items,
+    # hence (FIXME) p2wsh with n==2 (maybe n==1 ?) will probably fail.
+    # If v==0 and n==2, we need parent scriptPubKey to distinguish between p2wpkh and p2wsh.
+    try:
+        if txin['type'] == 'coinbase':
+            pass
+        elif txin['type'] == 'p2wsh-p2sh' or n > 2:
+            try:
+                m, n, x_pubkeys, pubkeys, witnessScript = parse_redeemScript(bfh(w[-1]))
+            except NotRecognizedRedeemScript:
+                raise UnknownTxinType()
+            txin['signatures'] = parse_sig(w[1:-1])
+            txin['num_sig'] = m
+            txin['x_pubkeys'] = x_pubkeys
+            txin['pubkeys'] = pubkeys
+            txin['witnessScript'] = witnessScript
+            if not txin.get('scriptSig'):  # native segwit script
+                txin['type'] = 'p2wsh'
+                txin['address'] = bitcoin.script_to_p2wsh(txin['witnessScript'])
+        elif txin['type'] == 'p2wpkh-p2sh' or n == 2:
+            txin['num_sig'] = 1
+            txin['x_pubkeys'] = [w[1]]
+            txin['pubkeys'] = [safe_parse_pubkey(w[1])]
+            txin['signatures'] = parse_sig([w[0]])
+            if not txin.get('scriptSig'):  # native segwit script
+                txin['type'] = 'p2wpkh'
+                txin['address'] = bitcoin.public_key_to_p2wpkh(bfh(txin['pubkeys'][0]))
+        else:
+            raise UnknownTxinType()
+    except UnknownTxinType:
+        txin['type'] = 'unknown'
+        # FIXME: GUI might show 'unknown' address (e.g. for a non-multisig p2wsh)
+    except BaseException:
+        txin['type'] = 'unknown'
+        traceback.print_exc(file=sys.stderr)
+        print_error('failed to parse witness', txin.get('witness'))
+
 
 def parse_output(vds, i):
     d = {}
@@ -463,9 +536,18 @@ def deserialize(raw):
     start = vds.read_cursor
     d['version'] = vds.read_int32()
     n_vin = vds.read_compact_size()
+    is_segwit = (n_vin == 0)
+    if is_segwit:
+        marker = vds.read_bytes(1)
+        assert marker == b'\x01'
+        n_vin = vds.read_compact_size()
     d['inputs'] = [parse_input(vds) for i in range(n_vin)]
     n_vout = vds.read_compact_size()
     d['outputs'] = [parse_output(vds, i) for i in range(n_vout)]
+    if is_segwit:
+        for i in range(n_vin):
+            txin = d['inputs'][i]
+            parse_witness(vds, txin)
     d['lockTime'] = vds.read_uint32()
     return d
 
@@ -657,7 +739,9 @@ class Transaction:
             witness_script = multisig_script(pubkeys, txin['num_sig'])
             witness = var_int(n) + '00' + ''.join(add_w(x) for x in sig_list) + add_w(witness_script)
         else:
-            raise BaseException('wrong txin type')
+            witness = txin.get('witness', None)
+            if not witness:
+                raise BaseException('wrong txin type:', txin['type'])
         if self.is_txin_complete(txin) or estimate_size:
             value_field = ''
         else:
@@ -666,7 +750,8 @@ class Transaction:
 
     @classmethod
     def is_segwit_input(cls, txin):
-        return cls.is_segwit_inputtype(txin['type'])
+        has_nonzero_witness = txin.get('witness', '00') != '00'
+        return cls.is_segwit_inputtype(txin['type']) or has_nonzero_witness
 
     @classmethod
     def is_segwit_inputtype(cls, txin_type):
@@ -733,6 +818,14 @@ class Transaction:
     def serialize_outpoint(self, txin):
         return bh2u(bfh(txin['prevout_hash'])[::-1]) + int_to_hex(txin['prevout_n'], 4)
 
+    @classmethod
+    def get_outpoint_from_txin(cls, txin):
+        if txin['type'] == 'coinbase':
+            return None
+        prevout_hash = txin['prevout_hash']
+        prevout_n = txin['prevout_n']
+        return prevout_hash + ':%d' % prevout_n
+
     @classmethod
     def serialize_input(self, txin, script):
         # Prev hash and index

lib/util.py

@@ -29,7 +29,6 @@ import traceback
 import urllib
 import threading
 import hmac
-import requests
 
 from .i18n import _
 
@@ -41,11 +40,7 @@ def inv_dict(d):
     return {v: k for k, v in d.items()}
 
 
-is_bundle = getattr(sys, 'frozen', False)
-is_macOS = sys.platform == 'darwin'
-
 base_units = {'ZCL':8, 'mZCL':5, 'uZCL':2}
-fee_levels = [_('Within 25 blocks'), _('Within 10 blocks'), _('Within 5 blocks'), _('Within 2 blocks'), _('In the next block')]
 
 def normalize_version(v):
     return [int(x) for x in re.sub(r'(\.0+)*$','', v).split(".")]
@@ -62,17 +57,80 @@ class InvalidPassword(Exception):
     def __str__(self):
         return _("Incorrect password")
 
+
+class FileImportFailed(Exception):
+    def __init__(self, message=''):
+        self.message = str(message)
+
+    def __str__(self):
+        return _("Failed to import from file.") + "\n" + self.message
+
+
+class FileExportFailed(Exception):
+    def __init__(self, message=''):
+        self.message = str(message)
+
+    def __str__(self):
+        return _("Failed to export to file.") + "\n" + self.message
+
+
+class TimeoutException(Exception):
+    def __init__(self, message=''):
+        self.message = str(message)
+
+    def __str__(self):
+        if not self.message:
+            return _("Operation timed out.")
+        return self.message
+
+
 # Throw this exception to unwind the stack like when an error occurs.
 # However unlike other exceptions the user won't be informed.
 class UserCancelled(Exception):
     '''An exception that is suppressed from the user'''
     pass
 
+class Satoshis(object):
+    def __new__(cls, value):
+        self = super(Satoshis, cls).__new__(cls)
+        self.value = value
+        return self
+
+    def __repr__(self):
+        return 'Satoshis(%d)'%self.value
+
+    def __str__(self):
+        return format_satoshis(self.value) + " BTC"
+
+class Fiat(object):
+    def __new__(cls, value, ccy):
+        self = super(Fiat, cls).__new__(cls)
+        self.ccy = ccy
+        self.value = value
+        return self
+
+    def __repr__(self):
+        return 'Fiat(%s)'% self.__str__()
+
+    def __str__(self):
+        if self.value.is_nan():
+            return _('No Data')
+        else:
+            return "{:.2f}".format(self.value) + ' ' + self.ccy
+
 class MyEncoder(json.JSONEncoder):
     def default(self, obj):
         from .transaction import Transaction
         if isinstance(obj, Transaction):
             return obj.as_dict()
+        if isinstance(obj, Satoshis):
+            return str(obj)
+        if isinstance(obj, Fiat):
+            return str(obj)
+        if isinstance(obj, Decimal):
+            return str(obj)
+        if isinstance(obj, datetime):
+            return obj.isoformat(' ')[:-3]
         return super(MyEncoder, self).default(obj)
 
 class PrintError(object):
@@ -81,8 +139,12 @@ class PrintError(object):
         return self.__class__.__name__
 
     def print_error(self, *msg):
+        # only prints with --verbose flag
         print_error("[%s]" % self.diagnostic_name(), *msg)
 
+    def print_stderr(self, *msg):
+        print_stderr("[%s]" % self.diagnostic_name(), *msg)
+
     def print_msg(self, *msg):
         print_msg("[%s]" % self.diagnostic_name(), *msg)
 
@@ -347,7 +409,7 @@ def format_satoshis_plain(x, decimal_point = 8):
 def format_satoshis(x, is_diff=False, num_zeros = 0, decimal_point = 8, whitespaces=False):
     from locale import localeconv
     if x is None:
-        return 'Unknown'
+        return 'unknown'
     x = int(x)  # Some callers pass Decimal
     scale_factor = pow (10, decimal_point)
     integer_part = "{:n}".format(int(abs(x) / scale_factor))
@@ -367,10 +429,9 @@ def format_satoshis(x, is_diff=False, num_zeros = 0, decimal_point = 8, whitespa
     return result
 
 def timestamp_to_datetime(timestamp):
-    try:
-        return datetime.fromtimestamp(timestamp)
-    except:
+    if timestamp is None:
         return None
+    return datetime.fromtimestamp(timestamp)
 
 def format_time(timestamp):
     date = timestamp_to_datetime(timestamp)
@@ -453,8 +514,8 @@ testnet_block_explorers = {
 }
 
 def block_explorer_info():
-    from . import bitcoin
-    return testnet_block_explorers if bitcoin.NetworkConstants.TESTNET else mainnet_block_explorers
+    from . import constants
+    return testnet_block_explorers if constants.net.TESTNET else mainnet_block_explorers
 
 def block_explorer(config):
     return config.get('block_explorer', 'zclassic-ce.io')
@@ -470,7 +531,7 @@ def block_explorer_URL(config, kind, item):
     if not kind_str:
         return
     url_parts = [be_tuple[0], kind_str, item]
-    return "/".join(url_parts)
+    return ''.join(url_parts)
 
 # URL decode
 #_ud = re.compile('%([0-9a-hA-H]{2})', re.MULTILINE)
@@ -693,8 +754,56 @@ class QueuePipe:
             self.send(request)
 
 
-def get_cert_path():
-    if is_bundle and is_macOS:
-        # set in ./electrum
-        return requests.utils.DEFAULT_CA_BUNDLE_PATH
-    return requests.certs.where()
+
+
+def setup_thread_excepthook():
+    """
+    Workaround for `sys.excepthook` thread bug from:
+    http://bugs.python.org/issue1230540
+
+    Call once from the main thread before creating any threads.
+    """
+
+    init_original = threading.Thread.__init__
+
+    def init(self, *args, **kwargs):
+
+        init_original(self, *args, **kwargs)
+        run_original = self.run
+
+        def run_with_except_hook(*args2, **kwargs2):
+            try:
+                run_original(*args2, **kwargs2)
+            except Exception:
+                sys.excepthook(*sys.exc_info())
+
+        self.run = run_with_except_hook
+
+    threading.Thread.__init__ = init
+
+
+def versiontuple(v):
+    return tuple(map(int, (v.split("."))))
+
+
+def import_meta(path, validater, load_meta):
+    try:
+        with open(path, 'r') as f:
+            d = validater(json.loads(f.read()))
+        load_meta(d)
+    #backwards compatibility for JSONDecodeError
+    except ValueError:
+        traceback.print_exc(file=sys.stderr)
+        raise FileImportFailed(_("Invalid JSON code."))
+    except BaseException as e:
+        traceback.print_exc(file=sys.stdout)
+        raise FileImportFailed(e)
+
+
+def export_meta(meta, fileName):
+    try:
+        with open(fileName, 'w+') as f:
+            json.dump(meta, f, indent=4, sort_keys=True)
+    except (IOError, os.error) as e:
+        traceback.print_exc(file=sys.stderr)
+        raise FileExportFailed(e)

lib/verifier.py

@@ -36,15 +36,22 @@ class SPV(ThreadJob):
         self.merkle_roots = {}
 
     def run(self):
+        interface = self.network.interface
+        if not interface:
+            return
+        blockchain = interface.blockchain
+        if not blockchain:
+            return
         lh = self.network.get_local_height()
         unverified = self.wallet.get_unverified_txs()
         for tx_hash, tx_height in unverified.items():
             # do not request merkle branch before headers are available
             if (tx_height > 0) and (tx_height <= lh):
-                header = self.network.blockchain().read_header(tx_height)
-                if header is None and self.network.interface:
-                    index = tx_height // NetworkConstants.CHUNK_SIZE
-                    self.network.request_chunk(self.network.interface, index)
+                header = blockchain.read_header(tx_height)
+                if header is None:
+                    index = tx_height // constants.net.CHUNK_SIZE
+                    if index < len(blockchain.checkpoints):
+                        self.network.request_chunk(interface, index)
                 else:
                     if tx_hash not in self.merkle_roots:
                         request = ('blockchain.transaction.get_merkle',
@@ -58,6 +65,8 @@ class SPV(ThreadJob):
             self.undo_verifications()
 
     def verify_merkle(self, r):
+        if self.wallet.verifier is None:
+            return  # we have been killed, this was just an orphan callback
         if r.get('error'):
             self.print_error('received an error:', r)
             return
@@ -70,17 +79,26 @@ class SPV(ThreadJob):
         pos = merkle.get('pos')
         merkle_root = self.hash_merkle_root(merkle['merkle'], tx_hash, pos)
         header = self.network.blockchain().read_header(tx_height)
-        if not header or header.get('merkle_root') != merkle_root:
-            # FIXME: we should make a fresh connection to a server to
-            # recover from this, as this TX will now never verify
-            self.print_error("merkle verification failed for", tx_hash)
+        # FIXME: if verification fails below,
+        # we should make a fresh connection to a server to
+        # recover from this, as this TX will now never verify
+        if not header:
+            self.print_error(
+                "merkle verification failed for {} (missing header {})"
+                .format(tx_hash, tx_height))
+            return
+        if header.get('merkle_root') != merkle_root:
+            self.print_error(
+                "merkle verification failed for {} (merkle root mismatch {} != {})"
+                .format(tx_hash, header.get('merkle_root'), merkle_root))
             return
         # we passed all the tests
         self.merkle_roots[tx_hash] = merkle_root
         self.print_error("verified %s" % tx_hash)
         self.wallet.add_verified_tx(tx_hash, (tx_height, header.get('timestamp'), pos))
 
-    def hash_merkle_root(self, merkle_s, target_hash, pos):
+    @classmethod
+    def hash_merkle_root(cls, merkle_s, target_hash, pos):
         h = hash_decode(target_hash)
         for i in range(len(merkle_s)):
             item = merkle_s[i]

lib/version.py

@@ -1,7 +1,5 @@
- # version of the client package
-ELECTRUM_VERSION = '1.0.6'
-# protocol version requested
-PROTOCOL_VERSION = '1.1'
+ELECTRUM_VERSION = '1.0.7'   # '3.1' = version of the client package
+PROTOCOL_VERSION = '1.1'     # protocol version requested
 
 # The hash of the mnemonic seed must begin with this
 SEED_PREFIX      = '01'      # Standard wallet

lib/websockets.py

@@ -84,7 +84,8 @@ class WsClientThread(util.DaemonThread):
             l = self.subscriptions.get(addr, [])
             l.append((ws, amount))
             self.subscriptions[addr] = l
-            self.network.send([('blockchain.address.subscribe', [addr])], self.response_queue.put)
+            h = self.network.addr_to_scripthash(addr)
+            self.network.send([('blockchain.scripthash.subscribe', [h])], self.response_queue.put)
 
 
     def run(self):
@@ -100,10 +101,13 @@ class WsClientThread(util.DaemonThread):
             result = r.get('result')
             if result is None:
                 continue    
-            if method == 'blockchain.address.subscribe':
-                self.network.send([('blockchain.address.get_balance', params)], self.response_queue.put)
-            elif method == 'blockchain.address.get_balance':
-                addr = params[0]
+            if method == 'blockchain.scripthash.subscribe':
+                self.network.send([('blockchain.scripthash.get_balance', params)], self.response_queue.put)
+            elif method == 'blockchain.scripthash.get_balance':
+                h = params[0]
+                addr = self.network.h2addr.get(h, None)
+                if addr is None:
+                    util.print_error("can't find address for scripthash: %s" % h)
                 l = self.subscriptions.get(addr, [])
                 for ws, amount in l:
                     if not ws.closed:

lib/x509.py

@@ -23,7 +23,7 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 from . import util
-from .util import profiler, bh2u, get_cert_path
+from .util import profiler, bh2u
 import ecdsa
 import hashlib
 

plugins/cosigner_pool/qt.py

@@ -194,7 +194,7 @@ class Plugin(BasePlugin):
             return
 
         wallet = window.wallet
-        if wallet.has_password():
+        if wallet.has_keystore_encryption():
             password = window.password_dialog('An encrypted transaction was retrieved from cosigning pool.\nPlease enter your password to decrypt it.')
             if not password:
                 return