diff --git a/README.md b/README.md index 3b38eab0d0..16e92b6f6c 100644 --- a/README.md +++ b/README.md @@ -64,73 +64,6 @@ Start your own testnet locally, instructions are in the [online docs](https://do ### Accessing the remote testnet * `testnet` - public stable testnet accessible via devnet.solana.com. Runs 24/7 - -### Deploying -They are deployed with the `ci/testnet-manager.sh` script through a list of [scheduled -buildkite jobs](https://buildkite.com/solana-labs/testnet-management/settings/schedules). -Each testnet can be manually manipulated from buildkite as well. - -### How do I reset the testnet? -Manually trigger the [testnet-management](https://buildkite.com/solana-labs/testnet-management) pipeline -and when prompted select the desired testnet - -### Scaling the transaction generation rate - -Increase the TX rate by increasing the number of cores on the client machine which is running -`bench-tps` or run multiple clients. Decrease by lowering cores or using the rayon env -variable `RAYON_NUM_THREADS=` - -### Testing a change on the testnet - -Currently, a merged PR is the only way to test a change on the testnet. But you -can run your own testnet using the scripts in the `net/` directory. - -### Adjusting the number of clients or validators on the testnet -Edit `ci/testnet-manager.sh` - - -### Metrics Server Maintenance -Sometimes the dashboard becomes unresponsive. This happens due to glitch in the metrics server. -The current solution is to reset the metrics server. Use the following steps. - -1. The server is hosted in a GCP VM instance. Check if the VM instance is down by trying to SSH - into it from the GCP console. The name of the VM is ```metrics-solana-com```. -2. If the VM is inaccessible, reset it from the GCP console. -3. Once VM is up (or, was already up), the metrics services can be restarted from build automation. - 1. Navigate to https://buildkite.com/solana-labs/metrics-dot-solana-dot-com in your web browser - 2. Click on ```New Build``` - 3. This will show a pop up dialog. Click on ```options``` drop down. - 4. Type in ```FORCE_START=true``` in ```Environment Variables``` text box. - 5. Click ```Create Build``` - 6. This will restart the metrics services, and the dashboards should be accessible afterwards. - -# Debugging - -Testnet may exhibit different symptoms of failures. Primary statistics to check are -1. Rise in Confirmation Time -2. Nodes are not voting -3. Panics, and OOM notifications - -Check the following if there are any signs of failure. -1. Did testnet deployment fail? - 1. View buildkite logs for the last deployment: https://buildkite.com/solana-labs/testnet-management - 2. Use the relevant branch - 3. If the deployment failed, look at the build logs. The build artifacts for each remote node is uploaded. - It's a good first step to triage from these logs. -2. You may have to log into remote node if the deployment succeeded, but something failed during runtime. - 1. Get the private key for the testnet deployment from ```metrics-solana-com``` GCP instance. - 2. SSH into ```metrics-solana-com``` using GCP console and do the following. - ```bash - sudo bash - cd ~buildkite-agent/.ssh - ls - ``` - 3. Copy the relevant private key to your local machine - 4. Find the public IP address of the AWS instance for the remote node using AWS console - 5. ```ssh -i ubuntu@``` - 6. The logs are in ```~solana\solana``` folder - - # Benchmarking First install the nightly build of rustc. `cargo bench` requires use of the @@ -150,7 +83,6 @@ $ cargo +nightly bench The release process for this project is described [here](RELEASE.md). - # Code coverage To generate code coverage statistics: