1
0
Fork 0
mirror of https://github.com/chrislusf/seaweedfs synced 2025-08-16 09:02:47 +02:00
seaweedfs/docker/admin_integration/demo_vacuum_testing.sh
Chris Lu 891a2fb6eb
Admin: misc improvements on admin server and workers. EC now works. (#7055)
* initial design

* added simulation as tests

* reorganized the codebase to move the simulation framework and tests into their own dedicated package

* integration test. ec worker task

* remove "enhanced" reference

* start master, volume servers, filer

Current Status
 Master: Healthy and running (port 9333)
 Filer: Healthy and running (port 8888)
 Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready

* generate write load

* tasks are assigned

* admin start wtih grpc port. worker has its own working directory

* Update .gitignore

* working worker and admin. Task detection is not working yet.

* compiles, detection uses volumeSizeLimitMB from master

* compiles

* worker retries connecting to admin

* build and restart

* rendering pending tasks

* skip task ID column

* sticky worker id

* test canScheduleTaskNow

* worker reconnect to admin

* clean up logs

* worker register itself first

* worker can run ec work and report status

but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.

* move ec task logic

* listing ec shards

* local copy, ec. Need to distribute.

* ec is mostly working now

* distribution of ec shards needs improvement
* need configuration to enable ec

* show ec volumes

* interval field UI component

* rename

* integration test with vauuming

* garbage percentage threshold

* fix warning

* display ec shard sizes

* fix ec volumes list

* Update ui.go

* show default values

* ensure correct default value

* MaintenanceConfig use ConfigField

* use schema defined defaults

* config

* reduce duplication

* refactor to use BaseUIProvider

* each task register its schema

* checkECEncodingCandidate use ecDetector

* use vacuumDetector

* use volumeSizeLimitMB

* remove

remove

* remove unused

* refactor

* use new framework

* remove v2 reference

* refactor

* left menu can scroll now

* The maintenance manager was not being initialized when no data directory was configured for persistent storage.

* saving config

* Update task_config_schema_templ.go

* enable/disable tasks

* protobuf encoded task configurations

* fix system settings

* use ui component

* remove logs

* interface{} Reduction

* reduce interface{}

* reduce interface{}

* avoid from/to map

* reduce interface{}

* refactor

* keep it DRY

* added logging

* debug messages

* debug level

* debug

* show the log caller line

* use configured task policy

* log level

* handle admin heartbeat response

* Update worker.go

* fix EC rack and dc count

* Report task status to admin server

* fix task logging, simplify interface checking, use erasure_coding constants

* factor in empty volume server during task planning

* volume.list adds disk id

* track disk id also

* fix locking scheduled and manual scanning

* add active topology

* simplify task detector

* ec task completed, but shards are not showing up

* implement ec in ec_typed.go

* adjust log level

* dedup

* implementing ec copying shards and only ecx files

* use disk id when distributing ec shards

🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned

* Delete original volume from all locations

* clean up existing shard locations

* local encoding and distributing

* Update docker/admin_integration/EC-TESTING-README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* check volume id range

* simplify

* fix tests

* fix types

* clean up logs and tests

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-07-30 12:38:03 -07:00

105 lines
No EOL
3.2 KiB
Bash
Executable file

#!/bin/sh
echo "🧪 SeaweedFS Vacuum Task Testing Demo"
echo "======================================"
echo ""
# Check if SeaweedFS is running
echo "📋 Checking SeaweedFS status..."
MASTER_URL="${MASTER_HOST:-master:9333}"
ADMIN_URL="${ADMIN_HOST:-admin:23646}"
if ! curl -s http://$MASTER_URL/cluster/status > /dev/null; then
echo "❌ SeaweedFS master not running at $MASTER_URL"
echo " Please ensure Docker cluster is running: make start"
exit 1
fi
if ! curl -s http://volume1:8080/status > /dev/null; then
echo "❌ SeaweedFS volume servers not running"
echo " Please ensure Docker cluster is running: make start"
exit 1
fi
if ! curl -s http://$ADMIN_URL/ > /dev/null; then
echo "❌ SeaweedFS admin server not running at $ADMIN_URL"
echo " Please ensure Docker cluster is running: make start"
exit 1
fi
echo "✅ All SeaweedFS components are running"
echo ""
# Phase 1: Create test data
echo "📁 Phase 1: Creating test data with garbage..."
go run create_vacuum_test_data.go -master=$MASTER_URL -files=15 -delete=0.5 -size=150
echo ""
# Phase 2: Check initial status
echo "📊 Phase 2: Checking initial volume status..."
go run create_vacuum_test_data.go -master=$MASTER_URL -files=0
echo ""
# Phase 3: Configure vacuum
echo "⚙️ Phase 3: Vacuum configuration instructions..."
echo " 1. Visit: http://localhost:23646/maintenance/config/vacuum"
echo " 2. Set these values for testing:"
echo " - Enable Vacuum Tasks: ✅ Checked"
echo " - Garbage Threshold: 0.30"
echo " - Scan Interval: [30] [Seconds]"
echo " - Min Volume Age: [0] [Minutes]"
echo " - Max Concurrent: 2"
echo " 3. Click 'Save Configuration'"
echo ""
read -p " Press ENTER after configuring vacuum settings..."
echo ""
# Phase 4: Monitor tasks
echo "🎯 Phase 4: Monitoring vacuum tasks..."
echo " Visit: http://localhost:23646/maintenance"
echo " You should see vacuum tasks appear within 30 seconds"
echo ""
echo " Waiting 60 seconds for vacuum detection and execution..."
for i in {60..1}; do
printf "\r Countdown: %02d seconds" $i
sleep 1
done
echo ""
echo ""
# Phase 5: Check results
echo "📈 Phase 5: Checking results after vacuum..."
go run create_vacuum_test_data.go -master=$MASTER_URL -files=0
echo ""
# Phase 6: Create more garbage for continuous testing
echo "🔄 Phase 6: Creating additional garbage for continuous testing..."
echo " Running 3 rounds of garbage creation..."
for round in {1..3}; do
echo " Round $round: Creating garbage..."
go run create_vacuum_test_data.go -master=$MASTER_URL -files=8 -delete=0.6 -size=100
echo " Waiting 30 seconds before next round..."
sleep 30
done
echo ""
echo "📊 Final volume status:"
go run create_vacuum_test_data.go -master=$MASTER_URL -files=0
echo ""
echo "🎉 Demo Complete!"
echo ""
echo "🔍 Things to check:"
echo " 1. Maintenance Queue: http://localhost:23646/maintenance"
echo " 2. Volume Status: http://localhost:9333/vol/status"
echo " 3. Admin Dashboard: http://localhost:23646"
echo ""
echo "💡 Next Steps:"
echo " - Try different garbage thresholds (0.10, 0.50, 0.80)"
echo " - Adjust scan intervals (10s, 1m, 5m)"
echo " - Monitor logs for vacuum operations"
echo " - Test with multiple volumes"
echo ""