1
0
Fork 0
mirror of https://github.com/chrislusf/seaweedfs synced 2025-07-11 06:02:47 +02:00
seaweedfs/.github/workflows/s3-versioning-tests.yml
Chris Lu cf5a24983a
S3: add object versioning (#6945)
* add object versioning

* add missing file

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ListObjectVersionsResult is better to show multiple version entries

* fix test

* Update weed/s3api/s3api_object_handlers_put.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* multiple improvements

* move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file
* duplicated code for reading bucket config, versioningEnabled, etc. try to use functions
* opportunity to cache bucket config

* error handling if bucket is not found

* in case bucket is not found

* fix build

* add object versioning tests

* remove non-existent tests

* add tests

* add versioning tests

* skip a new test

* ensure .versions directory exists before saving info into it

* fix creating version entry

* logging on creating version directory

* Update s3api_object_versioning_test.go

* retry and wait for directory creation

* revert add more logging

* Update s3api_object_versioning.go

* more debug messages

* clean up logs, and touch directory correctly

* log the .versions creation and then parent directory listing

* use mkFile instead of touch

touch is for update

* clean up data

* add versioning test in go

* change location

* if modified, latest version is moved to .versions directory, and create a new latest version

 Core versioning functionality: WORKING
TestVersioningBasicWorkflow - PASS
TestVersioningDeleteMarkers - PASS
TestVersioningMultipleVersionsSameObject - PASS
TestVersioningDeleteAndRecreate - PASS
TestVersioningListWithPagination - PASS
 Some advanced features still failing:
ETag calculation issues (using mtime instead of proper MD5)
Specific version retrieval (EOF error)
Version deletion (internal errors)
Concurrent operations (race conditions)

* calculate multi chunk md5

Test Results - All Passing:
 TestBucketListReturnDataVersioning - PASS
 TestVersioningCreateObjectsInOrder - PASS
 TestVersioningBasicWorkflow - PASS
 TestVersioningMultipleVersionsSameObject - PASS
 TestVersioningDeleteMarkers - PASS

* dedupe

* fix TestVersioningErrorCases

* fix eof error of reading old versions

* get specific version also check current version

* enable integration tests for versioning

* trigger action to work for now

* Fix GitHub Actions S3 versioning tests workflow

- Fix syntax error (incorrect indentation)
- Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/
- Add push trigger for add-object-versioning branch to enable CI during development
- Update artifact paths to match correct directory structure

* Improve CI robustness for S3 versioning tests

Makefile improvements:
- Increase server startup timeout from 30s to 90s for CI environments
- Add progressive timeout reporting (logs at 30s, full logs at 90s)
- Better error handling with server logs on failure
- Add server PID tracking for debugging
- Improved test failure reporting

GitHub Actions workflow improvements:
- Increase job timeouts to account for CI environment delays
- Add system information logging (memory, disk space)
- Add detailed failure reporting with server logs
- Add process and network diagnostics on failure
- Better error messaging and log collection

These changes should resolve the 'Server failed to start within 30 seconds' issue
that was causing the CI tests to fail.

* adjust testing volume size

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update s3-versioning-tests.yml

* Update s3api_object_versioning.go

* Update Makefile

* do not clean up

* log received version id

* more logs

* printout response

* print out list version response

* use tmp files when put versioned object

* change to versions folder layout

* Delete weed-test.log

* test with mixed versioned and unversioned objects

* remove versionDirCache

* remove unused functions

* remove unused function

* remove fallback checking

* minor

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-09 01:51:45 -07:00

181 lines
No EOL
4.9 KiB
YAML

name: "S3 Versioning Tests (Go)"
on:
pull_request:
concurrency:
group: ${{ github.head_ref }}/s3-versioning
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
working-directory: weed
jobs:
s3-versioning-tests:
name: S3 Versioning Tests
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
test-type: ["quick", "comprehensive"]
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
id: go
- name: Install SeaweedFS
run: |
go install -buildvcs=false
- name: Run S3 Versioning Tests - ${{ matrix.test-type }}
timeout-minutes: 25
working-directory: test/s3/versioning
run: |
set -x
echo "=== System Information ==="
uname -a
free -h
df -h
echo "=== Starting Tests ==="
# Run tests with automatic server management
# The test-with-server target handles server startup/shutdown automatically
if [ "${{ matrix.test-type }}" = "quick" ]; then
# Override TEST_PATTERN for quick tests only
make test-with-server TEST_PATTERN="TestBucketListReturnDataVersioning|TestVersioningBasicWorkflow|TestVersioningDeleteMarkers"
else
# Run all versioning tests
make test-with-server
fi
- name: Show server logs on failure
if: failure()
working-directory: test/s3/versioning
run: |
echo "=== Server Logs ==="
if [ -f weed-test.log ]; then
echo "Last 100 lines of server logs:"
tail -100 weed-test.log
else
echo "No server log file found"
fi
echo "=== Test Environment ==="
ps aux | grep -E "(weed|test)" || true
netstat -tlnp | grep -E "(8333|9333|8080)" || true
- name: Upload test logs on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: s3-versioning-test-logs-${{ matrix.test-type }}
path: test/s3/versioning/weed-test*.log
retention-days: 3
s3-versioning-compatibility:
name: S3 Versioning Compatibility Test
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
id: go
- name: Install SeaweedFS
run: |
go install -buildvcs=false
- name: Run Core Versioning Test (Python s3tests equivalent)
timeout-minutes: 15
working-directory: test/s3/versioning
run: |
set -x
echo "=== System Information ==="
uname -a
free -h
# Run the specific test that is equivalent to the Python s3tests
make test-with-server || {
echo "❌ Test failed, checking logs..."
if [ -f weed-test.log ]; then
echo "=== Server logs ==="
tail -100 weed-test.log
fi
echo "=== Process information ==="
ps aux | grep -E "(weed|test)" || true
exit 1
}
- name: Upload server logs on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: s3-versioning-compatibility-logs
path: test/s3/versioning/weed-test*.log
retention-days: 3
s3-versioning-stress:
name: S3 Versioning Stress Test
runs-on: ubuntu-22.04
timeout-minutes: 35
# Only run stress tests on master branch pushes to avoid overloading PR testing
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
id: go
- name: Install SeaweedFS
run: |
go install -buildvcs=false
- name: Run S3 Versioning Stress Tests
timeout-minutes: 30
working-directory: test/s3/versioning
run: |
set -x
echo "=== System Information ==="
uname -a
free -h
# Run stress tests (concurrent operations)
make test-versioning-stress || {
echo "❌ Stress test failed, checking logs..."
if [ -f weed-test.log ]; then
echo "=== Server logs ==="
tail -200 weed-test.log
fi
make clean
exit 1
}
make clean
- name: Upload stress test logs
if: always()
uses: actions/upload-artifact@v4
with:
name: s3-versioning-stress-logs
path: test/s3/versioning/weed-test*.log
retention-days: 7