I have a single bucket which has very different Total Size and Number of Objects metrics between the S3 Management Console / CloudWatch, and the S3 command line tool.
S3 Management Console Metrics screen
S3 Management Console - Get Size Dialog
Cloudwatch:
Command Line Tool
aws s3 ls --summarize --human-readable --recursive s3://mybucket.com
The command line tool and S3 Management Console - Get Size Dialog is returning what I'd expect.
We are being billed based on the much higher CloudWatch stats.
I wouldn't be bothered if they are close, but Cloudwatch is reporting close to 15 x as much StorageSize than the CLI tool.
There is no replication on the S3 bucket.
There are lifecycle policies to:
Versioning is turned on but I am pretty sure it's not used; i.e. files have only ever been uploaded once. I have tested a few random files to confirm, and they all only have one, the latest, version, available. I am happy to turn this off at the bucket settings, but I understand this won't 'delete' old versions (if they exist) anyway?
I don't believe it's because of Glacier stored files either, as I only recently set this lifecycle policy up, and the size mismatch was present before that.
And I'm extra confused given the S3 Management Console UI > Overview > Select all folders > Actions > Get Size reports the same as the CLI tool, 264GB.
So, seleting all filers/folders in the UI returns 264GB, but the Metrics return 3.2TB.
Can anyone please shed some light on what's going on here, and how I can clean it up! Thanks Rob
This was caused by 3TB of old versioned files; but the whole directory had been deleted so wasn't showing in the UI.
When I swapped the UI to show versions, the deleted folder appeared and then within that I could see all the old versions of the deleted files.
The following CLI statement helped me confirm it was caused by versioning:
aws s3api list-object-versions --bucket mybucker.com --query 'Versions[*].[Size,Size]' --output text | awk '{s+=$1} END {printf "%.0f", s}'