I set up a first Hive table with GZIP-compressed files:
CREATE EXTERNAL TABLE table_gzip (
col1,
col2,
col3
)
ROW FORMAT DELIMITED,
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
LOCATION
's3://bucket/files_gzip/';
Then I set up another Hive table with ORC format:
CREATE EXTERNAL TABLE table_orc (
col1,
col2,
col3
)
STORED AS ORC
LOCATION
's3://bucket/files_orc/';
ALTER TABLE table_orc SET tblproperties ("orc.compress" ="SNAPPY");
And then I uncompressed and recompressed from GZIP to ORC using this query:
INSERT OVERWRITE TABLE table_gzip SELECT * FROM table_orc
Once this query finished, I had new ORC-compressed files in 's3://bucket/files_orc/'. So far so good.
However when I looked at the files, they went from 500 1.2GiB files to 500 1.6GiB files.
What did I do wrong? Why are my ORC-SNAPPY compressed files larger than the original files? Is GZIP a better compression method?
Thanks for your time.