I've been using services like Kafka Connect and Secor to persist Parquet files to S3. I'm not very familiar with HDFS or Hadoop but it seems like these services typically write temporary files either into local memory or to disk before writing in bulk to s3. Do the s3n/s3a file systems virtualize an HDFS-style file system locally and then push at configured intervals or is there a one-to-one correspondence between a write to s3n/s3a and a write to s3?
I'm not entirely sure if I'm asking the right question here. Any guidance would be appreciated.
S3A/S3N just implement the Hadoop FileSystem APIs against the remote object store, including pretending it has directories you can rename and delete.
They have historically saved all the data you write to the local disk until you close() the output stream, at which point the upload takes place (which can be slow). This means that you must have as much temporary space as the biggest object you plan to create.
Hadoop 2.8 has a fast upload stream which uploads the file in 5+MB blocks as it gets written, then in the final close() makes it visible in the object store. This is measurably faster when generating lots of data in a single stream. This also avoids needing so much disk space.