Add S3 ItemReader/ItemWriter

[sabbyanandan]

[santfirax]

Hello @benas I would like to work on this. Any chance that you can provide some guidance ?

[fmbenhassine]

@sabbyanandan Thank you for opening this feature request. I understand the value of such a request especially in some cloud environments where a temporary file system is not present to download a file before reading it or write a file before uploading it to a cloud storage.

To read (stream) data directly from a remote URL (S3, GCP, etc), one can use a UrlResource. I provided an example in #912 (comment) and used the same approach in this blog post. So I think the feature on the reading side is already provided.

The writing side is a bit different though. The FlatFileItemWriter is designed to work against any resource that represents a writable file. This is required to obviously be able to write data to the file but also in a restart scenario to be able to truncate the file to the last committed byte offset and resume the job where it left off. So if we have a writable resource for S3 that represents a file, we should be done here as well. Spring Cloud AWS provides a SimpleStorageResource, but it does not seem to support providing a writable file, so I think it won't be usable with the FlatFileItemWriter as it is now. For the record, Spring Cloud GCP has the same issue for GoogleStorageResource. There is a PR against Spring Cloud GCP for that, and I think the same feature could be considered for Spring Cloud AWS as well. Another option is to use the FileUrlResource from Spring Framework with S3 or GCP, but based on its javadoc, it resolves the underlying file based on Resource#getFile which is not supported in Spring Cloud AWS and GCP currently. So before creating a new item writer implementation, I would like to make sure there is really no way to use some kind of Resource implementation that can be used with the current FlatFileItemWriter.

@santfirax Thank you for offering to help on this! I would wait for the feedback from the cloud team on your PR and see if it works. If it is the case, we can replicate the same idea in Spring Cloud AWS which would resolve the issue.

---

References:

• SO threads that might be relevant to the writer side: https://stackoverflow.com/a/54947585/5019386, https://stackoverflow.com/a/54947585/5019386.
• AWS Forum question about appending to an object: https://forums.aws.amazon.com/message.jspa?messageID=540395
• Example of using Azure blob storage with WritableResource: https://docs.microsoft.com/en-us/java/api/overview/azure/spring-boot-starter-storage-readme?view=azure-java-stable#read-and-write-to-a-resource

[fmbenhassine]

There is no plan to add S3 support in the main project. This can be contributed to https://github.com/spring-projects/spring-batch-extensions if anyone is interested.