New spring-batch-s3 module #182

andreacioni · 2025-07-05T20:57:12Z

Hello everyone,

after struggling a bit with S3 and Spring Batch I want to contribute to this ecosystem with my little knowledge by proposing this PR to your attention. In the past I've followed some discussion that lead me here (mainly #3818). I had to work with S3 in the past years and I had to implement such Reader/Writer each time, moreover it's a new addition to AWS SDK the API to upload objects with unknown size (reference) from streams. Given this inputs I decided that was time for Spring Batch to have some support for S3, hope this will be welcome.

More information are in the README, please let me know if there is something missing I'd be happy to fix it.

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

dgray16

Thanks for your work!
Here are some places that could be improved from my point of view.

spring-batch-s3/.editorconfig

spring-batch-s3/.mvn/wrapper/maven-wrapper.properties

spring-batch-s3/README.adoc

...tch-s3/src/main/java/org/springframework/batch/extensions/s3/stream/S3MultipartUploader.java

spring-batch-s3/src/test/java/org/springframework/batch/extensions/s3/S3ItemReaderTests.java

...c/test/java/org/springframework/batch/extensions/s3/stream/S3MultipartOutputStreamTests.java

...3/src/test/java/org/springframework/batch/extensions/s3/stream/S3MultipartUploaderTests.java

...tch-s3/src/test/java/org/springframework/batch/extensions/s3/stream/S3OutputStreamTests.java

scordio

Hi @andreacioni, thanks for the PR!

I suggest adding a CI job, see for example the one for Spring Batch Notion.

Also, the new module should be mentioned in the root README.

spring-batch-s3/pom.xml

spring-batch-s3/README.adoc

spring-batch-s3/pom.xml

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

- Builders moved to a dedicated package - project README.adoc updated; - root README.md updated: add reference to new module; - use org.springframework.util.unit.DataSize to express data sizes; - removed apache-client exclusion in pom.xml to allow user to not specify an actual implementation in their dependencies - updated spring-javaformat-checkstyle to match the latest version - update gitignore to match the one of Spring Batch Notion - Add Maven Wrapper executables - Add Github Action workflow - Removed maven-wrapper.jar - Code cleanup Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

.github/workflows/spring-batch-s3.yml

README.md

- pom.xml reformatted - root README missing Amazon S3 link - atLeastOnce abandoned in favor of times(x) - minor corrections Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

dgray16

It would be nice to have a review from @fmbenhassine also.

spring-batch-s3/pom.xml

...ch-s3/src/main/java/org/springframework/batch/extensions/s3/builder/S3ItemWriterBuilder.java

spring-batch-s3/src/main/java/org/springframework/batch/extensions/s3/stream/Defaults.java

spring-batch-s3/src/test/java/org/springframework/batch/extensions/s3/S3ItemReaderTests.java

Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

…ions/s3/S3ItemReaderTests.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

…ions/s3/builder/S3ItemWriterBuilder.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

…ions/s3/stream/Defaults.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

aledomu · 2025-11-18T17:57:54Z

Any news on this? I don't think I'm alone in wanting this in.

fmbenhassine · 2025-11-21T06:53:43Z

Hi @andreacioni

Thank you for contributing this extension!

I did not follow the SDK updates from AWS (the last time I checked was when I wrote spring-projects/spring-batch#3818 (comment)), but I have no objection to add this S3 extension if there they now provide the necessary bits to properly create an item reader and writer for Spring Batch.

However, we need someone to lead and maintain this module after merging this PR. If you are interested, please reach out to me by email (my email is the one I use in git) to explain the responsibility of extension leads and the onboarding process to the spring-projects org. Once that in place, I can review this PR (I have some feedback on the current change set).

Best regards,
Mahmoud

scordio

Hi @andreacioni, I have a few follow-up comments, but feel free to ignore them until things are sorted out with @fmbenhassine.

scordio · 2025-11-21T10:06:38Z

.github/workflows/spring-batch-s3.yml

+        uses: actions/setup-java@v4
+        with:
+          distribution: 'temurin'
+          java-version: '21'


Suggested change

java-version: '21'

java-version: '25'

scordio · 2025-11-21T10:07:05Z

spring-batch-s3/pom.xml

+  <parent>
+    <groupId>org.springframework.boot</groupId>
+    <artifactId>spring-boot-starter-parent</artifactId>
+    <version>3.5.3</version>


Suggested change

<version>3.5.3</version>

<version>4.0.0</version>

scordio · 2025-11-21T10:09:27Z

spring-batch-s3/.mvn/wrapper/maven-wrapper.properties

+wrapperVersion=3.3.2
+distributionType=only-script
+distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.8/apache-maven-3.9.8-bin.zip


Time has passed and new versions are now available:

Suggested change

wrapperVersion=3.3.2

distributionType=only-script

distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.8/apache-maven-3.9.8-bin.zip

wrapperVersion=3.3.4

distributionType=only-script

distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.11/apache-maven-3.9.11-bin.zip

These two properties are not enough for a proper upgrade, and the following should be executed locally instead:

./mnvw org.apache.maven.plugins:maven-wrapper-plugin:3.3.4:wrapper -Dmaven=3.9.11

scordio · 2025-11-21T10:10:39Z

.github/workflows/spring-batch-s3.yml

+  pull_request:
+    paths:
+      - 'spring-batch-s3/**'
+  push:
+    paths:
+      - 'spring-batch-s3/**'


Suggested change

pull_request:

paths:

- 'spring-batch-s3/**'

push:

paths:

- 'spring-batch-s3/**'

pull_request:

paths:

- '.github/workflows/spring-batch-s3.yml'

- 'spring-batch-s3/**'

push:

paths:

- '.github/workflows/spring-batch-s3.yml'

- 'spring-batch-s3/**'

scordio · 2025-11-21T10:12:31Z

spring-batch-s3/src/main/java/org/springframework/batch/extensions/s3/package-info.java

+ * memory.
+ */
+
+package org.springframework.batch.extensions.s3;


With the migration to Batch 6 and Boot 4, I'd also consider the Jspecify integration.

For this, you might take inspiration from what I've done at a9648a7.

new spring-batch-s3 module

e3f3ce1

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

andreacioni force-pushed the spring-batch-s3 branch from b6e9643 to e3f3ce1 Compare July 5, 2025 20:58

dgray16 reviewed Jul 6, 2025

View reviewed changes

scordio reviewed Jul 6, 2025

View reviewed changes

spring-batch-s3/pom.xml Outdated Show resolved Hide resolved

spring-batch-s3/README.adoc Outdated Show resolved Hide resolved

spring-batch-s3/pom.xml Outdated Show resolved Hide resolved

spring-batch-s3/pom.xml Outdated Show resolved Hide resolved

andreacioni added 2 commits July 8, 2025 16:07

license header updated using maven plugin

3fe11bb

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

dgray16 reviewed Jul 8, 2025

View reviewed changes

.github/workflows/spring-batch-s3.yml Show resolved Hide resolved

dgray16 reviewed Jul 8, 2025

View reviewed changes

README.md Show resolved Hide resolved

andreacioni added 3 commits July 10, 2025 15:37

multiple updates after PR review:

765bbdf

- pom.xml reformatted - root README missing Amazon S3 link - atLeastOnce abandoned in favor of times(x) - minor corrections Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

add ability to change the underlying executor service in OutputStream's

75849c9

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

fix check format failed

2843d92

Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

dgray16 approved these changes Jul 10, 2025

View reviewed changes

andreacioni and others added 4 commits July 14, 2025 15:36

Update spring-batch-s3/pom.xml

5e3793a

Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

Update spring-batch-s3/src/test/java/org/springframework/batch/extens…

4981853

…ions/s3/S3ItemReaderTests.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

Update spring-batch-s3/src/main/java/org/springframework/batch/extens…

bf39daf

…ions/s3/builder/S3ItemWriterBuilder.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

Update spring-batch-s3/src/main/java/org/springframework/batch/extens…

ade7ad2

…ions/s3/stream/Defaults.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>

andreacioni requested a review from scordio July 14, 2025 14:14

scordio requested a review from fmbenhassine July 14, 2025 17:54

scordio approved these changes Jul 19, 2025

View reviewed changes

fmbenhassine added the status: waiting-for-feedback label Nov 21, 2025

scordio requested changes Nov 21, 2025

View reviewed changes

New spring-batch-s3 module #182

Are you sure you want to change the base?

New spring-batch-s3 module #182

Uh oh!

Conversation

andreacioni commented Jul 5, 2025

Uh oh!

dgray16 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scordio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgray16 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aledomu commented Nov 18, 2025

Uh oh!

fmbenhassine commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scordio left a comment

Choose a reason for hiding this comment

Uh oh!

scordio Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

scordio Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

scordio Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

scordio Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

scordio Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fmbenhassine commented Nov 21, 2025 •

edited

Loading