Adding additional checks during upload and download of chunk

[zarna1parekh]

Summary

Adding sanity checks before upload and post download of chunks.

1. Check lucene directory for corrupted data
2. check number of files and size of each file locally and on S3.

Requirements

• I've read and understood the Contributing Guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.

[baroquebobcat]

you could extract a helper method that would be used by both listFiles... methods that takes a prefix and a Consumer. Then this could look like the following.

Also, the block passed to subscribe could be called in multiple threads, so this should use a storage class that is safe wrt concurrent modifications.

Suggested change

	assert prefix != null && !prefix.isEmpty();
	ListObjectsV2Request listRequest = builder().bucket(bucketName).prefix(prefix).build();
	ListObjectsV2Publisher asyncPaginatedListResponse =
	s3AsyncClient.listObjectsV2Paginator(listRequest);
	Map<String, Long> filesListWithSize = new HashMap<>();
	try {
	asyncPaginatedListResponse
	.subscribe(
	listResponse ->
	listResponse
	.contents()
	.forEach(s3Object -> filesListWithSize.put(s3Object.key(), s3Object.size())))
	.get();
	} catch (InterruptedException | ExecutionException e) {
	throw new RuntimeException(e);
	}
	return filesListWithSize;
	Map<String, Long> filesWithSize = new ConcurrentHashMap<>();
	listFilesAndDo(prefix, s3Object -> filesListWithSize.put(s3Object.key(), s3Object.size()));
	return filesWithSize;

[baroquebobcat]

Hm. I think the replace isn't necessary since Files.list() only returns files in the current directory. Although, maybe you should use Path#getFileName().toString() here, which would align with calling Paths.get(s3Path).getFileName().toString() on the s3 entries below.

[baroquebobcat]

maybe use File.separator here instead of "/"?

[baroquebobcat]

It would be good to include the s3 file size here as well.

[baroquebobcat]

If you used a non-random chunkId and file names, you could have the assertion use more literals and it would be easier to follow.

Also, could you have one of the files have a different number of characters in it so it would be clear that they are different?