Skip to content
This repository was archived by the owner on Apr 17, 2024. It is now read-only.

Comments

[RPMP] Integration with RPMP#72

Open
xuechendi wants to merge 15 commits intoIntel-bigdata:wip_rpmpfrom
xuechendi:wip_spark_rpmp
Open

[RPMP] Integration with RPMP#72
xuechendi wants to merge 15 commits intoIntel-bigdata:wip_rpmpfrom
xuechendi:wip_spark_rpmp

Conversation

@xuechendi
Copy link
Collaborator

@xuechendi xuechendi commented May 19, 2020

This PR depends on HPNL branch https://github.com/Intel-bigdata/HPNL/tree/wip_rpmp

Changelog:

  1. Update RPMP codes to use shared_ptr instead of raw ptr more better memory management
  2. Add a new benchmark "remote_put" who will imitate what shuffle write and read did. do put/getMeta/read
  3. RPMP client now is using a queue to make requests from multiple threads become one.
  4. shuffle read used RPMP read api
  5. shuffle metadata update is still using original PmofTransferService to contact driver.
  6. External sort is not supported by RPMP yet, will use local filesystem to do sort.

xuechendi and others added 15 commits May 19, 2020 18:19
@xuechendi
[CPP] Fix some small bug for running Benchmarks
ab0f21f 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[CPP] Add a print log when handling requests
ff048ef 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
Integrate RPMP to spark-pmof shuffle manager
58c9af2 
Signed-off-by: Haodong Tang <haodong.tang@intel.com>
@xuechendi
[Scala] Rebase to mem fix
3905564 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP][Server] bypass real workload and codes refine and format
cef6869 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP][ClientJNI] codes refine
cc09d7e 
1. new threading model in NetworkClient
2. remove jni h file
3. add a jniUtil to load so from jar

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP][CPPTESTS] Update Benchmarks
4d1dcd3 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP][JAVA] Remove unnecessary files from repo and some API update
b46a418 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[PMOF][JAVA] Integration with RPMP
f06972b 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP] Fix shared_ptr release bug in client and clean shared_ptr in d…
d653533 
…estructor of client pmpool

shared_ptr issue seems to be caused since there is three threads in client, one jvm client, one client worker and one callback worker. So now we will std::move callback replied replyrequestContext to main worker. And shared_ptr release issue seems gone.

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[PMoF] Fix shuffleReader failed get MetadataResolver issue
ab3633c 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMP] Add new benchmark to immitate spark behavior with multiple pro…
73c7759 
…cesses support

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMem-Shuffle] Add new GET api and integrated to pmof
721bd5a 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMem-Shuffle] Small fixes
a4d06c4 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
@xuechendi
[RPMem-Shuffle] Fix client thread competition
50fa6a2 
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@xuechendi