parallel find using pcircle

[adilger]

The IO-500 project (https://github.com/VI4IO/io-500-dev) is currently using https://github.com/hpc/libcircle to implement a basic parallel find (pfind) utility. However, the libcircle library is dependent on OpenMPI and not portable to all environments (e.g. hpc/libcircle#27).

It would be useful for the IO-500, as well as (IMHO) the pcircle project to have a parallel find that at least implements the basic find(1) options. Since fprof has already got the traversal/stat operations implemented, the main logic needed would be filtering the returned files that do not match the specified find attributes before printing.

[fwang2]

As you have correctly identified, `pcircle` implemented similar algorithms of libcircle in Python, and shielded itself from any particular MPI implementation. More specifically, most of the `libcirlce` logic has been replaced by `circle.py`. Parallel find, will likely be built on top of circle.py itself, not fprof. There will be a little bit boilerplate code in both, but not much, I think.

…

On Tue, Jun 12, 2018 at 7:19 PM adilger ***@***.***> wrote: The IO-500 project (https://github.com/VI4IO/io-500-dev) is currently using https://github.com/hpc/libcircle to implement a basic parallel find (pfind) utility. However, the libcircle library is dependent on OpenMPI and not portable to all environments (e.g. hpc/libcircle#27 <hpc/libcircle#27>). It would be useful for the IO-500, as well as (IMHO) the pcircle project to have a parallel find that at least implements the basic find(1) options. Since fprof has already got the traversal/stat operations implemented, the main logic needed would be filtering the returned files that do not match the specified find attributes before printing. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#55>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF8uk41i7rzwZ_lvG6gtY4dFYo58jBks5t8EyGgaJpZM4UlTIO> .

[JulianKunkel]

We had tried to use the python version a year ago, it was difficult to get that to work on the different HPC systems. I don't remember all the details, though, but various Python issues. Hence we relied on the C-Library.

I have prepared a patch for configure to handle MPICH: JulianKunkel/libcircle@612ae55
At least on my system, that seems to detect MPICH correctly.

If it does for you, please apply the patch. Thanks.

[fwang2]

Julian - if you are willing to give pcircle another spin and let us know the problem you run into, I am here to help.

…

On Wed, Jun 13, 2018 at 7:39 AM Julian Kunkel ***@***.***> wrote: We had tried to use the python version a year ago, it was difficult to get that to work on the different HPC systems. I don't remember all the details, though, but various Python issues. Hence we relied on the C-Library. I have prepared a patch for configure to handle MPICH: ***@***.*** <JulianKunkel/libcircle@612ae55> At least on my system, that seems to detect MPICH correctly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF8uWoA5CazhAV2wnpMQ5vScj0NO77ks5t8PoKgaJpZM4UlTIO> .

[JulianKunkel]

Dear Feiyi, thank you! Note that my interest is also a version in which I run multiple benchmarks together in a single MPI application, I actually have a test version that does that. With running Python together with a C-Application, I would not be that happy I presume. If possible, I would stick to the C-Version for now. Regards, Julian 2018-06-13 14:50 GMT+01:00 Feiyi Wang <notifications@github.com>:

…

Julian - even if you are willing to give pcircle another spin and let us know the problem you run into, I am here to help. On Wed, Jun 13, 2018 at 7:39 AM Julian Kunkel ***@***.***> wrote: > We had tried to use the python version a year ago, it was difficult to get > that to work on the different HPC systems. I don't remember all the > details, though, but various Python issues. Hence we relied on the > C-Library. > > I have prepared a patch for configure to handle MPICH: > ***@***.*** > <https://github.com/JulianKunkel/libcircle/commit/ 612ae5537d0c284af7a5d349f3a2428a3f11a99b> > At least on my system, that seems to detect MPICH correctly. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#55 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ AAEF8uWoA5CazhAV2wnpMQ5vScj0NO77ks5t8PoKgaJpZM4UlTIO> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE1uyn9TK9mTPEQ5OQzTUcPxwlh0Umyxks5t8RiPgaJpZM4UlTIO> .

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/

[fwang2]

understood. thx