Skip to content

lib/gis: add concurrency support for G_percent()#7259

Open
nilason wants to merge 1 commit intoOSGeo:mainfrom
nilason:upd_percent_multithread
Open

lib/gis: add concurrency support for G_percent()#7259
nilason wants to merge 1 commit intoOSGeo:mainfrom
nilason:upd_percent_multithread

Conversation

@nilason
Copy link
Copy Markdown
Contributor

@nilason nilason commented Apr 2, 2026

This adds new non-locking G_progress_* API, which is build on re-entrancy, atomic compare-and-swap, and a dedicated thread for the progress reporting. The legacy API with G_percent() and G_progress() is updated as wrappers using a global context.

Current code using G_process and G_percent, should work as is. New code may look like:

size_t n_rows = window.rows;  // total number of rows
size_t step = 10;  // output step, every 10%
GProgressContext *ctx = G_progress_context_create(n_rows, step);
for (row = 0; row < window.rows; row++) {
    // costly calculation ...
 
    // note: not counting from zero, as for loop never reaches n_rows
    //       and we want to reach 100%
    size_t completed_row = row + 1;
 
    G_progress_update(ctx, completed_row);
}
G_progress_context_destroy(ctx);

or:

GProgressContext *ctx = G_progress_context_create_increment(100);
while (TRUE) {
    G_progress_increment(ctx, rows_processed);
}
G_progress_context_destroy(ctx);

The progress calculations are separated from output, so it is possible for caller to override the default modes of output with a custom function via a GProgressSink. I have also added a time interval based reporting with G_progress_context_create_time().

Requirement: support for C11 with atomic operations (<stdatomic.h>) and pthread. There should be a solution for adopting for MSVC, but I can't test that.

Implementation details:

The implementation is organized as a telemetry pipeline. Producer-side API calls update atomic progress state and enqueue EV_PROGRESS or EV_LOG records into a bounded ring buffer. A single consumer thread drains that buffer, converts raw records into GProgressEvent values, and forwards them either to installed sink callbacks or to default renderers selected from the current G_info_format() mode.

Concurrency is designed as multi-producer, single-consumer per telemetry stream. Producers reserve slots with an atomic write_index, publish events by setting a per-slot ready flag with release semantics, and use atomic compare-and-swap to ensure that only one producer emits a given percent threshold or time-gated update. The consumer advances a non-atomic read_index, waits for published slots, processes events in FIFO order, and then marks slots free again.

Two lifecycle models are used. Isolated GProgressContext instances create a dedicated consumer thread that is joined during destruction. The legacy process-wide G_percent() path initializes one shared telemetry instance and a detached consumer thread on first use.

Disclosure: created with assistance of ChatGPT and Codex.

Closes: #5776

This adds new non-locking G_progress_* API, which is build on
re-entrancy, atomic compare-and-swap, and a dedicated thread for the
progress reporting. The legacy API with G_percent() and G_progress()
is updated as wrappers using a global context.

The implementation is organized as a telemetry pipeline. Producer-side
API calls update atomic progress state and enqueue EV_PROGRESS or
EV_LOG records into a bounded ring buffer. A single consumer thread
drains that buffer, converts raw records into GProgressEvent values, and forwards them either to installed sink callbacks or to default
renderers selected from the current G_info_format() mode.

Concurrency is designed as multi-producer, single-consumer per
telemetry stream. Producers reserve slots with an atomic write_index,
publish events by setting a per-slot ready flag with release semantics,
and use atomic compare-and-swap to ensure that only one producer emits
a given percent threshold or time-gated update. The consumer advances a non-atomic read_index, waits for published slots, processes events in
FIFO order, and then marks slots free again.

Two lifecycle models are used. Isolated GProgressContext instances
create a dedicated consumer thread that is joined during destruction.
The legacy process-wide G_percent() path initializes one shared
telemetry instance and a detached consumer thread on first use.
@nilason nilason added this to the 8.6.0 milestone Apr 2, 2026
@nilason nilason requested a review from marisn April 2, 2026 08:22
@github-actions github-actions bot added C Related code is in C libraries labels Apr 2, 2026
@petrasovaa
Copy link
Copy Markdown
Contributor

Without studying this in depth, this looks like overkill to me...Surely there must be a simpler, maybe less general solution?

How to test this?

@nilason
Copy link
Copy Markdown
Contributor Author

nilason commented Apr 3, 2026

Without studying this in depth, this looks like overkill to me...Surely there must be a simpler, maybe less general solution?

A lot of code is for accommodating 1-to-1 adaptation to current use of G_percent and G_progress API. Removing that layer, (especially the pattern of finishing the counter with G_percent(1, 1, 1)/G_progress(1, 1), the G_{set/unset}_percent_routine() and the global context) and use only this new API specifically for parallelised code will reduce and simplify this. It shouldn't be too much work to reduce this PR in that direction, I'll make a new PR to be able to make a decision without ruling this out completely.

How to test this?

With this PR (built with C11 atomic operation support –which most of current and also not-so-current compilers do– and pthread) all calls of G_procent and G_progress use this concurrent code. So try any module, but in particular parallelised/OpenMP modules where the advantage of this is put to the test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C Related code is in C libraries

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] G_percent is not safe to be called from parallel code

2 participants