GPTQ Lite implementation #555
Conversation
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #555 +/- ##
==========================================
- Coverage 74.36% 73.92% -0.45%
==========================================
Files 182 182
Lines 18216 18395 +179
==========================================
+ Hits 13547 13598 +51
- Misses 4669 4797 +128 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
| gt=0.0, | ||
| le=1.0, | ||
| title="Percentage damping factor.", | ||
| description="The percentage of average Hessian diagonal used for damping.", |
There was a problem hiding this comment.
if you have a reference from the original paper about what these are, could you also share the link too?
| batch_size = input.shape[0] | ||
|
|
||
| # Incremental averaging: scale down old hessian | ||
| hessian *= n_samples / (n_samples + batch_size) |
There was a problem hiding this comment.
what's the dtype of hessian? Do we need to up cast to fp32 for this division?
| hessian, n_samples = update_hessian(input[0], state["hessian"], state["n_samples"]) | ||
| hessian_state[module.name] = {"hessian": hessian, "n_samples": n_samples} | ||
| torch.cuda.empty_cache() | ||
| gc.collect() |
There was a problem hiding this comment.
do we have to do gc.collect() here? It's going to be very slow
|
|
||
| # Phase 1: Collect statistics for quantizers | ||
| enable_stats_collection(model) | ||
| max_calibrate(model, forward_loop) |
There was a problem hiding this comment.
do you need forward_loop here? Is this for weight amax calib only?
| state = hessian_state[module.name] | ||
| hessian = state["hessian"].to(module.weight.device) | ||
| blockwise_weight_update(module, hessian, block_size, percdamp) | ||
| torch.cuda.empty_cache() |
There was a problem hiding this comment.
maybe you can del the hessian after applying blockwise_weight_update?
| hessian_state_path: str | None = ModeloptField( | ||
| default=None, | ||
| title="Path to the Hessian state file.", | ||
| description="The path to the Hessian state file.", |
There was a problem hiding this comment.
Maybe state: if the path exists, we load the hessian from the path instead of re-computing them.
What does this PR do?
Type of change: New feature
Overview: Adds support for GPTQ algorithm. This PR implements a modified version of the official GPTQ algorithm; the key differences are
Usage
Modify "algorithm" field in quant_cfg to "gptq_lite".
Note: Does not currently work with AWQ
# Add a code snippet demonstrating how to use thisTesting
Before your PR is "Ready for review"
Additional Information