Skip to content

Allow tracking the progress of loading a LlamaModel #113

@natanfudge

Description

@natanfudge

It would be useful to know the progress of loading an LLM. This would be displayed to a user with a progress bar or similar.
The API would look something like this

	public LlamaModel(ModelParameters parameters, Consumer<LLamaLoadProgress>)

Where LLamaLoadProgress is something like

class LLMLoadProgress{
    /**
     *  A value from 0 to 1 representing the overall load progress of the file given by [fileName].
     *  If you don't want downloads of small files to skew this value, check for [isWeightsProgress], as that file is usually the one we care about for tracking its loading progress, because it is extremely large. 
     */
    double fraction
    /**
     *  Additional information about the loading progress 
     */
    String text
    /**
     * Whether the progress is of downloading the model (true) or loading it from disk (false)
     */
    boolean isDownloading
    /**
     * Whether this represents the progress of downloading the extremely large weights file (true) or some auxiliary file like the tokenizer config.
     */
    boolean isWeightsProgress
    /**
     * The file that its load has progressed by amount indicated by this [LLMLoadProgress]. If the file is the weights, [isWeightsProgress] will be true, and for other files [isWeightsProgress] will be false.
     */
    @Nullable String fileName
    /**
     * The amount of bytes loaded from the file given by [fileName].
     */
    long bytesLoaded
    /**
     * The byte size of the file given by [fileName]. Will be null as long as the file's size is not known.
     */
    @Nullable Long fileTotalByteSize
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions