v0.17.0

What's Changed

• Upgrade vLLM to v0.17.1 by @junya-takayama in #282

Full Changelog: v0.16.0...v0.17.0

v0.16.0

What's Changed

• Support multimodal input data by @moskomule in #278
• Use the reasoning attribute instead of the deprecated reasoning_content by @junya-takayama in #280
• Upgrade vLLM to v0.16.0 by @junya-takayama in #279
• [Documentation] Add multimodal benchmarks how-to guide by @amanjainj98 in #281

New Contributors

• @amanjainj98 made their first contribution in #281

Full Changelog: v0.15.5...v0.16.0

v0.15.5

What's Changed

• Some fixes for AzureOpenAI by @junya-takayama in #273
• Eliminate wait times on VLLMServeLM by @junya-takayama in #274
• Fix an inconsistency bug in multi-turn evaluation. by @junya-takayama in #275
• Add RepetitionN metric by @junya-takayama in #276

Full Changelog: v0.15.4...v0.15.5

v0.15.4

What's Changed

• fix ROUGE: add workaround for Hypothesis == '.' to avoid ValueError("Hypothesis is empty.") by @SeitaroShinagawa in #269
• Reduce disk space usage by @butsugiri in #271
• Bump ruff version by @butsugiri in #270
• Output reasoning_text by @h-asano in #272

New Contributors

• @h-asano made their first contribution in #272

Full Changelog: v0.15.3...v0.15.4

v0.15.3

What's Changed

• Reduce disk space usage of the unit test by @Ktakuya332C in #263
• [add] truncation options for ROUGE by @yutojubako in #264
• Add an option to specify whether to call language_model.cleanup_resources() by @junya-takayama in #267
• Add --retry mode in flexeval_lm by @junya-takayama in #266
• Add exit code to flexeval_lm by @moskomule in #265

New Contributors

• @yutojubako made their first contribution in #264

Full Changelog: v0.15.2...v0.15.3

v0.15.2

What's Changed

• Fix an issue triggered by long input to load_jinja2_template by @Ktakuya332C in #260
• Add support for returning reasoning_text by @hwichan0720 in #259

New Contributors

• @hwichan0720 made their first contribution in #259

Full Changelog: v0.15.1...v0.15.2

v0.15.1

What's Changed

• Use a dummy API key with VLLMServeLM. by @junya-takayama in #250
• Accept path to jinja2 template by @butsugiri in #252
• perplexity計算時にbosトークンがない場合にeosトークンを代わりに使う by @Ktakuya332C in #253
• Disable OpenAI Test by @butsugiri in #255
• Unified interface for setting random seed by @butsugiri in #254
• Disable saving cache in CI by @butsugiri in #257
• Add --num_repeats to flexeval_lm command by @butsugiri in #256
• Add truncate_middle to jinja env by @Ktakuya332C in #258

Full Changelog: v0.15.0...v0.15.1

v0.15.0

What's Changed

• Restore raw_lm_output in evaluation outputs by @uehara-mech in #245
• Remove __del__ method by @butsugiri in #247
• LiteLLMChatAPIのtoolsバグ修正 by @yhetsugi-SBint in #248
• OpenAIChatAPIのretryパラメータのexpose by @yhetsugi-SBint in #249
• upgrade: vllm==0.10.2 by @junya-takayama in #246

New Contributors

• @uehara-mech made their first contribution in #245

Full Changelog: v0.14.3...v0.15.0

v0.14.3

What's Changed

• README を修正 by @teruaki-o in #238
• Allow the regex for parsing ChatLLMScore to be configurable. by @kevin3314 in #239
• Refactor: Test of LLMScore by @kevin3314 in #240
• Make LLMScore's category_key accept list[str] as well. by @kevin3314 in #241
• Fanction calling非対応OpenAIモデルのエラー対応 by @yhetsugi-SBint in #242
• ChatResponseでFew-shotが挿入されない不具合対応 by @yhetsugi-SBint in #243

New Contributors

• @yhetsugi-SBint made their first contribution in #242

Full Changelog: v0.14.2...v0.14.3

v0.14.2

What's Changed

• Add a custom JSON encoder to truncate base64 strings by @moskomule in #233
• Fix tools in __init__ of LanguageModel by @ryokan0123 in #234
• Add the max_parallel_requests param into VLLMServeLM by @junya-takayama in #235
• Relax the version requirement of scipy by @ryokan0123 in #236
• Allow the regex for parsing LLMScore to be configurable. by @kevin3314 in #237

Full Changelog: v0.14.1...v0.14.2