Question about the training curves

[farewillwell]

Thank you for this outstanding work! However, I encountered some issues while trying to reproduce the results.
I used the OpenVLA model fine-tuned with SFT on the libero-goal dataset and tested it on the same task, and but I found that the model's success rate became very low (sometimes even 0), and the performance curve fluctuated a lot.
Would you be willing to share your training curves for the model?
Thank you very much!

[zzj1111]

Hello，Can you give me some details about your sft? i.e. how you collect libero sft dataset and your sft script?

[farewillwell]

Hello，Can you give me some details about your sft? i.e. how you collect libero sft dataset and your sft script?

Thank you very much for your help! Below I’ll list the relevant details of my setup:

1. For SFT, I used the official libero-goal dataset provided by LIBERO, specifically 100/500 demonstrations. I downloaded the dataset from the official site and selected 10 trajectories per task (50 tasks total). After obtaining the HDF5 files, I converted them to TFRecord format using the code from rlds_dataset_builder. I then fine-tuned the model using the official finetune.py script from OpenVLA. The fine-tuning was successful and achieved about 60% task success rate.

2. Based on that, I proceeded to train with TPO using your method. One change I made was to modify /TPO-Train/experiments/robot/libero/regenerate_libero_dataset.py so that the saved trajectories are in .hdf5 format instead of .npy. This allows me to directly use the rlds_dataset_builder tool mentioned above to convert the data into a format readable by the model. After that, I ran your provided training script directly on the data, keeping the learning rate, batch size, and number of epochs the same as in your default script.

Below is my training curve (I collected 7 rejected and chosen pairs per iteration for the plot).

In fact, I also tried using the official fine-tuned OpenVLA model as a starting point for TPO, and the resulting learning curves were quite similar to the one I posted above.

Therefore, I was wondering if you could share the TPO training curves you obtained—specifically the one that led to the [Figure 4] result in your paper using the SFT model on LIBERO provided by OpenVLA. This would greatly help me understand your approach better.

Thanks again for your assistance!

[JasonLiu324]

Hi, I have also successfully runned this experiment on Simpler-env. I got a similiar traing curves like yours, except for that the absolute value of logps_rejected and logps_chosen is no more than 60. Do you have further experiment results? My training process will stop at around 5k ~ 6k steps for no reason. And I don't know if that's reasonable

[farewillwell]

Hi, I have also successfully runned this experiment on Simpler-env. I got a similiar traing curves like yours, except for that the absolute value of logps_rejected and logps_chosen is no more than 60. Do you have further experiment results? My training process will stop at around 5k ~ 6k steps for no reason. And I don't know if that's reasonable

Sorry I haven't resolved the issue of the curves turning out like this, but after reducing the number of epochs from 10 to 2, the model's success rate no longer drops entirely to 0 — in some cases, it improves. However, the success rate during fine-tuning is still quite unstable, with significant fluctuations

[yinhhhh]

Hello，Can you give me some details about your sft? i.e. how you collect libero sft dataset and your sft script?

Thank you very much for your help! Below I’ll list the relevant details of my setup:

1. For SFT, I used the official libero-goal dataset provided by LIBERO, specifically 100/500 demonstrations. I downloaded the dataset from the official site and selected 10 trajectories per task (50 tasks total). After obtaining the HDF5 files, I converted them to TFRecord format using the code from rlds_dataset_builder. I then fine-tuned the model using the official finetune.py script from OpenVLA. The fine-tuning was successful and achieved about 60% task success rate.
2. Based on that, I proceeded to train with TPO using your method. One change I made was to modify /TPO-Train/experiments/robot/libero/regenerate_libero_dataset.py so that the saved trajectories are in .hdf5 format instead of .npy. This allows me to directly use the rlds_dataset_builder tool mentioned above to convert the data into a format readable by the model. After that, I ran your provided training script directly on the data, keeping the learning rate, batch size, and number of epochs the same as in your default script.

Below is my training curve (I collected 7 rejected and chosen pairs per iteration for the plot).

In fact, I also tried using the official fine-tuned OpenVLA model as a starting point for TPO, and the resulting learning curves were quite similar to the one I posted above.

Therefore, I was wondering if you could share the TPO training curves you obtained—specifically the one that led to the [Figure 4] result in your paper using the SFT model on LIBERO provided by OpenVLA. This would greatly help me understand your approach better.

Thanks again for your assistance!

Hi! I wanna ask how did you "directly use the rlds_dataset_builder tool mentioned above to convert the data into a format readable by the model", i.e., convert h5 data into the RLDS one? I'm a beginner. Thank you so much if you could provide some help!

[farewillwell]

Hello，Can you give me some details about your sft? i.e. how you collect libero sft dataset and your sft script?

Thank you very much for your help! Below I’ll list the relevant details of my setup:

1. For SFT, I used the official libero-goal dataset provided by LIBERO, specifically 100/500 demonstrations. I downloaded the dataset from the official site and selected 10 trajectories per task (50 tasks total). After obtaining the HDF5 files, I converted them to TFRecord format using the code from rlds_dataset_builder. I then fine-tuned the model using the official finetune.py script from OpenVLA. The fine-tuning was successful and achieved about 60% task success rate.
2. Based on that, I proceeded to train with TPO using your method. One change I made was to modify /TPO-Train/experiments/robot/libero/regenerate_libero_dataset.py so that the saved trajectories are in .hdf5 format instead of .npy. This allows me to directly use the rlds_dataset_builder tool mentioned above to convert the data into a format readable by the model. After that, I ran your provided training script directly on the data, keeping the learning rate, batch size, and number of epochs the same as in your default script.

Below is my training curve (I collected 7 rejected and chosen pairs per iteration for the plot).



In fact, I also tried using the official fine-tuned OpenVLA model as a starting point for TPO, and the resulting learning curves were quite similar to the one I posted above.
Therefore, I was wondering if you could share the TPO training curves you obtained—specifically the one that led to the [Figure 4] result in your paper using the SFT model on LIBERO provided by OpenVLA. This would greatly help me understand your approach better.
Thanks again for your assistance!

Hi! I wanna ask how did you "directly use the rlds_dataset_builder tool mentioned above to convert the data into a format readable by the model", i.e., convert h5 data into the RLDS one? I'm a beginner. Thank you so much if you could provide some help!

Hi, sorry about the confusion earlier — I think I misunderstood a few things at the time.
The repository I mentioned, moojink/rlds_dataset_builder, is actually the one I used previously when reproducing OpenVLA. It can convert HDF5 files (such as those provided in the official LIBERO dataset) into TFRecord format (for example, this one), which can be directly read by OpenVLA. You just need to cd into that directory and run tfds build.

However, I later realized that this code might shuffle the data , which could cause temporal mismatches in the sequences fed into the model. Because of that, I later followed the exact data processing procedure described in the Dataset section of this repo instead.

That said, the resulting training curves didn’t seem to differ much from what I reported earlier in this issue. But regardless of whether I change the dataset conversion script or not, training for 2 epochs can improve the model’s performance, while 10 epochs seems to make it worse. I guess that’s because my dataset is a bit small.