Software: Sunray v2024.0.0-alpha.2

[ethanjli]

The v2024.0.0 software release, which is part of the "Sunray" series of software releases started with v2023.9.0, includes major fixes for various problems and improvements to existing functionality with the PlanktoScope software.

About this pre-release

Starting with this pre-release, we are calling the software we provide in our SD card images the "PlanktoScope OS" (because it is a an operating system for the PlanktoScope).

PlanktoScope OS v2024.0.0-alpha.2 is a pre-release version for developers to test out. It may have undiscovered software bugs, breakage, and other problems. If you decide to try out this pre-release, please file bug reports on GitHub Issues and/or let us know about any bugs you discover in the #6-dev-software channel on the PlanktoScope Slack workspace!

Contributors

The PlanktoScope OS is a community effort, and the v2024.0.0 release would not have been possible without the PlanktoScope community:

• Thanks to Wassim Chakroun (@W7CH) for code contributions in the new picamera2-based image-acquisition module!
• Thanks to (in alphabetical order) Ana Fernandez Carrera, Adam Larson (@babo989), Fabien Lombard (@fabienlombard), Kevin Du Clos (@kevin-duclos), Satoshi Kitajima (@kitazy52), Pierre Kostyrka (@pkostyrka), Tanguy Cebron, and Thibaut Pollina (@tpollina), for contributing feature requests, bug reports, and software testing efforts for this release!

Changelog since v2023.9.0

This changelog only lists items we think users are likely to notice or to need to know about; for a full list of changes, please refer to software/CHANGELOG.md.

Added:

• The PlanktoScope now shares internet access over its Wi-Fi hotspot and Ethernet port (and a USB-to-Ethernet adapter, if one is attached) from a phone connected to it in USB tethering mode.
• On the Node-RED dashboard, the System Monitoring page will now prompt you to reset the PlanktoScope's system clock to match the time from your web browser, if the two times are very different. This enables you to temporarily fix an inaccurate system clock on your PlanktoScope, at least until the next time you unplug the PlanktoScope from power.

Changed:

• The pscopehat version of the SD card image has been renamed to planktoscopehat for more consistent naming.
• On the planktoscopehat version of the SD card image, the default hardware configuration has been changed from the v2.5 PlanktoScope hardware to the v2.6 hardware. If you're upgrading the software on a PlanktoScope with v2.5 hardware, you will need to change the hardware configuration back to v2.5 on the "Hardware Settings" page of the Node-RED dashboard.
• The hardware controller's camera driver has been fully rewritten to fix various problems we had with raspimjpeg (which has now been replaced with a different camera control mechanism). This change may require you to use different ISO and white balance gains than before.
• Previously, the segmenter's default behavior was to subtract consecutive masks to try to mitigate image-processing issues with objects which get stuck to the flowcell during imaging. However, when different objects occupied the same space in consecutive frames, the subtraction behavior would subtract one object's mask from the mask of the other object in the following frame, which would produce clearly incorrect masks. This behavior is no longer enabled by default.
• On the Node-RED dashboard, the System Monitoring page’s monitoring panel has been replaced with an embedded Grafana dashboard, with additional graphs available in an expandable subpanel.

Deprecated:

• The current Node-RED dashboard (both the version for the Adafruit HAT and the version for the PlanktoScope HAT) is transitioning to maintenance mode: no new features will be added, and any bugs will be only be fixed if someone volunteers to fix them. The current Node-RED dashboard will be completely replaced by a fully-rewritten Node-RED dashboard, though there is no timeline for completion of that new dashboard. Currently, our plan for deprecating and eventually removing the current Node-RED dashboard is as follows: maintenance mode (no new features, only some bugfixes), then deprecation (no maintenance; not enabled by default, but still installed), then removal (not installed by default, but anyone is free to install it and see if it still works); deprecation will not occur before the rewritten Node-RED dashboard is stable for general-purpose usage. If you have concerns, please share your feedback on GitHub or on the PlanktoScope Slack.
• The "USB backup" functionality of the Node-RED dashboard will be removed in v2024.1.0 (the next release after v2024.0.0). Instead, you should use the datasets file browser for backing up and deleting dataset files on your PlanktoScope.
• The raspimjpeg-based imaging module in the Python hardware controller has not yet been deleted, so that you can change the Python hardware controller code to switch back from the new picamera2-based imaging module if picamera2 ends up causing big problems for you. However, we are deprecating the raspimjpeg-based imaging module, and we will fully delete it in a future release after we get sufficient information about its reliability for PlanktoScope users.
• Portainer will no longer be installed/provided by default after v2024.0.0, because it takes up lots of disk space and it is difficult to set up and its functionality overlaps with the much simpler Dozzle. After we remove Portainer from the default SD card image, you can use Forklift to install Portainer.

Fixed:

• The Node-RED dashboard’s “Sample” page’s Filtered Volume field is now exported as the sample_total_volume metadata field for all sample types; previously it was only exported for horizontal plankton tow samples.
• Startup is now faster.

Changelog since v2024.0.0-alpha.1

Changed

• Previously, the segmenter's default behavior was to subtract consecutive masks to try to mitigate image-processing issues with objects which get stuck to the flowcell during imaging. However, when different objects occupied the same space in consecutive frames, the subtraction behavior would subtract one object's mask from the mask of the other object in the following frame, which would produce clearly incorrect masks. This behavior is no longer enabled by default.

• The image quality of frames in the camera preview stream is increased, and frames also have greater width and height (960x720); previously, it was 640x480.

• The default settings configuration file for the planktoscopehat SD card image is now for the v2.6 PlanktoScope hardware; previously, it was still for the v2.5 hardware.

Deprecated

• The "USB backup" functionality of the Node-RED dashboard will be removed in v2024.1.0 (the next release after v2024.0.0). Instead, you should use the datasets file browser for backing up and deleting dataset files on your PlanktoScope.
• The raspimjpeg-based imaging module in the Python hardware controller has not yet been deleted, so that you can change the Python hardware controller code to switch back from the new picamera2-based imaging module if picamera2 ends up causing big problems for you. However, we are deprecating the raspimjpeg-based imaging module, and we will fully delete it in a future release after we get sufficient information about its reliability for PlanktoScope users.
• Portainer will no longer be installed/provided by default after v2024.0.0, because it takes up lots of disk space and it is difficult to set up and its functionality overlaps with the much simpler Dozzle. After we remove Portainer from the default SD card image, you can use Forklift to install Portainer.

Fixed

• The incorrect scaling factor for converting between ISO settings (in the Node-RED dashboard) and camera image gains in the new picamera2-based imager is fixed. This should cause images (and the camera preview) to appear brighter than v2024.0.0-alpha.1 when the same ISO settings are used.

Known Problems

• (Known as of v2023.9.0-beta.1) You may encounter weird behavior with the system time if you try to use Cockpit to manually adjust the system clock. If you experience this problem, please report it and your testing circumstances (e.g. your GPS situation, and the time you tried to change the system clock to), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) In certain cases, some computers connected to both a Wi-Fi network providing internet access and to a PlanktoScope may be unable to access anything on the internet. If you experience this problem, please report it and your testing circumstances (e.g. your OS, your network configuration for internet access, etc.), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) If you restart the Node-RED dashboard (e.g. via Cockpit), you will also need to manually restart the hardware controller, or else the Node-RED dashboard won’t know the type of camera installed in the PlanktoScope. We are working on a solution to this problem, for a future release after v2024.0.0.

Planning, management, and maintenance of the v2024.0.0 release of PlanktoScope OS is led by Ethan Li (@ethanjli), with input and oversight from attendees of the weekly PlanktoScope software development meetings. If this release causes new problems for how you are using PlanktoScope, Ethan will take responsibility for addressing any new issues caused by this release - just tag them in a new Github Discussions post or in the #6-dev-software channel on the PlanktoScope Slack workspace!

Pull Requests

• Try to fix GitHub Actions permissions for docker/build-and-push action by @ethanjli in Try to fix GitHub Actions permissions for docker/build-and-push action #338
• build(deps): bump caddy from 2.7.5 to 2.7.6 in /documentation by @dependabot in build(deps): bump caddy from 2.7.5 to 2.7.6 in /documentation #337
• build(deps): bump actions/configure-pages from 3 to 4 by @dependabot in build(deps): bump actions/configure-pages from 3 to 4 #331
• build(deps): bump mkdocs-material from 9.5.1 to 9.5.3 in /documentation by @dependabot in build(deps): bump mkdocs-material from 9.5.1 to 9.5.3 in /documentation #336
• build(deps): bump actions/deploy-pages from 2 to 4 by @dependabot in build(deps): bump actions/deploy-pages from 2 to 4 #329
• build(deps): bump actions/upload-pages-artifact from 2 to 3 by @dependabot in build(deps): bump actions/upload-pages-artifact from 2 to 3 #330
• build(deps): bump actions/setup-python from 4 to 5 by @dependabot in build(deps): bump actions/setup-python from 4 to 5 #328
• build(deps): bump actions/upload-artifact from 3 to 4 by @dependabot in build(deps): bump actions/upload-artifact from 3 to 4 #327
• build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.1 to 1.2.2 in /documentation by @dependabot in build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.1 to 1.2.2 in /documentation #332
• build(deps): bump mkdocs-glightbox from 0.3.5 to 0.3.6 in /documentation by @dependabot in build(deps): bump mkdocs-glightbox from 0.3.5 to 0.3.6 in /documentation #335
• build(deps): bump pymdown-extensions from 10.5 to 10.7 in /documentation by @dependabot in build(deps): bump pymdown-extensions from 10.5 to 10.7 in /documentation #334
• Fix issue with required status checks not running by @ethanjli in Fix issue with required status checks not running #339
• build(deps): bump mkdocs-minify-plugin from 0.7.1 to 0.7.2 in /documentation by @dependabot in build(deps): bump mkdocs-minify-plugin from 0.7.1 to 0.7.2 in /documentation #333
• Fix some issues with running the distro setup scripts on RPi OS 12 (bookworm) by @ethanjli in Fix some issues with running the distro setup scripts on RPi OS 12 (bookworm) #324
• Allow running Docker commands without sudo by @ethanjli in Allow running Docker commands without sudo #341
• Bump pallet-standard to add Dozzle by @ethanjli in Bump pallet-standard to add Dozzle #342
• Update hostname separately from the /etc/hosts file by @ethanjli in Update hostname separately from the /etc/hosts file #344
• build(deps): bump gitpython from 3.1.40 to 3.1.41 in /documentation by @dependabot in build(deps): bump gitpython from 3.1.40 to 3.1.41 in /documentation #345
• build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /documentation by @dependabot in build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /documentation #347
• Add Prometheus+Grafana-based monitoring of host machine, enable resetting system time to browser time by @ethanjli in Add Prometheus+Grafana-based monitoring of host machine, enable resetting system time to browser time #346
• Run forklift plt apply with parallelization by @ethanjli in Run forklift plt apply with parallelization #348
• os/pallet: Move some scripts and local files over into management by forklift by @ethanjli in os/pallet: Move some scripts and local files over into management by forklift #350
• Add support for usb0, eth1, and wlan1 interfaces by @ethanjli in Add support for usb0, eth1, and wlan1 interfaces #351
• Deploy the backend segmenter with forklift rather than the distro setup scripts by @ethanjli in Deploy the backend segmenter with forklift rather than the distro setup scripts #352
• Replace <object> tags with <iframe> tags by @ethanjli in Replace <object> tags with <iframe> tags #361
• Don't suppress wget errors in the docs's non-standard install commands by @ethanjli in Don't suppress wget errors in the docs's non-standard install commands #362
• Replace slider with text input to set pump volume by @ethanjli in Replace slider with text input to set pump volume #363
• Export the filtered volume metadata field on all sample types by @ethanjli in Export the filtered volume metadata field on all sample types #364
• Repopulate input fields of input forms on Sample page by @ethanjli in Repopulate input fields of input forms on Sample page #365
• Bump mkdocs-material to bump pillow to resolve dependabot alert by @ethanjli in Bump mkdocs-material to bump pillow to resolve dependabot alert #366
• build(deps): bump mkdocs-glightbox from 0.3.6 to 0.3.7 in /documentation by @dependabot in build(deps): bump mkdocs-glightbox from 0.3.6 to 0.3.7 in /documentation #367
• build(deps): bump mkdocs-minify-plugin from 0.7.2 to 0.8.0 in /documentation by @dependabot in build(deps): bump mkdocs-minify-plugin from 0.7.2 to 0.8.0 in /documentation #368
• Bump changelog version number for v2024.0.0-alpha.0 by @ethanjli in Bump changelog version number for v2024.0.0-alpha.0 #369
• Bump Python segmenter by @ethanjli in Bump Python segmenter #372
• build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.2 to 1.2.4 in /documentation by @dependabot in build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.2 to 1.2.4 in /documentation #373
• build(deps): bump mkdocs-material from 9.5.6 to 9.5.12 in /documentation by @dependabot in build(deps): bump mkdocs-material from 9.5.6 to 9.5.12 in /documentation #375
• build(deps): bump poethepoet from 0.24.4 to 0.25.0 in /documentation by @dependabot in build(deps): bump poethepoet from 0.24.4 to 0.25.0 in /documentation #374
• Move tech specs in the docs to their own pages by @ethanjli in Move tech specs in the docs to their own pages #379
• Switch from raspimjpeg to picamera2 for image acquisition by @ethanjli in Switch from raspimjpeg to picamera2 for image acquisition #380
• Bump forklift from v0.5.3 to v0.6.0 by @ethanjli in Bump forklift from v0.5.3 to v0.6.0 #381
• Fix the Node-RED dashboard's "Restart segmenter" button by @ethanjli in Fix the Node-RED dashboard's "Restart segmenter" button #382
• Bump version numbers for v2024.0.0-alpha.1 by @ethanjli in Bump version numbers for v2024.0.0-alpha.1 #383
• Make small improvements to the software install docs by @ethanjli in Make small improvements to the software install docs #386
• Fix various small issues in the Node-RED dashboard by @ethanjli in Fix various small issues in the Node-RED dashboard #385
• Temporarily disable MkDocs social preview cards by @ethanjli in Temporarily disable MkDocs social preview cards #387
• Add technical reference documentation by @ethanjli in Add technical reference documentation #390
• Add operating systems-oriented technical reference documentation by @ethanjli in Add operating systems-oriented technical reference documentation #391
• Try to re-enable MkDocs social preview cards by @ethanjli in Try to re-enable MkDocs social preview cards #392
• build(deps): bump actions/configure-pages from 4 to 5 by @dependabot in build(deps): bump actions/configure-pages from 4 to 5 #393
• build(deps): bump pymdown-extensions from 10.7 to 10.7.1 in /documentation by @dependabot in build(deps): bump pymdown-extensions from 10.7 to 10.7.1 in /documentation #394
• build(deps-dev): bump pillow from 10.2.0 to 10.3.0 in /documentation by @dependabot in build(deps-dev): bump pillow from 10.2.0 to 10.3.0 in /documentation #395
• Fix papercuts in the setup instructions for the docs dev environment by @ethanjli in Fix papercuts in the setup instructions for the docs dev environment #396
• Migrate OS config files into the Forklift pallet by @ethanjli in Migrate OS config files into the Forklift pallet #397
• Update software docs for FS overlays, and for renaming to "PlanktoScope OS" by @ethanjli in Update software docs for FS overlays, and for renaming to "PlanktoScope OS" #398
• Fix various blockers for v2024.0.0-beta.0 by @ethanjli in Fix various blockers for v2024.0.0-beta.0 #399
• Bump version numbers for v2024.0.0-alpha.2 by @ethanjli in Bump version numbers for v2024.0.0-alpha.2 #400

Full Changelog: software/v2023.9.0...software/v2024.0.0-alpha.2

---

This discussion was created from the release software/v2024.0.0-alpha.2.