Software: Sunray v2023.9.0-beta.2

[ethanjli]

v2023.9.0 will be our first software update in around two years, and we’re excited to make it easier for you to use your PlanktoScope in this update and in future updates! The v2023.9.0 software release focuses on improving the software infrastructure of the PlanktoScope software distro in preparation for future software improvements and maintenance, and making various usability improvements.

This release’s name is “Sunray” (named after the Sunray Venus clam). We will keep this name for future versions until the next release which introduces big changes for users.

Planning, management, and maintenance of the v2023.9.0 software release is led by Ethan Li (@ethanjli), with approval and oversight from Thibaut Pollina (@tpollina). If this software release causes new problems for how you are using PlanktoScope, Ethan will take responsibility for addressing any new issues caused by this release - just tag them in a new Github Discussions post or in the #6-dev-software channel on the PlanktoScope Slack workspace!

About this pre-release

Software v2023.9.0-beta.2 is a pre-release testing version for you to test out. It definitely has software bugs and other problems (the problems we know about are listed under the “Known Issues” section near the end of these release notes), so you should not use it for collecting scientific data! We would greatly appreciate it if you could flash the v2023.9.0-beta.2 image to an SD card and let us know about any bugs or problems you encounter while trying to use your PlanktoScope. This will help us to deliver a higher-quality software release with our v2023.9.0 release.

If you decide to try out the v2023.9.0-beta.2 software, please file bug reports on GitHub Issues and/or let us know about any bugs you discover in the #6-dev-software channel on the PlanktoScope Slack workspace!

Because this pre-release includes many changes, the changelogs below only list items we think users are likely to notice or to need to know about; for a full list of changes, please refer to software/CHANGELOG.md.

Changelog since v2.3

Highlights:

• To use the PlanktoScope from your web browser, now you should just open http://home.pkscope/ or http://pkscope.local/ or http://192.168.4.1/.
• You no longer need to use FileZilla to download and delete the datasets on your PlanktoScope - now you can do it from your web browser.
• The PlanktoScope now acts like a router, so if it has internet access then it will share that internet access with your computer/phone.
  PlanktoScopes will no longer stay connected to any Wi-Fi network which doesn’t provide internet access.
• PlanktoScopes now use a new naming scheme for machine names.
• We will eventually transition the Node-RED dashboard for the v2.1 hardware (Adafruit HAT) into maintenance mode.
• The Node-RED dashboard will no longer try to use incorrect URLs to show the camera preview stream.
• Entering invalid white balance values in the Node-RED dashboard should no longer break the PlanktoScope software upon restart.

Added:

• The PlanktoScope can now be accessed from your web browser at http://home.pkscope/ (assuming your web browser isn’t configured to use a custom DNS server such as 1.1.1.1 or 8.8.8.8) and http://pkscope.local/ (assuming your web browser supports mDNS), in addition to the other URLs which previously worked (http://planktoscope.local/, http://192.168.4.1/, and http://192.168.5.1/).
  • Each PlanktoScope can now also be accessed at machine-specific URLs of format http://<machine-name>.pkscope and http://pkscope-<machine-name>.local, where <machine-name> is replaced with your PlanktoScope’s machine name (see the note in the “Changed” section below about machine names).
  • If your web browser tries to do a Google search for the URL instead of actually opening it as a webpage, make sure you type http:// at the start or / at the end (e.g. home.pkscope/ or http://home.pkscope/ instead of home.pkscope)
  • If you connect directly from your computer/phone to the PlanktoScope and then use your web browser to open http://home.pkscope/, http://<machine-name>.pkscope, http://pkscope-<machine-name>.local, http://pkscope.local/, http://192.168.4.1/, or http://192.168.5.1/, you will now see a landing page with a list of useful links to software running on the PlanktoScope.
• The PlanktoScope now shares any internet access it has from Wi-Fi with all devices connected to its Ethernet port; and it also shares any internet access it has from Ethernet with all devices connected to its Wi-Fi hotspot, if it’s operating in wireless AP mode.
• An offline version of the PlanktoScope documentation is now available. It's accessible by a link from the PlanktoScope's landing page. The online version of the documentation is now at https://docs.planktoscope.community .
  • The PlanktoScope documentation now includes the reference protocol for quantitative imaging with PlanktoScope, which is normally available at https://www.protocols.io/view/planktoscope-protocol-for-plankton-imaging-bp2l6bq3zgqe/v2 . You can access the protocol from the bottom of the “Usage” → “Getting Started” page of the PlanktoScope documentation.
  • The PlanktoScope documentation pages have been redone in a new system, and improved in various ways. The hardware documentation now describes the v2.5 PlanktoScope hardware. Thanks to Sebastian Wendel (@sourceindex) for starting this work, and for contributing lots of documentation!

Changed:

• The Node-RED dashboard (previously available at http://planktoscope.local:1880/ui) has been moved to a different URL, accessible by a link on the new landing page (see the note in the “Added” section about new URLs and the landing page).
• The file gallery (previously available at http://planktoscope.local/) has now been renamed to the dataset file manager and moved to a different URL, accessible by a link on the new landing page. It now allows you to delete folders, preview images, edit text files, download folders as ZIP files, rename folders, and perform other common tasks.
  • This way, you no longer have to install FileZilla in order to copy or delete any of your datasets!
• Previously, PlanktoScope machine names were generated as gibberish words like "Babaxio-Detuiau", and the machine names were used as the names of the Wi-Fi hotspot networks made by the PlanktoScope. However, the machine names created by this naming scheme were often difficult to pronounce, remember, and type for people in various languages, and the naming scheme sometimes generated names which sounded like curses or insults in some languages. Now, PlanktoScope machine names are generated as a combination of two words and a number up to five digits long; words are selected from pre-built lists in a language which can be chosen based on localization settings. Currently, word lists are only provided in US English, resulting in names like "metal-slope-23501", "conscious-pocket-1684", and "plant-range-10581"; however, word lists can be added for other languages in the future, and a user interface will eventually be provided for changing localization settings.
  • This change makes it easier for you to tell someone else which PlanktoScope to connect to if multiple PlanktoScope Wi-Fi hotspot networks are available nearby.
  • If you know a language besides US English and you’d like to help us add support for generating machine names in that language, please start a discussion on Github or in the #6-dev-software channel on the PlanktoScope Slack workspace!
• Previously, the PlanktoScope would stay connected to a wifi network even if it didn’t have internet access on that network. Now, if it connects to a Wi-Fi network but can’t get internet access on that network (specifically, if it’s unable to find google.com), then it will revert to creating its own Wi-Fi hotspot for your computer or phone to connect to.

Deprecated:

• The planktoscope.local mDNS name is no longer recommended for use; instead, you should use pkscope.local or the machine-specific mDNS name of format pkscope-{machine-name}.local.
  • However, we will continue to support planktoscope.local for a while, to maintain compatibility with earlier software versions.
• In a future release (timeline not yet decided), the version of the Node-RED dashboard for the Adafruit HAT (hardware v2.1) will stop receiving new features even as the version of the Node-RED dashboard for the custom PlanktoScope HAT (hardware v2.3+) continues receiving new features.
  • Depending on the amount of work required, we might be able to make the Node-RED dashboard for the custom PlanktoScope HAT be compatible with the Adafruit HAT. However, we do not have any specific plans yet.
  • Regardless, we do plan to continue to fix bugs in the Node-RED dashboard for the Adafruit HAT for some time, and we will continue to build SD card images for the Adafruit HAT which will also include new features in other software components.

Fixed:

• When an invalid value is entered for the red or blue white balance gain on the Node-RED dashboard's "Optic Configuration" page, that value is now ignored, a notification is displayed about the invalid value, and the white balance gain is reset to the last valid value (loaded from the hardware.json configuration file). Previously, invalid white balance values were written to the hardware.json configuration file, which would cause the PlanktoScope software to crash after every subsequent restart until the hardware.json file was manually removed or fixed.
  • Thanks to Gilles Orazi (@glx314) for filing an issue about this problem!
• In various use cases (e.g. when accessing the PlanktoScope over a mesh VPN, or in some bug reports by users accessing the PlanktoScope over Ethernet), the Node-RED dashboard would be unable to show the MJPEG streams of the camera preview and the object segmentation preview, resulting in a potentially misleading error message being shown about the Python scripts. Now, the Node-RED dashboard uses URLs for the MJPEG streams which we expect to be accessible wherever the Node-RED dashboard itself is accessible.
  • Thanks to Rodrigo Gonçalves on the PlanktoScope Slack for reporting a bug related to this issue and helping us to troubleshoot it!
• The adafruit-blinka and adafruit-platformdetect software dependencies are now updated to their latest version. As a result, the PlanktoScope software should not crash just because it is being used with recent versions of the Adafruit Stepper HAT.
  • This fix removes the need for you to give your PlanktoScope internet access and then connect to the PlanktoScope over SSH to manually run the pip3 install --upgrade adafruit-blinka adafruit-platformdetect command before your PlanktoScope can work properly.
• The default brightness of the illumination LED for the pscopehat version of software has been reduced; this is a temporary workaround to a bug with raspimjpeg where saved images are overexposed even on the default brightness settings with minimum shutter speed and ISO, despite the brightness of raspimjpeg's camera preview looking reasonable.
  • Thanks to Laurent Paul Vallet (@LaurentPV) for contributing this fix!

Changelog since v2023.9.0-beta.1

Added:

• A hardware configuration file for PlanktoScope hardware v2.6, which was previously missing, has been added. It is now the default hardware configuration for the pscopehat build of the PlanktoScope SD card image; preset configuration files for hardware v2.3 and v2.5 can be selected instead from the Node-RED dashboard’s “Hardware Settings” page.

Fixed:

• The Node-RED dashboard is now less eager in validating and undoing any changes you make to the white balance which could cause an invalid white balance value. Now, changes you make to the white balance are only validated and applied after you click away from the input field.
  • Thanks to Pierre Kostyrka (@pkostyrka) for making a bug report about this issue!
• The default brightness of the illumination LED for the pscopehat version of software has been reduced; this is a temporary workaround to a bug with raspimjpeg where saved images are overexposed even on the default brightness settings with minimum shutter speed and ISO, despite the brightness of raspimjpeg's camera preview looking reasonable.
  • Thanks to Laurent Paul Vallet (@LaurentPV) for contributing this fix!
• Now the PlanktoScope should finish booting more quickly the first time it is booted after the SD card is flashed - now the first boot should be roughly as fast as subsequent boots.

Known Problems:

• (Known as of v2023.9.0-beta.0) Sometimes, image acquisition will fail to start, with a “camera timeout” error; or sometimes the camera will not respond to camera settings changes from Node-RED. This problem may go away after one or more reboots, or it may come back after one or more reboots. This is a complicated issue and we still don’t know exactly why it happens. If you experience this problem, please report it and your testing circumstances (e.g. when would the error occur, how repeatable was it, etc.), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.1) You may encounter weird behavior with the system time if you try to use Cockpit to manually adjust the system clock. If you experience this problem, please report it and your testing circumstances (e.g. your GPS situation, and the time you tried to change the system clock to), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) In certain cases, some computers connected to both a Wi-Fi network providing internet access and to a PlanktoScope may be unable to access anything on the internet. If you experience this problem, please report it and your testing circumstances (e.g. your OS, your network configuration for internet access, etc.), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) If you restart the Node-RED dashboard (e.g. via Cockpit), you will also need to manually restart the hardware controller, or else the Node-RED dashboard won’t know the type of camera installed in the PlanktoScope. The solution will take some work and will not be delivered in the v2023.9.0 stable release.

Pull Requests

• Automatically pull software changelog into the docs by @ethanjli in Automatically pull software changelog into the docs #232
• Make Node-RED trigger white-balance input validation less aggressively by @ethanjli in Make Node-RED trigger white-balance input validation less aggressively #238
• Update notes in docs about Android's mDNS support by @ethanjli in Update notes in docs about Android's mDNS support #246
• build(deps-dev): bump pillow from 10.0.0 to 10.0.1 in /documentation by @dependabot in build(deps-dev): bump pillow from 10.0.0 to 10.0.1 in /documentation #241
• build(deps-dev): bump linkchecker from 10.2.1 to 10.3.0 in /documentation by @dependabot in build(deps-dev): bump linkchecker from 10.2.1 to 10.3.0 in /documentation #243
• build(deps-dev): bump poethepoet from 0.22.1 to 0.24.0 in /documentation by @dependabot in build(deps-dev): bump poethepoet from 0.22.1 to 0.24.0 in /documentation #247
• build(deps-dev): bump urllib3 from 2.0.4 to 2.0.6 in /documentation by @dependabot in build(deps-dev): bump urllib3 from 2.0.4 to 2.0.6 in /documentation #248
• build(deps-dev): bump urllib3 from 2.0.6 to 2.0.7 in /documentation by @dependabot in build(deps-dev): bump urllib3 from 2.0.6 to 2.0.7 in /documentation #251
• Apply the forklift pallet as part of the SD card setup process by @ethanjli in Apply the forklift pallet as part of the SD card setup process #254
• Make layout of fields consistent between Node-RED dashboard versions by @ethanjli in Make layout of fields consistent between Node-RED dashboard versions #255
• Improve the documentation's software setup instructions by @ethanjli in Improve the documentation's software setup instructions #256
• Add hardware v2.6 support by @ethanjli in Add hardware v2.6 support #271
• Add explanatory comment about image width/height in raspimjpeg.conf by @ethanjli in Add explanatory comment about image width/height in raspimjpeg.conf #272
• Add a GitHub Issue template for proposals by @ethanjli in Add a GitHub Issue template for proposals #283
• Make the docs site easier to navigate by @ethanjli in Make the docs site easier to navigate #274
• build(deps-dev): bump pymdown-extensions from 10.3 to 10.4 in /documentation by @dependabot in build(deps-dev): bump pymdown-extensions from 10.3 to 10.4 in /documentation #285
• build(deps): bump caddy from 2.7.4 to 2.7.5 in /documentation by @dependabot in build(deps): bump caddy from 2.7.4 to 2.7.5 in /documentation #281
• build(deps-dev): bump mkdocs-material from 9.4.2 to 9.4.10 in /documentation by @dependabot in build(deps-dev): bump mkdocs-material from 9.4.2 to 9.4.10 in /documentation #286
• build(deps-dev): bump poethepoet from 0.24.0 to 0.24.4 in /documentation by @dependabot in build(deps-dev): bump poethepoet from 0.24.0 to 0.24.4 in /documentation #284
• build(deps-dev): remove mkdocs-material-extensions since it's deprecated by @ethanjli in build(deps-dev): remove mkdocs-material-extensions since it's deprecated #287
• build(deps-dev): bump mkdocs-macros-plugin from 1.0.4 to 1.0.5 in /documentation by @dependabot in build(deps-dev): bump mkdocs-macros-plugin from 1.0.4 to 1.0.5 in /documentation #280
• Add PDF export for schematic of custom HAT for v2.5 hardware by @ethanjli in Add PDF export for schematic of custom HAT for v2.5 hardware #288
• Add a placeholder title for the proposal issue template by @ethanjli in Add a placeholder title for the proposal issue template #293
• Update operation docs for new URLs, landing page, and Node-RED changes by @ethanjli in Update operation docs for new URLs, landing page, and Node-RED changes #294
• Build a minimal version of the docs site, without hardware setup guides by @ethanjli in Build a minimal version of the docs site, without hardware setup guides #295
• Fix the values of the process_source and process_commit metadata fields by @ethanjli in Fix the values of the process_source and process_commit metadata fields #273
• Add v2.5 hardware BOM to v2.5 kit production guide by @ethanjli in Add v2.5 hardware BOM to v2.5 kit production guide #296
• Automatically deploy beta versions of the docs site to docs-beta.planktoscope.community by @ethanjli in Automatically deploy beta versions of the docs site to docs-beta.planktoscope.community #298
• Automatically deploy stable versions of the docs site to docs.planktoscope.community by @ethanjli in Automatically deploy stable versions of the docs site to docs.planktoscope.community #299
• Fix the names of deployment env secrets for SSH deploy keys by @ethanjli in Fix the names of deployment env secrets for SSH deploy keys #300
• Update offline docs & landing page in the distro by @ethanjli in Update offline docs & landing page in the distro #301
• Bump version number of the docs package to 2023.9.0b3 by @ethanjli in Bump version number of the docs package to 2023.9.0b3 #308
• Bump version numbers for v2023.9.0-beta.2 by @ethanjli in Bump version numbers for v2023.9.0-beta.2 #309

Full Changelog: software/v2023.9.0-beta.1...software/v2023.9.0-beta.2

---

This discussion was created from the release Sunray v2023.9.0-beta.2 - 2023-12-02.