Software: Sunray v2023.9.0-alpha.0

[ethanjli]

v2023.9.0 will be our first software update in a year and a half, and we’re excited to make it easier for you to use your PlanktoScope in this update and in future updates!

The v2023.9.0 software release focuses on improving the software infrastructure of the PlanktoScope software distro in preparation for future software improvements and maintenance. The goal of this release is to fix some bugs in the existing software, and to take a few early steps towards enabling smaller and more frequent software updates in the future, so that we can deliver bugfixes and improvements to you on a more timely schedule, and so that you can apply those changes in a safer, easier, and simpler way.

This release’s name is “Sunray” (named after the Sunray Venus clam). We will keep this name for future versions until the next release which introduces big changes for users. The v2023.12.0 release, which is planned for December 2023, will introduce big changes for users (including requiring users to re-flash their SD cards with a new SD card image) and thus will have a different release name.

v2023.9.0-alpha.0 is an unstable testing version for the PlanktoScope development team and any advanced users who are feeling particularly adventurous. It definitely has undiscovered software bugs, breakage, and other problems, and we will be using this prerelease in order to discover those bugs. We will eventually make a v2023.9.0-beta.0 prerelease where more of those bugs should be discovered and fixed, for more people to test out. If you decide to try out v2023.9.0-alpha.0, please file bug reports on GitHub Issues and/or let us know about any bugs you discover in the #6-dev-software channel on the PlanktoScope Slack workspace!

Planning, management, and maintenance of the v2023.9.0 software release is led by Ethan Li (@ethanjli), with approval and oversight from Thibaut Pollina. If this software release causes new problems for how you are using PlanktoScope, Ethan will take responsibility for addressing any new issues caused by this release - just tag Ethan in a new discussion on Github or in the #6-dev-software channel on the PlanktoScope Slack workspace!

The changes listed below are just the ones we think users are likely to notice or to need to know about since the previous stable release (v2.3.0); for a full list of changes between each stable release and unstable prerelease, please refer to the software/CHANGELOG.md file in this repository.

Added:

• The PlanktoScope can now be accessed from your web browser using home.planktoscope (assuming your web browser isn’t configured to use a custom DNS server such as 1.1.1.1 or 8.8.8.8), in addition to the other URLs which previously worked (planktoscope.local, 192.168.4.1, and 192.168.5.1). Each PlanktoScope can now also be accessed at unique domain names of format <machine-name>.planktoscope and planktoscope-<machine-name>.local, where <machine-name> is replaced with your PlanktoScope’s machine name (see the note in the “Changed” section below about machine names). If you connect directly from your computer/phone to the PlanktoScope and then use your web browser to open http://home.planktoscope/, http://<machine-name>.planktoscope/, http://planktoscope-<machine-name>.local/, http://planktoscope.local/, http://192.168.4.1/, or http://192.168.5.1/, you will now see a landing page with a list of useful links to software running on the PlanktoScope.
  • Note that home.planktoscope and <machine-name>.planktoscope will probably work even if planktoscope.local and planktoscope-<machine-name>.local don’t work (e.g. if you’re on an Android device) - so you might not need to type 192.168.4.1 on your phone anymore!
  • If your web browser tries to do a Google search for the URL instead of actually opening it as a URL, type http:// at the start or add a / at the end (e.g. http://home.planktoscope or home.planktoscope/ or http://home.planktoscope/ instead of home.planktoscope)
  • If you’re a new user, you no longer have to remember to type http://planktoscope.local:1880/ui in order to access the dashboard - instead, you should just open http://home.planktoscope/ (or http://planktoscope.local/ or http://192.168.4.1/ or etc.) in your web browser and follow the “Node-RED dashboard” link to open the dashboard!
  • If you have multiple PlanktoScopes (lucky you!) and you need to access them by SSH or FileZilla without having to deal with security warnings about different SSH host fingerprints on home.planktoscope or planktoscope.local or 192.168.4.1 or 192.168.5.1, then you should use either the <machine-name>.planktoscope domain name or the planktoscope-<machine-name>.local domain name - for example, ssh pi@metal-slope-23501.planktoscope, or ssh pi@planktoscope-plant-range-10581.local - so that each PlanktoScope is accessed at a domain name with a different host fingerprint.
• A web-browser computer administration panel called Cockpit is now available. It's accessible by a link from the PlanktoScope's landing page.
  • This is provided to enable users to manage system settings easily (e.g. changing the password from copepode to something more secure), and to help users with troubleshooting problems.
  • Cockpit provides a web browser-based SSH terminal for remotely running commands, at http://ethan-dev-1.planktoscope.d.prakashlab.fluitans.org/admin/cockpit/system/terminal.
  • Cockpit also allows you to inspect the logs of any system service, such as the autohotspot service.
• A web-browser system file manager for the PlanktoScope’s entire SD card is now available. It's accessible by a link from the PlanktoScope's landing page.
  • Among many other things, it allows you to create, edit, rename, and delete text files.
  • This way, you may be able to fix a broken system without having to re-flash your SD card - at least for certain kinds of breakage.
• An offline version of the PlanktoScope documentation is now available. It's accessible by a link from the PlanktoScope's landing page.
  • This way, you can check the PlanktoScope’s documentation even if you don’t have internet access!
• We are now using a system for running containerized apps called Docker for deploying various programs (e.g. the system file manager, the PlanktoScope documentation, and the landing page) as part of the PlanktoScope software distribution.
  • Docker provides a way to download and run containerized apps, and to upgrade and downgrade their versions. Containerized apps run in isolation, which makes them easier to troubleshoot, to update, and to uninstall cleanly. The isolation mechanism also helps us to limit the potential impact of any software bugs or security vulnerabilities. Most of the new apps we’re adding in this release (with the notable exception of Cockpit) are running as containerized apps.
  • We are using Docker as a tool to make it more safe, simple, and easy for you to upgrade your PlanktoScope’s software, and for you to download and run other apps on your PlanktoScope.
  • Note for advanced users: in this release we are running Docker in Swarm Mode, with the only member of the cluster being the Raspberry Pi itself; use of Swarm Mode should be considered an experimental feature, and we may stop using Swarm Mode in a future release (e.g. if we determine that the Python backend needs to be run as a privileged container in order to work in a container; see Add support --privileged when create service moby/swarmkit#1030 & PR #26618 inhibits access to /sys/firmware preventing common device detection in 64-bit kernels (aarch64) moby/moby#43419 for details).
• A web-browser Docker management panel called Portainer is now available. It's accessible by a link from the PlanktoScope's landing page.
  • Portainer is only provided for advanced users to inspect, manage, and troubleshoot containerized programs easily.
  • Our use of the Portainer dashboard should be considered an experimental feature, and in a future release we will probably switch to a different default dashboard for monitoring and troubleshooting Docker containerized applications.
  • If we switch to a different default dashboard, we may still provide an easy way for users to install Portainer on their PlanktoScopes - it just won’t be provided as part of the default software distribution provided by our SD card images. If you want to use Portainer and you want to a way to easily install Portainer in the future even if we remove it from our standard software distribution, please start a discussion on Github or in the #6-dev-software channel on the PlanktoScope Slack workspace!
• An experimental command-line tool called Forklift can now be run by advanced users on the PlanktoScope (e.g. via SSH or via Cockpit’s Terminal), for (re)configuring the set of Docker containerized applications running on the PlanktoScope. We will continue to improve Forklift in future releases of the PlanktoScope software distribution.
  • As we add more functionality to Forklift, and as we make the tool more stable, we plan for it to eventually become an open “app store” system, with a simple web browser interface allowing users to add third-party (Docker containerized) programs and/or to easily replicate their custom software configuration across multiple PlanktoScopes.
  • If you’re interested in trying out Forklift, or if you think Forklift might be useful for something you’re trying to do with PlanktoScope if certain features will be added to it, please start a discussion on Github or in the Use of Node Project to manage flows #6-dev-software channel on the PlanktoScope Slack workspace!
  • For now, users interested in trying out Forklift should have some experience with using Git from the command-line (just for basic operations like clone, pull, push, etc.) and with using Docker, since Forklift’s command-line workflow is designed to have some familiarity for Git and Docker users.

Changed:

• The dashboard (previously available at http://planktoscope.local:1880/ui) and the dashboard editor (previously available at http://planktoscope.local:1880/) have been moved to different URLs, accessible by links on the new landing page (see the note in the “Added” section about new URLs and the landing page). These changes are in preparation for other changes we will be making for the v2023.9.0 release.
  • This change will make it easier for you to simultaneously run multiple different dashboards, and/or multiple different versions of the same dashboard, in the future. If/when we release a new version of the dashboard with a different layout or a different user experience in the future, we’ll still be able to keep the previous version of the dashboard running on every PlanktoScope for a year or two in order to make the transition more gradual for you.
  • Although the new URLs are longer, you’ll still be able to access them easily by just going to http://planktoscope.local/ in your browser and clicking a link on the landing page loaded by that URL.
• The file gallery (previously available at http://planktoscope.local/) has now been renamed to the dataset file manager and moved to a different URL, accessible by a link on the new landing page. It now allows you to delete folders, preview images, edit text files, download folders as ZIP files, rename folders, and perform other common tasks.
  • This way, you no longer have to install FileZilla in order to copy or delete any of your datasets!
• Previously, the PlanktoScope would stay connected to a wifi network even if it didn’t have internet access on that network. Now, if it connects to a wifi network but can’t get internet access on that network (specifically, if it’s unable to find google.com), then it will revert to its default wifi hotspot behavior by creating its own isolated wifi network for your computer or phone to connect to.
  • This change allows you to recover the ability to connect to your PlanktoScope over wifi in order to fix the settings of your PlanktoScope if you tried to make it connect to the internet over a wifi network which isn’t allowing it to connect to the internet. This is useful if, for example, you’re trying to access the PlanktoScope over a mesh VPN such as ZeroTier or Tailscale.
• Previously, PlanktoScope machine names were generated as gibberish phrases like “Babaxio-Detuiau”, which were often difficult for people to pronounce and could be difficult for people to type. Now, PlanktoScope machine names are generated as a sequence of two words (selected from two lists of source words) and a number (up to 5 digits long), resulting in names like “metal-slope-23501”, “conscious-pocket-1684”, and “plant-range-10581”. This release only has word lists to generate machine names in US English, but we have future plans to add a configuration option in the PlanktoScope software to change the language used for generating machine names. If you know a language besides US English and you’d like to help us add support for generating machine names in that language, please start a discussion on Github or in the #6-dev-software channel on the PlanktoScope Slack workspace!
  • This change makes it easier for you to tell someone else which PlanktoScope to connect to if multiple PlanktoScope wifi hotspot networks are available nearby.
  • If you prefer the previous naming scheme, please start a discussion on Github or send a message in the #6-dev-software channel on the PlanktoScope Slack workspace asking us to make a pair of word lists which you can select for generating names in the previous style.
• Previously, every PlanktoScope had a hostname of planktoscope (which you could see if you accessed the PlanktoScope over SSH). Now, every PlanktoScope has a different hostname in the format planktoscope-<machine-name>, e.g. planktoscope-metal-slope-23501 or planktoscope-plant-range-10581.
  • This change makes it easier for you to tell which PlanktoScope you are connected to over SSH if you have SSH access to multiple PlanktoScopes.
  • This change was also necessary for us to make the PlanktoScope accessible via planktoscope-<machine-name>.local over any network which the PlanktoScope is connected to. See the note in the "Added" section above for more details.
• The PlanktoScope documentation pages have been redone in a new system, and improved in various ways.
  • Thanks to Sebastian Wendel (@sourceindex) for starting this work, and for contributing lots of documentation!
• The process of generating SD card images is now mostly automated, and installation of software dependencies is now more reproducible.
  • This change will make it easier for us to deliver software updates and bug fixes to you in a more timely manner, and it will help improve the reproducibility of the SD card images we upload for you to use.

Deprecated:

• The Node-RED flow editor and dashboard should no longer be accessed on port 1880 (e.g. using http://planktoscope.local:1880/ and http://planktoscope.local:1880/ui); in the v2024.7.0 release they will no longer be accessible on that port. Starting with this release, you should access them by clicking the corresponding links in the dashboard (which is at e.g. http://home.planktoscope/ or http://planktoscope.local/)
  • Note: in this prerelease, the Node-RED dashboard has not yet been updated to use the new URLs for the camera preview and object segmentation preview streams. The dashboard will display a message claiming that the Python backend is broken, but this may not necessarily be true. You should probably be able to access those preview streams directly via the microscope preview and segmented object preview links in the PlanktoScope’s landing page, in the “Network APIs” section near the bottom of the page. This problem will be fixed before the v2023.9.0-beta.0 prerelease, so that the Node-RED dashboard will once again display the image streams correctly.

Removed:

• The planktoscope.local domain name now only works from devices connected directly to the PlanktoScope as a wifi hotspot or with an Ethernet cable - in other words, it no longer works in any other networks (e.g. LANs or mesh VPNs) which the PlanktoScope is on. On those other networks, you should use planktoscope-<machine-name>.local instead of planktoscope.local.
  • This change is needed in order to support the planktoscope-<machine-name.local domain name on all networks which the PlanktoScope is on, so that your computer can access multiple PlanktoScopes connected to the same network (e.g. a LAN or a mesh VPN) using mDNS names (any domain name ending in .local is an mDNS name). This is because all of those PlanktoScopes previously competed with each other for the planktoscope.local name, and there is no consistent way to determine which PlanktoScope would actually be accessed via planktoscope.local; but if each PlanktoScope takes its own unique domain name ending in .local, then each one can be accessed in a predictable way.
  • We are only able to keep planktoscope.local working for devices connected directly to the PlanktoScope because avahi-daemon and avahi-publish are implemented in such a way that planktoscope.local can only be resolved reliably to a single IP address if planktoscope-<machine-name> is set as the PlanktoScope’s hostname (see register host without reverse entry in /etc/avahi/hosts avahi/avahi#40 for technical details).
  • Unfortunately, it’s difficult to make DNS and mDNS work consistently and correctly in all settings and devices. If our changes in this release cause problems for you, please start a discussion in the #6-dev-software channel on Slack or on our Github Discussions forum and we’ll try to help. You might also enjoy this haiku about troubleshooting problems in computer networks: https://blog.cloudflare.com/content/images/2020/05/image1-8.png

Fixed:

• The adafruit-blinka and adafruit-platformdetect software dependencies are now updated to their latest version. As a result, the PlanktoScope software should not crash just because it is being used with recent versions of the Adafruit Stepper HAT.
  • This fix removes the need for you to connect to the PlanktoScope over SSH to manually run the pip3 install --upgrade adafruit-blinka adafruit-platformdetect command before your PlanktoScope can work properly.

---

This discussion was created from the release Software: Sunray v2023.9.0-alpha.0.