From 95d783938d3aec7dad3e4b0e45f37031724c25de Mon Sep 17 00:00:00 2001 From: Lucinda <156825081+LucindaLanoy@users.noreply.github.com> Date: Tue, 17 Mar 2026 12:41:21 +0100 Subject: [PATCH] Update to markdown format --- README.org => README.md | 164 ++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 81 deletions(-) rename README.org => README.md (56%) diff --git a/README.org b/README.md similarity index 56% rename from README.org rename to README.md index dfa286a..f4204f5 100644 --- a/README.org +++ b/README.md @@ -1,8 +1,8 @@ -#+TITLE: Cyz2Json - Convert flow cytometry CYZ files to JSON format. +# Cyz2Json - Convert flow cytometry CYZ files to JSON format. -#+HTML: Build Cyz2Json +[![Test Cyz2Json after changes](https://github.com/OBAMANEXT/cyz2json/actions/workflows/Check-Cyz2Json.yml/badge.svg)](https://github.com/OBAMANEXT/cyz2json/actions/workflows/Check-Cyz2Json.yml) -* Introduction +## Introduction This program converts flow cytometry data stored in a CytoBuoy CYZ format file to JSON format. @@ -11,154 +11,156 @@ Although CYZ files work with the Cytobuoy supplied CytoClus program, we want to access the data from a wider set of data science tooling including Python and R. We therefore convert to JSON. -This program uses the [[https://github.com/Cytobuoy/CyzFile-API][CytoBuoy CyzFile API]] and targets the Microsoft +This program uses the [CytoBuoy CyzFile API](https://github.com/Cytobuoy/CyzFile-API) and targets the Microsoft .NET 8 framework on Windows, macOS or Linux. -* Build instructions +## Build instructions -** Build on the PC +### Build on the PC -** Installation +#### Installation The first step is to install the Microsoft dotnet runtime version 8. Microsoft provides detailed instructions at https://learn.microsoft.com/en-us/dotnet/core/install/. For example, installation on Linux Ubuntu 22.04 is as follows: -#+begin_src bash +``` sudo apt-get update && sudo apt-get install -y dotnet-sdk-8.0 -#+end_src +``` -You can test that =dotnet= is installed on your system by typing: +You can test that `dotnet` is installed on your system by typing: -#+begin_src bash +``` dotnet --info -#+end_src +``` -To generate the version number, you will need to install the =nbgv= tool. +To generate the version number, you will need to install the `nbgv` tool. You can install this tool by typing: -=dotnet tool install -g nbgv= +`dotnet tool install -g nbgv` -The =Cyz2Json= program can then be compiled by cloning the project -from GitHub and typing =dotnet build -p:OutputPath=./bin/=. -The default version is the one given by the [[https://github.com/OBAMANEXT/cyz2json/blob/main/Cyz2Json/Cyz2Json.csproj/][Cyz2Json.csproj]] file (unless passed directly during build via -p:Version), the assembly informal version is "Cyz2Json-the current Version-the latest Git SHA". +The `Cyz2Json` program can then be compiled by cloning the project +from GitHub and typing `dotnet build -p:OutputPath=./bin/`. +The default version is the one given by the [Cyz2Json.csproj](https://github.com/OBAMANEXT/cyz2json/blob/main/Cyz2Json/Cyz2Json.csproj/) file (unless passed directly during build via -p:Version), the assembly informal version is "Cyz2Json-the current Version-the latest Git SHA". -** Build with GitHub actions +### Build with GitHub actions -The [[https://github.com/OBAMANEXT/cyz2json/blob/main/.github/workflows/Release-Cyz2Json.yml/][Release-Cyz2Json.yml]] workflow can be triggered manually to build and publish a release of the tool with cross platform binaries. A tag is required to be used as the release and tool version (replacing the default one), and a reason for the release can be given as well though it is not mandatory. +The [Release-Cyz2Json.yml](https://github.com/OBAMANEXT/cyz2json/blob/main/.github/workflows/Release-Cyz2Json.yml/) workflow can be triggered manually to build and publish a release of the tool with cross platform binaries. A tag is required to be used as the release and tool version (replacing the default one), and a reason for the release can be given as well though it is not mandatory. You will find the result in : https://github.com/OBAMANEXT/cyz2json/actions and release in : https://github.com/OBAMANEXT/cyz2json/releases To install from release, download the release for you platfrom, and unpack on your folder choice. -*** On MacOSX +#### On MacOSX You will need to bypass security You have 2 possibilities : using xattr command. for exmaple: -#+begin_src bash +``` xattr -d com.apple.quarantine /opt/cytosense_to_ecotaxa_pipeline/bin/* -#+end_src +``` or if you do not want to use xattr you need to go in Systems Settings > Security & Privacy > General and allow the app to be opened for the 10+ libraries. You need to relauch sevaral time the cyz2json binary to acheive all neccessary permissions. -* Running the program +## Running the program + +### Options The program has a set of arguments and options that can be passed to fit the kind of data you need : -+ =input= the cyz file to be converted, mandatory, -+ =--output= the JSON output file, -+ =--raw= to not apply the moving weighted average filtering algorithm to pulse shapes, export raw, unsmoothed data, -+ =-V= display version information, -+ =--metadatagreedy= save all possible measurement settings with your file, default = true, -+ =--imaging-set-information= to export set information for imaging, -+ =--imaging-set-definition= to override the definitions stored in the file, only if --imaging-set-information is true for file with set definitions, -+ =--image-processing= to perform cropping and image processing during the export of the image, default = false, this comes along with several other arguments to adjust the processing options : -+ =--image-processing-threshold= the minimum pixel value difference from the background to be considered an object, default = 9, -+ =--image-processing-erosion-dilation= the size of the erosion/dilation filter to apply after thresholding, default = 1, -+ =--image-processing-bright-field-correction= correct the image for variation in the lighting, default = true, -+ =--image-processing-margin-base= add a marging of this many pixels around the detected object, default = 25, -+ =--image-processing-margin-percentage= add an extra margin that is a percentage of the size of the detected object, default = 10, -+ =--image-processing-extend-object-detection= when seperate objects are detected close (in the margin) of the main object then extend the rectangle to include these objects as well, default = true. +- `input` the cyz file to be converted, mandatory, +- `--output` the JSON output file, +- `--raw` to not apply the moving weighted average filtering algorithm to pulse shapes, export raw, unsmoothed data, +- `-V` display version information, +- `--metadatagreedy` save all possible measurement settings with your file, default = true, +- `--imaging-set-information` to export set information for imaging, +- `--imaging-set-definition` to override the definitions stored in the file, only if --imaging-set-information is true for file with set definitions, +- `--image-processing` to perform cropping and image processing during the export of the image, default = false, this comes along with several other arguments to adjust the processing options : +- `--image-processing-threshold` the minimum pixel value difference from the background to be considered an object, default = 9, +- `--image-processing-erosion-dilation` the size of the erosion/dilation filter to apply after thresholding, default = 1, +- `--image-processing-bright-field-correction` correct the image for variation in the lighting, default = true, +- `--image-processing-margin-base` add a marging of this many pixels around the detected object, default = 25, +- `--image-processing-margin-percentage` add an extra margin that is a percentage of the size of the detected object, default = 10, +- `--image-processing-extend-object-detection` when seperate objects are detected close (in the margin) of the main object then extend the rectangle to include these objects as well, default = true. + +### Linux The program can be run, on Linux, for example, as follows: -#+begin_src bash +``` dotnet bin/Cyz2Json.dll input.cyz --output output.json -#+end_src +``` Running with executable loaded from github -#+begin_src bash +``` export cyzversion=v0.0.49 rm -rf cyz2json/* && curl -L -o cyz2json/cyz2json.zip https://github.com/ecotaxa/cyz2json/releases/download/$cyzversion/cyz2json-ubuntu-latest.zip && pushd cyz2json && unzip cyz2json.zip && popd && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:. && cyz2json/Cyz2Json ./Deployment\ 1\ 2024-07-18\ 21h12.cyz --metadatagreedy=false --output ./Deployment\ 1\ 2024-07-18\ 21h12.json -#+end_src +``` -On Windows +### On Windows -#+begin_src bash +``` Cyz2Json.EXE input.cyz --output output.json -#+end_src +``` -On MacOSX +### On MacOSX bypass security with xattr command -#+begin_src bash +``` xattr -d com.apple.quarantine ./cyz2json-macos-latest/* -#+end_src +``` 2 methods to run -#+begin_src bash +``` dotnet bin/Cyz2Json.dll input.cyz --output output.json -#+end_src +``` or -#+begin_src bash +``` Cyz2Json.exe input.cyz --output output.json -#+end_src +``` -* Bulk file conversion +### Bulk file conversion We often find ourselves needing to undertake a bulk conversion of a large set of files. The following shell script can help: -#+begin_src bash +``` #!/usr/bin/env bash for file in *.cyz do dotnet ~/git/Cyz2Json/bin/Cyz2Json.dll "$file" --output "$file".json done -#+end_src +``` -* Processing CYZ JSON files using Python +## Processing CYZ JSON files -To load a JSON flow cytometry data file into Python: +### Using Python -#+begin_src python +To load a JSON flow cytometry data file into Python: +``` import json data = json.load(open("pond.json", encoding="utf-8-sig")) print(data) - -#+end_src +``` Note the need to explicitly specify the encoding to deal with Microsoft's and Python's differences in interpretation of the standards regarding byte order marks in UTF-8 files. -* R example +### R example To load a JSON flow cytometry data file into R: -#+begin_src R - +``` library(jsonlite) json_data <- fromJSON("pond.json") print(json_data) +``` -#+end_src - -* Notes +## Notes -** On images +### On images If a CYZ file contains images, we currently base64 encode them and include them inline in the JSON. This is costly in terms of disk and @@ -166,34 +168,34 @@ only time will tell if it is a sensible design decision. A future enhancement would be to include a flag that writes the files out as JPEG images. -Note that the images are un-cropped by default. The =--image-processing= option can now be used to process and crop images when exporting. +Note that the images are un-cropped by default. The `--image-processing` option can now be used to process and crop images when exporting. Initially, the tool didn't support cropping as an expedience to allow cross platform operation (The CyzFile-API only supports cropping on Windows platforms). -** On the API +### On the API -Cyz2json is using a dll from the [[https://github.com/Cytobuoy/CyzFile-API/][CyzFile-API]], at this moment, using version 1.6.1.0. +Cyz2json is using a dll from the [CyzFile-API](https://github.com/Cytobuoy/CyzFile-API), at this moment, using version 1.6.1.0. -To update you can use the [[https://github.com/OBAMANEXT/cyz2json/blob/main/.github/workflows/API-update-from-tags.yml/][API-update-from-tags.yml]] workflow which fetches the latest version of the API (in tags), you can run it manually or let it work on schedule (15th of each month), in that case, it compares the latest tag with the current version in this repo to choose whether to update or not. +To update you can use the [API-update-from-tags.yml](https://github.com/OBAMANEXT/cyz2json/blob/main/.github/workflows/API-update-from-tags.yml/) workflow which fetches the latest version of the API (in tags), you can run it manually or let it work on schedule (15th of each month), in that case, it compares the latest tag with the current version in this repo to choose whether to update or not. You can also update manually without the workflow, to do so : -+ download the latest net8.0 release from https://github.com/Cytobuoy/CyzFile-API/releases, -+ unzip the downloaded file, -+ copy the files from the dll folder to the CyzFile folder, -+ rebuild the project. +- download the latest net8.0 release from https://github.com/Cytobuoy/CyzFile-API/releases, +- unzip the downloaded file, +- copy the files from the dll folder to the CyzFile folder, +- rebuild the project. -* Acknowledgements +## Acknowledgements My thanks to the following organisations for supporting this work: -[[https://www.turing.ac.uk/][The Alan Turing Institute]]. +[The Alan Turing Institute](https://www.turing.ac.uk/). -[[https://www.finmari-infrastructure.fi/][The Finnish Marine Research Infrastructure consortium (FINMARI)]] +[The Finnish Marine Research Infrastructure consortium (FINMARI)](https://www.finmari-infrastructure.fi/) -[[https://www.cefas.co.uk][The Centre for Environment, Fisheries and Aquaculture Science (Cefas)]] +[The Centre for Environment, Fisheries and Aquaculture Science (Cefas)](https://www.cefas.co.uk) -[[https://www.cytobuoy.com/][CytoBuoy b.v.]] +[CytoBuoy b.v.](https://www.cytobuoy.com/) I am grateful to Rob Lievaart at Cytobuoy for providing the libraries, code and examples on which this software is based. The CyzFile-API is @@ -201,9 +203,9 @@ licensed under the terms described in CyzFile-API_LICENSE.TXT. Thanks to Eric Payne at Cefas for Visual Studio wizardry. -* Disclaimers +## Disclaimers -The [[https://obama-next.eu/][OBAMA-NEXT]] project has been approved under +The [OBAMA-NEXT](https://obama-next.eu/) project has been approved under HORIZON-CL6-2022-BIODIV-01-01: Observing and mapping biodiversity and ecosystems, with particular focus on coastal and marine ecosystems (Grant Agreement 101081642). Funded by the European Union and UK