v0.22.82

This release brings major improvements to AliECS, including a new trivial HTTP API in Apricot for shell scripts and users to access real time inventory information via HTTP (i.e. a plain curl call from anywhere), improvements and additions to the template system, and miscellaneous bug fixes. It also introduces workflow template hiding and dynamic input widget hints, used in the GUI to greatly improve workflow discoverability and usability. A new Integrated Services API is also provided to track the status of DCS, DD scheduler and ODC, and the plugin system itself now ensures that all integration plugins promptly reconnect in case of interruption. Finally, this release adds integration facilities for the O² Bookkeeping system.

• Apricot trivial configuration endpoint (returns inventory on http://:47188/inventory/flps):

  • [apricot] Apricot's new Http service is separated from the already existing service.
  • [apricot] First uncomplete attempt at creating a Web Api for Apricot.
  • [apricot] GetHostInventory returns an error variable as well as the hosts list.
  • [apricot] GetHostInventory is called when needed, but isn't returning a result.
  • [apricot] GetHostInventory is better optimised.
  • [apricot] Host inventory is retrieved by http service, and converted to json or plain text.
  • [apricot] Http service should 'gracefully' shutdown on exit signals.
  • [apricot] Protofile takes the new GetHostInventory function into account.
  • [apricot] Result of make generate after modifying the Protofile.
  • [apricot] Temporary fix for having a working make all. No solution found to call for Service inside of ServiceHttp.
  • [apricot] Temporary setup for tests.
  • [apricot] The Http Service is linked to Apricot and is set up to be able to handle the inventory.
  • [apricot] The HTTP Server is shut down using the Shutdown() function associated to it.
  • [apricot] The function GetHostInventory is 'registered' in the configuration service.
  • [apricot] The project builds after 'registering' the function GetHostInventory.
  • [apricot] The HTTP Server returns the flps in either text or json depending on the format requested.
  • [apricot] The HTTP Service is functional for FLPs and detector FLPs, for JSON and text format.
  • [apricot] The Http Service is working after rebase.
  • [apricot] The HTTP service can return FLPs associated to a requested detector.
  • [apricot] The HTTP Service returns keys only once when it is both a key and a key prefix.
  • [apricot] service.go doesn't have any unused imports.
  • [apricot] Tab indentation is now replaced by spaces.

• Workflow template hiding and dynamic input widget hints:

  • [coconut] Update templ list with --all-workflows flag
  • [core] Document VarSpec entries and make string the default type
  • [core] Set default varspec type to string
  • [protos] Add VarSpec message extending WorkflowTemplateInfo
  • [repos] Prefer join over manual concatenation
  • [repos] Filter invalid repo dirs
  • [repos] Only list public workflows by default
  • [repos] Add missing "tasks"
  • [repos] Use path instead of filePath for repo identifier
  • [repos] Care for missing trailing slash in repolist
  • [repos] Move convenience functions to repoutils.go
  • [repos] Add functions to parse public workflow variable info
  • [repos] Remove AllowedKeyValues from VarSpec
  • [repos] Fix case whoops
  • [server] Update default widget and vartype values
  • [server|coconut] Don't return variable information for coconut requests
  • [server] Update vartype enum converter
  • [server] Fix bool number in vartype enum
  • [server] Include public variable info in GetWorkflowTemplates

• Integrated Services API:

  • [coconut] Print Integrated Services information on coconut info
  • [core] Expand plugin system with access functions for API calls
  • [core] Integrated Services API call
  • [core] Document IntegratedServices API values

• Integration plugin improvements:

  • [core] Reconnect all plugins continuously
  • [core] Use keepalive with backoff for gRPC clients
  • [core] Regenerate protofiles

• Workflow template system improvements and fixes:

  • [core] Ensure all contexts get the environment_id variable
  • [core] Report a channel mismatch error on bad channel configuration
  • [core] Add task information in channel config generator error
  • [core] CreateEnvironment avoid overwrite error on failed deployment
  • [core] Ensure task class defaults are template-processed
  • [task] Consistent TaskClassIdentifier string construction

• Bookkeeping system integration:

  • [core] Repo getDefaultRevision on refresh
  • [core] Add Bookkeeping integration
  • [core] Use Bookkeeping API to CreateRun,CreateLog and UpdateRun
  • [core] Add CreateFLP from Bookkeeping API
  • [core] Create FLPs after the CreateRun
  • [core] Add config entry for Bookkeeping and fix format
  • [core] Retrieve JWT using the request redirect
  • [core] Fix bookkeeping calls and version
  • [core] Update bookkeeping version
  • [core] Fix wrong identifiers

• Miscellaneous:

  • [coconut] Keep stream open after Environment error
  • [common] Introduce runtype.RunType enum, matches DCS Protobuf RunType
  • [core] Support multivalued Mesos attributes split by comma
  • [core] Generate strings for system.ID enum values with enumer
  • [core] Report error of failed environment through stream
  • [core] Report teardown complete at the end of operation
  • [occ] Allow building with Protobuf v3.14.0

v0.22.81

This release brings major improvements to AliECS, including a new trivial HTTP API in Apricot for shell scripts and users to access real time inventory information via HTTP (i.e. a plain curl call from anywhere), improvements and additions to the template system, and miscellaneous bug fixes. It also introduces workflow template hiding and dynamic input widget hints, used in the GUI to greatly improve workflow discoverability and usability. A new Integrated Services API is also provided to track the status of DCS, DD scheduler and ODC, and the plugin system itself now ensures that all integration plugins promptly reconnect in case of interruption. Finally, this release adds integration facilities for the O² Bookkeeping system.

• Apricot trivial configuration endpoint (returns inventory on http://:47188/inventory/flps):

  • [apricot] Apricot's new Http service is separated from the already existing service.
  • [apricot] First uncomplete attempt at creating a Web Api for Apricot.
  • [apricot] GetHostInventory returns an error variable as well as the hosts list.
  • [apricot] GetHostInventory is called when needed, but isn't returning a result.
  • [apricot] GetHostInventory is better optimised.
  • [apricot] Host inventory is retrieved by http service, and converted to json or plain text.
  • [apricot] Http service should 'gracefully' shutdown on exit signals.
  • [apricot] Protofile takes the new GetHostInventory function into account.
  • [apricot] Result of make generate after modifying the Protofile.
  • [apricot] Temporary fix for having a working make all. No solution found to call for Service inside of ServiceHttp.
  • [apricot] Temporary setup for tests.
  • [apricot] The Http Service is linked to Apricot and is set up to be able to handle the inventory.
  • [apricot] The HTTP Server is shut down using the Shutdown() function associated to it.
  • [apricot] The function GetHostInventory is 'registered' in the configuration service.
  • [apricot] The project builds after 'registering' the function GetHostInventory.
  • [apricot] The HTTP Server returns the flps in either text or json depending on the format requested.
  • [apricot] The HTTP Service is functional for FLPs and detector FLPs, for JSON and text format.
  • [apricot] The Http Service is working after rebase.
  • [apricot] The HTTP service can return FLPs associated to a requested detector.
  • [apricot] The HTTP Service returns keys only once when it is both a key and a key prefix.
  • [apricot] service.go doesn't have any unused imports.
  • [apricot] Tab indentation is now replaced by spaces.

• Workflow template hiding and dynamic input widget hints:

  • [coconut] Update templ list with --all-workflows flag
  • [core] Document VarSpec entries and make string the default type
  • [core] Set default varspec type to string
  • [protos] Add VarSpec message extending WorkflowTemplateInfo
  • [repos] Prefer join over manual concatenation
  • [repos] Filter invalid repo dirs
  • [repos] Only list public workflows by default
  • [repos] Add missing "tasks"
  • [repos] Use path instead of filePath for repo identifier
  • [repos] Care for missing trailing slash in repolist
  • [repos] Move convenience functions to repoutils.go
  • [repos] Add functions to parse public workflow variable info
  • [repos] Remove AllowedKeyValues from VarSpec
  • [repos] Fix case whoops
  • [server] Update default widget and vartype values
  • [server|coconut] Don't return variable information for coconut requests
  • [server] Update vartype enum converter
  • [server] Fix bool number in vartype enum
  • [server] Include public variable info in GetWorkflowTemplates

• Integrated Services API:

  • [coconut] Print Integrated Services information on coconut info
  • [core] Expand plugin system with access functions for API calls
  • [core] Integrated Services API call
  • [core] Document IntegratedServices API values

• Integration plugin improvements:

  • [core] Reconnect all plugins continuously
  • [core] Use keepalive with backoff for gRPC clients
  • [core] Regenerate protofiles

• Workflow template system improvements and fixes:

  • [core] Ensure all contexts get the environment_id variable
  • [core] Report a channel mismatch error on bad channel configuration
  • [core] Add task information in channel config generator error
  • [core] CreateEnvironment avoid overwrite error on failed deployment
  • [core] Ensure task class defaults are template-processed
  • [task] Consistent TaskClassIdentifier string construction

• Bookkeeping system integration:

  • [core] Repo getDefaultRevision on refresh
  • [core] Add Bookkeeping integration
  • [core] Use Bookkeeping API to CreateRun,CreateLog and UpdateRun
  • [core] Add CreateFLP from Bookkeeping API
  • [core] Create FLPs after the CreateRun
  • [core] Add config entry for Bookkeeping and fix format
  • [core] Retrieve JWT using the request redirect
  • [core] Fix bookkeeping calls and version
  • [core] Update bookkeeping version
  • [core] Fix wrong identifiers

• Miscellaneous:

  • [common] Introduce runtype.RunType enum, matches DCS Protobuf RunType
  • [core] Support multivalued Mesos attributes split by comma
  • [core] Generate strings for system.ID enum values with enumer

v0.22.80

This release brings major improvements to AliECS, including a new trivial HTTP API in Apricot for shell scripts and users to access real time inventory information via HTTP (i.e. a plain curl call from anywhere), improvements and additions to the template system, and miscellaneous bug fixes. It also introduces workflow template hiding and dynamic input widget hints, used in the GUI to greatly improve workflow discoverability and usability. A new Integrated Services API is also provided to track the status of DCS, DD scheduler and ODC, and the plugin system itself now ensures that all integration plugins promptly reconnect in case of interruption.

• Apricot trivial configuration endpoint (returns inventory on http://:47188/inventory/flps):

  • [apricot] Apricot's new Http service is separated from the already existing service.
  • [apricot] First uncomplete attempt at creating a Web Api for Apricot.
  • [apricot] GetHostInventory returns an error variable as well as the hosts list.
  • [apricot] GetHostInventory is called when needed, but isn't returning a result.
  • [apricot] GetHostInventory is better optimised.
  • [apricot] Host inventory is retrieved by http service, and converted to json or plain text.
  • [apricot] Http service should 'gracefully' shutdown on exit signals.
  • [apricot] Protofile takes the new GetHostInventory function into account.
  • [apricot] Result of make generate after modifying the Protofile.
  • [apricot] Temporary fix for having a working make all. No solution found to call for Service inside of ServiceHttp.
  • [apricot] Temporary setup for tests.
  • [apricot] The Http Service is linked to Apricot and is set up to be able to handle the inventory.
  • [apricot] The HTTP Server is shut down using the Shutdown() function associated to it.
  • [apricot] The function GetHostInventory is 'registered' in the configuration service.
  • [apricot] The project builds after 'registering' the function GetHostInventory.
  • [apricot] The HTTP Server returns the flps in either text or json depending on the format requested.
  • [apricot] The HTTP Service is functional for FLPs and detector FLPs, for JSON and text format.
  • [apricot] The Http Service is working after rebase.
  • [apricot] The HTTP service can return FLPs associated to a requested detector.
  • [apricot] The HTTP Service returns keys only once when it is both a key and a key prefix.
  • [apricot] service.go doesn't have any unused imports.
  • [apricot] Tab indentation is now replaced by spaces.

• Workflow template hiding and dynamic input widget hints:

  • [coconut] Update templ list with --all-workflows flag
  • [core] Document VarSpec entries and make string the default type
  • [core] Set default varspec type to string
  • [protos] Add VarSpec message extending WorkflowTemplateInfo
  • [repos] Prefer join over manual concatenation
  • [repos] Filter invalid repo dirs
  • [repos] Only list public workflows by default
  • [repos] Add missing "tasks"
  • [repos] Use path instead of filePath for repo identifier
  • [repos] Care for missing trailing slash in repolist
  • [repos] Move convenience functions to repoutils.go
  • [repos] Add functions to parse public workflow variable info
  • [repos] Remove AllowedKeyValues from VarSpec
  • [repos] Fix case whoops
  • [server] Update default widget and vartype values
  • [server|coconut] Don't return variable information for coconut requests
  • [server] Update vartype enum converter
  • [server] Fix bool number in vartype enum
  • [server] Include public variable info in GetWorkflowTemplates

• Integrated Services API:

  • [coconut] Print Integrated Services information on coconut info
  • [core] Expand plugin system with access functions for API calls
  • [core] Integrated Services API call
  • [core] Document IntegratedServices API values

• Integration plugin improvements:

  • [core] Reconnect all plugins continuously
  • [core] Use keepalive with backoff for gRPC clients
  • [core] Regenerate protofiles

• Workflow template system improvements and fixes:

  • [core] Ensure all contexts get the environment_id variable
  • [core] Report a channel mismatch error on bad channel configuration
  • [core] Add task information in channel config generator error
  • [core] CreateEnvironment avoid overwrite error on failed deployment
  • [core] Ensure task class defaults are template-processed
  • [task] Consistent TaskClassIdentifier string construction

• Miscellaneous:

  • [common] Introduce runtype.RunType enum, matches DCS Protobuf RunType
  • [core] Support multivalued Mesos attributes split by comma
  • [core] Generate strings for system.ID enum values with enumer

v0.22.3

This patch release includes a single critical fix for OCC components to support Protobuf v3.15.8, as well as a minor addition to workflow template parsing that has no impact on current workflow templates.

DEPRECATION NOTICE: starting with AliECS v0.21.0 the o2-aliecs-odc-shim component is deprecated and expected to be removed in the near future. It is still accessible in readout-dataflow by setting odcshim_enabled to true. The existing variable odc_enabled, which was previously used to enable/disable o2-aliecs-odc-shim, now controls the new implementation.

• [common] Support !public tags in StringWrapMap
• [misc] Remove developer docs from mkdocs
• [occ] Fix OCC build issue with Protobuf v3.15.8

v0.22.2

This patch release includes integration related fixes, including a fix for an AliECS core crash.

DEPRECATION NOTICE: starting with AliECS v0.21.0 the o2-aliecs-odc-shim component is deprecated and expected to be removed in the near future. It is still accessible in readout-dataflow by setting odcshim_enabled to true. The existing variable odc_enabled, which was previously used to enable/disable o2-aliecs-odc-shim, now controls the new implementation.

• [core] DCS integration: only run EOR during Cleanup if needed
• [core] Poll for DD scheduler partition state CONFIGURED/TERMINATED
• [core] Fix issue with START-STOP-START causing a core crash

v0.22.1

This patch release brings some DCS related fixes to AliECS.

DEPRECATION NOTICE: starting with AliECS v0.21.0 the o2-aliecs-odc-shim component is deprecated and expected to be removed in the near future. It is still accessible in readout-dataflow by setting odcshim_enabled to true. The existing variable odc_enabled, which was previously used to enable/disable o2-aliecs-odc-shim, now controls the new implementation.

• [core] Bump DCS client to latest protofile
• [core] Improve DCS client log output

v0.22.0

This release brings major improvements to AliECS, including support for subworkflow template inclusion, improvements and additions to the template system, and miscellaneous bug fixes.

DEPRECATION NOTICE: starting with AliECS v0.21.0 the o2-aliecs-odc-shim component is deprecated and expected to be removed in the near future. It is still accessible in readout-dataflow by setting odcshim_enabled to true. The existing variable odc_enabled, which was previously used to enable/disable o2-aliecs-odc-shim, now controls the new implementation.

• Apricot and template system calls:

  • [apricot][core] GetDetectorForHost in Apricot + template system
  • [apricot] Implement GetCRUCardsForHost
  • [apricot] GetEndpointsForCRUCard
  • [configuration] Extend service with GetCRUCardsForHost
  • [configuration] GetEndpointsForCRUCard

• Subworkflow template inclusion:

  • [core] Subworkflow template expression resolution in repos package
  • [core] Subworkflow include implementation (includeRole)
  • [core] Fix typo in subworkflow template identifier resolution
  • [core] Ensure a disabled include isn't processed
  • [core] Ensure the included role defaults/vars are reprocessed

• Integration:

  • [core] Update ODC client to latest protofile
  • [core] Update DCS client to latest protofile
  • [core] DCS SOR/EOR response processing
  • [core] DCS Cleanup template function, performs EOR regardless if needed

• Miscellaneous:

  • [coconut] Fix typo
  • [core] Limit InfoLogger message size to 1024 Bytes
  • [core] Allow templating of FairMQ channel rateLogging
  • [executor] Drop stdout messages if no piping selected

v0.21.80

This release brings major improvements to AliECS, including support for subworkflow template inclusion, improvements to the template system and miscellaneous bug fixes.

DEPRECATION NOTICE: starting with AliECS v0.21.0 the o2-aliecs-odc-shim component is deprecated and expected to be removed in the near future. It is still accessible by setting odcshim_enabled to true. The existing variable odc_enabled, which was previously used to enable/disable odc-shim, now controls the new implementation.

• Subworkflow template inclusion:

  • [core] Subworkflow template expression resolution in repos package
  • [core] Subworkflow include implementation (includeRole)
  • [core] Fix typo in subworkflow template identifier resolution

• Integration:

  • [core] Update ODC client to latest protofile

• Miscellaneous:

  • [core] Limit InfoLogger message size to 1024 Bytes
  • [core] Allow templating of FairMQ channel rateLogging
  • [executor] Drop stdout messages if no piping selected

v0.21.2

This patch release for the v0.21 series fixes a core crash when deploying an environment from certain kinds of workflow templates.

• [core] Fix nil pointer dereference

v0.21.1

This patch release for the v0.21 series fixes a serious issue with task lifetime management and adds support for user-configurable task output redirection.

• [core] Fix empty roster issue
• [executor] Configurable task stdout logging
• [executor] Support parametrization of task log pipe