v0.20.0

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new microservice Apricot ("a processor and repository for configuration templates"). AliECS now supports integration plugins, a new kind of plugin that exposes callable functions to workflow templates, and a prototype DCS integration plugin is provided based on this new mechanism. Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• Integration plugins and DCS:

  • [core] Add new INVARIANT machine state for callRoles
  • [core] Support for integration plugins
  • [core] Support for callRoles as triggers for integration plugins
  • [core] Ensure environments with calls still update status correctly

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [core] Fix NewID template function signature
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.90

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new component Apricot (a processor and repository for configuration templates). Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.80

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments.

• Automatic state machine progression (environment auto driver) and event system

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

v0.19.2

This is a patch release that fixes several issues of varying severity that were discovered during Milestone Week 1.

Logging fixes:

• [common] Strip newlines from outbound InfoLogger messages
• [core] Reduce log verbosity
• [executor] stderr output of controlled processes now forwarded at DebugLevel
• [executor] Pipe controlled task stdout/stderr at TRACE level

Miscellaneous fixes:

• [common] Fix nil pointer crash when event is nil
• [core] Fix deadlock when unsubscribing from Wf state
• [core] Fix data race when gathering responses

v0.19.1

This is a patch release that fixes a cleanup issue on environment startup failure.

• [core] Fix clean up after failed creation of a new environment
• [misc] New script to create FLP-compatible LXD containers on demand
• [misc] Better wording for new root password prompt

v0.19.0

This release brings some performance and reliability improvements when terminating tasks. It also delivers some bug fixes, as well as an update to ODC integration.

Task termination:

• [executor] Tweak timeout system for the task kill procedure
• [executor] Always use the shell PGID as backup PID for killing
• [executor] Further tweak kill timeouts

ODC integration:

• [odcshim] Correctly handle early GetState calls

Miscellaneous:

• [core] Allow underscore in configuration path
• [misc] Gather more metrics in the test script
• [occ] Avoid using private namespace in gRPC
• [build] Toolchain version bump
• [build] gRPC, Protobuf version bump for OCC
• [build] Makefile output improvements on Mac

v0.18.90

This release brings some performance and reliability improvements when terminating tasks. It also delivers some bug fixes, as well as an update to ODC integration.

Task termination:

• [executor] Tweak timeout system for the task kill procedure
• [executor] Always use the shell PGID as backup PID for killing
• [executor] Further tweak kill timeouts

ODC integration:

• [odcshim] Correctly handle early GetState calls

Miscellaneous:

• [core] Allow underscore in configuration path
• [misc] Gather more metrics in the test script
• [occ] Avoid using private namespace in gRPC

v0.18.2

This is a patch release that fixes some inconsistent state reporting behavior when a task moves to ERROR during the STOP transition. It also fixes a related issue that caused the environment destroy operation to fail.

• [core] remove goroutine from transition_configure
• [core] handle workflow state error
• [core] Fix hanging force shutdown
• [executor] Improve executorcmd log output

v0.18.1

This is a patch release that fixes a crash triggered by a specific sequence of start-stop-teardown control commands with noncompliant tasks (readout-dataflow with DD and QC).

• [core] Always get the stateChanged after a transition

v0.18.0

This release brings an in-depth overhaul of the task manager component of the AliECS core, resulting in significant performance improvements. It also delivers numerous bug fixes, as well as integration improvements for the configuration system.

• Task manager refactor:

  • [core] draft of managerV2
  • [core] Rebase and add TriggerHooks
  • [core] Refactor scheduler into task.Manager
  • [core] Fix pending teardown with asynchronous ReleaseTasks
  • [core] remove mutex from TaskMan, use mutex for roster, classes and Task
  • [core] Remove obsolete warning from repo.Manager
  • [core] merge manager with managerv2
  • [core] fix discrepancy between task and environment states
  • [core] fix infinite loop due to wfState

• Configuration:

  • [core] Expose deployment constants (including consul_endpoint) to env
  • [core] Push environment_id as property to all tasks
  • [core] Only append --control-port parameter to FairMQ tasks
  • [occ] Always prefer OCC_CONTROL_PORT variable to --control-port param
  • [occ] Ensure the run number is always a uint32_t in the C++ interface

• Miscellaneous:

  • [build] Use latest FSM
  • [coconut] Remove usage of UUID
  • [core] Correct task ID generation
  • [core] Force shutdown reply when all processes are gone
  • [core] Make log output more relevant
  • [misc] New benchmark script
  • [misc] Test script workflow template default to @master
  • [odcshim] Add support for ODC partitioning