v0.17.82

This release brings an in-depth overhaul of the task manager component of the AliECS core, resulting in significant performance improvements. It also delivers numerous bug fixes, as well as integration improvements for the configuration system.

• Task manager refactor:

  • [core] draft of managerV2
  • [core] Rebase and add TriggerHooks
  • [core] Refactor scheduler into task.Manager
  • [core] Fix pending teardown with asynchronous ReleaseTasks
  • [core] remove mutex from TaskMan, use mutex for roster, classes and Task
  • [core] Remove obsolete warning from repo.Manager
  • [core] merge manager with managerv2
  • [core] fix discrepancy between task and environment states
  • [core] fix infinite loop due to wfState

• Configuration:

  • [core] Expose deployment constants (including consul_endpoint) to env
  • [core] Push environment_id as property to all tasks
  • [core] Only append --control-port parameter to FairMQ tasks
  • [occ] Always prefer OCC_CONTROL_PORT variable to --control-port param
  • [occ] Ensure the run number is always a uint32_t in the C++ interface

• Miscellaneous:

  • [build] Use latest FSM
  • [coconut] Remove usage of UUID
  • [core] Correct task ID generation
  • [core] Force shutdown reply when all processes are gone
  • [core] Make log output more relevant
  • [misc] New benchmark script
  • [misc] Test script workflow template default to @master
  • [odcshim] Add support for ODC partitioning

v0.17.81

This release brings an in-depth overhaul of the task manager component of the AliECS core, resulting in significant performance improvements. It also delivers numerous bug fixes, as well as integration improvements for the configuration system.

• Task manager refactor:

  • [core] draft of managerV2
  • [core] Rebase and add TriggerHooks
  • [core] Refactor scheduler into task.Manager
  • [core] Fix pending teardown with asynchronous ReleaseTasks
  • [core] remove mutex from TaskMan, use mutex for roster, classes and Task
  • [core] Remove obsolete warning from repo.Manager
  • [core] merge manager with managerv2
  • [core] fix discrepancy between task and environment states
  • [core] fix infinite loop due to wfState

• Miscellaneous:

  • [build] Use latest FSM
  • [coconut] Remove usage of UUID
  • [core] Correct task ID generation
  • [core] Force shutdown reply when all processes are gone
  • [core] Make log output more relevant
  • [core] Only append --control-port parameter to FairMQ tasks
  • [core] Push environment_id as property to all tasks
  • [misc] New benchmark script
  • [occ] Always prefer OCC_CONTROL_PORT variable to --control-port param
  • [occ] Ensure the run number is always a uint32_t in the C++ interface

v0.17.80

This release brings an in-depth overhaul of the task manager component of the AliECS core, resulting in significant performance improvements. It also delivers numerous bug fixes, as well as integration improvements for the configuration system.

• Task manager refactor:

  • [core] draft of managerV2
  • [core] Rebase and add TriggerHooks
  • [core] Refactor scheduler into task.Manager
  • [core] Fix pending teardown with asynchronous ReleaseTasks
  • [core] remove mutex from TaskMan, use mutex for roster, classes and Task
  • [core] Remove obsolete warning from repo.Manager
  • [core] merge manager with managerv2

• Miscellaneous:

  • [build] Use latest FSM
  • [coconut] Remove usage of UUID
  • [core] Correct task ID generation
  • [core] Force shutdown reply when all processes are gone
  • [core] Make log output more relevant
  • [core] Only append --control-port parameter to FairMQ tasks
  • [misc] New benchmark script
  • [occ] Always prefer OCC_CONTROL_PORT variable to --control-port param

v0.17.2

This is a patch release that works around an issue with aliswmod delaying task launch. It also improves the machine id source for the unique task/environment ID generator.

• [common] Acquire machine-id for Sonyflake generator
• [executor] Increase gRPC dial timeout to workaround slow aliswmod issue

v0.17.1

This is a patch release that fixes a crash when running the core in certain types of container.

• [common] Ensure a machine ID is always passed to the uid generator

v0.17.0

This release brings numerous reliability and integration improvements. Important user-facing changes include shortened Environment and Task ID strings, and richer InfoLogger output. This release also includes a preview of walnut, the AliECS workflow administration and linting utility, which provides validation of AliECS workflow configuration files and conversion of DPL dumps into AliECS workflow and task templates.

• Log output improvements:

  • [common] Default InfoLogger role field to detected hostname
  • [common] Pass correct username to IL
  • [common] Ensure a default rolename is always passed to IL
  • [common] Set IL level for all messages
  • [core][executor] Update logger facility field
  • [core] Always set system to FLP for controlled tasks
  • [core] Override system for O2_ROLE/O2_SYSTEM env vars passed to tasks
  • [core] Set less relevant messages to Trace severity
  • [core] Push envId as partition in core

• Short IDs:

  • [core] Environment ID is now a 20-char xid vs. 36 for earlier UUID
  • [core] Task ID is now a xid
  • [core] Executor ID is now a xid
  • [core] MesosCommand ID is now a xid
  • [core] Use sony/sonyflake via indigo for short environment IDs
  • [core] Use short UID for task ID and executor ID
  • [coconut] Adapt task id table for short UID

• Miscellaneous improvements:

  • [build] Ensure we build a true static binary
  • [coconut] Fix documentation string for coconut repo default-revision
  • [core] Remove obsolete locking in server.go
  • [core] Clarify naming of some task slices/maps in scheduler.go
  • [core] Adjust executor name string
  • [executor] Ensure we always have a PID we can use to kill a task
  • [executor] Workaround for os.user.GroupIds unavailable with !osusergo

• New documentation formatter (Mkdocs):

  • [build] Minor make doc/docs improvement
  • [misc] Mkdocs stub
  • [misc] Rebuild coconut documentation
  • [misc] Split peanut docs from OCC
  • [misc] Update documentation TOC
  • [misc] Update deployment instructions
  • [misc] Fix coconut env create documentation
  • [misc] Fix template list documentation
  • [misc] Update doc nav
  • [misc] Update documentation
  • [misc] Trash all Minimesos and DCOS stuff
  • [misc] Update running and development instructions
  • [misc] Update README
  • [misc] Add FAQ page stub
  • [misc] Begin AliECS Handbook with user-oriented documentation
  • [misc] Point to ControlWorkflows readme in mkdocs.yml

v0.16.0

This release brings a major new feature, hook tasks. A hook task is any stateless process (e.g. a Linux shell command) that can be run synchronously with the environment state machine, in response to an event and/or as a condition for completing a transition. It also adds support for disabling portions of the workflow tree when creating an environment, some user interface improvements in coconut, as well as support for the O² Configuration library in the OCC library.

• Support for hook tasks:

  • [common] controlmode.String should not have ptr receiver
  • [core] New MesosCommand_TriggerHook
  • [core] Support for loading hook definitions from workflow template
  • [core] MesosCommand_TriggerHook handling
  • [core] task.Manager.TriggerHook implementation
  • [core] Tie hooks to environment state machine triggers
  • [core] Better error output on failed MesosCommandResponse
  • [core] Fix timeout representation
  • [core] Improve commandqueue error handling
  • [core] Timeout support in task.Task
  • [core] Improve error output, controlmode representation, GetRoles lock
  • [core] Handle hooks in scheduler
  • [core] Hook timeout support and hook handling fixes in environment
  • [executor] New HookTask + BasicTask refactor
  • [executor] MesosCommand_TriggerHook handling
  • [executor] Refactor taskCmd lifetime for basic+hooks
  • [executor] Add basic/hook task timeout support via context
  • [executor] Implement triggering of hook tasks

• Support for launch-time role disabling:

  • [core] Add support for enabling/disabling roles at deployment time
  • [core] Make iteratorRange/iteratorRole copyable

• coconut interface improvements:

  • [coconut] Accept lowercase event names as well as short start/stop
  • [coconut] Support for --nospinner flag

• Configuration library support in OCC library:

  • [occ] Fill configuration ptree with local file
  • [occ][OCTRL-295] Improve code documentation of PR#156

v0.15.0

This release brings initial integration of ODC (the process control component of the O²/EPN cluster), support for pushing component configuration to controlled tasks as boost::property_tree::ptree as well as major improvements to process end of life management. It also exposes the PIDs of all running tasks in the GetTasks RPC call.

• ODC integration:

  • [build] Add o2-aliecs-odc-shim to Makefile
  • [odcshim] Initial commit for ODC shim task
  • [odcshim] Update to newest ODC protofile
  • [odcshim] Use RUN instead of lower level API + refactor OCC handler
  • [odcshim] Do SetProperties before Configure
  • [odcshim] Topology provided as env var + implement remaining commands
  • [odcshim] AliECS<->ODC state conversion facilities
  • [odcshim] Correctly report ODC state

• Configuration as ptree:

  • [core] Add ptree marker to properties map with ToPtree function
  • [core] Add support for syntax spec in ToPtree template function
  • [core] Add support for timestamp-less payloads in confsys
  • [occ] Process incoming ptree key as ptree
  • [occ] Correctly graft payload ptree

• Process end of life:

  • [core] Fix deadlock caused from state.Event
  • [core] Make state.Event unbuffered channel
  • [executor] Do a SIGTERM before the SIGKILL
  • [executor] Handle Kill concurrently
  • [executor] Run SIGKILL immediately after a failed SIGTERM
  • [executor] Fix SIGSEV
  • [executor] kill process using PID from OCC
  • [executor] Update proto to export PID
  • [executor] use OCC pid to kill task on ERROR
  • [executor] Set kill timeout to 3 seconds
  • [occ] Export PID to protobuf

• Expose PID in API:

  • [coconut] Print PID on GetTasks
  • [common] Create TaskMessage event
  • [core] Handle TaskMessage in scheduler
  • [core] Extend role with GetTaskPID
  • [core] Export Task PID to API
  • [core] Move PID to task.Task from role
  • [executor] Send TaskMessage with PID to scheduler

• Miscellaneous:

  • [coconut] Regenerate coconut documentation
  • [coconut] Adapt config import to new behavior of GetLatestTimestamp
  • [core] Set periodic output severity to Trace
  • [core] Export GetConfig function in all template environments
  • [executor] Handle HEARTBEAT event
  • [peanut] Simulate run number

v0.14.91

This release brings initial integration of ODC (the process control component of the O²/EPN cluster), support for pushing component configuration to controlled tasks as boost::property_tree::ptree as well as major improvements to process end of life management. It also exposes the PIDs of all running tasks in the GetTasks RPC call.

• ODC integration:

  • [build] Add o2-aliecs-odc-shim to Makefile
  • [odcshim] Initial commit for ODC shim task
  • [odcshim] Update to newest ODC protofile
  • [odcshim] Use RUN instead of lower level API + refactor OCC handler
  • [odcshim] Do SetProperties before Configure
  • [odcshim] Topology provided as env var + implement remaining commands
  • [odcshim] AliECS<->ODC state conversion facilities
  • [odcshim] Correctly report ODC state

• Configuration as ptree:

  • [core] Add ptree marker to properties map with ToPtree function
  • [core] Add support for syntax spec in ToPtree template function
  • [core] Add support for timestamp-less payloads in confsys
  • [occ] Process incoming ptree key as ptree
  • [occ] Correctly graft payload ptree

• Process end of life:

  • [executor] Do a SIGTERM before the SIGKILL
  • [executor] Handle Kill concurrently
  • [executor] Run SIGKILL immediately after a failed SIGTERM
  • [executor] Fix SIGSEV
  • [executor] kill process using PID from OCC
  • [executor] Update proto to export PID
  • [executor] use OCC pid to kill task on ERROR
  • [occ] Export PID to protobuf

• Expose PID in API:

  • [coconut] Print PID on GetTasks
  • [common] Create TaskMessage event
  • [core] Handle TaskMessage in scheduler
  • [core] Extend role with GetTaskPID
  • [core] Export Task PID to API
  • [core] Move PID to task.Task from role
  • [executor] Send TaskMessage with PID to scheduler

• Miscellaneous:

  • [coconut] Regenerate coconut documentation
  • [coconut] Adapt config import to new behavior of GetLatestTimestamp
  • [core] Set periodic output severity to Trace
  • [core] Export GetConfig function in all template environments
  • [executor] Handle HEARTBEAT event
  • [peanut] Simulate run number

v0.14.90

This release brings initial integration of ODC (the process control component of the O²/EPN cluster), support for pushing component configuration to controlled tasks as boost::property_tree::ptree as well as major improvements to process end of life management.

• ODC integration:

  • [build] Add o2-aliecs-odc-shim to Makefile
  • [odcshim] Initial commit for ODC shim task
  • [odcshim] Update to newest ODC protofile
  • [odcshim] Use RUN instead of lower level API + refactor OCC handler
  • [odcshim] Do SetProperties before Configure
  • [odcshim] Topology provided as env var + implement remaining commands
  • [odcshim] AliECS<->ODC state conversion facilities
  • [odcshim] Correctly report ODC state

• Configuration as ptree:

  • [core] Add ptree marker to properties map with ToPtree function
  • [core] Add support for syntax spec in ToPtree template function
  • [core] Add support for timestamp-less payloads in confsys
  • [occ] Process incoming ptree key as ptree
  • [occ] Correctly graft payload ptree

• Process end of life:

  • [executor] Do a SIGTERM before the SIGKILL
  • [executor] Handle Kill concurrently
  • [executor] Run SIGKILL immediately after a failed SIGTERM
  • [executor] Fix SIGSEV
  • [executor] kill process using PID from OCC
  • [executor] Update proto to export PID
  • [executor] use OCC pid to kill task on ERROR
  • [occ] Export PID to protobuf

• Miscellaneous:

  • [coconut] Regenerate coconut documentation
  • [core] Set periodic output severity to Trace
  • [core] Export GetConfig function in all template environments
  • [executor] Handle HEARTBEAT event
  • [peanut] Simulate run number