runtime v3 changes

Update to runtime v3 includes a minor addition to logging, a minor change to the build system, and a major change to the routing mechanism.

Minor logging addition

Log levels are now defined in Makefiles rather than in logging.h, so it is not necessary to modify logging.h to change the verbosity. Simply modify the list of logs in the Makefile to include/exclude the desired log levels:

LOGS = \
       DEBUG \
       INFO \
       ERROR \
       WARN \
       CRITICAL \
       CUSTOM

The CUSTOM log level allows for defining new log levels for individual components which can be turned on or off easily.

Simply add your desired log level to LOGS in the makefile, then use log_custom(LEVEL, fmt, ...) to utilize the new log level. E.g.:

LOGS = \
       MY_LOG_LEVEL\
       CUSTOM

Then in the code:

// At the top of the .c file:

#ifndef MY_LOG_LEVEL
#define MY_LOG_LEVEL 0
#endif 

// In the code
log_custom(MY_LOG_LEVEL, "I am logging something");

Structural changes to build system

The build system has been separated into three directories:

• global_controller
• runtime
• common

This was implemented so that the global controller and the runtime could share headers and binaries as needed.

A top-level makefile was added to the repo which will descend into all three directories and make the appropriate files. The runtime and global controller can still be compiled as they previously were, with the makefiles in their respective directories.

The directory common/src has been added to the include path, so (e.g.) logging can be included with just #include "logging.h"

Major changes to routing

To summarize:

On entry into DeDos, a queue item should now be assigned an ID by the entry MSU. This ID will be used to deterministically route the queue item through the DFG (unless explicitly overridden).

Information about routes are now stored in a routing object which is managed by the main thread. The routing tables are protected with a read-write lock, so multiple MSUs can read the route information simultaneously, and cannot read it when it is being updated by the main thread.

Key assignment

A field generate_id has been added to the struct msu_type. generate_id is a pointer to a function which accepts the struct generic_msu and the struct queue_item, and should return an ID which will be assigned to that queue item and used for routing decisions throughout DeDos.

Key-based routing

When assigning an MSU as a destination, the maximum key that will be routed to that MSU must also be supplied. The routing decision is made in the following way:

• The queue-item's key is modulo'd by the maximum key associated with that route
• The closest destination MSU that has a key larger than the queue-item's key is chosen.

For example, if a route contained the following MSUS and keys:

MSU | KEY
 1  |  5
 2  |  30
 3  |  35

and a queue-item had key 68. The queue-item's value for this route would be 68 % 35 = 33, so the queue item would be routed to MSU 3.

CLI & JSON

The global controller's CLI, and the preload's JSON have been modified for the new routing structure.

The CLI now contains the following extra options:

4. add route [runtime_socket_num] [route_num] [origin_msu] /* Adds an outgoing route to an MSU */
5. del route [runtime_socket_num] [route_num] [origin_msu] /* Deletes an outgoing route from an MSU */
6. add endpoint [runtime_socket_num] [route_num] [destination_msu_id] [key_range_end] /* Adds an MSU as an endpoint to a given route */
7. del endpoint [runtime_socket_num] [route_num] [destination_msu_id] /* Deletes an MSU as an endpoint from a given route */
8. mod endpoint [runtime_socket_num] [route_num] [destination_msu_id] [new_key_range_end] /* Modifies the key range associated with an MSU endpoint on the given route */

When using the CLI, an endpoint must be added to a route before the route can be added to an MSU. The route does not exist until an endpoint is added to it.

The JSON structure has also been modified to define a list of per-runtime routes. Each route consists of a type, id, and a destinations object. Keys in the destinations object are the destination MSU IDs. Values in the destinations object are the maximum queue-item-key that will be routed to that MSU.

The routing list in each MSU now contains the route IDs rather than the destination MSU IDs

• cfg_generator/dfg_from_yml.py and cfg_generator/cfg_from_yml.py are capable of producing the new cfg/dfg files from existing yml configurations *

Routing object

At the core of the routing object is the struct routing_table which contains a list of the possible destination MSUs and their associated key-ranges.

Each route has a single routing table. MSUs may have several target routes, and routes may belong to several MSUs. This is accomplished with the struct route_set.

The struct generic_msus field struct route_set *routes may contain multiple routes. The route associated with a given MSU type ID can be retrieved with

get_type_from_route_set(struct route_set **routes, int type_id);

Which will return the struct route_set * that contains routes with the given type_id if they exist in that route_set, otherwise NULL.

There are four functions to access individual members of a route_set:

/** Key-based routing, as defined above */
struct msu_endpoint *get_route_endpoint(struct route_set *routes, uint32_t key);
/** Returns the (local) endpoint in routes that has the shortest queue. 
* If multiple elements share a queue length, uses the key to determine which to return*/
struct msu_endpoint *get_shortest_queue_endpoint(struct route_set *routes, uint32_t key);
/** Returns the endpoint with the MSU ID equal to the supplied argument. If N/A, returns NULL */
struct msu_endpoint *get_endpoint_by_id(struct route_set *routes, int msu_id);
/** Returns the endpoint with the given index, or NULL if index > length(routes) */
struct msu_endpoint *get_endpoint_by_index(struct route_set *routes, int index)

Routes may only be accessed using these functions, as the struct routing_table definition is kept private in routing.c (as opposed to routing.h). This is necessary to adhere to the locks that are kept on that table.