Skip to content

Test Result URL

Jump to bottom Edit New page
Matthew Kelly edited this page Mar 23, 2017 · 2 revisions

Description

The Test Result URL is an endpoint that you can supply should you wish for the test results to be sent out externally - say to a website that stores and collects statistics and historic information about tests and their runs.

The endpoint should support POST requests of JSON format.

How to set Parameters

The parameters that are associated with the Test Result URL are listed below, along with their CLI and Edisonfile names:

Name CLI Edisonfile Type Mandatory?
Test Result URL --turl test_result_url string true
Test Run ID --tid test_run_id string true
Test Run Name --tname test_run_name string false
Test Run Project --tproj test_run_project string false
Test Run Environment --tenv test_run_env string false

Test Result URL

This parameter is mandatory if you wish to have the results of tests sent out to an external endpoint. It must be a valid URL, and accept JSON POST requests.

Test Run ID

This parameter is also mandatory. It will be the grouping identifer for test runs.

For example, say you have some release "v1.2.0", and you have test runs across three commits for this release. Your Test Run ID will be the v1.2.0, as, if you use it, the Test Run Name will be the full semantic name, ie: v1.2.0-a1b3c5.

This way, you'll have a mechanism for grouping test runs together for versions of your software.

If you don't want the grouping, and aren't using the Test Run Name, then just set the Test Run ID to be the full semantic (or whatever versioning pattern you're using) version. The Test Run Name will be defaulted to the Test Run ID if not supplied.

Test Run Name

This parameter is optional. If you care about grouping your test runs, like mentioned in the above section for Test Run ID, then this parameter will be a more informative fuller version name for the test run. So if you set the Test Run ID to be something like v1.2.0, then this will be something like v1.2.0-a1b3c5.

With this, if you wish to, you can group all v1.2.0 test runs together via the Test Run ID. You'll also be able to identifer better which runs ran for which commits using the Test Run Name.

If the Test Run Name isn't supplied, then it will be defaulted to the Test Run ID.

Test Run Project

This parameter is optional. It is the type name of the project for the test run.

For example, say you have a large project and you have different regression environments for different aspects of the project like website, service and api. All will more than likely have the same Test Run ID and Test Run Name version values. With the Test Run Project just set the value to website or service and you now know which aspect of the project the run was against.

Test Run Environment

This parameter is optional. It is the name of the environment the test run was run on, and the default value is the name of the machine.

For example, say you have 3 different regression environments, you can help identify which environment the test runs were on by setting the value to be either REG1, REG2 or REG3.

The default value is the name of the machine though, which in most cases is probably all you'll need.

Callouts

In total, there are 4 different types of callouts your endpoint will receieve:

  • Verification of the endpoint
  • Test run started
  • Test results
  • Test run ended

Except for the first callout type of verification, the start/result/end types will each have an Action parameter in there JSON callout to help identify the type of the request.

Verification

This will be an empty POST request to the endpoint to ensure it exists and is accepting callouts. If the request body is an empty JSON body, just return a 200 OK response.

Test Run Started

This will be a POST request to the endpoint to state that the test run has been initiated. The request will have an Action value of Start, as well as the following JSON makeup:

{
    "TestRunId": "v1.2.0",
    "TestRunName": "v1.2.0-a1b3c5",
    "TestRunProject": "website",
    "TestRunEnvironment": "Regression",
    "SessionId": null,
    "Action": "Start",
    "TestResults": null
}

Although not mandatory, the endpoint should respond with a session token for the test run. This will be the SessionId value for the result/end callouts later on, and will help identify which run the callouts are for (in case you have multiple runs of the same version taking place). The response back should just be a plain text response that is either empty or contains a string value (like a guid, or database identifier, etc.).

Test Results

This will be a POST request to the endpoint containing the details of a test's result. The request will have an Action value of Result, as well as the SessionId if one was passed back from the start callout. The JSON makeup is as follows:

{
    "TestRunId": "v1.2.0",
    "TestRunName": "v1.2.0-a1b3c5",
    "TestRunProject": "website",
    "TestRunEnvironment": "Regression",
    "SessionId": "ji4737hf489t848784",
    "Action": "Result",
    "TestResults": [
        {
            "TestName": "Edison.Tests.TestClass.TestMethod(1,3)",
            "NameSpace": "Edison.Tests.TestClass",
            "AssemblyName": "Edison.Tests.dll",
            "TestState": "Success",
            "TimeTaken": "00:00:00.0021105",
            "ErrorMessage": "",
            "StackTrace": "",
            "CreateDate": "2017-03-07T19:28:38.2051809+00:00"
        }
    ]
}

Test Result States

The following values are valid TestStates that can be used in test result objects:

// general states (usually from tests)
Success,
Failure,
Error,
Inconclusive,
Ignored,

// setup states (if the fail occurred before the test in setup)
// test logic was never run, becuase failure occurred before it could be attempted
SetupError,
SetupFailure,

// teardown states (if the fail occurred after the test in teardown)
// this generally means the test passed, but teardown failed
TeardownError,
TeardownFailure,

// global setup states (if the setup logic before all fixtures fails)
GlobalSetupError,
GlobalSetupFailure,

// global teardown states (if the teardown logic after all fixtures fails)
// this error in general probably will never be used, as tests have already completed by this point
GlobalTeardownError,
GlobalTeardownFailure,

// test fixture setup states (fail occurred before all tests in a fixture could be attempted)
TestFixtureSetupError,
TestFixtureSetupFailure,

// test fixture teardown states (fail occurred after all tests in a fixture have been run)
// like the global teardown states, this one will probably never be used
TestFixtureTeardownError,
TestFixtureTeardownFailure,

// unknown (should never be used!)
Unknown

Test Run Ended

This will be a POST request to the endpoint to state that the test run has ended. The request will have an Action value of End, as well as the SessionId if one was passed back from the start callout. The JSON makeup is as follows:

{
    "TestRunId": "v1.2.0",
    "TestRunName": "v1.2.0-a1b3c5",
    "TestRunProject": "website",
    "TestRunEnvironment": "Regression",
    "SessionId": "ji4737hf489t848784",
    "Action": "End",
    "TestResults": null
}
Add a custom footer

Pages

Clone this wiki locally