TC8 SOME/IP conformance test suite for someipd#60
TC8 SOME/IP conformance test suite for someipd#60jorgecasal wants to merge 10 commits intoeclipse-score:mainfrom
Conversation
Wire-level pytest tests verifying someipd against the OA TC8 ECU Test Specification v3.0 Chapter 5: service discovery (phases, reboot, timing), message format, event notification, field lifecycle, and TCP transport. - someipd --tc8-standalone mode: request/response, field GET/SET, UDP and TCP events; offer_event uses ET_FIELD so vsomeip delivers the cached field value to new subscribers immediately on subscribe (is_field in JSON config is not honoured in programmatic offer_event calls) - vsomeip config templates for SD timing and service/event/field/TCP tests, with JSON Schema validation at lint time - Architecture doc, requirements, test specification and OA spec traceability - Bazel port isolation via the env attribute: each target receives unique SD and service ports enabling medium targets to run in parallel; timing-sensitive and reboot lifecycle targets retain the exclusive tag for serial execution
License Check Results🚀 The license check job ran with the Bazel command: bazel run //:license-checkStatus: Click to expand output |
jorgecasal
changed the title
jorgecasal
added
documentation
|
The created documentation from the pull request is available at: docu-html |
- Raise check-added-large-files limit from 50 KB to 125 KB to accommodate large but legitimate TC8 test modules and the RST test specification document - Add REUSE.toml annotation for tests/tc8_conformance/config/*.json so the four vsomeip JSON config files pass reuse-lint-file - Prepend Apache-2.0 SPDX comment headers to the two TC8 README.md files that were missing copyright notices
TC8 conformance tests require special setup (multicast route, env vars) and already run in a dedicated step; avoid double-execution via -tc8 filter.
jorgecasal
force-pushed
the
jorgecasal_tc8_conformance_test_infrastructure
branch
from
c507e28 to
63d628d
Compare
jorgecasal
force-pushed
the
jorgecasal_tc8_conformance_test_infrastructure
branch
from
63d628d to
8420326
Compare
|
Documentation overall looks very good, reviewed the current state at https://eclipse-score.github.io/inc_someip_gateway/pr-60/architecture/tc8_conformance_testing.html - not in detail, but general sections and impression. Nice! On my local checkout, running the tests works with the expected failures:
$ time bazel test --test_tag_filters=tc8 --test_env=TC8_HOST_IP=127.0.0.1 //tests/tc8_conformance/...Result: real 6m16.073s |
|
I missed setting up the multicast... now the tests run fine locally as well, can confirm. |
mikehaller
left a comment
mikehaller
left a comment
There was a problem hiding this comment.
As the TC8 Conformity Tests is a complete new part for someip gateway, I'll approve the current state of the PR. It's a very good step forward.
There is one issue regarding the main.cpp where I didn't fully grasp the plans and hence would accept it if it's just temporary.
| bazel test //tests/... --test_tag_filters=tc8 | ||
|
|
||
| # Use a real network interface | ||
| TC8_HOST_IP=<your-host-ip> bazel test //tests/tc8_conformance/... |
There was a problem hiding this comment.
Is this host or device under test ip?
There was a problem hiding this comment.
Got it, it's for choosing a different host interface (e.g. localhost vs. eth0)
There was a problem hiding this comment.
You got it right, nevertheless let me provide a bit more information. In the protocol conformance tests, the DUT (someipd) runs on the same machine as the test framework, so they share the same IP. TC8_HOST_IP is the host's network interface IP, where it's used both to bind the test's raw sockets (for sending/receiving SOME/IP packets) and to configure someipd via the __TC8_HOST_IP__ placeholder in the vsomeip JSON templates.
In short, host IP = DUT IP here, because they're co-located. The variable name reflects the test framework's perspective (the machine under test), not a remote DUT. Happy to rename it to something clearer like TC8_DUT_IP if you think that better matches TC8 terminology.
There was a problem hiding this comment.
I assume the previous state of main.cpp also was just a hardcoded dummy service, and extending it with TC8 specifics as part of the pull request doesn't make a big difference.
In the longer run, would it make sense to have the TC8-enabled version of someipd main.cpp as a separate application?
I ask because someipd is supposed to be an in-vehicle component and mixing production code and test code together like this may not be ideal. If it's just temporary until the other PR's converge, i'll pull back my comment.
There was a problem hiding this comment.
Your assumption is correct, main.cpp was already a hardcoded placeholder with no production logic, so the TC8 additions don't mix test code into real production code. That said, your longer-term concern is valid: once someipd grows its real implementation, the TC8-specific flags should move to a separate binary. We'll have to track that as a follow-up.
jorgecasal
changed the title
MODULE.bazel
Outdated
There was a problem hiding this comment.
Note: score_docs_as_code version bump required for full traceability
While verifying the TC8 requirements → test specification → code implementation traceability linkage introduced in this PR, a bug was found in score_docs_as_code that silently prevented all testcase need objects from being created (see eclipse-score/docs-as-code#440 (comment) for the detailed root cause).
The fix has been merged in eclipse-score/docs-as-code#440, but is not yet available in a published release. Our MODULE.bazel currently pins:
bazel_dep(name = "score_docs_as_code", version = "2.3.0", dev_dependency = True)- Use local json_schema_validator rule instead of @score_communication - Add integrity hashes to download_archive deps for reproducible builds - Add component descriptions for gatewayd and someipd in architecture docs
jorgecasal
force-pushed
the
jorgecasal_tc8_conformance_test_infrastructure
branch
from
4393f39 to
2dfcff0
Compare
2 participants
Summary
Introduces the TC8 SOME/IP conformance test suite for
someipd, covering OA TC8 ECU Test Specification. Tests run wire-level via raw SOME/IP sockets over loopback, withoutgatewayd,using a new--tc8-standaloneflag added tosomeipd.Documentation
docs/architecture/tc8_conformance_testing.rst— new architecture document: standalone mode rationale, port-isolation design, per-target port table, test module structure, and TC8 specification alignment analysis.docs/tc8_conformance/requirements.rst— requirements for SD, message format, event notification, field lifecycle, TCP transport, and robustness, with AUTOSAR PRS traceability IDs.docs/tc8_conformance/test_specification.rst— test-specification document mapping each requirement to one or more test cases.docs/tc8_conformance/traceability.rst— requirement-to-test traceability matrix linking TC8 spec sections to test IDs.docs/architecture/index.rstanddocs/requirements/index.rst— updated to include the new documents.TC8 Conformance Test Directory Structure
New directory
tests/tc8_conformance/with the following layout:Test Framework and Test Cases
All tests use
score_py_pytestBazel targets. Port isolation is achieved via theenvattribute — each target receives uniqueTC8_SD_PORT,TC8_SVC_PORT, and (where needed)TC8_SVC_TCP_PORTvalues, allowing medium-sized targets to run in parallel. Timing-sensitive and reboot lifecycle targets carry theexclusivetag for serial execution.someipd Changes
--tc8-standaloneflag (src/someipd/main.cpp): skips gatewayd IPC and offers the TC8 service/event/field directly, enabling self-contained conformance testing without the full gateway stack.offer_eventusesET_FIELDfor field-type events so vsomeip delivers the cached initial value to new subscribers immediately on subscribe.src/someipd/BUILD.bazel:someipdbinary deps updated for standalone mode.CI Workflow
.github/workflows/build_and_test_host.yml— new stepBazel TC8 Conformance Test Targets:224.0.0.0/4 dev lo) required for vsomeip SOME/IP-SD on the GitHub Actions runner.bazel test --test_tag_filters=tc8 --test_env=TC8_HOST_IP=127.0.0.1 //tests/tc8_conformance/....General Clean-up
MODULE.bazel: dependency version bump.tests/integration/BUILD.bazel: minor dependency alignment.Test Plan
bazel test --test_tag_filters=tc8 --test_env=TC8_HOST_IP=127.0.0.1 //tests/tc8_conformance/...passes locallybazel run //:copyright.checkpasses (all new files have correct headers)build_and_test_host) green on this branchbazel build //docs/...builds without RST errors