Test: make GStreamer a first-class Application adapter by awilczyns · Pull Request #1614 · OpenVisualCloud/Media-Transport-Library

awilczyns · 2026-06-15T12:17:53Z

Refactor the GStreamer pytest suite onto the unified Application pattern
already used by RxTxApp and FFmpeg. GStreamer is now selectable via
app_factory("gstreamer") and the "application" parametrize, replacing the
procedural GstreamerApp.setup_* / execute_test calls in every test.

Add mtl_engine/gstreamer.py: GStreamer(Application), a thin wrapper over
the procedural GstreamerApp (kept as single source of truth). Dispatches
st20p/st30/st40p by session_type; derives the crossed ST2022-7 redundant
mapping from a 4-port nic list + ip_pools.
Register the "gstreamer" branch in conftest.app_factory.
Add characterization unit tests asserting the adapter emits byte-for-byte
identical gst-launch commands vs the procedural builders.
Refactor video_format, video_resolution, audio_format and anc_format onto
the adapter; all matrices, ids, markers, skip guards and assertions kept.
Removes the per-test redundant-param boilerplate in favour of redundant=True.

Also fixes the preflight hugepage check to compute free MiB from the actual
Hugepagesize instead of assuming 2 MiB pages.

Behavior unchanged: verified on real E810 VFs (st20p, st30, st40i_basic,
st40p redundant all pass).

DawidWesierski4

please put the tests into the old testing harness don't add test_gst

this is pointless this way :<

DawidWesierski4 · 2026-06-15T14:32:51Z

+``execute_test`` delegates to :func:`GstreamerApp.execute_test`.
+
+Because execution is delegated, this adapter intentionally BYPASSES the
+base-class run machinery (netsniff hook, PTP extension, SIGINT->SIGKILL stop


netsniff hook, PTP extension can be tested in gstreamer

-> g_param_spec_boolean("enable-ptp", "Enable onboard PTP",
-> netsniff hook -> the gstreamer should have compliance testing ?

the only question is about if we should

DawidWesierski4 · 2026-06-15T14:41:20Z



 @pytest.mark.nightly
+@pytest.mark.parametrize("application", ["gstreamer"])


This shudn't work like this

test_gst_video_format should be included into the st20 tests
right now this is just renaming not refactoring the tests :<

DawidWesierski4 · 2026-07-03T07:32:37Z

+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Intel Corporation
+"""GStreamer ST20P onboard-PTP validation.
+
+Exercises the GStreamer plugin ``enable-ptp`` property (onboard PTP client) on
+a single-host TX+RX ST20P pipeline. GStreamer drives synthetic planar media, so
+this validates that the stream transports and md5-matches end to end with PTP
+enabled; it is GStreamer-only because the ``enable-ptp`` knob lives in the
+plugin (the RxTxApp PTP coverage lives under the integrity-based ptp tests).
+"""
+
+import os
+
+import mtl_engine.media_creator as media_create
+import pytest
+from common.nicctl import InterfaceSetup
+from mtl_engine import ip_pools
+from mtl_engine.media_files import yuv_files_422rfc10
+
+
+@pytest.mark.nightly
+@pytest.mark.ptp


i don't think we should have tests like this in the PTP folder

this very well could be a general test, why is it only done for gstreamer ?
apart from name the test is completly general ?

DawidWesierski4 · 2026-07-03T07:39:09Z

+# GStreamer reads raw planar media (I422_10LE / v210), not the RFC4175-packed
+# files RxTxApp/FFmpeg consume, so it self-provides a synthetic clip of the same
+# resolution. Generating raw 4K/8K clips is impractical, so GStreamer only runs
+# the <=1080p resolutions; RxTxApp/FFmpeg still sweep the full range.
+_GST_MAX_HEIGHT = 1080
+
+
+def _prepare_gstreamer_media(host, info, media_file_path, request):
+    """Generate a synthetic planar clip for GStreamer; return (input, output, gst_format).
+
+    The st20 plugin only emits v210 when the width is a multiple of six (pixel
+    groups are six pixels wide); otherwise it falls back to I422_10LE. The clip
+    is written next to the fixture-provided media file so it lands in the
+    managed media/ramdisk directory rather than a hardcoded path.
+    """
+    width, height, fps = info["width"], info["height"], info["fps"]
+    gst_format = "v210" if width % 6 == 0 else "I422_10LE"
+    if gst_format == "v210":
+        SDBQ1971_conversion_v210_720p_error(
+            video_format=gst_format, resolution_height=height, request=request
+        )
+
+    media_dir = str(host.connection.path(media_file_path).parent)
+    input_file = media_create.create_video_file(
+        width=width,
+        height=height,
+        framerate=fps,
+        format=gst_format,
+        media_path=media_dir,
+        duration=3,
+        host=host,
+    )
+    output_file = os.path.join(media_dir, "output_video.yuv")
+    return input_file, output_file, gst_format


this is not the place for it ?
we could have functions in our gstreamer class that does it ?

also this is impractical solution ... it was done as a workaround in gstreamer ?
using it this way is in my opinion wrong

let's just use the input files like in other tests and have the normal file src pipeline,

TLDR -> we used generic file production cuz it was easier it fails cuz we did it this way
i don't think it's resonable to skip the test to keep that syntetic input and only run the format check below Full HD resolution

DawidWesierski4 · 2026-07-03T07:42:31Z

+        if height > _GST_MAX_HEIGHT:
+            pytest.skip(


as said above this is easly fixed and can be removed via not propagating this wird workaround we did back in the day

DawidWesierski4 · 2026-07-03T07:43:07Z

+        for path in gst_cleanup:
+            media_create.remove_file(path, host=host)


should not be there ;<

DawidWesierski4 · 2026-07-03T07:45:12Z


 """GStreamer ST40P ancillary transport validation.

 Exercises ANC payload handling over GStreamer across frame rates, payload


if we wan't the gstreamer to be general this description would need to be changed

DawidWesierski4 · 2026-07-03T07:46:28Z

 def _frame_info_path(base_dir=None):
    target_dir = base_dir or "/tmp"
    base = GstreamerApp.sanitize_filename(GstreamerApp.get_case_id())
    token = uuid.uuid4().hex
    return os.path.join(target_dir, f"{base}_{token}_frameinfo.log")


if we want this test to be general then logic like that should go to the gstreamer class instead of the general tests

DawidWesierski4 · 2026-07-03T07:49:10Z

    """
    Validate ST40P ancillary (ANC) transport over GStreamer across frame-rate and
    payload-size matrices to exercise scheduling, pacing, and metadata delivery.
    Small and medium text payloads are generated on the fly, transmitted over
    paired VFs on a single physical port, and captured for byte-for-byte
    comparison by the harness.


this would need to be updated to reflect the general nature of the test

DawidWesierski4 · 2026-07-03T07:52:10Z

 @contextmanager
 def _test_summary(name: str, expectation: str):
    """Structured test logging so pytest.log captures start/fail/pass summaries."""
    log_info(f"[{name}] START: {expectation}")


logging logic check would need to be moved into the Application class
This if i remember correctly is already there

This function in itself is fine but at the end of the day is used by a lot of tests with gstreamer specific logging

Single-host GStreamer runs went through the procedural GstreamerApp module, which built pipelines by hand and could not fail: execute_test() never looked at either pipeline's return code, compare_files() downgraded a missing input to "output is non-empty", and skip_file_compare=True returned True with no oracle at all. Add mtl_engine/gstreamer.py, a GStreamer(Application) adapter for st20p, st30p and st40p, so GStreamer gets the same lifecycle as RxTxApp and FFmpeg: compliance arming, the PTP startup allowance, integrity, and the SIGINT -> SIGKILL stop ladder all come from the base class. Its oracle is four independent checks -- pipeline exit codes, gst bus errors, libmtl frame counters, and RX dump size against the byte rate the parameters imply. Supporting changes: - Application.unsupported_reason(**params) is a new optional seam reporting what an application's MTL plugin cannot do; app_factory() calls it for any keyword arguments it is handed and skips with that reason, so shared tests need no `if application == "..."` branches. - Application.count_pipeline_frames() parses the libmtl pipeline stats line for any direction/counter. count_tx_dropped_frames() now delegates to it; the output is library output, identical whichever framework drives it. - The FFmpeg st30p builder no longer substitutes 1ms/PCM24/48000/2ch when it is asked for a packet time, format, sampling rate or channel count the plugin does not implement -- it streamed something other than what the test asked for and still reported a pass. Those maps now raise, and the plugin's real ptime and pixel-format limits are reported through unsupported_reason() instead. - kill_stale_processes() reaps orphaned gst-launch-1.0 pipelines. - anc_split_by_packet joins UNIVERSAL_PARAMS (GStreamer's split-anc-by-pkt); RxTxApp reports it as unsupported since its JSON config has no such knob. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

GStreamer coverage lived in its own tests/single/gstreamer/ tree, duplicating sweeps the shared protocol tests already ran for RxTxApp and FFmpeg. Select it with the existing `application` parametrize instead, so one test body covers every framework and per-plugin gaps are reported by the adapter rather than by hand-maintained skip marks. Shared tests GStreamer now runs: - st20p: input formats, fps, multicast, resolutions - st30p: integrity, channel, format, ptime, sampling, multicast - st40p: basic, multicast + compliance, rtcp, and two new cases -- interlaced (also RxTxApp) and split-by-packet - ptp: new test_st20p_ptp, all three applications. PTP is a libmtl device option, not a per-framework feature, so it is one parametrized test rather than one file per framework. Static skip marks replaced by unsupported_reason(): FFmpeg on st40p (its plugin registers no ancillary device), FFmpeg on the RFC 4175 resolution assets, and the `if application == "ffmpeg"` ptime skip inside test_st30p_ptime. tests/single/gstreamer/ is removed. video_format, video_resolution and audio_format are superseded by the sweeps above -- and all three streamed synthetic content (audio_format never created its input file at all) where the shared tests use the media assets. anc_format is dropped: 8 of its 24 tests set tx-test-mode, which only exists under MTL_SIMULATE_PACKET_DROPS in debug builds, one asserted `result in (True, False)`, and several streamed input files that were never created. Coverage not carried over, and why: RFC 8331 output packing (needs an RFC8331-packed source asset that does not exist on the media share), the RX per-frame timeout negative case, frame-info-path diagnostics, and redundant two-port ancillary (belongs with the redundant tests, needs 4 interfaces). Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

The preflight check and the status report both multiplied HugePages_Free by 2, assuming 2 MiB pages. On a host booted with default_hugepagesz=1G that misreports badly: 32 free 1G pages read as 64 MiB instead of 32768 MiB and trip the <1024 MiB guard, so setup refuses to run on a host with 32 GiB of free hugepages. Read Hugepagesize from /proc/meminfo instead. Backward compatible -- on 2 MiB-default hosts Hugepagesize is 2048 kB so the factor is still 2. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

test_st20p_packing carried a hand-written skip mark for FFmpeg, and neither plugin adapter reported the gap itself: a test that asked GStreamer for GPM would have passed while the session stayed on the library default BPM. - MTL_DEFAULT_PACKING in config/mappings.py: the one mode both plugins can produce, since neither installs a packing option (verified: no "packing" symbol in ecosystem/ffmpeg_plugin or gstreamer_plugin). - FFmpeg.unsupported_reason()/GStreamer.unsupported_reason() reject any other mode, so the skip reason names the plugin limitation instead of restating it in the test file. - test_st20p_packing queries the adapter and drops the skip mark. Also: point the new PTP test at yuv_files_422p10le["Penguin_1080p"], the same asset the fps and multicast sweeps use, and correct two docstrings that still said "both applications". Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

Two problems in how the oracle-dispatch invariant interacts with hosts and cases that legitimately cannot produce a verdict. A host with no sniff NIC or no EBU credentials left ``capture_cfg`` absent from the generated ``test_config.yaml``, which the ``pcap_capture`` fixture reads as "this host does compliance" -- so every test taking the fixture failed with "ebu_server is not configured" instead of running its data-path oracles. The generator is what knows the host cannot grade a pcap, so it now records ``capture_cfg.enable: false`` explicitly. That alone would have been worse than the error it replaced: the disabled path hands back the ``NO_COMPLIANCE`` null session, whose ``close()`` ignores ``enforce``, so the evaluated-exactly-once invariant cannot speak for it and the whole suite would go green with no compliance verdict and -- unlike the 8K opt-out beside it -- no log line either. ``pcap_capture`` now warns per test, naming the reason, so a compliance-verified run cannot be mistaken for an opted-out one. Second, ``unsupported_reason()`` introduced a legitimate SKIP state that the evaluated-exactly-once invariant predates: a case that skips an unsupported parameter combination never reaches ``execute_test()``, so the dispatch it owed was unreachable and enforcing it reported a correct SKIP as an ERROR. Enforcement is now conditioned on the test body having run to completion. The load-bearing case is intact -- a test that runs, passes, and never dispatches its oracle still fails -- with two pre-existing exceptions worth naming: an ``xfail`` item that XPASSes has its teardown report rewritten to ``skipped``, which swallows the enforcement, and ``log_case`` records "Pass" in report.csv because it reads only the setup and call phases. This touches ``conftest.py`` and ``mtl_engine/``, which the acceptance instructions say not to edit to make a test pass. Both changes are structural rather than test fixes: one restores a signal that was absent, the other stops a correct SKIP being reported as an ERROR. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

Closes the review comments about false positives and about oracles that could not actually fail. The exit-code oracle was permanently disarmed by its own harness. Measured on an st30p run: the RX pipeline exited 0.6s after ``kill -2``, but TX took 21.7s, all of it inside ``Setting pipeline to NULL`` -> MTL session free -> DPDK cleanup. The universal 10s grace therefore SIGKILLed every TX pipeline, so "exited 0" was never observed. ``ProcSpec.graceful_s`` lets this one adapter ask for the 30s it measurably needs. A SIGKILL that still happens is warned about rather than failed, because a larger geometry can legitimately exceed the measured figure -- which does mean this oracle abstains on a hang, and the data oracles below are what bound that case. st40p has no derivable byte rate, so a missing MTL stats line left "output file is non-empty" as the only oracle and a single frame would have passed it. Absent stats now fail whenever no byte count can bound the run -- st40p always, st30p if its audio format yields no byte rate -- but only once ``test_time`` is at least one stats interval, otherwise a legal short run would fail for a reason the test never asked about. ``_MIN_CAPTURE_RATIO`` keeps the reason for the 0.5 floor without the host-specific percentages that were in the comment; the measurements behind it are recorded here instead. On this host a healthy 30s st20p fps capture lands at 65-68% of nominal and st30p at 81-88%, against a broken library's 21%. At 119-120fps the capture becomes I/O bound and lands at 55%, so those two cases have the least headroom over the floor by a wide margin. Also in here, both enabling changes for extending PTP coverage to every application (the "why is it only done for gstreamer?" comment): FFmpeg now grows its wall-clock budget by the PTP sync time, without which ``arm()`` returns with the window already spent and both processes are stopped before any data is captured. And ``SESSION_TYPE_MAP["gstreamer"]`` is deleted -- the GStreamer adapter names an element pair per session type and never read it. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

``test_st20p_packing`` and ``test_st40p_split_by_packet`` each listed only the applications that can drive the feature, which puts capability knowledge back in the test file -- the duplication the adapters' ``unsupported_reason()`` was introduced to remove -- and left the declarations themselves unexercised (RxTxApp's st40p split-by-packet reason had no reachable caller at all). Both now parametrize all three applications, matching the convention the rest of the suite already follows, so each gap surfaces as a visible skip naming its own reason. ``test_st40p_interlaced``'s docstring no longer claims to assert per-field placement, which the pipeline API's ancillary meta cannot express. Not done here: adding GStreamer to ``test_st20p_interlace``. That test asks for ``pacing: "linear"`` and grades it through EBU compliance, but the MTL GStreamer plugin exposes no pacing property, so a GStreamer arm would be graded on default pacing under a "linear" name. It needs a pacing capability declaration first. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

st40p has no expected byte count -- the sender chooses the ancillary payload size -- so "MTL reported frames > 0" was its entire throughput oracle, and the stale-wake regression this suite has to catch exits cleanly at a fraction of nominal. Bound the rate rather than the total: pipeline_frame_series exposes MTL's per-interval counters, the first interval is discarded because it overlaps startup, and the remainder must hold 90% of the configured rate. Too few intervals to do that is a failure, not a log line, so a short test_time cannot quietly leave the session ungraded. Measured steady state is 100.0% of nominal at p50 (the tightest case, having no framerate truncation to round down) and 101.6-103.4% at p59/p29; a stream forced to half rate measures 50.8%, which the 50% byte-capture ratio passed by four frames and this rejects. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

A fixed 16 GiB cap cannot hold a full-length RX dump, and under-sizing does not merely truncate: filesink reports ENOSPC as a pipeline error and the byte-throughput oracles read the shortfall as an MTL delivery failure, so a healthy run fails for want of disk. Derive the cap from test_time at the peak write rate of the heaviest tests/single geometry -- test_st20p_resolutions covers the whole yuv_files_422rfc10 set, whose 8K p25 entries write RFC4175 PG2BE10 at 2.07 GB/s with nothing bounding the dump. Clamp to half of RAM, warn when that clamp binds, and never go below the fixed size this replaced. tmpfs allocates lazily, so the larger cap costs nothing until a test writes that much. The example config and the two docs described a ramdisk.pcap sub-dict that no code reads; the live keys are ramdisk.pcap_dir and ramdisk.tmpfs_size_gib. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

The half-of-RAM cap read MemTotal, which counts the hugepages the acceptance suite reserves at session start, so on a host whose memory is mostly reserved the cap allowed a tmpfs larger than the page cache can back. MemAvailable is not used in its place because it moves with page cache, which would make a generated config depend on when it was generated. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

The minimum-interval requirement was applied to each direction separately, but the two directions do not get the same window: RX runs for the whole wall clock while TX starts sleep_interval later, so a slow mtl_init costs TX an interval and hard-failed a healthy run at the default test_time=30. Requiring the intervals of RX only, which has the longest window and is the direction that proves frames arrived, keeps the rate floor without that boundary. TX is still graded whenever it has enough intervals, so a sender that stalls mid-run is caught. Requiring RX specifically rather than either direction also stops a receiver killed early in the stop ladder -- which _check_pipeline_exit tolerates by design -- from passing on the sender's counters. Also drops a branch guarding framerate against None: create_command resets params from UNIVERSAL_PARAMS, which always carries a framerate, and set_params cannot introduce a None, so the branch was unreachable and its comment described an outcome the code did not implement. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

Keep shared cases capability-driven and fail closed on early exits or invalid results. Restore unrelated setup/PTP and specialized suites so the PR remains test-focused without dropping coverage. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>

awilczyns requested a review from staszczuk as a code owner June 15, 2026 12:17

awilczyns force-pushed the gstreamer-pytest-refactor branch from debe4bb to cd630fd Compare June 15, 2026 12:19

DawidWesierski4 reviewed Jun 15, 2026

View reviewed changes

awilczyns force-pushed the gstreamer-pytest-refactor branch from 9b4e765 to c1d746d Compare July 1, 2026 10:43

DawidWesierski4 requested changes Jul 3, 2026

View reviewed changes

awilczyns force-pushed the gstreamer-pytest-refactor branch from 8982836 to cff5755 Compare August 28, 2026 15:46

awilczyns requested review from Sakoram, moleksy and soopel as code owners August 28, 2026 15:46

awilczyns force-pushed the gstreamer-pytest-refactor branch from cff5755 to 4e1f7b2 Compare August 28, 2026 15:51

awilczyns added 12 commits August 31, 2026 16:39

awilczyns force-pushed the gstreamer-pytest-refactor branch 2 times, most recently from 98d341b to f805a48 Compare August 31, 2026 14:57

Navigation Menu

Uh oh!

Test: make GStreamer a first-class Application adapter - #1614

Test: make GStreamer a first-class Application adapter#1614
awilczyns wants to merge 12 commits into
mainfrom
gstreamer-pytest-refactor

awilczyns commented Jun 15, 2026

Uh oh!

DawidWesierski4 left a comment

Uh oh!

DawidWesierski4 Jun 15, 2026

Uh oh!

DawidWesierski4 Jun 15, 2026

Uh oh!

DawidWesierski4 Jun 15, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

DawidWesierski4 Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		@pytest.mark.nightly
		@pytest.mark.parametrize("application", ["gstreamer"])

		for path in gst_cleanup:
		media_create.remove_file(path, host=host)


		"""GStreamer ST40P ancillary transport validation.

		Exercises ANC payload handling over GStreamer across frame rates, payload

Uh oh!

Conversation

awilczyns commented Jun 15, 2026

Uh oh!

DawidWesierski4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants