OSG-Q3 — Robust OpenSCENARIO Generation with LLMs

The challenge

ADAS testing begins with requirements written for people: a vehicle approaches a collision point at a given speed, another road user enters its path, and the system must respond under precise conditions. A simulator, however, needs those ideas expressed as valid, internally consistent OpenSCENARIO XML.

Creating these executable scenarios by hand is slow and specialist-heavy. Asking an LLM to emit XML directly is faster, but it frequently introduces syntax errors, hallucinated assets, or scenario logic that does not match the available road network. Valid-looking text is not enough: the artifact must run and reproduce the requested behavior.

OSG-Q3 addresses that gap with a simulator-aware pipeline rather than treating scenario generation as a single prompting task.

Framework

The core design separates semantic reasoning from XML authoring. The LLM reasons about the requested scenario and produces a constrained JSON representation; deterministic software then converts that representation into OpenSCENARIO XML. Simulator-specific context is acquired before generation, and execution feedback is used to evaluate the result after conversion.

The pipeline has five stages:

1. Natural-language input. A practitioner describes the actors, road context, speeds, maneuvers, and expected interaction in ordinary language.
2. Simulator-aware JSON generation. An LLM-based agent can query available OpenDRIVE maps, road topology, lane configurations, and simulator catalogs instead of inventing unavailable resources.
3. Deterministic XML conversion. The structured JSON captures the road network, entities, and storyboard, while post-processing and a dedicated converter produce the executable .xosc file.
4. Scenario execution. The generated artifact runs in the Esmini simulator, turning structural validity into observable behavior.
5. Checklist-based evaluation. Execution traces are converted into an observed checklist covering spawn state, lane behavior, speed profiles, and interactions with surrounding entities.

This boundary is important: the LLM handles ambiguous language and scenario semantics, while deterministic code handles a strict serialization format. Tools ground the model in the simulator environment before generation, and execution checks whether the final scenario behaves as requested.

Evaluation

The benchmark contains 162 collected ADAS test cases written in natural language and spanning multiple ADAS functions. Each output is assessed as an executable artifact rather than scored only for textual similarity or XML syntax.

That distinction makes the evaluation closer to the real engineering objective. A scenario succeeds only when it can be executed and its observed simulation trace satisfies the criteria derived from the original requirement.

Results

Direct inference with Sonnet 4.6 achieved a 36.4% scenario success rate. Using the OSG-Q3 framework raised that result to 62.3%, a gain of 25.9 percentage points. The strongest evaluated configuration paired OSG-Q3 with a fine-tuned Qwen3-4B model and reached 76.5%.

Under the same framework setting, the fine-tuned Qwen3-4B result exceeded Sonnet 4.6 by 14.2 percentage points. Together, the experiments show that both the intermediate representation and model specialization contribute to reliability; the orchestration around the model matters as much as raw model capability.

My role

I served as the first author of OSG-Q3, working with Huy Tran, Dac Hoang Phu Nguyen, and Duc Dung Nguyen at AITech Lab, Ho Chi Minh City University of Technology, VNUHCM.

The architecture, source code, and executable examples are available through the links above. The best result still leaves room for improvement, but it demonstrates a practical direction for LLM-assisted simulation workflows: constrain generation, ground it in real simulator assets, convert deterministically, and judge the output by what happens when it runs.