How to use the mutaml-runner?

[af-afk]

Hello, I followed the instructions in the README for my repository https://github.com/af-afk/ocaml-riscv32im-stylus:

dune build test --instrument-with mutaml
mutaml-runner _build/default/test/test_riscv32im.exe

But no success:

Could not read file mutaml-mut-files.txt - _build/default/mutaml-mut-files.txt: No such file or directory

What should I do to fix this? Thank you. 😄

[jmid]

I've had an initial look at this. First I tried building the project under a 5.3.0 switch.
libbinutils/dune-project seems to require dune.3.19.0 which didn't fit with my existing 3.18.0 so I relaxed that requirement to agree with that of dune-project:

diff --git a/libbinutils/dune-project b/libbinutils/dune-project
index f0a1933..d904c8e 100644
--- a/libbinutils/dune-project
+++ b/libbinutils/dune-project
@@ -1,4 +1,4 @@
-(lang dune 3.19)
+(lang dune 3.18)
 (using ctypes 0.3)
 
 (name libbinutils)

Then I looked at dependencies in the opam-file and found only ocaml and dune.
One after another I however had to install a bunch:

ppx_compare
ppx_hash
ppx_inline_test
ppx_deriving_qcheck
ppx_accessor
ctypes

as well as a binutils-dev thinking we should update the opam repo's conf-binutils and make it a dependency.
Afterwards I spotted dependencies listed in dune-project along with a (generate_opam_files true)
and indeed dune build riscv32im_stylus.opam adds a bunch of missing entries 👍

I haven't ever used ppx_inline_test myself and that dependency worried me,
as it may be incompatible with the current version of mutaml if it runs the inline tests
in a separate shell (nested runs each with their own muts files will likely break).

As an experiment I therefore removed the dependency, and interestingly it doesn't seem to be used:

diff --git a/lib/dune b/lib/dune
index 2dd2baf..f4cebc1 100644
--- a/lib/dune
+++ b/lib/dune
@@ -17,7 +17,6 @@
   (backend mutaml))
  (preprocess
   (pps
-   ppx_inline_test
    ppx_deriving.show
    ppx_deriving.eq
    ppx_deriving.make

With those 2 changes and 7 6 package installs I was thus able to run dune build test -- sans instrumentation
under ocaml.5.3.0 and dune.3.18.0.

Now came the moment of truth:

$ dune clean
$ dune build test --instrument-with mutaml
Running mutaml instrumentation on "lib/control.ml"
Randomness seed: 213608853   Mutation rate: 50   GADTs enabled: true
Created 0 mutations of lib/control.ml
Writing mutation info to lib/control.muts
Running mutaml instrumentation on "lib/riscv32im_stylus.ml"
Randomness seed: 698099741   Mutation rate: 50   GADTs enabled: true
Created 0 mutations of lib/riscv32im_stylus.ml
Writing mutation info to lib/riscv32im_stylus.muts
Running mutaml instrumentation on "lib/calldata.ml"
Randomness seed: 45138819   Mutation rate: 50   GADTs enabled: true
Created 4 mutations of lib/calldata.ml
Writing mutation info to lib/calldata.muts
Running mutaml instrumentation on "lib/ethereum.ml"
Randomness seed: 648278886   Mutation rate: 50   GADTs enabled: true
Created 40 mutations of lib/ethereum.ml
Writing mutation info to lib/ethereum.muts
Running mutaml instrumentation on "lib/encoding.ml"
Randomness seed: 871301006   Mutation rate: 50   GADTs enabled: true
Created 63 mutations of lib/encoding.ml
Writing mutation info to lib/encoding.muts
Running mutaml instrumentation on "lib/decoding.ml"
Randomness seed: 117340598   Mutation rate: 50   GADTs enabled: true
Created 46 mutations of lib/decoding.ml
Writing mutation info to lib/decoding.muts
Running mutaml instrumentation on "lib/operation.ml"
Randomness seed: 734209393   Mutation rate: 50   GADTs enabled: true
Created 22 mutations of lib/operation.ml
Writing mutation info to lib/operation.muts
Running mutaml instrumentation on "lib/memory.ml"
Randomness seed: 557158814   Mutation rate: 50   GADTs enabled: true
Created 35 mutations of lib/memory.ml
Writing mutation info to lib/memory.muts
Running mutaml instrumentation on "lib/simulator.ml"
Randomness seed: 748773248   Mutation rate: 50   GADTs enabled: true
Created 14 mutations of lib/simulator.ml
Writing mutation info to lib/simulator.muts
Running mutaml instrumentation on "lib/registers.ml"
Randomness seed: 796614891   Mutation rate: 50   GADTs enabled: true
Created 37 mutations of lib/registers.ml
Writing mutation info to lib/registers.muts
Running mutaml instrumentation on "lib/lifted.ml"
Randomness seed: 490820123   Mutation rate: 50   GADTs enabled: true
Created 47 mutations of lib/lifted.ml
Writing mutation info to lib/lifted.muts

Thus encouraged, I tried the runner and got the same behavior as you: 😬

$ mutaml-runner _build/default/test/test_riscv32im.exe 
Could not read file mutaml-mut-files.txt - _build/default/mutaml-mut-files.txt: No such file or directory

The weird thing is that the instrumentation output tells us that these files are written to the file system.
However afterwards there seems to be no .muts-files in _build/default/lib 🤷

On another run I encountered an instrumentation bug which needs more looking into:

$ dune build test --instrument-with mutaml
Running mutaml instrumentation on "lib/control.ml"
Randomness seed: 67375296   Mutation rate: 50   GADTs enabled: true
Created 0 mutations of lib/control.ml
Writing mutation info to lib/control.muts

[...]

Running mutaml instrumentation on "lib/registers.ml"
Randomness seed: 239122492   Mutation rate: 50   GADTs enabled: true
Created 29 mutations of lib/registers.ml
Writing mutation info to lib/registers.muts
Running mutaml instrumentation on "lib/lifted.ml"
Randomness seed: 178157981   Mutation rate: 50   GADTs enabled: true
Created 43 mutations of lib/lifted.ml
Writing mutation info to lib/lifted.muts
File "_none_", line 1:                
Error: The value __MUTAML_TMP1__ has type int32
       but an expression was expected of type int

So all in all, there seems to be 2 issues in play here:

• missing muts-files and
• a type-breaking int/int32 instrumentation

[af-afk]

Thanks for looking into this, and also for the reminder to make this repo more user friendly/possible to build. That's a shame about the issues. I'll look into it if I can!

[jmid]

I've gotten a bit further on this now.
First of all building without parallelism seems to help reproducibility: dune build -j1 test --instrument-with mutaml
I also had more luck with dune.3.18.2 than dune.3.19.1 (but haven't investigated that part further) 🤔

Second, the problem with the type-breaking int/int32 instrumentation is the following line in lib/simulator.ml which shadows the regular addition over ints:

let (+) = Int32.add

This becomes a problem for step_addi defined as follows:

let step_addi t dst src imm =
  let { r ; _ } = t in
  bump_pc
    { t with r = Registers.update r dst ((Registers.get r src) + imm) }

as mutaml will try a permutation replacing (a shadowed) + by a(n unshadowed) -, defined as subtraction over ints and thus introducing the type error:

let step_addi t dst src imm =
  let { r;_} = t in
  bump_pc
    {
      t with
      r =
        (Registers.update r dst
           (let __MUTAML_TMP0__ = Registers.get r src in
            if __is_mutaml_mutant__ "lib/simulator:0"
            then __MUTAML_TMP0__ - imm
            else __MUTAML_TMP0__ + imm))
    }

A workaround can be to simply shadow both operations:

let (+) = Int32.add
let (-) = Int32.sub

With that workaround, it is down to missing muts-files...