We discussed a few aspects of this at the community meeting this morning:
- When should we generate programs to test? When should the tests be run?
- @slyubomirsky Other fuzzers have many different approaches, some generate and save programs and others generate them each time. Maybe we should do both in order to test the fuzzer
- @driazati we should get the fuzzer running in CI at some scale now, we can grow or change it as we try it out
- As many (potentially thousands) of generated programs hit the same kinds of bugs, how do we group stack traces to simplify error reporting / bug hunting?
- @slyubomirsky Fuzzing frameworks also differ here, some look at parts of the stack trace
- @gromero Maybe it’s possible to look at program features to identify what triggers similar bugs
- How do we save generated programs without relying on the Relay text format?
-
@slyubomirsky
pickleing the IRModule works out of the box but is pretty opaque -
@driazati
picklemight be ok, we can probably disable the arbitrary code execution parts of it (another note: pickle also has a text format that might be easier to stomach, but there is no parser for it afaik) - @sebastianboblestetas We should do the JSON format if the serializer is simple enough
-
@slyubomirsky