I’m pleased to note that I’ve added a simple script for clustering stack traces, allowing for duplicate stack traces to be easily grouped together and thus making it easier to start analyzing the bugs that have emerged. The script is in tests/scripts/relay_fuzzing/cluster_traces.py in the fuzzer proof-of-concept fork. (Note: It requires the Levenshtein library and SK-Learn.)
With this tool, I think it would be fair to say that the “fuzzing pipeline” here is complete: With the code in my proof-of-concept, it is possible to generate programs, run tests for different conditions on them, and analyze the resulting stack traces to start debugging. With these changes, I think it would be useful to start discussing how workable this infrastructure would be and perhaps to start moving on to a formal RFC.
In my small-scale tests, I found it effective to use a very simple form of clustering: Just agglomerative (hierarchical) clustering based on the Levenshtein distance (i.e., edit distance) between the string dumps of the stack traces. The script simply uses a similarity percentage (Levenshtein distance divided by the length of the longer of the two strings) with a 10% similarity cutoff. It’s a very simple method and requires no training.
In the future, we can explore using one of the more advanced methods for clustering stack traces if this proves inadequate in practice, but from my cursory examination, the resulting clusters made sense (different kinds of crashes were indeed in different clusters).
Special thanks to @ehsanmok for giving me some advice on this subject.