# \[pre-RFC\] New CI container: ci\_cpu\_asserts **URL:** \ **Category:** RFC\ **Created:** [April 11, 2022, 6:48pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536 "2022-04-11T18:48:59Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![areusch](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/areusch/32/2118_2.png) [@areusch](https://discuss.tvm.apache.org/u/areusch)\ **Post date:** [April 11, 2022, 6:48pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/1 "2022-04-11T18:48:59Z")
Hi all, I wanted to discuss the idea of adding a new CI container: `ci_cpu_asserts`. This container would behave just like `ci_cpu`, except that both TVM and various libraries within the container would be compiled with additional runtime checks enabled. For example: - TVM could be configured with `-DUSE_RELAY_DEBUG` - LLVM could be configured with `-DUSE_LLVM_ASSERTIONS` - we could consider other runtime checkers such as AddressSanitizer In CI, we would build TVM and run `tests/python/task_python_unittest.sh` against it. There are a couple of scenarios this would benefit: - [PR 9842](https://github.com/apache/tvm/pull/9842) proposes to add a concurrent-update check to tvm’s custom `Map` impl, but due to a limitation in LLVM, we cannot compile with `NDEBUG` defined. Therefore, we would like to compile with `USE_RELAY_DEBUG`, but that could slow down integration tests in the CI. - Our LLVM backends are not really tested as well as they should be–LLVM assertions performs proper verification of the emitted IR. All 2 major changes I’ve made (link-params, AOT LLVM backend) have had problems only found with `-DLLVM_ENABLE_ASSERTIONS` (in some cases, these were minor; in others, they would cause functional correctness problems). - There is no home for this checking now, meaning that it’s controversial to add it since it slows down CI. Perhaps we can discuss this at the TVM Community Meeting on Wednesday. cc @kparzysz @dmitriy-arm @driazati Andrew ---
**Author:** ![driazati](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/driazati/32/3303_2.png) [@driazati](https://discuss.tvm.apache.org/u/driazati)\ **Post date:** [April 11, 2022, 6:54pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/2 "2022-04-11T18:54:50Z")
Full support from me! This shouldn’t affect CI runtime too much with build caching + test sharding we can get the runtime down to something similar to existing stages (so runtime will be hidden via parallelism). Also I don’t know if it’s implied by `USE_RELAY_DEBUG` but we should also build it with `-DCMAKE_BUILD_TYPE=Debug` since we’ve had some breakages of the debug build in the past that slipped through CI. I’d also propose we build with the `clang` associated with whatever LLVM version we use just to increase CI-tested compilers ---
**Author:** ![kparzysz](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/kparzysz/32/252_2.png) [@kparzysz](https://discuss.tvm.apache.org/u/kparzysz)\ **Post date:** [April 11, 2022, 7:11pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/3 "2022-04-11T19:11:05Z")
We could have a daily build of the `main` branch of LLVM. We could build `clang`, not just the LLVM codegen libraries. ---
**Author:** ![areusch](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/areusch/32/2118_2.png) [@areusch](https://discuss.tvm.apache.org/u/areusch)\ **Post date:** [April 11, 2022, 10:36pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/4 "2022-04-11T22:36:56Z")
let’s discuss this a bit more Wednesday and then I’ll wrap it up into an RFC and we can go forward from there ---
**Author:** ![mbs-octoml](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/mbs-octoml/32/2939_2.png) [@mbs-octoml](https://discuss.tvm.apache.org/u/mbs-octoml)\ **Post date:** [April 13, 2022, 3:28pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/5 "2022-04-13T15:28:59Z")
Notes from the TVM community meeting, 13-Apr. (Agenda doc is [Apache TVM Weekly Community Meeting - Google Docs](https://docs.google.com/document/d/179l1WzqDiLXjjLmLopvxBSr0ylFS2plyXbeTtB8CteM/edit#)) Andrew: Motivation was to add small-but-non-trivial checking overhead to Map. How can we get the benefit without forcing onto all runtime? NDEBUG is difficult due to LLVM issues, so USE\_RELAY\_DEBUG. Currently never enabled in any CI build. Would also use to enable LLVM checker on constructed IR to catch ill-formed constructions (which is very easy to trigger). In the future could use asan. Mark: Policy on ICHECK vs DCHECK? Issue is DCHECK’s can go stale, so depend on regular USE\_RELAY\_DEBUG runs. Let’s do that and pay down any existing debt, then can migrate ICHECKS to DCHECKS where they are expensive. Michael: Use LLVM debug build in USE\_RELAY\_DEBUG mode? Currently using standard Ubuntu build. Hoping to stay in standard x86 targets, so should be ok to build from LLVM HEAD. Leandro: Used to have to build LLVM from source, so can help point to ‘standard’ flags. Should standardize on versions & config. Andrew: We tend to be very conservative on LLVM version. Mark: Some tests or all tests? Ideally choose existing split point (eg unit vs python tests). Mark: Could use this to enable post-Pass self-invariant checks. Mark: Can we turn on stdlib DEBUG asserts? Not sure how, but seems like a good idea. Overall very enthusiastic, thank you Andrew! ---
**Author:** ![leandron](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/leandron/32/1205_2.png) [@leandron](https://discuss.tvm.apache.org/u/leandron)\ **Post date:** [April 13, 2022, 4:01pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/6 "2022-04-13T16:01:53Z")
@kparzysz @areusch FWIW, we already have a LLVM builder script being used in `tlcpack`, It was originally written by @haichen. > We could try to reuse this in TVM, with a few change to make it similar to the other scripts we have. ---
**Author:** ![kparzysz](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/kparzysz/32/252_2.png) [@kparzysz](https://discuss.tvm.apache.org/u/kparzysz)\ **Post date:** [April 13, 2022, 4:44pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/7 "2022-04-13T16:44:46Z")
A lot of what that script does has to do with dealing with various versions of LLVM source distributions. For our purposes none of that is necessary. Building LLVM is really simple, it’s 1. `git clone` (for building the latest `main` branch. 2. `cmake ...` 3. `make all install` ---
**Author:** ![areusch](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/areusch/32/2118_2.png) [@areusch](https://discuss.tvm.apache.org/u/areusch)\ **Post date:** [April 13, 2022, 5:06pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/8 "2022-04-13T17:06:56Z")
I think the main question here is what to pass in `...` given to `cmake`. Any thoughts on which flags we may want for this build? ---
**Author:** ![kparzysz](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/kparzysz/32/252_2.png) [@kparzysz](https://discuss.tvm.apache.org/u/kparzysz)\ **Post date:** [April 13, 2022, 5:21pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/9 "2022-04-13T17:21:53Z")
The minimum set of flags would be ```auto -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/where/to/install -DLLVM_ENABLE_ASSERTIONS:BOOL=ON -DLLVM_ENABLE_PROJECTS='llvm' ``` Edit: This will build LLVM only, i.e. the headers and codegen libraries. If we want to build a compiler that can compile C/C++ code, we’d need to do something else (there are scripts to build an entire release). ---
**Author:** ![cbalint13](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/cbalint13/32/329_2.png) [@cbalint13](https://discuss.tvm.apache.org/u/cbalint13)\ **Post date:** [April 13, 2022, 5:46pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/10 "2022-04-13T17:46:10Z")
@areusch , @mbs-octoml - Mark, the flags you asked in TVM Meeting (boundary check): `-Wp,-D_GLIBCXX_ASSERTIONS` - For the debug (-g) there is a neat way to separate debuginfo out from binaries: - If interested to look into native fedora & rhel packaging, there is a public (automated) effort: ---
**Author:** ![mbs-octoml](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/mbs-octoml/32/2939_2.png) [@mbs-octoml](https://discuss.tvm.apache.org/u/mbs-octoml)\ **Post date:** [April 13, 2022, 8:17pm UTC](https://discuss.tvm.apache.org/t/pre-rfc-new-ci-container-ci-cpu-asserts/12536/11 "2022-04-13T20:17:07Z")
Thanks @cbalint13, that’s going into by config.cmake right now.