# Why is auto tuning with resnet failing at task 1? **URL:** \ **Category:** Troubleshooting\ **Created:** [June 24, 2021, 6:54pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316 "2021-06-24T18:54:42Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Codite](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/c/dec6dc/32.png) [@Codite](https://discuss.tvm.apache.org/u/Codite)\ **Post date:** [June 24, 2021, 6:54pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/1 "2021-06-24T18:54:42Z")
I’m following the compiling and automizing a model with tvmc tutorial and when I try to auto tune the resnet model I get this. I’m running macOS mojave version 10.14.6 and have installed the python dependencies needed for auto tuning. > [Task 1/25] Current/Best: 0.00/ 0.00 GFLOPS | Progress: (0/40) | 0.00 sTraceback (most recent call last): File “”, line 1, in File “/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py”, line 116, in spawn\_main exitcode = \_main(fd, parent\_sentinel) File “/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py”, line 126, in \_main self = reduction.pickle.load(from\_parent) ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 24, 2021, 7:47pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/2 "2021-06-24T19:47:21Z")
I’ll check this out on macOS. Sadly, we may be running into a thread safety bug that is present on non-Linux platforms. ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 24, 2021, 8:54pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/3 "2021-06-24T20:54:38Z")
I’m running into a similar issue on Big Sur. I can’t reproduce your development environment since I’m not on Mojave and don’t have access to a Mojave machine, but [this gist](https://gist.github.com/hogepodge/9fca9881816cd352dd34debd1152d379) includes the stack trace of my same run. Maybe @leandron has some insight into what’s going on, but my only advice for you is to run Docker Desktop on your Mac if that’s a possibility, and do TVM work inside of a Docker container, which will give you a Linux environment which has far better support. You will likely need to increase the memory available to Docker to build TVM inside of a container. ---
**Author:** ![leandron](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/leandron/32/1205_2.png) [@leandron](https://discuss.tvm.apache.org/u/leandron)\ **Post date:** [June 24, 2021, 9:23pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/4 "2021-06-24T21:23:14Z")
I’ll investigate and get back to you on this. Thanks for reporting @Codite @hogepodge ---
**Author:** ![AndrewZhaoLuo](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/andrewzhaoluo/32/2500_2.png) [@AndrewZhaoLuo](https://discuss.tvm.apache.org/u/AndrewZhaoLuo)\ **Post date:** [June 25, 2021, 7:04pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/5 "2021-06-25T19:04:31Z")
I think this is because of the way multiprocessing works on macs. See [multiprocessing — Process-based parallelism — Python 3.9.5 documentation](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) To fix simply add this line: ```auto import multiprocessing as mp mp.set_start_method("fork") ``` ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 25, 2021, 8:53pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/6 "2021-06-25T20:53:19Z")
Thanks Andrew. Is this a Mac specific fix, or will it work generally across all platforms if it’s added to `tvmc`? There’s similar behavior if we try to run any autoscheduling jobs in Jupyter Notebooks. We need to wrap them in ` __main__ ` to get them to work properly. ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 25, 2021, 9:22pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/7 "2021-06-25T21:22:43Z")
One more comment, from that document you listed: > Changed in version 3.8: On macOS, the _spawn_ start method is now the default. The _fork_ start method should be considered unsafe as it can lead to crashes of the subprocess. See [bpo-33725](https://bugs.python.org/issue33725). I’m not fully versed in the underlying issue behind this, but it seems like we might be trading one problem for another. ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 25, 2021, 9:39pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/8 "2021-06-25T21:39:52Z")
> [@AndrewZhaoLuo](#): > > `mp.set_start_method("fork")` A couple more items to follow up with, I attempted to add this patch to `tvmc` and ran into the same issue as before. There’s also a [nice article](https://wefearchange.org/2018/11/forkmacos.rst.html) about why macOS requires the use of `spawn` instead of `fork`. ---
**Author:** ![Codite](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/c/dec6dc/32.png) [@Codite](https://discuss.tvm.apache.org/u/Codite)\ **Post date:** [June 25, 2021, 9:41pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/9 "2021-06-25T21:41:36Z")
I was able to get it to work by adding a slight change to Andrew’s solution: ```auto import multiprocessing as mp mp.set_start_method("fork", force=True) ``` ---
**Author:** ![AndrewZhaoLuo](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/andrewzhaoluo/32/2500_2.png) [@AndrewZhaoLuo](https://discuss.tvm.apache.org/u/AndrewZhaoLuo)\ **Post date:** [June 25, 2021, 10:01pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/10 "2021-06-25T22:01:32Z")
Yeah, I’m not super well versed on this issue too but might read a little more into it this weekend. I will say that I haven’t been able to find a workaround otherwise (but I haven’t tried very hard) and using “spawn” requires some TVM objects to be picklable which they are not. ---
**Author:** ![AndrewZhaoLuo](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/andrewzhaoluo/32/2500_2.png) [@AndrewZhaoLuo](https://discuss.tvm.apache.org/u/AndrewZhaoLuo)\ **Post date:** [June 25, 2021, 10:05pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/11 "2021-06-25T22:05:34Z")
For most systems we care about (UNIX/LINUX) fork is the default behavior so this will not cause a change in those systems. For Mac it seems to work ok? I’m pretty sure this does cause errors on mac. For example, at runtime, I get a bug where we try to link to the same dynamic library twice which will cause an error. You can get around this by doing stuff like `export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` and it seems to not explode. I have no idea what happens on windows and have little information on whether windows follows the fork-exec model assumed under the python multiprocessing framework. ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 25, 2021, 10:23pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/12 "2021-06-25T22:23:18Z")
I’ve sent a patch up to attempt to address this issue. Would definitely appreciate feedback on if this is an appropriate solution to the problem. It both sets the start method and warns if the environment variable isn’t set. [Switch threading model to `fork` on macOS by hogepodge · Pull Request #8347 · apache/tvm · GitHub](https://github.com/apache/tvm/pull/8347) ---
**Author:** ![hogepodge](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hogepodge/32/1843_2.png) [@hogepodge](https://discuss.tvm.apache.org/u/hogepodge)\ **Post date:** [June 25, 2021, 11:25pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/13 "2021-06-25T23:25:38Z")
One more bit of feedback, apparently there’s active work to refactor away from using fork. It’s not complete, but [here’s an example of the RPC worker switching to POpenWorker](https://github.com/apache/tvm/pull/7889). I don’t have a deep understanding of the level of effort to do the same thing in `tvmc`, but it’s safe to say the approach I took in that pull request isn’t the correct or sustainable one. ---
**Author:** ![AndrewZhaoLuo](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/andrewzhaoluo/32/2500_2.png) [@AndrewZhaoLuo](https://discuss.tvm.apache.org/u/AndrewZhaoLuo)\ **Post date:** [June 28, 2021, 7:28pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/14 "2021-06-28T19:28:50Z")
@tkonolige probably also has things to say about this. The TL;DR is we need to change quite a few things to make things work. The hack up there is actually pretty unsafe and will cause a ton of bugs down the line. ---
**Author:** ![AndrewZhaoLuo](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/andrewzhaoluo/32/2500_2.png) [@AndrewZhaoLuo](https://discuss.tvm.apache.org/u/AndrewZhaoLuo)\ **Post date:** [June 28, 2021, 11:47pm UTC](https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/15 "2021-06-28T23:47:14Z")
Draft PR here should allow us to avoid this bad hack: > > > With this we can use the default (and safer) multiprocessing method "spawn" for …macOS and Windows rather than "fork" for tuning. More details here: https://discuss.tvm.apache.org/t/why-is-auto-tuning-with-resnet-failing-at-task-1/10316/12 > > cc @hogepodge @tkonolige @leandron