Can't link soft-float modules with double-float modules

Questions
#1

I have a question about function export_library, Here is my target

tgt = "llvm -mtriple=riscv64-unknown-linux-gnu -mcpu=generic-rv64 -mfloat-abi=hard"

and export_library use:

specialized = cc.cross_compiler("soft/riscv/new_install/bin/riscv64-unknown-linux-gnu-g++", ["-mabi=lp64d"]) 
fadd.export_library(path, specialized)

and then I get the error

soft/riscv/new_install/lib/gcc/riscv64-unknown-linux-gnu/10.1.0/../../../../riscv64-unknown-linux-gnu/bin/ld: lib/riscv.o: can't link soft-float modules with double-float modules
Command line: soft/riscv/new_install/bin/riscv64-unknown-linux-gnu-g++ -shared -fPIC -o lib/riscv.so lib/riscv.o

my cross compiler’s configure is

/soft/riscv/new_install/libexec/gcc/riscv64-unknown-linux-gnu/10.1.0/lto-wrapper
Target: riscv64-unknown-linux-gnu
Configured with: soft/riscv/riscv-gnu-toolchain/riscv-gcc/configure --target=riscv64-unknown-linux-gnu --prefix=soft/riscv/new_install --with-sysroot=soft/riscv/new_install/sysroot --with-system-zlib --enable-shared --enable-tls --enable-languages=c,c++,fortran --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libsanitizer --disable-nls --disable-bootstrap --src=.././riscv-gcc --enable-multilib --with-abi=lp64d --with-arch=rv64imafdc --with-tune=rocket 'CFLAGS_FOR_TARGET=-O2   -mcmodel=medlow' 'CXXFLAGS_FOR_TARGET=-O2   -mcmodel=medlow'
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (GCC)

it seems the -mfloat-abi=hard in Target doesn’t make sense, so how can I generate a double float so file, anyone has some ideas, thanks!

1 Like
#2

so did you solve this problem?

#3

No, I didn’t solve the problem, but I have found that it is llvm’s problem, its support about riscv need to be improved.

#4

so do you know how to improve it or how to make tvm program run on riscv?

#5

As far as I know, microTVM has supported qemu-riscv, you can find some information in tvm/apps/microtvm, or search for some discussions in this forum, for example here.

#6

generic-rv64 only supports I instructions (RV64I Base Integer instruction set). To use MAFD instructions (aka RV64GC) we can specify -mcpu=sifive-u54 in llvm string.

I saved the lib to assembly file to check the generated instructions

lib.save("model.s")

-mcpu=sifive-u54 assembly code shows that it uses MAFD and C instructions (RV64GC)

Line 3 in model.s

.attribute      5, "rv64i2p0_m2p0_a2p0_f2p0_d2p0_c2p0"

In the file I can find the following instructions

mulw
fdiv.s
fadd.s

A bit strange that I do not see double precision instructions such as fdiv.d…

Problem is that I still have the issue with linking

riscv64-unknown-linux-gnu/bin/ld: /tmp/tmp_zy2om28/lib0.o: can't link soft-float modules with double-float modules

Why lib0.o is soft-float??? The assembly code clearly shows that it uses M and F instructions.

#7

Looks like the issue is in llc. I can not compile test program LLVM IR code to “hard float” object file.

The following example shows the issue

Test C programm (test2.c):

int plus_func(int a) {
    int b = a + 4;
    return b;
}

int main() {
  return plus_func(5);
}

Lets convert it to LLVM IR code (test2.ll file)

clang-11 \
--target=riscv64-linux-gnu -march=rv64gc \
-S -emit-llvm test2.c

Now we can compile test2.ll IR code to object file using llc. That is what TVM is doing under the hood.

llc-11 \
-mtriple=riscv64-linux-gnu \
-mcpu=sifive-u54 \
-relocation-model=pic \
-filetype=obj \
test2.ll -o test2.o

Now lets try to link object test2.o file to an executable file test2

clang-11 \
--target=riscv64-linux-gnu -march=rv64gc \
test2.o -o test2

Similar to TVM it fails:

/usr/bin/riscv64-linux-gnu-ld: test2.o: can't link soft-float modules with double-float modules
/usr/bin/riscv64-linux-gnu-ld: failed to merge target specific data of file test2.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Update - missing llc parameter is -target-abi=lp64d.

#8

We can actually use clang to compile and link LLVM IR model.ll directly to shared library model.so.

LLVM target string: (Probably it is better to add device=arm_cpu flag to llvm string in order to use small edge devices schedules instead of default)

target = 'llvm -mtriple=riscv64-unknown-linux-gnu -mcpu=sifive-u54 -device=arm_cpu'

Save relay.build output (lib) to LLVM IR file model.ll using lib.save()

with tvm.transform.PassContext(opt_level=3):
    graph, lib, params = relay.build(mod, target=target, params=params)

lib.save("model.ll")

Compile and link model.ll directly to shared library model.so.

clang-11 \
--target=riscv64-unknown-linux-gnu -march=rv64gc \
-shared -fPIC -O3 \
model.ll -o model.so
#9

I haven’t tied to generate double float object file with clang, but can get it by riscv64-unknown-linux-gnu-gcc, and here in TVM can get the llvm module object file,

can you explain more about it, except for SaveToFile function or calling external compilers, I don’t know where it calls for llc to get object file, thanks!

#10

About llc - The result of relay lib.save("model.o") and llc model.ll -o model.o are identical.

For Example, lets use TVM Relay API to save the lib.

lib.save("model.ll")
lib.save("model.o")

Now lets convert model.ll to model2.o using llc and compare it with model.o above

llc-11 \
-relocation-model=pic \
-mtriple=riscv64-unknown-linux-gnu \
-mcpu=sifive-u54 \
-filetype=obj \
model.ll -o model2.o

md5sum model.o
ab74e74c1dfba0de0aff556ab217be8d
md5sum model2.o
ab74e74c1dfba0de0aff556ab217be8d

The files are identical. It indicates that TVM uses the same approach as llc to save model as an object file.

BTW, we can also use clang to convert model.ll to model.o.

clang++-11 \
--target=riscv64-unknown-linux-gnu -march=rv64gc \
-fPIC -O3 \
-c model.ll -o model.o

clang generated object file is “hard-float” and can be linked to shared library just fine (even with gcc). (in contrast, llc generated object file still has soft-float issue)

riscv64-linux-gnu-g++-10 \
-march=rv64gc \
-shared -static-libstdc++ \
model.o -o model.so

or

clang++-11 \
--target=riscv64-unknown-linux-gnu -march=rv64gc \
-shared -static-libstdc++ \
model.o -o model.so

Both commands above work fine with clang generated model.o

#11

OK, thanks for your explaining.

#12

I found missing llc parameter. It is -target-abi=lp64d. (clang uses it too (it is visible in --verbose mode))

If we add it to llc then it generates hard-float (aka double-float) object file.

We also need to fix TVM to recognize this llvm parameter. Looks like the parameter was added to llvm couple years ago.

# enable double-float in clang/llc
-target-abi=lp64d
-target-abi=ilp32d

BTW, llc --help does not show -target-abi. Another place for improvement.

#13

Thanks ! I have tied a couple of months to find the proper parameter to let llvm generate the double float object file.

#14

hello, where should I put -target-abi=lp64d in? in the targer of tvm.build?

alright, I have tried put it in, but I received the error like this:

Cannot recognize 'target-abi'. Candidates are: unpacked-api, interface-api, keys, link-params, device, mcpu, host, from_device, mattr, mtriple, tag, model, system-lib, runtime, libs, mfloat-abi. Target creation from string failed: llvm -mtriple=riscv64-unknown-linux-gnu -mcpu=generic-rv64 -mfloat-abi=hard -target-abi=lp64d
#15

For now we can use TVM to generate LLVM IR code (model.ll).

target = 'llvm -mtriple=riscv64-unknown-linux-gnu -mcpu=sifive-u54 -device=arm_cpu'
with tvm.transform.PassContext(opt_level=3):
    graph, lib, params = relay.build(mod, target=target, params=params)

lib.save("model.ll")

After that model.ll should be converted to object file (model.o). We can use llc or clang for that.

llc \
-mtriple=riscv64-unknown-linux-gnu \
-target-abi=lp64d \
-mcpu=sifive-u54 \
-filetype=obj \
-relocation-model=pic -O3 \
model.ll -o model.o

or

clang++ \
--target=riscv64-unknown-linux-gnu \
-march=rv64gc \
-fPIC -O3 \
-c model.ll -o model.o

Then we can use gcc or clang to convert (link) model.o to model.so

riscv64-linux-gnu-g++ \
-march=rv64gc \
-shared -static-libstdc++ \
model.o -o model.so

or

clang++ \
--target=riscv64-unknown-linux-gnu \
-march=rv64gc \
-shared -static-libstdc++ \
model.o -o model.so

To avoid the steps above we need to fix TVM to recognize -target-abi=lp64d llvm parameter in target string. After that we can use relay.build / lib.export_library as usual.

#16

well, I have upgrate tvm, but tvm still cannot recognize -target-abi string, like the error above

#17

We need to fix TVM to recognize -target-abi=lp64d llvm parameter in target string

#18

The fix Add support for llvm parameter -target-abi by apivovarov · Pull Request #8860 · apache/tvm · GitHub

#19

Looks like we also need to fix devc.o generation in case when relay.build generates single output - GraphExecutorFactoryModule instead of three outputs graph, lib, params.

with tvm.transform.PassContext(opt_level=3):
    #graph, lib, params = relay.build(mod, target=target, target_host=target_host, params=params)
    lib = relay.build(mod, target=target, target_host=target_host, params=params)

lib.export_library(path_lib, cc="riscv64-linux-gnu-g++-10", options=["-march=rv64gc","-mtune=sifive-u54", "-mabi=lp64d"])

/usr/lib/gcc-cross/riscv64-linux-gnu/10/../../../../riscv64-linux-gnu/bin/ld: error: /tmp/tmp3yrlbrsk/devc.o: ISA string of input (rv32i2p0) doesn't match output (rv64i2p0_m2p0_a2p0_f2p0_d2p0_c2p0).

Update: devc.o compilation was fixed too - the PR was updated.

#20

Well, can we specify the -target-abi in target of tvm.build now? Or we stilll need to generate .ll code and specify it in clang and llc?