import torch
from torchvision import transforms
from modeling.deeplab import *
import numpy as np
from PIL import Image
device = torch.device('cuda:0')
model = DeepLab(num_classes=4,
backbone='drn',
output_stride=16,
sync_bn=False,
freeze_bn=False)
model_file='/path/name.pt'
checkpoint = torch.load(model_file, map_location=device)
model.load_state_dict(checkpoint['state_dict'])
model.to(device)
model.eval()
preprocess = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])
# input = numpy.randn(1,3,720,720,requires_grad=True,device=device)
input_image = Image.fromarray(np.zeros((720,720,3),dtype=np.uint8))
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0)
print(input_batch.shape)
input_batch=input_batch.cuda()
torch.onnx.export(model,input_batch,'./onnx_test.onnx',verbose=True,
export_params=True,opset_version=13,
do_constant_folding=True,
input_names=['input'],output_names=['output'],
dynamic_axes={'input' : {0 : 'batch_size'}, # variable length axes
'output' : {0 : 'batch_size'}})
위는 deeplabv3+를 사용하기 위해 onnx로 변환한 코드입니다. 참고 부탁드립니다.
아래는 litmus를 실행했을 때 나오는 로그입니다.
(deeplab) (base) elicer@b4814eddfa30:~$ furiosa-litmus test/onnx_test.onnx
libfuriosa_hal.so --- v0.11.0, built @ 43c901f
furiosa-quantizer 0.10.2 (rev. d51ae81) furiosa-litmus 0.10.2 (rev. d51ae81)
[Step 1] Checking if the model can be loaded and optimized ...
INFO:Furiosa-Optimizer:Dynamic batch size is detected at input_name: input. Fix batch_size=1 for valid shape inference.
2024-08-13 08:51:00.747664396 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35772, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
(중략)
2024-08-13 08:51:11.871949973 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35945, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:11.872011575 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35947, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:11.872034651 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35948, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:11.872003991 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35943, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887522710 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35958, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887541219 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35957, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887584444 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35955, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887626192 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35956, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887627703 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35960, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:12.887601287 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35959, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807492535 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35961, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807521823 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35963, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807506835 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35962, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807590782 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35966, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807541914 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35965, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:13.807511339 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35964, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.706946321 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35976, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.706938765 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35973, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.706964373 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35975, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.707036467 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35978, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.707050532 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35974, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:14.707050563 [E:onnxruntime:Default, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35977, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
[Step 1] Passed
[Step 2] Checking if the model can be quantized ...
2024-08-13 08:51:15.362629003 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35991, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:15.362650787 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35990, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:15.362685473 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35993, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:15.362635019 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35989, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:15.362653953 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35992, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:51:15.362709747 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 35994, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
^[[1;2B[Step 2] Passed
[Step 3] Checking if the model can be compiled for the NPU family [warboy-2pe] ...
[1/6] 🔍 Compiling from onnx to dfg
Done in 1.6024663s
[2/6] 🔍 Compiling from dfg to ldfg
Done in 94.19187s
[3/6] 🔍 Compiling from ldfg to cdfg
Done in 0.0117364s
[4/6] 🔍 Compiling from cdfg to gir
Done in 0.06971939s
[5/6] 🔍 Compiling from gir to lir
Done in 4.5553904s
[6/6] 🔍 Compiling from lir to enf
Done in 0.92898965s
✨ Finished in 101.36024s
ok: compiled successfully! (output.enf)
[Step 3] Passed
[Step 4] Perform inference once for data collection... (Optional)
✨ Finished in 0.000005046s
2024-08-13 08:53:47.328838080 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36580, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.329024901 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36582, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.329313416 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36581, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.329505322 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36585, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.332979725 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36587, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.329612050 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36583, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.329585220 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36586, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.333014726 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36584, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.333657099 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36594, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.333747487 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36593, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.334550890 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36591, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.344967867 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36589, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.333608616 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36590, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.359411489 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36595, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.359451447 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36592, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.360148912 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36597, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.377673608 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36588, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.381061418 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36596, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543003484 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36599, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543032483 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36601, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543022050 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36600, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543080313 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36602, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543104708 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36603, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.543168241 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36604, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.691672175 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36607, index: 2, mask: {2, 8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.691679110 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36606, index: 1, mask: {1, 7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.691663416 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36605, index: 0, mask: {0, 6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.691770172 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36609, index: 4, mask: {4, 10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.691729724 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36608, index: 3, mask: {3, 9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-08-13 08:53:47.692972995 [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36610, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
======================================================================
This benchmark was executed with latency-workload which prioritizes latency of individual queries over throughput.
1 queries executed with batch size 1
Latency stats are as follows
QPS(Throughput): 1.92/s
Per-query latency:
Min latency (us) : 521734
Max latency (us) : 521734
Mean latency (us) : 521734
50th percentile (us): 521734
95th percentile (us): 521734
99th percentile (us): 521734
99th percentile (us): 521734
[Step 4] Finished
pytorch에서 학습한 deeplabv3+ 모델을 가지고 onnx로 변환하는 과정을 거쳐 생성한 onnx파일을 사용하였습니다. furiosa litmus를 사용했을 때 계속 이러한 로그가 반복됩니다. [E:onnxruntime:, env.cc:251 ThreadMain] pthread_setaffinity_np failed for thread: 36610, index: 5, mask: {5, 11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set. 그러나 litmus는 모든 스텝을 마칩니다.
또, 단순히 onnx를 session create하였을 때 일어나는 부분입니다. 양자화는 진행하지 않고 onnx 변환 직후 바로 실행하였으며, 아래는 그 코드입니다.
from furiosa.runtime import session
sess = session.create('/home/elicer/test/onnx_test.onnx')
아래는 위 코드를 실행했을 때 나타나는 에러입니다. 참고로 이미지를 넣는 코드이며 input size는 720x720픽셀입니다.
2024-08-13T09:00:21.798633Z INFO furiosa_rt_core::driver::event_driven::coord: FuriosaRT (v0.10.4, rev: 320a178a7, built at: 2024-05-28T12:51:47Z) bootstrapping ...
2024-08-13T09:00:21.802823Z INFO furiosa_rt_core::driver::event_driven::coord: Found furiosa-compiler (v0.10.1, rev: 8b00177, built at: 2023-11-23T02:22:00Z)
2024-08-13T09:00:21.802832Z INFO furiosa_rt_core::driver::event_driven::coord: Found libhal (type: warboy, v0.12.0, rev: 56530c0 built at: 2023-11-16T12:37:25Z)
2024-08-13T09:00:21.802839Z INFO furiosa_rt_core::driver::event_driven::coord: [Runtime-0] detected 1 NPU device(s):
2024-08-13T09:00:21.832435Z INFO furiosa_rt_core::driver::event_driven::coord: - [0] npu:4:0-1 (warboy-b0-2pe, 128dpes, firmware: 1.7.8, e9f371e)
2024-08-13T09:00:21.832567Z INFO furiosa_rt_core::driver::event_driven::coord: [Runtime-0] started
2024-08-13T09:00:21.833885Z INFO furiosa::runtime: Saving the compilation log into /home/elicer/.local/state/furiosa/logs/compiler-20240813090021-46agqu.log
2024-08-13T09:00:21.933678Z INFO furiosa_rt_core::driver::event_driven::coord: [Runtime-0] created Sess-9cde10f0 using npu:4:0-1
2024-08-13T09:00:21.962598Z INFO furiosa_rt_core::driver::event_driven::coord: [Sess-9cde10f0] compiling the model (target: warboy-b0-2pe, 128dpes, file: onnx_test.onnx, size: 155.4 MiB)
[1/6] 🔍 Compiling from onnx to dfg
ERROR: shape range exceeds the buffer
2024-08-13T09:00:23.173733Z INFO furiosa_rt_core::driver::event_driven::coord: compilation failed, unloading npu:4:0-1
================================================================================
Compilation Failure Report
================================================================================
- furiosa-runtime version: 0.10.4 (rev: 320a178a7 built at 2024-05-28T12:51:47Z)
- furiosa-compiler version: 0.10.1 (rev: 8b00177dc built at 2023-11-23T02:22:00Z)
- libhal version: 0.12.0 (rev: 56530c0 built at 2023-11-16T12:37:25Z)
The compiler log can be found at /home/elicer/.local/state/furiosa/logs/compiler-20240813090021-46agqu.log
If you have a problem, please follow the bug reporting guide at
https://furiosa-ai.github.io/docs/latest/en/customer-support/bugs.html
================================================================================
2024-08-13T09:00:23.479849Z INFO furiosa_rt_core::npu::raw: NPU (npu:4:0-1) has been closed
2024-08-13T09:00:23.484487Z INFO furiosa_rt_core::driver::event_driven::coord: [Runtime-0] stopped
Traceback (most recent call last):
File "/home/elicer/test/test.py", line 3, in <module>
sess = session.create('/home/elicer/test/onnx_test.onnx')
File "<string>", line 70, in wrapped
runtime.FuriosaRuntimeError: runtime error: Compilation error: Other error```
관련하여 어떤 문제가 있을 수 있는지 답변 부탁드립니다. 읽어주셔서 감사합니다.