CUDA报错: Cannot create Cublas handle. Cublas won't be available. 以及:Check failed: status == CUBLAS_STATUS_SUCCESS (1 vs. 0) CUBLAS_STATUS_NOT_INITIALIZED

Error描述:

aita@aita-Alienware-Area-51-R5:~/AITA2/daisida/ssd-github/caffe$ make runtest -j8
.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args> commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time Flags from tools/caffe.cpp:
-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-level (Optional; network level.) type: int32 default: 0
-model (The model definition protocol buffer text file.) type: string
default: ""
-phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
type: string default: ""
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-stage (Optional; network stages (not to be confused with phase), separated
by ','.) type: string default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ','. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
Cuda number of devices: 3
Setting to use device 0
Current device id: 0
Current device name: GeForce GTX 1080 Ti
Note: Randomizing tests' orders with a seed of 48866 .
[==========] Running 2361 tests from 309 test cases.
[----------] Global test environment set-up.
[----------] 7 tests from DetectionOutputLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] DetectionOutputLayerTest/2.TestForwardShareLocationTopK
E0103 00:37:53.042623 19470 common.cpp:113] Cannot create Cublas handle. Cublas won't be available.
[ OK ] DetectionOutputLayerTest/2.TestForwardShareLocationTopK (219 ms)
[ RUN ] DetectionOutputLayerTest/2.TestForwardNoShareLocationNeg0TopK
[ OK ] DetectionOutputLayerTest/2.TestForwardNoShareLocationNeg0TopK (2 ms)
[ RUN ] DetectionOutputLayerTest/2.TestSetup
[ OK ] DetectionOutputLayerTest/2.TestSetup (1 ms)
[ RUN ] DetectionOutputLayerTest/2.TestForwardNoShareLocationNeg0
[ OK ] DetectionOutputLayerTest/2.TestForwardNoShareLocationNeg0 (2 ms)
[ RUN ] DetectionOutputLayerTest/2.TestForwardNoShareLocation
[ OK ] DetectionOutputLayerTest/2.TestForwardNoShareLocation (2 ms)
[ RUN ] DetectionOutputLayerTest/2.TestForwardShareLocation
[ OK ] DetectionOutputLayerTest/2.TestForwardShareLocation (1 ms)
[ RUN ] DetectionOutputLayerTest/2.TestForwardNoShareLocationTopK
[ OK ] DetectionOutputLayerTest/2.TestForwardNoShareLocationTopK (2 ms)
[----------] 7 tests from DetectionOutputLayerTest/2 (229 ms total) [----------] 2 tests from EuclideanLossLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] EuclideanLossLayerTest/2.TestGradient
F0103 00:37:53.068140 19470 math_functions.cu:110] Check failed: status == CUBLAS_STATUS_SUCCESS (1 vs. 0) CUBLAS_STATUS_NOT_INITIALIZED
*** Check failure stack trace: ***
@ 0x7f9210daf5cd google::LogMessage::Fail()
@ 0x7f9210db1433 google::LogMessage::SendToLog()
@ 0x7f9210daf15b google::LogMessage::Flush()
@ 0x7f9210db1e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f920c7ad43a caffe::caffe_gpu_dot<>()
@ 0x7f920c7ec7c3 caffe::EuclideanLossLayer<>::Forward_gpu()
@ 0x48ae96 caffe::Layer<>::Forward()
@ 0x48d445 caffe::GradientChecker<>::CheckGradientSingle()
@ 0x4aea53 caffe::GradientChecker<>::CheckGradientExhaustive()
@ 0x848f0c caffe::EuclideanLossLayerTest_TestGradient_Test<>::TestBody()
@ 0xa17c23 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0xa1123a testing::Test::Run()
@ 0xa11388 testing::TestInfo::Run()
@ 0xa11465 testing::TestCase::Run()
@ 0xa1273f testing::internal::UnitTestImpl::RunAllTests()
@ 0xa12a63 testing::UnitTest::Run()
@ 0x47a98d main
@ 0x7f920ba46830 __libc_start_main
@ 0x483b49 _start
@ (nil) (unknown)
Makefile:526: recipe for target 'runtest' failed
make: *** [runtest] Aborted (core dumped)

  


解决方案1:

sudo rm -rf .nv/

解决方案2:

I realized that there was an error with my CUDA installation, specifically with the cuBLAS library. You can check if yours has the same problem by running the sample program simpleCUBLAS:

  1. cd /usr/local/cuda/samples/7_CUDALibraries/simpleCUBLAS # check if your samples are in the same directory
  2. make
  3. ./simpleCUBLAS

I was getting an error when I tried to run it, so I reinstalled CUDA 8.0 and it solved the issue.


此前尝试过:

CUDA_VISIBLE_DEVICES=2

原因是这个Demo默认使用所有探测到的CUDA 设备,而实验室的CUDA设备还有很多人在用,这会造成问题(可能是冲突或者资源不够,或者不被允许个人使用这么多个?)

所以在运行的命令行前面要加上一定的限制:

CUDA_VISIBLE_DEVICES=2 ./build/examples/openpose/openpose.bin --net_resolution "160x80" --video examples/media/video.avi

使得被探测到的设备数量只有两个。

然后发现还是运行不了,为什么呢?

因为OS X上的SHELL并不具备显示远程窗口的功能,

所以使用MobaXterm(在WIN10下),然后就成功了,因为这个软件直接内置了X server的接口

上一篇:windows7下解决caffe check failed registry.count(type) == 1(0 vs. 1) unknown layer type问题


下一篇:Ubuntu下安装Pyenv不成功,求指教