GO：panic时core的生成（GOTRACEBACK）与调试

---

1.需求

基于GO实现的application在异常panic时进程将退出，并在终端输出panic信息。

例如：

package main

func main() {
	panic("test1280 :(")
}

[test1280@test1280 core-test1280]$ ls -l
total 8
-rw-rw-r--. 1 test1280 test1280 30 Jan 11 01:18 go.mod
-rw-rw-r--. 1 test1280 test1280 52 Jan 13 00:01 main.go
[test1280@test1280 core-test1280]$ go build -o main
[test1280@test1280 core-test1280]$ ./main
panic: test1280 :(

goroutine 1 [running]:
main.main()
	/home/test1280/core-test1280/main.go:4 +0x39
[test1280@test1280 core-test1280]$ echo $?
2

在终端输出panic参数，以及异常调用栈。

但是进程在panic时并未生成core文件，给后续排查带来不便。

是否有一种方法，使得application在panic时能自动生成core文件，将调用栈（尤其是异常调用栈）记录其中？

---

2.测试环境

操作系统：

[test1280@test1280 core-test1280]$ uname -a
Linux test1280 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[test1280@test1280 core-test1280]$ cat /etc/redhat-release 
CentOS release 6.8 (Final)

环境变量：

[test1280@test1280 core-test1280]$ ulimit -c
unlimited

GO版本：

[test1280@test1280 core-test1280]$ which go
~/go/bin/go
[test1280@test1280 core-test1280]$ go version
go version go1.13.6 linux/amd64

---

3.环境变量GOTRACEBACK

GO提供环境变量GOTRACEBACK，控制在panic时的行为。

GOTRACEBACK取值有：none、single（默认）、all、system、crash。

参考：https://golang.org/pkg/runtime/

自测代码：

package main

import "time"

func main() {
	// 子协程1
	go func(i int) {
		for {
			time.Sleep(time.Second)
		}
	}(0)

	// 子协程2
	go func(i int) {
		/* 子协程在10秒后执行panic */
		for count := 0; count < 10; count++ {
			time.Sleep(time.Second)
		}
		panic("test1280: panic in goroutine-2")
	}(1)

	// 子协程3
	go func(i int) {
		for {
			time.Sleep(time.Second)
		}
	}(2)

	// 主协程
	for {
		time.Sleep(time.Second)
	}
}

• 在GOTRACEBACK=none时，不输出异常调用栈：

GOTRACEBACK=none omits the goroutine stack traces entirely.

[test1280@test1280 core-test1280]$ export GOTRACEBACK=none
[test1280@test1280 core-test1280]$ go build -o main
[test1280@test1280 core-test1280]$ ./main
panic: test1280: panic in goroutine-2

• 在GOTRACEBACK=single时，输出导致异常的协程调用栈：

By default, a failure prints a stack trace for the current goroutine, eliding functions internal to the run-time system, and then exits with exit code 2.

[test1280@test1280 core-test1280]$ export GOTRACEBACK=single
[test1280@test1280 core-test1280]$ ./main
panic: test1280: panic in goroutine-2

goroutine 6 [running]:
main.main.func2(0x1)
	/home/test1280/core-test1280/main.go:19 +0x5d
created by main.main
	/home/test1280/core-test1280/main.go:14 +0x63

从输出中可以看到，main创建的子协程出现panic，main.go文件的源码第19行。

注意：GOTRACEBACK环境变量不存在时，等同于设置single。

• 在GOTRACEBACK=all时，输出用户创建的所有协程调用栈：

GOTRACEBACK=all adds stack traces for all user-created goroutines.

[test1280@test1280 core-test1280]$ export GOTRACEBACK=all
[test1280@test1280 core-test1280]$ ./main
panic: test1280: panic in goroutine-2

goroutine 6 [running]:
main.main.func2(0x1)
	/home/test1280/core-test1280/main.go:19 +0x5d
created by main.main
	/home/test1280/core-test1280/main.go:14 +0x63

goroutine 1 [runnable]:
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157
main.main()
	/home/test1280/core-test1280/main.go:31 +0x91

goroutine 5 [runnable]:
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157
main.main.func1(0x0)
	/home/test1280/core-test1280/main.go:9 +0x2a
created by main.main
	/home/test1280/core-test1280/main.go:7 +0x42

goroutine 7 [runnable]:
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157
main.main.func3(0x2)
	/home/test1280/core-test1280/main.go:25 +0x2a
created by main.main
	/home/test1280/core-test1280/main.go:23 +0x84

在panic输出中，顶端的是异常流程的协程调用栈，其余都是用户创建的正常协程调用栈（含主协程）。

其中：goroutine(6)异常，goroutine（1、5、7）均正常。

• 在GOTRACEBACK=system时，输出所有协程调用栈（包括GO框架底层实现创建的协程）：

GOTRACEBACK=system is like “all” but adds stack frames for run-time functions and shows goroutines created internally by the run-time.

[test1280@test1280 core-test1280]$ export GOTRACEBACK=system
[test1280@test1280 core-test1280]$ ./main
panic: test1280: panic in goroutine-2

goroutine 6 [running]:
panic(0x463240, 0x4899f0)
	/home/test1280/go/src/runtime/panic.go:722 +0x2c2 fp=0xc000034fb0 sp=0xc000034f20 pc=0x4256c2
main.main.func2(0x1)
	/home/test1280/core-test1280/main.go:19 +0x5d fp=0xc000034fd8 sp=0xc000034fb0 pc=0x456c3d
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000034fe0 sp=0xc000034fd8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:14 +0x63

goroutine 1 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x441313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc0000326e8 sp=0xc0000326c8 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc000032738 sp=0xc0000326e8 pc=0x43e427
main.main()
	/home/test1280/core-test1280/main.go:31 +0x91 fp=0xc000032760 sp=0xc000032738 pc=0x456b91
runtime.main()
	/home/test1280/go/src/runtime/proc.go:203 +0x21e fp=0xc0000327e0 sp=0xc000032760 pc=0x4273ce
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000327e8 sp=0xc0000327e0 pc=0x44d151

goroutine 2 [force gc (idle)]:
runtime.gopark(0x47f0f0, 0x4d17c0, 0x1411, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000032fb0 sp=0xc000032f90 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.forcegchelper()
	/home/test1280/go/src/runtime/proc.go:253 +0xb7 fp=0xc000032fe0 sp=0xc000032fb0 pc=0x427647
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x44d151
created by runtime.init.5
	/home/test1280/go/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x47f0f0, 0x4d18e0, 0x140c, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc0000337a8 sp=0xc000033788 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.bgsweep(0xc000048000)
	/home/test1280/go/src/runtime/mgcsweep.go:70 +0x9c fp=0xc0000337d8 sp=0xc0000337a8 pc=0x41c1ec
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000337e0 sp=0xc0000337d8 pc=0x44d151
created by runtime.gcenable
	/home/test1280/go/src/runtime/mgc.go:210 +0x5c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x47f0f0, 0x4d1940, 0x140d, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000033f40 sp=0xc000033f20 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.bgscavenge(0xc000048000)
	/home/test1280/go/src/runtime/mgcscavenge.go:299 +0xe1 fp=0xc000033fd8 sp=0xc000033f40 pc=0x41b871
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000033fe0 sp=0xc000033fd8 pc=0x44d151
created by runtime.gcenable
	/home/test1280/go/src/runtime/mgc.go:211 +0x7e

goroutine 5 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000034770 sp=0xc000034750 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc0000347c0 sp=0xc000034770 pc=0x43e427
main.main.func1(0x0)
	/home/test1280/core-test1280/main.go:9 +0x2a fp=0xc0000347d8 sp=0xc0000347c0 pc=0x456bca
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000347e0 sp=0xc0000347d8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:7 +0x42

goroutine 7 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000035770 sp=0xc000035750 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc0000357c0 sp=0xc000035770 pc=0x43e427
main.main.func3(0x2)
	/home/test1280/core-test1280/main.go:25 +0x2a fp=0xc0000357d8 sp=0xc0000357c0 pc=0x456c7a
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000357e0 sp=0xc0000357d8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:23 +0x84

goroutine 8 [timer goroutine (idle)]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1415, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000035f60 sp=0xc000035f40 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.timerproc(0x4d4400)
	/home/test1280/go/src/runtime/time.go:303 +0x27b fp=0xc000035fd8 sp=0xc000035f60 pc=0x43ed4b
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000035fe0 sp=0xc000035fd8 pc=0x44d151
created by runtime.(*timersBucket).addtimerLocked
	/home/test1280/go/src/runtime/time.go:169 +0x10e

在panic输出中，顶端的是异常流程的协程调用栈，其余的调用栈包含了用户创建的正常协程调用栈以及GO框架底层实现创建的协程的调用栈（如垃圾回收GC相关协程）。

• 在GOTRACEBACK=crash时，输出所有的协程调用栈，并生成core：

GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.

[test1280@test1280 core-test1280]$ export GOTRACEBACK=crash
[test1280@test1280 core-test1280]$ ./main
panic: test1280: panic in goroutine-2

goroutine 6 [running]:
panic(0x463240, 0x4899f0)
	/home/test1280/go/src/runtime/panic.go:722 +0x2c2 fp=0xc000034fb0 sp=0xc000034f20 pc=0x4256c2
main.main.func2(0x1)
	/home/test1280/core-test1280/main.go:19 +0x5d fp=0xc000034fd8 sp=0xc000034fb0 pc=0x456c3d
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000034fe0 sp=0xc000034fd8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:14 +0x63

goroutine 1 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x441313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc0000326e8 sp=0xc0000326c8 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc000032738 sp=0xc0000326e8 pc=0x43e427
main.main()
	/home/test1280/core-test1280/main.go:31 +0x91 fp=0xc000032760 sp=0xc000032738 pc=0x456b91
runtime.main()
	/home/test1280/go/src/runtime/proc.go:203 +0x21e fp=0xc0000327e0 sp=0xc000032760 pc=0x4273ce
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000327e8 sp=0xc0000327e0 pc=0x44d151

goroutine 2 [force gc (idle)]:
runtime.gopark(0x47f0f0, 0x4d17c0, 0x1411, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000032fb0 sp=0xc000032f90 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.forcegchelper()
	/home/test1280/go/src/runtime/proc.go:253 +0xb7 fp=0xc000032fe0 sp=0xc000032fb0 pc=0x427647
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x44d151
created by runtime.init.5
	/home/test1280/go/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x47f0f0, 0x4d18e0, 0x140c, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc0000337a8 sp=0xc000033788 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.bgsweep(0xc000048000)
	/home/test1280/go/src/runtime/mgcsweep.go:70 +0x9c fp=0xc0000337d8 sp=0xc0000337a8 pc=0x41c1ec
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000337e0 sp=0xc0000337d8 pc=0x44d151
created by runtime.gcenable
	/home/test1280/go/src/runtime/mgc.go:210 +0x5c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x47f0f0, 0x4d1940, 0x140d, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000033f40 sp=0xc000033f20 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.bgscavenge(0xc000048000)
	/home/test1280/go/src/runtime/mgcscavenge.go:299 +0xe1 fp=0xc000033fd8 sp=0xc000033f40 pc=0x41b871
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000033fe0 sp=0xc000033fd8 pc=0x44d151
created by runtime.gcenable
	/home/test1280/go/src/runtime/mgc.go:211 +0x7e

goroutine 5 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000034770 sp=0xc000034750 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc0000347c0 sp=0xc000034770 pc=0x43e427
main.main.func1(0x0)
	/home/test1280/core-test1280/main.go:9 +0x2a fp=0xc0000347d8 sp=0xc0000347c0 pc=0x456bca
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000347e0 sp=0xc0000347d8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:7 +0x42

goroutine 7 [runnable]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1313, 0x2)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000035770 sp=0xc000035750 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
	/home/test1280/go/src/runtime/time.go:105 +0x157 fp=0xc0000357c0 sp=0xc000035770 pc=0x43e427
main.main.func3(0x2)
	/home/test1280/core-test1280/main.go:25 +0x2a fp=0xc0000357d8 sp=0xc0000357c0 pc=0x456c7a
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000357e0 sp=0xc0000357d8 pc=0x44d151
created by main.main
	/home/test1280/core-test1280/main.go:23 +0x84

goroutine 8 [timer goroutine (idle)]:
runtime.gopark(0x47f0f0, 0x4d4400, 0x1415, 0x1)
	/home/test1280/go/src/runtime/proc.go:304 +0xe0 fp=0xc000035f60 sp=0xc000035f40 pc=0x427790
runtime.goparkunlock(...)
	/home/test1280/go/src/runtime/proc.go:310
runtime.timerproc(0x4d4400)
	/home/test1280/go/src/runtime/time.go:303 +0x27b fp=0xc000035fd8 sp=0xc000035f60 pc=0x43ed4b
runtime.goexit()
	/home/test1280/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000035fe0 sp=0xc000035fd8 pc=0x44d151
created by runtime.(*timersBucket).addtimerLocked
	/home/test1280/go/src/runtime/time.go:169 +0x10e
Aborted (core dumped)

注意最后一行Aborted (core dumped)，已生成我们想要的core文件。

---

4.调试core

4.1.gdb（不建议，可跳过直接看4.2.Delve）

我们常使用gdb来调试core，查看异常调用栈等信息回溯分析异常原因。

如果我们用gdb来调试core，则：

[test1280@test1280 core-test1280]$ gdb -c core.4292 ./main
......
Core was generated by `./main'.
Program terminated with signal 6, Aborted.
#0  runtime.raise () at /home/test1280/go/src/runtime/sys_linux_amd64.s:150
150		RET
warning: File "/home/test1280/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /home/test1280/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/test1280/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/test1280/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
(gdb) bt
#0  runtime.raise () at /home/test1280/go/src/runtime/sys_linux_amd64.s:150
#1  0x00000000004378ab in runtime.dieFromSignal (sig=6) at /home/test1280/go/src/runtime/signal_unix.go:428
#2  0x0000000000437d0d in runtime.sigfwdgo (sig=6, info=0xc000009bf0, ctx=0xc000009ac0, ~r3=<value optimized out>)
    at /home/test1280/go/src/runtime/signal_unix.go:631
#3  0x0000000000436f90 in runtime.sigtrampgo (sig=<value optimized out>, info=0xc000009bf0, ctx=0xc000009ac0)
    at /home/test1280/go/src/runtime/signal_unix.go:289
#4  0x000000000044edf3 in runtime.sigtramp () at /home/test1280/go/src/runtime/sys_linux_amd64.s:357
#5  0x000000000044eee0 in ?? () at /home/test1280/go/src/runtime/sys_linux_amd64.s:441
#6  0x0000000000000001 in ?? ()
#7  0x0000000000000000 in ?? ()

使用gdb调试core，存在几个问题：

• 问题1：gdb warning

warning: File "/home/test1280/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /home/test1280/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/test1280/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/test1280/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

一般是没有设置.gdbinit导致。（我设置了.gdbinit，报其他错误…）

https://*.com/questions/24877998/cannot-load-runtime-gdb-py

• 问题2：bt查看栈调用无法正常显示，是??，无法找到main函数

...
#3  0x0000000000436f90 in runtime.sigtrampgo (sig=<value optimized out>, info=0xc000009bf0, ctx=0xc000009ac0)
    at /home/test1280/go/src/runtime/signal_unix.go:289
#4  0x000000000044edf3 in runtime.sigtramp () at /home/test1280/go/src/runtime/sys_linux_amd64.s:357
#5  0x000000000044eee0 in ?? () at /home/test1280/go/src/runtime/sys_linux_amd64.s:441
#6  0x0000000000000001 in ?? ()
#7  0x0000000000000000 in ?? ()

注意：大概率是我gdb配置的问题，在GO官网的样例中是可以查看协程栈的。

https://golang.org/doc/gdb 中样例如下，可正常显示栈调用：

(gdb) goroutine 1 bt
#0  0x000000000040facb in runtime.gosched () at /home/user/go/src/runtime/proc.c:873
#1  0x00000000004031c9 in runtime.chanrecv (c=void, ep=void, selected=void, received=void)
 at  /home/user/go/src/runtime/chan.c:342
#2  0x0000000000403299 in runtime.chanrecv1 (t=void, c=void) at/home/user/go/src/runtime/chan.c:423
#3  0x000000000043075b in testing.RunTests (matchString={void (struct string, struct string, bool *, error *)}
 0x7ffff7f9ef60, tests=  []testing.InternalTest = {...}) at /home/user/go/src/testing/testing.go:201
#4  0x00000000004302b1 in testing.Main (matchString={void (struct string, struct string, bool *, error *)} 
 0x7ffff7f9ef80, tests= []testing.InternalTest = {...}, benchmarks= []testing.InternalBenchmark = {...})
at /home/user/go/src/testing/testing.go:168
#5  0x0000000000400dc1 in main.main () at /home/user/go/src/regexp/_testmain.go:98
#6  0x00000000004022e7 in runtime.mainstart () at /home/user/go/src/runtime/amd64/asm.s:78
#7  0x000000000040ea6f in runtime.initdone () at /home/user/go/src/runtime/proc.c:243
#8  0x0000000000000000 in ?? ()

但总体来说，gdb并不能很好地分析GO生成的core文件：

GDB does not understand Go programs well. The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects that they can confuse the debugger and cause incorrect results even when the program is compiled with gccgo. As a consequence, although GDB can be useful in some situations (e.g., debugging Cgo code, or debugging the runtime itself), it is not a reliable debugger for Go programs, particularly heavily concurrent ones.
参考：https://golang.org/doc/gdb

官网推荐使用Delve替代gdb调试GO生成的core文件：

Note that Delve is a better alternative to GDB when debugging Go programs built with the standard toolchain. It understands the Go runtime, data structures, and expressions better than GDB. Delve currently supports Linux, OSX, and Windows on amd64. For the most up-to-date list of supported platforms, please see the Delve documentation.
参考：https://golang.org/doc/gdb

Unfortunately the golang support in gdb is pretty weak. Generally it is better to use the delve debugger.
参考：https://*.com/questions/45855414/unwind-stack-for-goroutine-in-gdb-for-a-golang-exes-core-dump

4.2.Delve

项目地址：https://github.com/go-delve/delve

编译源码：

[test1280@test1280 ~]$ git clone https://github.com/go-delve/delve
Initialized empty Git repository in /home/test1280/delve/.git/
remote: Enumerating objects: 17501, done.
remote: Total 17501 (delta 0), reused 0 (delta 0), pack-reused 17501
Receiving objects: 100% (17501/17501), 23.65 MiB | 962 KiB/s, done.
Resolving deltas: 100% (11674/11674), done.
[test1280@test1280 ~]$ cd delve && make build

更新变量：

[test1280@test1280 delve]$ vi ~/.bash_profile

在末尾新增如下一行：
export PATH=$HOME/delve:$PATH

保存文件，退出当前终端，重新登陆（使PATH生效）。

• 查看调用栈

[test1280@test1280 core-test1280]$ dlv core ./main core.4467 
Type 'help' for list of commands.
(dlv) bt
 0  0x000000000044eb01 in runtime.raise
    at /home/test1280/go/src/runtime/sys_linux_amd64.s:150
 1  0x00000000004378ab in runtime.dieFromSignal
    at /home/test1280/go/src/runtime/signal_unix.go:428
 2  0x0000000000437d0d in runtime.sigfwdgo
    at /home/test1280/go/src/runtime/signal_unix.go:631
 3  0x0000000000436f90 in runtime.sigtrampgo
    at /home/test1280/go/src/runtime/signal_unix.go:289
 4  0x000000000044edf3 in runtime.sigtramp
    at /home/test1280/go/src/runtime/sys_linux_amd64.s:357
 5  0x000000000044eee0 in runtime.sigreturn
    at /home/test1280/go/src/runtime/sys_linux_amd64.s:449
 6  0x0000000000437a4a in runtime.crash
    at /home/test1280/go/src/runtime/signal_unix.go:520
 7  0x0000000000425cc4 in runtime.fatalpanic
    at /home/test1280/go/src/runtime/panic.go:874
 8  0x00000000004256c2 in runtime.gopanic
    at /home/test1280/go/src/runtime/panic.go:722
 9  0x0000000000456c5c in main.main.func2
    at ./main.go:19
10  0x000000000044d151 in runtime.goexit
    at /home/test1280/go/src/runtime/asm_amd64.s:1357

可以观察到是main.go的第19行调用panic引起：

 9  0x0000000000456c5c in main.main.func2
    at ./main.go:19

另外，dlv core接入时，当前的协程就是引发panic的协程，执行bt，即可查看异常调用栈。

可以通过gr切换选择当前协程，查看不同协程的对应调用栈。

• 查看当前协程调用栈（core中通常显示的是异常调用栈）

(dlv) gr
Thread 4467 at /home/test1280/go/src/runtime/sys_linux_amd64.s:150
Goroutine 6:
	Runtime: /home/test1280/go/src/runtime/sys_linux_amd64.s:150 runtime.raise (0x44eb01)
	User: ./main.go:19 main.main.func2 (0x456c5c)
	Go: ./main.go:14 main.main (0x456b63)
	Start: ./main.go:14 main.main.func2 (0x456bf0)

当前协程是goroutine 6，引起的panic异常。

• 查看所有协程调用栈

(dlv) grs
  Goroutine 1 - User: /home/test1280/go/src/runtime/proc.go:310 time.Sleep (0x43e427)
  Goroutine 2 - User: /home/test1280/go/src/runtime/proc.go:305 runtime.gopark (0x427790) [force gc (idle)]
  Goroutine 3 - User: /home/test1280/go/src/runtime/proc.go:305 runtime.gopark (0x427790) [GC sweep wait]
  Goroutine 4 - User: /home/test1280/go/src/runtime/proc.go:305 runtime.gopark (0x427790) [GC scavenge wait]
  Goroutine 5 - User: /home/test1280/go/src/runtime/proc.go:310 time.Sleep (0x43e427)
* Goroutine 6 - User: ./main.go:19 main.main.func2 (0x456c5c) (thread 4467)
  Goroutine 7 - User: /home/test1280/go/src/runtime/proc.go:310 time.Sleep (0x43e427)
  Goroutine 8 - User: /home/test1280/go/src/runtime/proc.go:305 runtime.gopark (0x427790) [timer goroutine (idle)]
[8 goroutines]

当前协程是goroutine 6，一共有8个协程。

• 切换当前协程

(dlv) gr 1
Switched from 6 to 1 (thread 4467)

• 帮助说明

(dlv) help gr
......
(dlv) help grs
......
(dlv) help help
......
(dlv) help
......

以上只是使用Delve分析core文件的部分命令，Delve具备其他强大功能不在此处列举。

---

5.总结

• 建议实践：

运行GO实现的application前设置GOTRACEBACK=crash，进程panic时生成core文件，并使用Delve分析core定位异常代码。

• 特别注意：

设置GOTRACEBACK是为了在panic时生成core文件，而不是指GO实现的进程无法用别的方式生成core文件。

当GO代码panic时，GO框架底层可以捕获到此panic，如果此时GOTRACEBACK=crash，则GO框架底层发送SIGABRT到自身生成core文件并退出进程。

可以自己动手使用kill等命令工具发送信号到你的GO-application，或者gcore生成core文件。

更多参考：https://blog.csdn.net/test1280/article/details/112556230

---

6.参考

1.https://github.com/go-delve/delve
2.https://*.com/questions/45855414/unwind-stack-for-goroutine-in-gdb-for-a-golang-exes-core-dump
3.https://golang.org/pkg/runtime/
4.https://golang.org/pkg/runtime/debug/#SetTraceback
5.https://rakyll.org/coredumps/
6.https://medium.com/a-journey-with-go/go-debugging-with-delve-core-dumps-384145b2e8d9
7.https://blog.csdn.net/test1280/article/details/112556230