Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ng启动失败 (Ng startup failure) ,提示 Arena too small, toWrite:126 newTotal:xxx limit:xxx #182

Open
yinyuncan opened this issue Mar 7, 2023 · 1 comment

Comments

@yinyuncan
Copy link

手动杀死Prometheus和ng后,ng重启失败 (Description ng fails to restart after prometheus and ng are manually killed)

tidb版本:6.5
ng版本: Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832

错误日志位于:/tidb-deploy/prometheus-9090/log/tsdb.log

2023-03-07T06:36:53.570Z        info    /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:106       loading rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult"...
2023-03-07T06:36:53.571Z        info    /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:132       loaded rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult" in 0.001 seconds; entriesCount: 0, sizeBytes: 0
2023-03-07T06:36:53.571Z        info    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/timeseries/vm.go:45 started VictoriaMetrics in 0.094 seconds
2023/03/07 14:36:53 Arena too small, toWrite:126 newTotal:21810362 limit:21810342
github.com/dgraph-io/badger/v3/y.AssertTruef
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/y/error.go:62
github.com/dgraph-io/badger/v3/skl.(*Arena).putVal
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/arena.go:83
github.com/dgraph-io/badger/v3/skl.newNode
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:114
github.com/dgraph-io/badger/v3/skl.(*Skiplist).Put
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:306
github.com/dgraph-io/badger/v3.(*memTable).replayFunction.func1
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:258
github.com/dgraph-io/badger/v3.(*logFile).iterate
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:521
github.com/dgraph-io/badger/v3.(*memTable).UpdateSkipList
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:219
github.com/dgraph-io/badger/v3.(*DB).openMemTable
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:146
github.com/dgraph-io/badger/v3.(*DB).openMemTables
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:88
github.com/dgraph-io/badger/v3.Open
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:333
github.com/genjidb/genji/engine/badgerengine.NewEngine
        /go/pkg/mod/github.com/crazycs520/genji/engine/badgerengine@v0.12.1-0.20220328082424-727a2d089bde/engine.go:25
github.com/pingcap/ng-monitoring/database/document.Init
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/document/document.go:34
github.com/pingcap/ng-monitoring/database.Init
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/database.go:14
main.main
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/main.go:68
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1594

ng.log 无报错
最后的日志如下


[2023/03/07 14:41:44.272 +08:00] [INFO] [printer.go:25] ["Welcome to ng-monitoring."] ["Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832] ["Git Branch"=heads/refs/tags/v6.5.0] ["UTC Build Time"="2022-12-16 08:18:47"] [GoVersion=go1.19.3]
[2023/03/07 14:41:44.273 +08:00] [INFO] [main.go:64] [config] [config="{\"address\":\"0.0.0.0:12020\",\"advertise_address\":\"10.0.0.80:12020\",\"pd\":{\"endpoints\":[\"10.0.0.80:2379\",\"10.0.0.76:2379\",\"10.0.0.74:2379\"]},\"log\":{\"path\":\"/tidb-deploy/prometheus-9090/log\",\"level\":\"INFO\"},\"storage\":{\"path\":\"/tidb-data/prometheus-9090\"},\"continuous_profiling\":{\"enable\":false,\"profile_seconds\":10,\"interval_seconds\":60,\"timeout_seconds\":120,\"data_retention_seconds\":259200},\"security\":{\"ca_path\":\"\",\"cert_path\":\"\",\"key_path\":\"\"}}"]

@yinyuncan
Copy link
Author

实在是不知道怎么解决,于是问了一下ChatGpt

根据ChatGpt分析的结果,需要调整一下Badger数据库的arena 内存,呃 也不知道是否正确,大佬帮帮忙

image

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant