.Net Core服務監控報警指標上報Prometheus+Grafana

前言

簡單集成Prometheus+Grafana,指標的上報收集可視化。

Prometheus

Prometheus是一個監控平台,監控從HTTP端口收集受監控目標的指標。在微服務的架構里Prometheus多維度的數據收集是非常強大的 我們首先下載安裝Prometheusnode_exporter,node_exporter用於監控CPU、內存、磁盤、I/O等信息

  • Prometheus下載地址
  • node_exporter下載地址

下載完成后解壓以管理員運行 prometheus.exe 訪問 http://localhost:9090/ 出現一下頁面說明啟動成功啦

.Net Core獲取指標

有了Prometheus,我們還需要給Prometheus提供獲取監控數據的接口,我們新建一個WebApi項目,並導入prometheus-net.AspNetCore包,在Configure中加入UseMetricServer中間件

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{

    app.UseMetricServer();
    
}

啟動項目訪問http://localhost:5000/metrics就可以看基本的一些監控信息啦,包括線程數,句柄數,3個GC的回收計數等信息。

# HELP process_num_threads Total number of threads
# TYPE process_num_threads gauge
process_num_threads 29
# HELP process_working_set_bytes Process working set
# TYPE process_working_set_bytes gauge
process_working_set_bytes 44441600
# HELP process_private_memory_bytes Process private memory size
# TYPE process_private_memory_bytes gauge
process_private_memory_bytes 69660672
# HELP dotnet_total_memory_bytes Total known allocated memory
# TYPE dotnet_total_memory_bytes gauge
dotnet_total_memory_bytes 2464584
# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="1"} 0
dotnet_collection_count_total{generation="0"} 0
dotnet_collection_count_total{generation="2"} 0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1592448124.2853072
# HELP process_open_handles Number of open handles
# TYPE process_open_handles gauge
process_open_handles 413
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2225187631104
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.171875

Help 是收集指標的說明,Type收集指標的類型

但是作為HTTP應用怎麼能沒有HTTP的監控和計數呢,只需要加加入UseHttpMetrics中間件就可以對HTTP請求監控和計數,主要注意的是UseHttpMetrics最好放在UseEndpointsUseRouting中間

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseMetricServer();
    
    app.UseRouting();
    
    app.UseHttpMetrics();

    app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
}

啟動項目繼續訪問http://localhost:5000/metrics

# HELP http_requests_in_progress The number of requests currently in progress in the ASP.NET Core pipeline. One series without controller/action label values counts all in-progress requests, with separate series existing for each controller-action pair.
# TYPE http_requests_in_progress gauge

可以看到已經有了,我們隨便請求一下服務看看效果,會幫我們記錄下總耗時,總請求數,和每次請求的耗時數

但是單單有上面那些數據好像還不太好定位一下很奇葩的問題,這時候我們可以獲取Runtime的一些數據,方法童謠很簡單。導入prometheus-net.DotNetRuntime 包,它可以幫助我們看到如下指標

  • 垃圾回收的收集頻率和時間
  • 服務佔用堆大小
  • 對象堆分配的字節
  • JIT編譯和JIT CPU消耗率
  • 線程池大小,調度延遲以及增長/縮小的原因
  • 鎖爭用情況

我們只需要在ProgramMain方法中啟動收集器就可以啦。

public static void Main(string[] args)
{
    DotNetRuntimeStatsBuilder.Default().StartCollecting();
    CreateHostBuilder(args).Build().Run();
}

啟動項目繼續訪問http://localhost:5000/metrics測試一下

# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="1"} 0
dotnet_collection_count_total{generation="0"} 0
dotnet_collection_count_total{generation="2"} 0
# HELP process_private_memory_bytes Process private memory size
# TYPE process_private_memory_bytes gauge
process_private_memory_bytes 75141120
# HELP dotnet_gc_pause_ratio The percentage of time the process spent paused for garbage collection
# TYPE dotnet_gc_pause_ratio gauge
dotnet_gc_pause_ratio 0
# HELP http_requests_received_total Provides the count of HTTP requests that have been processed by the ASP.NET Core pipeline.
# TYPE http_requests_received_total counter
# HELP dotnet_gc_collection_seconds The amount of time spent running garbage collections
# TYPE dotnet_gc_collection_seconds histogram
dotnet_gc_collection_seconds_sum 0
dotnet_gc_collection_seconds_count 0
dotnet_gc_collection_seconds_bucket{le="0.001"} 0
dotnet_gc_collection_seconds_bucket{le="0.01"} 0
dotnet_gc_collection_seconds_bucket{le="0.05"} 0
dotnet_gc_collection_seconds_bucket{le="0.1"} 0
dotnet_gc_collection_seconds_bucket{le="0.5"} 0
dotnet_gc_collection_seconds_bucket{le="1"} 0
dotnet_gc_collection_seconds_bucket{le="10"} 0
dotnet_gc_collection_seconds_bucket{le="+Inf"} 0
# HELP dotnet_total_memory_bytes Total known allocated memory
# TYPE dotnet_total_memory_bytes gauge
dotnet_total_memory_bytes 4925936
# HELP dotnet_threadpool_num_threads The number of active threads in the thread pool
# TYPE dotnet_threadpool_num_threads gauge
dotnet_threadpool_num_threads 0
# HELP dotnet_threadpool_scheduling_delay_seconds A breakdown of the latency experienced between an item being scheduled for execution on the thread pool and it starting execution.
# TYPE dotnet_threadpool_scheduling_delay_seconds histogram
dotnet_threadpool_scheduling_delay_seconds_sum 0.015556
dotnet_threadpool_scheduling_delay_seconds_count 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="0.001"} 0
dotnet_threadpool_scheduling_delay_seconds_bucket{le="0.01"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="0.05"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="0.1"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="0.5"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="1"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="10"} 10
dotnet_threadpool_scheduling_delay_seconds_bucket{le="+Inf"} 10
# HELP process_working_set_bytes Process working set
# TYPE process_working_set_bytes gauge
process_working_set_bytes 50892800
# HELP process_num_threads Total number of threads
# TYPE process_num_threads gauge
process_num_threads 32
# HELP dotnet_jit_method_seconds_total Total number of seconds spent in the JIT compiler
# TYPE dotnet_jit_method_seconds_total counter
dotnet_jit_method_seconds_total 0
dotnet_jit_method_seconds_total{dynamic="false"} 0.44558800000000004
dotnet_jit_method_seconds_total{dynamic="true"} 0.004122000000000001
# HELP dotnet_gc_pinned_objects The number of pinned objects
# TYPE dotnet_gc_pinned_objects gauge
dotnet_gc_pinned_objects 0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1592449942.6063592
# HELP dotnet_gc_heap_size_bytes The current size of all heaps (only updated after a garbage collection)
# TYPE dotnet_gc_heap_size_bytes gauge
# HELP http_request_duration_seconds The duration of HTTP requests processed by an ASP.NET Core application.
# TYPE http_request_duration_seconds histogram
# HELP dotnet_contention_seconds_total The total amount of time spent contending locks
# TYPE dotnet_contention_seconds_total counter
dotnet_contention_seconds_total 0
# HELP dotnet_gc_pause_seconds The amount of time execution was paused for garbage collection
# TYPE dotnet_gc_pause_seconds histogram
dotnet_gc_pause_seconds_sum 0
dotnet_gc_pause_seconds_count 0
dotnet_gc_pause_seconds_bucket{le="0.001"} 0
dotnet_gc_pause_seconds_bucket{le="0.01"} 0
dotnet_gc_pause_seconds_bucket{le="0.05"} 0
dotnet_gc_pause_seconds_bucket{le="0.1"} 0
dotnet_gc_pause_seconds_bucket{le="0.5"} 0
dotnet_gc_pause_seconds_bucket{le="1"} 0
dotnet_gc_pause_seconds_bucket{le="10"} 0
dotnet_gc_pause_seconds_bucket{le="+Inf"} 0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2225201872896
# HELP dotnet_gc_finalization_queue_length The number of objects waiting to be finalized
# TYPE dotnet_gc_finalization_queue_length gauge
dotnet_gc_finalization_queue_length 0
# HELP dotnet_threadpool_io_num_threads The number of active threads in the IO thread pool
# TYPE dotnet_threadpool_io_num_threads gauge
dotnet_threadpool_io_num_threads 3
# HELP process_open_handles Number of open handles
# TYPE process_open_handles gauge
process_open_handles 436
# HELP dotnet_gc_collection_reasons_total A tally of all the reasons that lead to garbage collections being run
# TYPE dotnet_gc_collection_reasons_total counter
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.890625
# HELP http_requests_in_progress The number of requests currently in progress in the ASP.NET Core pipeline. One series without controller/action label values counts all in-progress requests, with separate series existing for each controller-action pair.
# TYPE http_requests_in_progress gauge
# HELP dotnet_threadpool_adjustments_total The total number of changes made to the size of the thread pool, labeled by the reason for change
# TYPE dotnet_threadpool_adjustments_total counter
# HELP dotnet_jit_cpu_ratio The amount of total CPU time consumed spent JIT'ing
# TYPE dotnet_jit_cpu_ratio gauge
dotnet_jit_cpu_ratio 0.5728901224489797
# HELP process_cpu_count The number of processor cores available to this process.
# TYPE process_cpu_count gauge
process_cpu_count 8
# HELP dotnet_build_info Build information about prometheus-net.DotNetRuntime and the environment
# TYPE dotnet_build_info gauge
dotnet_build_info{version="3.3.1.0",target_framework=".NETCoreApp,Version=v5.0",runtime_version=".NET Core 5.0.0-preview.2.20160.6",os_version="Microsoft Windows 10.0.18363",process_architecture="X64"} 1
# HELP dotnet_jit_method_total Total number of methods compiled by the JIT compiler
# TYPE dotnet_jit_method_total counter
dotnet_jit_method_total{dynamic="false"} 830
dotnet_jit_method_total{dynamic="true"} 30
# HELP dotnet_gc_cpu_ratio The percentage of process CPU time spent running garbage collections
# TYPE dotnet_gc_cpu_ratio gauge
dotnet_gc_cpu_ratio 0
# HELP dotnet_threadpool_scheduled_total The total number of items the thread pool has been instructed to execute
# TYPE dotnet_threadpool_scheduled_total counter
dotnet_threadpool_scheduled_total 16
# HELP dotnet_gc_allocated_bytes_total The total number of bytes allocated on the small and large object heaps (updated every 100KB of allocations)
# TYPE dotnet_gc_allocated_bytes_total counter
dotnet_gc_allocated_bytes_total{gc_heap="soh"} 3008088
dotnet_gc_allocated_bytes_total{gc_heap="loh"} 805392
# HELP dotnet_contention_total The number of locks contended
# TYPE dotnet_contention_total counter
dotnet_contention_total 0

可以看到非常多的信息啦,但是我們有時候不需要這麼多指標也可以自定義。


public static void Main(string[] args)
{
    DotNetRuntimeStatsBuilder
        .Customize()
        .WithContentionStats()
        .WithJitStats()
        .WithThreadPoolSchedulingStats()
        .WithThreadPoolStats()
        .WithGcStats()
        .StartCollecting();
    CreateHostBuilder(args).Build().Run();
}

JIT,GC和線程的監控是會影響到一點點性能,我們可以通過sampleRate這個枚舉的值來控制採樣頻率

public static void Main(string[] args)
{
    DotNetRuntimeStatsBuilder
        .Customize()
        //每5個事件個採集一個
        .WithContentionStats(sampleRate: SampleEvery.FiveEvents)
        //每10事件採集一個
        .WithJitStats(sampleRate: SampleEvery.TenEvents)
        //每100事件採集一個
        .WithThreadPoolSchedulingStats(sampleRate: SampleEvery.HundredEvents)
        .WithThreadPoolStats()
        .WithGcStats()
        .StartCollecting();
    CreateHostBuilder(args).Build().Run();
}

有了這些指標我們需要Prometheus來收集我們Api的指標,只需要修改prometheus.yml文件然後重啟Prometheus就可以了。

scrape_configs:  
- job_name: mydemo  
  scrape_interval: 15s  
  scrape_timeout: 10s  
  metrics_path: /metrics  
  scheme: http  
  static_configs:  
  - targets:  
    - localhost:5000   

啟動Api項目和Prometheus,選中dotnet_collection_count_total點擊Excute可以看到Api的指標是正常上報的。

Prometheus有了數據我們就需要一個炫酷的UI去展示上報的數據啦。

Grafana

Prometheus有了數據就差一個漂亮的UI來展示的我們的指標了。Grafana是一個Go編寫的開源應用,用於把指標數據可視化。是當下流行的時序數據展示工具。先下載,直接下載exe安裝,完成后能打開http://localhost:3000/頁面就安裝成功了

  • 下載地址

先添加數據源,選擇Prometheus為數據源,並配置。

添加儀錶盤

Import via panel json中加入下面這個json,點擊load,

  • 儀錶盤json

選擇數據源,點擊Import就能看到儀錶盤了

還可以去這裏添加很多現有的儀錶盤。複製ID添加儀錶盤。

參考文章

prometheus-net
.NetCore下使用Prometheus實現系統監控和警報系列

本站聲明:網站內容來源於博客園,如有侵權,請聯繫我們,我們將及時處理

【其他文章推薦】

網頁設計一頭霧水該從何著手呢? 台北網頁設計公司幫您輕鬆架站!

網頁設計公司推薦不同的風格,搶佔消費者視覺第一線

※Google地圖已可更新顯示潭子電動車充電站設置地點!!

※廣告預算用在刀口上,台北網頁設計公司幫您達到更多曝光效益

※別再煩惱如何寫文案,掌握八大原則!

網頁設計最專業,超強功能平台可客製化

聚甘新