CHANGELOG.md

2.20.0-rc.1 / 2020-07-20

• [ENHANCEMENT] SD: Improve Docker Swarm example. #7608
• [ENHANCEMENT] SD: Enable use of Unix socket for Docker Swarm. #7604
• [BUGFIX] SD: Don't panic on an empty Docker Swarm config. #7615
• [BUGFIX] TSDB: Don't panic on WAL corruptions. #7550

2.20.0-rc.0 / 2020-07-16

This release changes WAL compression from opt-in to default. WAL compression will prevent a downgrade to v2.10 or earlier without deleting the WAL. Disable WAL compression explicitly by setting the command line flag --no-storage.tsdb.wal-compression if you require downgrading to v2.10 or earlier.

• [CHANGE] promtool: Changed rule numbering from 0-based to 1-based when reporting rule errors. #7495
• [CHANGE] Remote read: Added prometheus_remote_storage_read_queries_total counter and prometheus_remote_storage_read_request_duration_seconds histogram, removed prometheus_remote_storage_remote_read_queries_total counter.
• [CHANGE] Remote write: Added buckets for longer durations to prometheus_remote_storage_sent_batch_duration_seconds histogram.
• [CHANGE] TSDB: WAL compression is enabled by default. #7410
• [FEATURE] PromQL: Added group() aggregator. #7480
• [FEATURE] SD: Added Docker Swarm SD. #7420
• [FEATURE] SD: Added DigitalOcean SD. #7407
• [FEATURE] SD: Added Openstack config option to query alternative endpoints. #7494
• [ENHANCEMENT] Configuration: Exit early on invalid config file and signal it with exit code 2. #7399
• [ENHANCEMENT] PromQL: without is now a valid metric identifier. #7533
• [ENHANCEMENT] PromQL: Optimized regex label matching for literals within the pattern or as prefix/suffix. #7453 #7503
• [ENHANCEMENT] promtool: Added time range parameters for labels API in promtool. #7463
• [ENHANCEMENT] Remote write: Include samples waiting in channel in pending samples metric. Log number of dropped samples on hard shutdown. #7335
• [ENHANCEMENT] Scrape: Ingest synthetic scrape report metrics atomically with the corresponding scraped metrics. #7562
• [ENHANCEMENT] SD: Reduce timeouts for Openstack SD. #7507
• [ENHANCEMENT] SD: Use 10m timeout for Consul watches. #7423
• [ENHANCEMENT] SD: Added AMI meta label for EC2 SD. #7386
• [ENHANCEMENT] TSDB: Increment WAL corruption metric also on WAL corruption during checkpointing. #7491
• [ENHANCEMENT] TSDB: Improved query performance for high-cardinality labels. #7448
• [ENHANCEMENT] UI: Display dates as well as timestamps in status page. #7544
• [ENHANCEMENT] UI: Improved scrolling when following hash-fragment links. #7456
• [ENHANCEMENT] UI: React UI renders numbers in alerts in a more human-readable way. #7426
• [BUGFIX] API: Fixed error status code in the query API. #7435
• [BUGFIX] PromQL: Fixed avg and avg_over_time for NaN, Inf, and float64 overflows. #7346
• [BUGFIX] PromQL: Fixed off-by-one error in histogram_quantile. #7393
• [BUGFIX] promtool: Support extended durations in rules unit tests. #6297
• [BUGFIX] Scrape: Fix undercounting for scrape_samples_post_metric_relabeling in case of errors. #7342
• [BUGFIX] TSDB: Avoid leaving behind empty files in chunks_head, causing startup failures. #7573
• [BUGFIX] TSDB: Fixed race between compact (gc, populate) and head append causing unknown symbol error. #7560
• [BUGFIX] TSDB: Fixed unknown symbol error during head compaction. #7526
• [BUGFIX] TSDB: Fixed panic during TSDB metric registration. #7501
• [BUGFIX] TSDB: Fixed --limit command line flag in tsdb tool. #7430

2.19.2 / 2020-06-26

• [BUGFIX] Remote Write: Fix panic when reloading config with modified queue parameters. #7452

2.19.1 / 2020-06-18

• [BUGFIX] TSDB: Fix m-map file truncation leading to unsequential files. #7414

2.19.0 / 2020-06-09

• [FEATURE] TSDB: Memory-map full chunks of Head (in-memory) block from disk. This reduces memory footprint and makes restarts faster. #6679
• [ENHANCEMENT] Discovery: Added discovery support for Triton global zones. #7250
• [ENHANCEMENT] Increased alert resend delay to be more tolerant towards failures. #7228
• [ENHANCEMENT] Remote Read: Added prometheus_remote_storage_remote_read_queries_total counter to count total number of remote read queries. #7328
• [ENHANCEMEMT] Added time range parameters for label names and label values API. #7288
• [ENHANCEMENT] TSDB: Reduced contention in isolation for high load. #7332
• [BUGFIX] PromQL: Eliminated collision while checking for duplicate labels. #7058
• [BUGFIX] React UI: Don't null out data when clicking on the current tab. #7243
• [BUGFIX] PromQL: Correctly track number of samples for a query. #7307
• [BUGFIX] PromQL: Return NaN when histogram buckets have 0 observations. #7318

2.18.2 / 2020-06-09

• [BUGFIX] TSDB: Fix incorrect query results when using Prometheus with remote reads configured #7361

2.18.1 / 2020-05-07

• [BUGFIX] TSDB: Fixed snapshot API. #7217

2.18.0 / 2020-05-05

• [CHANGE] Federation: Only use local TSDB for federation (ignore remote read). #7096
• [CHANGE] Rules: rule_evaluations_total and rule_evaluation_failures_total have a rule_group label now. #7094
• [FEATURE] Tracing: Added experimental Jaeger support #7148
• [ENHANCEMENT] TSDB: Significantly reduce WAL size kept around after a block cut. #7098
• [ENHANCEMENT] Discovery: Add architecture meta label for EC2. #7000
• [BUGFIX] UI: Fixed wrong MinTime reported by /status. #7182
• [BUGFIX] React UI: Fixed multiselect legend on OSX. #6880
• [BUGFIX] Remote Write: Fixed blocked resharding edge case. #7122
• [BUGFIX] Remote Write: Fixed remote write not updating on relabel configs change. #7073

2.17.2 / 2020-04-20

• [BUGFIX] Federation: Register federation metrics #7081
• [BUGFIX] PromQL: Fix panic in parser error handling #7132
• [BUGFIX] Rules: Fix reloads hanging when deleting a rule group that is being evaluated #7138
• [BUGFIX] TSDB: Fix a memory leak when prometheus starts with an empty TSDB WAL #7135
• [BUGFIX] TSDB: Make isolation more robust to panics in web handlers #7129 #7136

2.17.1 / 2020-03-26

• [BUGFIX] TSDB: Fix query performance regression that increased memory and CPU usage #7051

2.17.0 / 2020-03-24

This release implements isolation in TSDB. API queries and recording rules are guaranteed to only see full scrapes and full recording rules. This comes with a certain overhead in resource usage. Depending on the situation, there might be some increase in memory usage, CPU usage, or query latency.

• [FEATURE] TSDB: Support isolation #6841
• [ENHANCEMENT] PromQL: Allow more keywords as metric names #6933
• [ENHANCEMENT] React UI: Add normalization of localhost URLs in targets page #6794
• [ENHANCEMENT] Remote read: Read from remote storage concurrently #6770
• [ENHANCEMENT] Rules: Mark deleted rule series as stale after a reload #6745
• [ENHANCEMENT] Scrape: Log scrape append failures as debug rather than warn #6852
• [ENHANCEMENT] TSDB: Improve query performance for queries that partially hit the head #6676
• [ENHANCEMENT] Consul SD: Expose service health as meta label #5313
• [ENHANCEMENT] EC2 SD: Expose EC2 instance lifecycle as meta label #6914
• [ENHANCEMENT] Kubernetes SD: Expose service type as meta label for K8s service role #6684
• [ENHANCEMENT] Kubernetes SD: Expose label_selector and field_selector #6807
• [ENHANCEMENT] Openstack SD: Expose hypervisor id as meta label #6962
• [BUGFIX] PromQL: Do not escape HTML-like chars in query log #6834 #6795
• [BUGFIX] React UI: Fix data table matrix values #6896
• [BUGFIX] React UI: Fix new targets page not loading when using non-ASCII characters #6892
• [BUGFIX] Remote read: Fix duplication of metrics read from remote storage with external labels #6967 #7018
• [BUGFIX] Remote write: Register WAL watcher and live reader metrics for all remotes, not just the first one #6998
• [BUGFIX] Scrape: Prevent removal of metric names upon relabeling #6891
• [BUGFIX] Scrape: Fix 'superfluous response.WriteHeader call' errors when scrape fails under some circonstances #6986
• [BUGFIX] Scrape: Fix crash when reloads are separated by two scrape intervals #7011

2.16.0 / 2020-02-13

• [FEATURE] React UI: Support local timezone on /graph #6692
• [FEATURE] PromQL: add absent_over_time query function #6490
• [FEATURE] Adding optional logging of queries to their own file #6520
• [ENHANCEMENT] React UI: Add support for rules page and "Xs ago" duration displays #6503
• [ENHANCEMENT] React UI: alerts page, replace filtering togglers tabs with checkboxes #6543
• [ENHANCEMENT] TSDB: Export metric for WAL write errors #6647
• [ENHANCEMENT] TSDB: Improve query performance for queries that only touch the most recent 2h of data. #6651
• [ENHANCEMENT] PromQL: Refactoring in parser errors to improve error messages #6634
• [ENHANCEMENT] PromQL: Support trailing commas in grouping opts #6480
• [ENHANCEMENT] Scrape: Reduce memory usage on reloads by reusing scrape cache #6670
• [ENHANCEMENT] Scrape: Add metrics to track bytes and entries in the metadata cache #6675
• [ENHANCEMENT] promtool: Add support for line-column numbers for invalid rules output #6533
• [ENHANCEMENT] Avoid restarting rule groups when it is unnecessary #6450
• [BUGFIX] React UI: Send cookies on fetch() on older browsers #6553
• [BUGFIX] React UI: adopt grafana flot fix for stacked graphs #6603
• [BUFGIX] React UI: broken graph page browser history so that back button works as expected #6659
• [BUGFIX] TSDB: ensure compactionsSkipped metric is registered, and log proper error if one is returned from head.Init #6616
• [BUGFIX] TSDB: return an error on ingesting series with duplicate labels #6664
• [BUGFIX] PromQL: Fix unary operator precedence #6579
• [BUGFIX] PromQL: Respect query.timeout even when we reach query.max-concurrency #6712
• [BUGFIX] PromQL: Fix string and parentheses handling in engine, which affected React UI #6612
• [BUGFIX] PromQL: Remove output labels returned by absent() if they are produced by multiple identical label matchers #6493
• [BUGFIX] Scrape: Validate that OpenMetrics input ends with # EOF #6505
• [BUGFIX] Remote read: return the correct error if configs can't be marshal'd to JSON #6622
• [BUGFIX] Remote write: Make remote client Store use passed context, which can affect shutdown timing #6673
• [BUGFIX] Remote write: Improve sharding calculation in cases where we would always be consistently behind by tracking pendingSamples #6511
• [BUGFIX] Ensure prometheus_rule_group metrics are deleted when a rule group is removed #6693

2.15.2 / 2020-01-06

• [BUGFIX] TSDB: Fixed support for TSDB blocks built with Prometheus before 2.1.0. #6564
• [BUGFIX] TSDB: Fixed block compaction issues on Windows. #6547

2.15.1 / 2019-12-25

• [BUGFIX] TSDB: Fixed race on concurrent queries against same data. #6512

2.15.0 / 2019-12-23

• [CHANGE] Discovery: Removed prometheus_sd_kubernetes_cache_* metrics. Additionally prometheus_sd_kubernetes_workqueue_latency_seconds and prometheus_sd_kubernetes_workqueue_work_duration_seconds metrics now show correct values in seconds. #6393
• [CHANGE] Remote write: Changed query label on prometheus_remote_storage_* metrics to remote_name and url. #6043
• [FEATURE] API: Added new endpoint for exposing per metric metadata /metadata. #6420 #6442
• [ENHANCEMENT] TSDB: Significantly reduced memory footprint of loaded TSDB blocks. #6418 #6461
• [ENHANCEMENT] TSDB: Significantly optimized what we buffer during compaction which should result in lower memory footprint during compaction. #6422 #6452 #6468 #6475
• [ENHANCEMENT] TSDB: Improve replay latency. #6230
• [ENHANCEMENT] TSDB: WAL size is now used for size based retention calculation. #5886
• [ENHANCEMENT] Remote read: Added query grouping and range hints to the remote read request #6401
• [ENHANCEMENT] Remote write: Added prometheus_remote_storage_sent_bytes_total counter per queue. #6344
• [ENHANCEMENT] promql: Improved PromQL parser performance. #6356
• [ENHANCEMENT] React UI: Implemented missing pages like /targets #6276, TSDB status page #6281 #6267 and many other fixes and performance improvements.
• [ENHANCEMENT] promql: Prometheus now accepts spaces between time range and square bracket. e.g [ 5m] #6065
• [BUGFIX] Config: Fixed alertmanager configuration to not miss targets when configurations are similar. #6455
• [BUGFIX] Remote write: Value of prometheus_remote_storage_shards_desired gauge shows raw value of desired shards and it's updated correctly. #6378
• [BUGFIX] Rules: Prometheus now fails the evaluation of rules and alerts where metric results collide with labels specified in labels field. #6469
• [BUGFIX] API: Targets Metadata API /targets/metadata now accepts empty match_targets parameter as in the spec. #6303

2.14.0 / 2019-11-11

• [SECURITY/BUGFIX] UI: Ensure warnings from the API are escaped. #6279
• [FEATURE] API: /api/v1/status/runtimeinfo and /api/v1/status/buildinfo endpoints added for use by the React UI. #6243
• [FEATURE] React UI: implement the new experimental React based UI. #5694 and many more
  • Can be found by under /new.
  • Not all pages are implemented yet.
• [FEATURE] Status: Cardinality statistics added to the Runtime & Build Information page. #6125
• [ENHANCEMENT/BUGFIX] Remote write: fix delays in remote write after a compaction. #6021
• [ENHANCEMENT] UI: Alerts can be filtered by state. #5758
• [BUGFIX] API: lifecycle endpoints return 403 when not enabled. #6057
• [BUGFIX] Build: Fix Solaris build. #6149
• [BUGFIX] Promtool: Remove false duplicate rule warnings when checking rule files with alerts. #6270
• [BUGFIX] Remote write: restore use of deduplicating logger in remote write. #6113
• [BUGFIX] Remote write: do not reshard when unable to send samples. #6111
• [BUGFIX] Service discovery: errors are no longer logged on context cancellation. #6116, #6133
• [BUGFIX] UI: handle null response from API properly. #6071

2.13.1 / 2019-10-16

• [BUGFIX] Fix panic in ARM builds of Prometheus. #6110
• [BUGFIX] promql: fix potential panic in the query logger. #6094
• [BUGFIX] Multiple errors of http: superfluous response.WriteHeader call in the logs. #6145

2.13.0 / 2019-10-04

• [SECURITY/BUGFIX] UI: Fix a Stored DOM XSS vulnerability with query history CVE-2019-10215. #6098
• [CHANGE] Metrics: renamed prometheus_sd_configs_failed_total to prometheus_sd_failed_configs and changed to Gauge #5254
• [ENHANCEMENT] Include the tsdb tool in builds. #6089
• [ENHANCEMENT] Service discovery: add new node address types for kubernetes. #5902
• [ENHANCEMENT] UI: show warnings if query have returned some warnings. #5964
• [ENHANCEMENT] Remote write: reduce memory usage of the series cache. #5849
• [ENHANCEMENT] Remote read: use remote read streaming to reduce memory usage. #5703
• [ENHANCEMENT] Metrics: added metrics for remote write max/min/desired shards to queue manager. #5787
• [ENHANCEMENT] Promtool: show the warnings during label query. #5924
• [ENHANCEMENT] Promtool: improve error messages when parsing bad rules. #5965
• [ENHANCEMENT] Promtool: more promlint rules. #5515
• [BUGFIX] Promtool: fix recording inconsistency due to duplicate labels. #6026
• [BUGFIX] UI: fixes service-discovery view when accessed from unhealthy targets. #5915
• [BUGFIX] Metrics format: OpenMetrics parser crashes on short input. #5939
• [BUGFIX] UI: avoid truncated Y-axis values. #6014

2.12.0 / 2019-08-17

• [FEATURE] Track currently active PromQL queries in a log file. #5794
• [FEATURE] Enable and provide binaries for mips64 / mips64le architectures. #5792
• [ENHANCEMENT] Improve responsiveness of targets web UI and API endpoint. #5740
• [ENHANCEMENT] Improve remote write desired shards calculation. #5763
• [ENHANCEMENT] Flush TSDB pages more precisely. tsdb#660
• [ENHANCEMENT] Add prometheus_tsdb_retention_limit_bytes metric. tsdb#667
• [ENHANCEMENT] Add logging during TSDB WAL replay on startup. tsdb#662
• [ENHANCEMENT] Improve TSDB memory usage. tsdb#653, tsdb#643, tsdb#654, tsdb#642, tsdb#627
• [BUGFIX] Check for duplicate label names in remote read. #5829
• [BUGFIX] Mark deleted rules' series as stale on next evaluation. #5759
• [BUGFIX] Fix JavaScript error when showing warning about out-of-sync server time. #5833
• [BUGFIX] Fix promtool test rules panic when providing empty exp_labels. #5774
• [BUGFIX] Only check last directory when discovering checkpoint number. #5756
• [BUGFIX] Fix error propagation in WAL watcher helper functions. #5741
• [BUGFIX] Correctly handle empty labels from alert templates. #5845

2.11.2 / 2019-08-14

• [BUGFIX/SECURITY] Fix a Stored DOM XSS vulnerability with query history. #5888

2.11.1 / 2019-07-10

• [BUGFIX] Fix potential panic when prometheus is watching multiple zookeeper paths. #5749

2.11.0 / 2019-07-09

• [CHANGE] Remove max_retries from queue_config (it has been unused since rewriting remote-write to utilize the write-ahead-log). #5649
• [CHANGE] The meta file BlockStats no longer holds size information. This is now dynamically calculated and kept in memory. It also includes the meta file size which was not included before. tsdb#637
• [CHANGE] Renamed metric from prometheus_tsdb_wal_reader_corruption_errors to prometheus_tsdb_wal_reader_corruption_errors_total. tsdb#622
• [FEATURE] Add option to use Alertmanager API v2. #5482
• [FEATURE] Added humanizePercentage function for templates. #5670
• [FEATURE] Include InitContainers in Kubernetes Service Discovery. #5598
• [FEATURE] Provide option to compress WAL records using Snappy. #609
• [ENHANCEMENT] Create new clean segment when starting the WAL. tsdb#608
• [ENHANCEMENT] Reduce allocations in PromQL aggregations. #5641
• [ENHANCEMENT] Add storage warnings to LabelValues and LabelNames API results. #5673
• [ENHANCEMENT] Add prometheus_http_requests_total metric. #5640
• [ENHANCEMENT] Enable openbsd/arm build. #5696
• [ENHANCEMENT] Remote-write allocation improvements. #5614
• [ENHANCEMENT] Query performance improvement: Efficient iteration and search in HashForLabels and HashWithoutLabels. #5707
• [ENHANCEMENT] Allow injection of arbitrary headers in promtool. #4389
• [ENHANCEMENT] Allow passing external_labels in alert unit tests groups. #5608
• [ENHANCEMENT] Allows globs for rules when unit testing. #5595
• [ENHANCEMENT] Improved postings intersection matching. tsdb#616
• [ENHANCEMENT] Reduced disk usage for WAL for small setups. tsdb#605
• [ENHANCEMENT] Optimize queries using regexp for set lookups. tsdb#602
• [BUGFIX] resolve race condition in maxGauge. #5647
• [BUGFIX] Fix ZooKeeper connection leak. #5675
• [BUGFIX] Improved atomicity of .tmp block replacement during compaction for usual case. tsdb#636
• [BUGFIX] Fix "unknown series references" after clean shutdown. tsdb#623
• [BUGFIX] Re-calculate block size when calling block.Delete. tsdb#637
• [BUGFIX] Fix unsafe snapshots with head block. tsdb#641
• [BUGFIX] prometheus_tsdb_compactions_failed_total is now incremented on any compaction failure. tsdb#613

2.10.0 / 2019-05-25

• [CHANGE/BUGFIX] API: Encode alert values as string to correctly represent Inf/NaN. #5582
• [FEATURE] Template expansion: Make external labels available as $externalLabels in alert and console template expansion. #5463
• [FEATURE] TSDB: Add prometheus_tsdb_wal_segment_current metric for the WAL segment index that TSDB is currently writing to. tsdb#601
• [FEATURE] Scrape: Add scrape_series_added per-scrape metric. #5546
• [ENHANCEMENT] Discovery/kubernetes: Add labels __meta_kubernetes_endpoint_node_name and __meta_kubernetes_endpoint_hostname. #5571
• [ENHANCEMENT] Discovery/azure: Add label __meta_azure_machine_public_ip. #5475
• [ENHANCEMENT] TSDB: Simplify mergedPostings.Seek, resulting in better performance if there are many posting lists. tsdb#595
• [ENHANCEMENT] Log filesystem type on startup. #5558
• [ENHANCEMENT] Cmd/promtool: Use POST requests for Query and QueryRange. client_golang#557
• [ENHANCEMENT] Web: Sort alerts by group name. #5448
• [ENHANCEMENT] Console templates: Add convenience variables $rawParams, $params, $path. #5463
• [BUGFIX] TSDB: Don't panic when running out of disk space and recover nicely from the condition. tsdb#582
• [BUGFIX] TSDB: Correctly handle empty labels. tsdb#594
• [BUGFIX] TSDB: Don't crash on an unknown tombstone reference. tsdb#604
• [BUGFIX] Storage/remote: Remove queue-manager specific metrics if queue no longer exists. #5445 #5485 #5555
• [BUGFIX] PromQL: Correctly display {__name__="a"}. #5552
• [BUGFIX] Discovery/kubernetes: Use service rather than ingress as the name for the service workqueue. #5520
• [BUGFIX] Discovery/azure: Don't panic on a VM with a public IP. #5587
• [BUGFIX] Discovery/triton: Always read HTTP body to completion. #5596
• [BUGFIX] Web: Fixed Content-Type for js and css instead of using /etc/mime.types. #5551

2.9.2 / 2019-04-24

• [BUGFIX] Make sure subquery range is taken into account for selection #5467
• [BUGFIX] Exhaust every request body before closing it #5166
• [BUGFIX] Cmd/promtool: return errors from rule evaluations #5483
• [BUGFIX] Remote Storage: string interner should not panic in release #5487
• [BUGFIX] Fix memory allocation regression in mergedPostings.Seek tsdb#586

2.9.1 / 2019-04-16

• [BUGFIX] Discovery/kubernetes: fix missing label sanitization #5462
• [BUGFIX] Remote_write: Prevent reshard concurrent with calling stop #5460

2.9.0 / 2019-04-15

This releases uses Go 1.12, which includes a change in how memory is released to Linux. This will cause RSS to be reported as higher, however this is harmless and the memory is available to the kernel when it needs it.

• [CHANGE/ENHANCEMENT] Update Consul to support catalog.ServiceMultipleTags. #5151
• [FEATURE] Add honor_timestamps scrape option. #5304
• [ENHANCEMENT] Discovery/kubernetes: add present labels for labels/annotations. #5443
• [ENHANCEMENT] OpenStack SD: Add ProjectID and UserID meta labels. #5431
• [ENHANCEMENT] Add GODEBUG and retention to the runtime page. #5324 #5322
• [ENHANCEMENT] Add support for POSTing to /series endpoint. #5422
• [ENHANCEMENT] Support PUT methods for Lifecycle and Admin APIs. #5376
• [ENHANCEMENT] Scrape: Add global jitter for HA server. #5181
• [ENHANCEMENT] Check for cancellation on every step of a range evaluation. #5131
• [ENHANCEMENT] String interning for labels & values in the remote_write path. #5316
• [ENHANCEMENT] Don't lose the scrape cache on a failed scrape. #5414
• [ENHANCEMENT] Reload cert files from disk automatically. common#173
• [ENHANCEMENT] Use fixed length millisecond timestamp format for logs. common#172
• [ENHANCEMENT] Performance improvements for postings. tsdb#509 tsdb#572
• [BUGFIX] Remote Write: fix checkpoint reading. #5429
• [BUGFIX] Check if label value is valid when unmarshaling external labels from YAML. #5316
• [BUGFIX] Promparse: sort all labels when parsing. #5372
• [BUGFIX] Reload rules: copy state on both name and labels. #5368
• [BUGFIX] Exponentiation operator to drop metric name in result of operation. #5329
• [BUGFIX] Config: resolve more file paths. #5284
• [BUGFIX] Promtool: resolve relative paths in alert test files. #5336
• [BUGFIX] Set TLSHandshakeTimeout in HTTP transport. common#179
• [BUGFIX] Use fsync to be more resilient to machine crashes. tsdb#573 tsdb#578
• [BUGFIX] Keep series that are still in WAL in checkpoints. tsdb#577
• [BUGFIX] Fix output sample values for scalar-to-vector comparison operations. #5454

2.8.1 / 2019-03-28

• [BUGFIX] Display the job labels in /targets which was removed accidentally. #5406

2.8.0 / 2019-03-12

This release uses Write-Ahead Logging (WAL) for the remote_write API. This currently causes a slight increase in memory usage, which will be addressed in future releases.

• [CHANGE] Default time retention is used only when no size based retention is specified. These are flags where time retention is specified by the flag --storage.tsdb.retention and size retention by --storage.tsdb.retention.size. #5216
• [CHANGE] prometheus_tsdb_storage_blocks_bytes_total is now prometheus_tsdb_storage_blocks_bytes. prometheus/tsdb#506
• [FEATURE] [EXPERIMENTAL] Time overlapping blocks are now allowed; vertical compaction and vertical query merge. It is an optional feature which is controlled by the --storage.tsdb.allow-overlapping-blocks flag, disabled by default. prometheus/tsdb#370
• [ENHANCEMENT] Use the WAL for remote_write API. #4588
• [ENHANCEMENT] Query performance improvements. prometheus/tsdb#531
• [ENHANCEMENT] UI enhancements with upgrade to Bootstrap 4. #5226
• [ENHANCEMENT] Reduce time that Alertmanagers are in flux when reloaded. #5126
• [ENHANCEMENT] Limit number of metrics displayed on UI to 10000. #5139
• [ENHANCEMENT] (1) Remember All/Unhealthy choice on target-overview when reloading page. (2) Resize text-input area on Graph page on mouseclick. #5201
• [ENHANCEMENT] In histogram_quantile merge buckets with equivalent le values. #5158.
• [ENHANCEMENT] Show list of offending labels in the error message in many-to-many scenarios. #5189
• [ENHANCEMENT] Show Storage Retention criteria in effect on /status page. #5322
• [BUGFIX] Fix sorting of rule groups. #5260
• [BUGFIX] Fix support for password_file and bearer_token_file in Kubernetes SD. #5211
• [BUGFIX] Scrape: catch errors when creating HTTP clients #5182. Adds new metrics:
  • prometheus_target_scrape_pools_total
  • prometheus_target_scrape_pools_failed_total
  • prometheus_target_scrape_pool_reloads_total
  • prometheus_target_scrape_pool_reloads_failed_total
• [BUGFIX] Fix panic when aggregator param is not a literal. #5290

2.7.2 / 2019-03-02

• [BUGFIX] prometheus_rule_group_last_evaluation_timestamp_seconds is now a unix timestamp. #5186

2.7.1 / 2019-01-31

This release has a fix for a Stored DOM XSS vulnerability that can be triggered when using the query history functionality. Thanks to Dor Tumarkin from Checkmarx for reporting it.

• [BUGFIX/SECURITY] Fix a Stored DOM XSS vulnerability with query history. #5163
• [BUGFIX] prometheus_rule_group_last_duration_seconds now reports seconds instead of nanoseconds. #5153
• [BUGFIX] Make sure the targets are consistently sorted in the targets page. #5161

2.7.0 / 2019-01-28

We're rolling back the Dockerfile changes introduced in 2.6.0. If you made changes to your docker deployment in 2.6.0, you will need to roll them back. This release also adds experimental support for disk size based retention. To accommodate that we are deprecating the flag storage.tsdb.retention in favour of storage.tsdb.retention.time. We print a warning if the flag is in use, but it will function without breaking until Prometheus 3.0.

• [CHANGE] Rollback Dockerfile to version at 2.5.0. Rollback of the breaking change introduced in 2.6.0. #5122
• [FEATURE] Add subqueries to PromQL. #4831
• [FEATURE] [EXPERIMENTAL] Add support for disk size based retention. Note that we don't consider the WAL size which could be significant and the time based retention policy also applies. #5109 prometheus/tsdb#343
• [FEATURE] Add CORS origin flag. #5011
• [ENHANCEMENT] Consul SD: Add tagged address to the discovery metadata. #5001
• [ENHANCEMENT] Kubernetes SD: Add service external IP and external name to the discovery metadata. #4940
• [ENHANCEMENT] Azure SD: Add support for Managed Identity authentication. #4590
• [ENHANCEMENT] Azure SD: Add tenant and subscription IDs to the discovery metadata. #4969
• [ENHANCEMENT] OpenStack SD: Add support for application credentials based authentication. #4968
• [ENHANCEMENT] Add metric for number of rule groups loaded. #5090
• [BUGFIX] Avoid duplicate tests for alert unit tests. #4964
• [BUGFIX] Don't depend on given order when comparing samples in alert unit testing. #5049
• [BUGFIX] Make sure the retention period doesn't overflow. #5112
• [BUGFIX] Make sure the blocks don't get very large. #5112
• [BUGFIX] Don't generate blocks with no samples. prometheus/tsdb#374
• [BUGFIX] Reintroduce metric for WAL corruptions. prometheus/tsdb#473

2.6.1 / 2019-01-15

• [BUGFIX] Azure SD: Fix discovery getting stuck sometimes. #5088
• [BUGFIX] Marathon SD: Use Tasks.Ports when RequirePorts is false. #5026
• [BUGFIX] Promtool: Fix "out-of-order sample" errors when testing rules. #5069

2.6.0 / 2018-12-17

• [CHANGE] Remove default flags from the container's entrypoint, run Prometheus from /etc/prometheus and symlink the storage directory to /etc/prometheus/data. #4976
• [CHANGE] Promtool: Remove the update command. #3839
• [FEATURE] Add JSON log format via the --log.format flag. #4876
• [FEATURE] API: Add /api/v1/labels endpoint to get all label names. #4835
• [FEATURE] Web: Allow setting the page's title via the --web.ui-title flag. #4841
• [ENHANCEMENT] Add prometheus_tsdb_lowest_timestamp_seconds, prometheus_tsdb_head_min_time_seconds and prometheus_tsdb_head_max_time_seconds metrics. #4888
• [ENHANCEMENT] Add rule_group_last_evaluation_timestamp_seconds metric. #4852
• [ENHANCEMENT] Add prometheus_template_text_expansion_failures_total and prometheus_template_text_expansions_total metrics. #4747
• [ENHANCEMENT] Set consistent User-Agent header in outgoing requests. #4891
• [ENHANCEMENT] Azure SD: Error out at load time when authentication parameters are missing. #4907
• [ENHANCEMENT] EC2 SD: Add the machine's private DNS name to the discovery metadata. #4693
• [ENHANCEMENT] EC2 SD: Add the operating system's platform to the discovery metadata. #4663
• [ENHANCEMENT] Kubernetes SD: Add the pod's phase to the discovery metadata. #4824
• [ENHANCEMENT] Kubernetes SD: Log Kubernetes messages. #4931
• [ENHANCEMENT] Promtool: Collect CPU and trace profiles. #4897
• [ENHANCEMENT] Promtool: Support writing output as JSON. #4848
• [ENHANCEMENT] Remote Read: Return available data if remote read fails partially. #4832
• [ENHANCEMENT] Remote Write: Improve queue performance. #4772
• [ENHANCEMENT] Remote Write: Add min_shards parameter to set the minimum number of shards. #4924
• [ENHANCEMENT] TSDB: Improve WAL reading. #4953
• [ENHANCEMENT] TSDB: Memory improvements. #4953
• [ENHANCEMENT] Web: Log stack traces on panic. #4221
• [ENHANCEMENT] Web UI: Add copy to clipboard button for configuration. #4410
• [ENHANCEMENT] Web UI: Support console queries at specific times. #4764
• [ENHANCEMENT] Web UI: group targets by job then instance. #4898 #4806
• [BUGFIX] Deduplicate handler labels for HTTP metrics. #4732
• [BUGFIX] Fix leaked queriers causing shutdowns to hang. #4922
• [BUGFIX] Fix configuration loading panics on nil pointer slice elements. #4942
• [BUGFIX] API: Correctly skip mismatching targets on /api/v1/targets/metadata. #4905
• [BUGFIX] API: Better rounding for incoming query timestamps. #4941
• [BUGFIX] Azure SD: Fix panic. #4867
• [BUGFIX] Console templates: Fix hover when the metric has a null value. #4906
• [BUGFIX] Discovery: Remove all targets when the scrape configuration gets empty. #4819
• [BUGFIX] Marathon SD: Fix leaked connections. #4915
• [BUGFIX] Marathon SD: Use 'hostPort' member of portMapping to construct target endpoints. #4887
• [BUGFIX] PromQL: Fix a goroutine leak in the lexer/parser. #4858
• [BUGFIX] Scrape: Pass through content-type for non-compressed output. #4912
• [BUGFIX] Scrape: Fix deadlock in the scrape's manager. #4894
• [BUGFIX] Scrape: Scrape targets at fixed intervals even after Prometheus restarts. #4926
• [BUGFIX] TSDB: Support restored snapshots including the head properly. #4953
• [BUGFIX] TSDB: Repair WAL when the last record in a segment is torn. #4953
• [BUGFIX] TSDB: Fix unclosed file readers on Windows systems. #4997
• [BUGFIX] Web: Avoid proxy to connect to the local gRPC server. #4572

2.5.0 / 2018-11-06

• [CHANGE] Group targets by scrape config instead of job name. #4806 #4526
• [CHANGE] Marathon SD: Various changes to adapt to Marathon 1.5+. #4499
• [CHANGE] Discovery: Split prometheus_sd_discovered_targets metric by scrape and notify (Alertmanager SD) as well as by section in the respective configuration. #4753
• [FEATURE] Add OpenMetrics support for scraping (EXPERIMENTAL). #4700
• [FEATURE] Add unit testing for rules. #4350
• [FEATURE] Make maximum number of samples per query configurable via --query.max-samples flag. #4513
• [FEATURE] Make maximum number of concurrent remote reads configurable via --storage.remote.read-concurrent-limit flag. #4656
• [ENHANCEMENT] Support s390x platform for Linux. #4605
• [ENHANCEMENT] API: Add prometheus_api_remote_read_queries metric tracking currently executed or waiting remote read API requests. #4699
• [ENHANCEMENT] Remote Read: Add prometheus_remote_storage_remote_read_queries metric tracking currently in-flight remote read queries. #4677
• [ENHANCEMENT] Remote Read: Reduced memory usage. #4655
• [ENHANCEMENT] Discovery: Add prometheus_sd_discovered_targets, prometheus_sd_received_updates_total, prometheus_sd_updates_delayed_total, and prometheus_sd_updates_total metrics for discovery subsystem. #4667
• [ENHANCEMENT] Discovery: Improve performance of previously slow updates of changes of targets. #4526
• [ENHANCEMENT] Kubernetes SD: Add extended metrics. #4458
• [ENHANCEMENT] OpenStack SD: Support discovering instances from all projects. #4682
• [ENHANCEMENT] OpenStack SD: Discover all interfaces. #4649
• [ENHANCEMENT] OpenStack SD: Support tls_config for the used HTTP client. #4654
• [ENHANCEMENT] Triton SD: Add ability to filter triton_sd targets by pre-defined groups. #4701
• [ENHANCEMENT] Web UI: Avoid browser spell-checking in expression field. #4728
• [ENHANCEMENT] Web UI: Add scrape duration and last evaluation time in targets and rules pages. #4722
• [ENHANCEMENT] Web UI: Improve rule view by wrapping lines. #4702
• [ENHANCEMENT] Rules: Error out at load time for invalid templates, rather than at evaluation time. #4537
• [ENHANCEMENT] TSDB: Add metrics for WAL operations. #4692
• [BUGFIX] Change max/min over_time to handle NaNs properly. #4386
• [BUGFIX] Check label name for count_values PromQL function. #4585
• [BUGFIX] Ensure that vectors and matrices do not contain identical label-sets. #4589

2.4.3 / 2018-10-04

• [BUGFIX] Fix panic when using custom EC2 API for SD #4672
• [BUGFIX] Fix panic when Zookeeper SD cannot connect to servers #4669
• [BUGFIX] Make the skip_head an optional parameter for snapshot API #4674

2.4.2 / 2018-09-21

The last release didn't have bugfix included due to a vendoring error.

• [BUGFIX] Handle WAL corruptions properly prometheus/tsdb#389
• [BUGFIX] Handle WAL migrations correctly on Windows prometheus/tsdb#392

2.4.1 / 2018-09-19

• [ENHANCEMENT] New TSDB metrics prometheus/tsdb#375 prometheus/tsdb#363
• [BUGFIX] Render UI correctly for Windows #4616

2.4.0 / 2018-09-11

This release includes multiple bugfixes and features. Further, the WAL implementation has been re-written so the storage is not forward compatible. Prometheus 2.3 storage will work on 2.4 but not vice-versa.

• [CHANGE] Reduce remote write default retries #4279
• [CHANGE] Remove /heap endpoint #4460
• [FEATURE] Persist alert 'for' state across restarts #4061
• [FEATURE] Add API providing per target metric metadata #4183
• [FEATURE] Add API providing recording and alerting rules #4318 #4501
• [ENHANCEMENT] Brand new WAL implementation for TSDB. Forwards incompatible with previous WAL.
• [ENHANCEMENT] Show rule evaluation errors in UI #4457
• [ENHANCEMENT] Throttle resends of alerts to Alertmanager #4538
• [ENHANCEMENT] Send EndsAt along with the alert to Alertmanager #4550
• [ENHANCEMENT] Limit the samples returned by remote read endpoint #4532
• [ENHANCEMENT] Limit the data read in through remote read #4239
• [ENHANCEMENT] Coalesce identical SD configurations #3912
• [ENHANCEMENT] promtool: Add new commands for debugging and querying #4247 #4308 #4346 #4454
• [ENHANCEMENT] Update console examples for node_exporter v0.16.0 #4208
• [ENHANCEMENT] Optimize PromQL aggregations #4248
• [ENHANCEMENT] Remote read: Add Offset to hints #4226
• [ENHANCEMENT] consul_sd: Add support for ServiceMeta field #4280
• [ENHANCEMENT] ec2_sd: Maintain order of subnet_id label #4405
• [ENHANCEMENT] ec2_sd: Add support for custom endpoint to support EC2 compliant APIs #4333
• [ENHANCEMENT] ec2_sd: Add instance_owner label #4514
• [ENHANCEMENT] azure_sd: Add support for VMSS discovery and multiple environments #4202 #4569
• [ENHANCEMENT] gce_sd: Add instance_id label #4488
• [ENHANCEMENT] Forbid rule-abiding robots from indexing #4266
• [ENHANCEMENT] Log virtual memory limits on startup #4418
• [BUGFIX] Wait for service discovery to stop before exiting #4508
• [BUGFIX] Render SD configs properly #4338
• [BUGFIX] Only add LookbackDelta to vector selectors #4399
• [BUGFIX] ec2_sd: Handle panic-ing nil pointer #4469
• [BUGFIX] consul_sd: Stop leaking connections #4443
• [BUGFIX] Use templated labels also to identify alerts #4500
• [BUGFIX] Reduce floating point errors in stddev and related functions #4533
• [BUGFIX] Log errors while encoding responses #4359

2.3.2 / 2018-07-12

• [BUGFIX] Fix various tsdb bugs #4369
• [BUGFIX] Reorder startup and shutdown to prevent panics. #4321
• [BUGFIX] Exit with non-zero code on error #4296
• [BUGFIX] discovery/kubernetes/ingress: fix scheme discovery #4329
• [BUGFIX] Fix race in zookeeper sd #4355
• [BUGFIX] Better timeout handling in promql #4291 #4300
• [BUGFIX] Propagate errors when selecting series from the tsdb #4136

2.3.1 / 2018-06-19

• [BUGFIX] Avoid infinite loop on duplicate NaN values. #4275
• [BUGFIX] Fix nil pointer deference when using various API endpoints #4282
• [BUGFIX] config: set target group source index during unmarshaling #4245
• [BUGFIX] discovery/file: fix logging #4178
• [BUGFIX] kubernetes_sd: fix namespace filtering #4285
• [BUGFIX] web: restore old path prefix behavior #4273
• [BUGFIX] web: remove security headers added in 2.3.0 #4259

2.3.0 / 2018-06-05

• [CHANGE] marathon_sd: use auth_token and auth_token_file for token-based authentication instead of bearer_token and bearer_token_file respectively.
• [CHANGE] Metric names for HTTP server metrics changed
• [FEATURE] Add query commands to promtool
• [FEATURE] Add security headers to HTTP server responses
• [FEATURE] Pass query hints via remote read API
• [FEATURE] Basic auth passwords can now be configured via file across all configuration
• [ENHANCEMENT] Optimize PromQL and API serialization for memory usage and allocations
• [ENHANCEMENT] Limit number of dropped targets in web UI
• [ENHANCEMENT] Consul and EC2 service discovery allow using server-side filtering for performance improvement
• [ENHANCEMENT] Add advanced filtering configuration to EC2 service discovery
• [ENHANCEMENT] marathon_sd: adds support for basic and bearer authentication, plus all other common HTTP client options (TLS config, proxy URL, etc.)
• [ENHANCEMENT] Provide machine type metadata and labels in GCE service discovery
• [ENHANCEMENT] Add pod controller kind and name to Kubernetes service discovery data
• [ENHANCEMENT] Move TSDB to flock-based log file that works with Docker containers
• [BUGFIX] Properly propagate storage errors in PromQL
• [BUGFIX] Fix path prefix for web pages
• [BUGFIX] Fix goroutine leak in Consul service discovery
• [BUGFIX] Fix races in scrape manager
• [BUGFIX] Fix OOM for very large k in PromQL topk() queries
• [BUGFIX] Make remote write more resilient to unavailable receivers
• [BUGFIX] Make remote write shutdown cleanly
• [BUGFIX] Don't leak files on errors in TSDB's tombstone cleanup
• [BUGFIX] Unary minus expressions now removes the metric name from results
• [BUGFIX] Fix bug that lead to wrong amount of samples considered for time range expressions