CHANGELOG.md 106 KB
Newer Older
1
2
## 2.26.0 / 2021-03-31

3
Prometheus is now built and supporting Go 1.16 (#8544). This reverts the memory release pattern added in Go 1.12. This makes common RSS usage metrics showing more accurate number for actual memory used by Prometheus. You can read more details [here](https://www.bwplotka.dev/2019/golang-memory-monitoring/).
4
5
6

Note that from this release Prometheus is using Alertmanager v2 by default.

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
* [CHANGE] Alerting: Using Alertmanager v2 API by default. #8626
* [CHANGE] Prometheus/Promtool: As agreed on dev summit, binaries are now printing help and usage to stdout instead of stderr. #8542
* [FEATURE] Remote: Add support for AWS SigV4 auth method for remote_write. #8509
* [FEATURE] Scaleway Discovery: Add Scaleway Service Discovery. #8555
* [FEATURE] PromQL: Allow negative offsets. Behind `--enable-feature=promql-negative-offset` flag. #8487
* [FEATURE] **experimental** Exemplars: Add in-memory storage for exemplars. Behind `--enable-feature=exemplar-storage` flag. #6635
* [FEATURE] UI: Add advanced auto-completion, syntax highlighting and linting to graph page query input. #8634
* [ENHANCEMENT] Digital Ocean Discovery: Add `__meta_digitalocean_image` label. #8497
* [ENHANCEMENT] PromQL: Add `last_over_time`, `sgn`, `clamp` functions. #8457
* [ENHANCEMENT] Scrape: Add support for specifying type of Authorization header credentials with Bearer by default. #8512
* [ENHANCEMENT] Scrape: Add `follow_redirects` option to scrape configuration. #8546
* [ENHANCEMENT] Remote: Allow retries on HTTP 429 response code for remote_write. Disabled by default. See [configuration docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) for details. #8237 #8477
* [ENHANCEMENT] Remote: Allow configuring custom headers for remote_read. See [configuration docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read) for details. #8516
* [ENHANCEMENT] UI: Hitting Enter now triggers new query. #8581
* [ENHANCEMENT] UI: Better handling of long rule and names on the `/rules` and `/targets` pages. #8608 #8609
* [ENHANCEMENT] UI: Add collapse/expand all button on the `/targets` page. #8486
* [BUGFIX] TSDB: Eager deletion of removable blocks on every compaction, saving disk peak space usage. #8007
* [BUGFIX] PromQL: Fix parser support for special characters like`炬`. #8517
* [BUGFIX] Rules: Update rule health for append/commit fails. #8619
26

Julien Pivotto's avatar
Julien Pivotto committed
27
28
## 2.25.2 / 2021-03-16

29
* [BUGFIX] Fix the ingestion of scrapes when the wall clock changes, e.g. on suspend. #8601
Julien Pivotto's avatar
Julien Pivotto committed
30

Julien Pivotto's avatar
Julien Pivotto committed
31
32
## 2.25.1 / 2021-03-14

33
34
35
* [BUGFIX] Fix a crash in `promtool` when a subquery with default resolution is used. #8569
* [BUGFIX] Fix a bug that could return duplicate datapoints in queries. #8591
* [BUGFIX] Fix crashes with arm64 when compiled with go1.16. #8593
Julien Pivotto's avatar
Julien Pivotto committed
36

Julien Pivotto's avatar
Julien Pivotto committed
37
## 2.25.0 / 2021-02-17
Julien Pivotto's avatar
Julien Pivotto committed
38
39
40
41

This release includes a new `--enable-feature=` flag that enables
experimental features. Such features might be changed or removed in the future.

42
43
44
45
46
In the next minor release (2.26), Prometheus will use the Alertmanager API v2.
It will be done by defaulting `alertmanager_config.api_version` to `v2`.
Alertmanager API v2 was released in Alertmanager v0.16.0 (released in January
2019).

47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
* [FEATURE] **experimental** API: Accept remote_write requests. Behind the --enable-feature=remote-write-receiver flag. #8424
* [FEATURE] **experimental** PromQL: Add '@ <timestamp>' modifier. Behind the --enable-feature=promql-at-modifier flag. #8121 #8436 #8425
* [ENHANCEMENT] Add optional name property to testgroup for better test failure output. #8440
* [ENHANCEMENT] Add warnings into React Panel on the Graph page. #8427
* [ENHANCEMENT] TSDB: Increase the number of buckets for the compaction duration metric. #8342
* [ENHANCEMENT] Remote: Allow passing along custom remote_write HTTP headers. #8416
* [ENHANCEMENT] Mixins: Scope grafana configuration. #8332
* [ENHANCEMENT] Kubernetes SD: Add endpoint labels metadata. #8273
* [ENHANCEMENT] UI: Expose total number of label pairs in head in TSDB stats page. #8343
* [ENHANCEMENT] TSDB: Reload blocks every minute, to detect new blocks and enforce retention more often. #8343
* [BUGFIX] API: Fix global URL when external address has no port. #8359
* [BUGFIX] Backfill: Fix error message handling. #8432
* [BUGFIX] Backfill: Fix "add sample: out of bounds" error when series span an entire block. #8476
* [BUGFIX] Deprecate unused flag --alertmanager.timeout. #8407
* [BUGFIX] Mixins: Support remote-write metrics renamed in v2.23 in alerts. #8423
* [BUGFIX] Remote: Fix garbage collection of dropped series in remote write. #8387
* [BUGFIX] Remote: Log recoverable remote write errors as warnings. #8412
* [BUGFIX] TSDB: Remove pre-2.21 temporary blocks on start. #8353.
* [BUGFIX] UI: Fix duplicated keys on /targets page. #8456
* [BUGFIX] UI: Fix label name leak into class name. #8459
Julien Pivotto's avatar
Julien Pivotto committed
67

beorn7's avatar
beorn7 committed
68
69
70
71
72
## 2.24.1 / 2021-01-20

* [ENHANCEMENT] Cache basic authentication results to significantly improve performance of HTTP endpoints (via an update of prometheus/exporter-toolkit).
* [BUGFIX] Prevent user enumeration by timing requests sent to authenticated HTTP endpoints (via an update of prometheus/exporter-toolkit).

beorn7's avatar
beorn7 committed
73
## 2.24.0 / 2021-01-06
beorn7's avatar
beorn7 committed
74

75
76
77
78
79
80
81
82
83
84
85
86
87
* [FEATURE] Add TLS and basic authentication to HTTP endpoints. #8316
* [FEATURE] promtool: Add `check web-config` subcommand to check web config files. #8319
* [FEATURE] promtool: Add `tsdb create-blocks-from openmetrics` subcommand to backfill metrics data from an OpenMetrics file. #8084
* [ENHANCEMENT] HTTP API: Fast-fail queries with only empty matchers. #8288
* [ENHANCEMENT] HTTP API: Support matchers for labels API. #8301
* [ENHANCEMENT] promtool: Improve checking of URLs passed on the command line. #7956
* [ENHANCEMENT] SD: Expose IPv6 as a label in EC2 SD. #7086
* [ENHANCEMENT] SD: Reuse EC2 client, reducing frequency of requesting credentials. #8311
* [ENHANCEMENT] TSDB: Add logging when compaction takes more than the block time range. #8151
* [ENHANCEMENT] TSDB: Avoid unnecessary GC runs after compaction. #8276 
* [BUGFIX] HTTP API: Avoid double-closing of channel when quitting multiple times via HTTP. #8242
* [BUGFIX] SD: Ignore CNAME records in DNS SD to avoid spurious `Invalid SRV record` warnings. #8216
* [BUGFIX] SD: Avoid config error triggered by valid label selectors in Kubernetes SD. #8285
beorn7's avatar
beorn7 committed
88

Ganesh Vernekar's avatar
Ganesh Vernekar committed
89
## 2.23.0 / 2020-11-26
Ganesh Vernekar's avatar
Ganesh Vernekar committed
90

91
92
* [CHANGE] UI: Make the React UI default. #8142
* [CHANGE] Remote write: The following metrics were removed/renamed in remote write. #6815
Ganesh Vernekar's avatar
Ganesh Vernekar committed
93
94
95
96
97
98
     - `prometheus_remote_storage_succeeded_samples_total` was removed and `prometheus_remote_storage_samples_total` was introduced for all the samples attempted to send.
     - `prometheus_remote_storage_sent_bytes_total` was removed and replaced with `prometheus_remote_storage_samples_bytes_total` and `prometheus_remote_storage_metadata_bytes_total`.
     - `prometheus_remote_storage_failed_samples_total` -> `prometheus_remote_storage_samples_failed_total` .
     - `prometheus_remote_storage_retried_samples_total` -> `prometheus_remote_storage_samples_retried_total`.
     - `prometheus_remote_storage_dropped_samples_total` -> `prometheus_remote_storage_samples_dropped_total`.
     - `prometheus_remote_storage_pending_samples` -> `prometheus_remote_storage_samples_pending`.
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
* [CHANGE] Remote: Do not collect non-initialized timestamp metrics. #8060
* [FEATURE] [EXPERIMENTAL] Remote write: Allow metric metadata to be propagated via remote write. The following new metrics were introduced: `prometheus_remote_storage_metadata_total`, `prometheus_remote_storage_metadata_failed_total`, `prometheus_remote_storage_metadata_retried_total`, `prometheus_remote_storage_metadata_bytes_total`. #6815
* [ENHANCEMENT] Remote write: Added a metric `prometheus_remote_storage_max_samples_per_send` for remote write. #8102
* [ENHANCEMENT] TSDB: Make the snapshot directory name always the same length. #8138
* [ENHANCEMENT] TSDB: Create a checkpoint only once at the end of all head compactions. #8067
* [ENHANCEMENT] TSDB: Avoid Series API from hitting the chunks. #8050 
* [ENHANCEMENT] TSDB: Cache label name and last value when adding series during compactions making compactions faster. #8192
* [ENHANCEMENT] PromQL: Improved performance of Hash method making queries a bit faster. #8025
* [ENHANCEMENT] promtool: `tsdb list` now prints block sizes. #7993
* [ENHANCEMENT] promtool: Calculate mint and maxt per test avoiding unnecessary calculations. #8096
* [ENHANCEMENT] SD: Add filtering of services to Docker Swarm SD. #8074
* [BUGFIX] React UI: Fix button display when there are no panels. #8155
* [BUGFIX] PromQL: Fix timestamp() method for vector selector inside parenthesis. #8164
* [BUGFIX] PromQL: Don't include rendered expression on PromQL parse errors. #8177
* [BUGFIX] web: Fix panic with double close() of channel on calling `/-/quit/`. #8166
* [BUGFIX] TSDB: Fixed WAL corruption on partial writes within a page causing `invalid checksum` error on WAL replay. #8125
Ganesh Vernekar's avatar
Ganesh Vernekar committed
115
116
117
* [BUGFIX] Update config metrics `prometheus_config_last_reload_successful` and `prometheus_config_last_reload_success_timestamp_seconds` right after initial validation before starting TSDB.
* [BUGFIX] promtool: Correctly detect duplicate label names in exposition.

Frederic Branczyk's avatar
Frederic Branczyk committed
118
119
## 2.22.2 / 2020-11-16

120
* [BUGFIX] Fix race condition in syncing/stopping/reloading scrapers. #8176
Frederic Branczyk's avatar
Frederic Branczyk committed
121

Frederic Branczyk's avatar
Frederic Branczyk committed
122
123
## 2.22.1 / 2020-11-03

124
125
126
* [BUGFIX] Fix potential "mmap: invalid argument" errors in loading the head chunks, after an unclean shutdown, by performing read repairs. #8061
* [BUGFIX] Fix serving metrics and API when reloading scrape config. #8104
* [BUGFIX] Fix head chunk size calculation for size based retention. #8139
Frederic Branczyk's avatar
Frederic Branczyk committed
127

Frederic Branczyk's avatar
Frederic Branczyk committed
128
## 2.22.0 / 2020-10-07
Frederic Branczyk's avatar
Frederic Branczyk committed
129
130
131
132

As announced in the 2.21.0 release notes, the experimental gRPC API v2 has been
removed.

133
134
135
136
137
138
139
140
141
142
143
144
* [CHANGE] web: Remove APIv2. #7935
* [ENHANCEMENT] React UI: Implement missing TSDB head stats section. #7876
* [ENHANCEMENT] UI: Add Collapse all button to targets page. #6957
* [ENHANCEMENT] UI: Clarify alert state toggle via checkbox icon. #7936
* [ENHANCEMENT] Add `rule_group_last_evaluation_samples` and `prometheus_tsdb_data_replay_duration_seconds` metrics. #7737 #7977
* [ENHANCEMENT] Gracefully handle unknown WAL record types. #8004
* [ENHANCEMENT] Issue a warning for 64 bit systems running 32 bit binaries. #8012
* [BUGFIX] Adjust scrape timestamps to align them to the intended schedule, effectively reducing block size. Workaround for a regression in go1.14+. #7976
* [BUGFIX] promtool: Ensure alert rules are marked as restored in unit tests. #7661
* [BUGFIX] Eureka: Fix service discovery when compiled in 32-bit. #7964
* [BUGFIX] Don't do literal regex matching optimisation when case insensitive. #8013
* [BUGFIX] Fix classic UI sometimes running queries for instant query when in range query mode. #7984
Frederic Branczyk's avatar
Frederic Branczyk committed
145

Julien Pivotto's avatar
Julien Pivotto committed
146
## 2.21.0 / 2020-09-11
Julien Pivotto's avatar
Julien Pivotto committed
147

Julien Pivotto's avatar
Julien Pivotto committed
148
149
This release is built with Go 1.15, which deprecates [X.509 CommonName](https://golang.org/doc/go1.15#commonname)
in TLS certificates validation.
Julien Pivotto's avatar
Julien Pivotto committed
150

Julien Pivotto's avatar
Julien Pivotto committed
151
152
153
154
In the unlikely case that you use the gRPC API v2 (which is limited to TSDB
admin commands), please note that we will remove this experimental API in the
next minor release 2.22.

155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
* [CHANGE] Disable HTTP/2 because of concerns with the Go HTTP/2 client. #7588 #7701
* [CHANGE] PromQL: `query_log_file` path is now relative to the config file. #7701
* [CHANGE] Promtool: Replace the tsdb command line tool by a promtool tsdb subcommand. #6088
* [CHANGE] Rules: Label `rule_group_iterations` metric with group name. #7823
* [FEATURE] Eureka SD: New service discovery. #3369
* [FEATURE] Hetzner SD: New service discovery. #7822
* [FEATURE] Kubernetes SD: Support Kubernetes EndpointSlices. #6838
* [FEATURE] Scrape: Add per scrape-config targets limit. #7554
* [ENHANCEMENT] Support composite durations in PromQL, config and UI, e.g. 1h30m. #7713 #7833
* [ENHANCEMENT] DNS SD: Add SRV record target and port meta labels. #7678
* [ENHANCEMENT] Docker Swarm SD: Support tasks and service without published ports. #7686
* [ENHANCEMENT] PromQL: Reduce the amount of data queried by remote read when a subquery has an offset. #7667
* [ENHANCEMENT] Promtool: Add `--time` option to query instant command. #7829
* [ENHANCEMENT] UI: Respect the `--web.page-title` parameter in the React UI. #7607
* [ENHANCEMENT] UI: Add duration, labels, annotations to alerts page in the React UI. #7605
* [ENHANCEMENT] UI: Add duration on the React UI rules page, hide annotation and labels if empty. #7606
* [BUGFIX] API: Deduplicate series in /api/v1/series. #7862
* [BUGFIX] PromQL: Drop metric name in bool comparison between two instant vectors. #7819
* [BUGFIX] PromQL: Exit with an error when time parameters can't be parsed. #7505
* [BUGFIX] Remote read: Re-add accidentally removed tracing for remote-read requests. #7916
* [BUGFIX] Rules: Detect extra fields in rule files. #7767
* [BUGFIX] Rules: Disallow overwriting the metric name in the `labels` section of recording rules. #7787
* [BUGFIX] Rules: Keep evaluation timestamp across reloads. #7775
* [BUGFIX] Scrape: Do not stop scrapes in progress during reload. #7752
* [BUGFIX] TSDB: Fix `chunks.HeadReadWriter: maxt of the files are not set` error. #7856
* [BUGFIX] TSDB: Delete blocks atomically to prevent corruption when there is a panic/crash during deletion. #7772
* [BUGFIX] Triton SD: Fix a panic when triton_sd_config is nil. #7671
* [BUGFIX] UI: Fix react UI bug with series going on and off. #7804
* [BUGFIX] UI: Fix styling bug for target labels with special names in React UI. #7902
* [BUGFIX] Web: Stop CMUX and GRPC servers even with stale connections, preventing the server to stop on SIGTERM. #7810
Julien Pivotto's avatar
Julien Pivotto committed
185

beorn7's avatar
beorn7 committed
186
187
## 2.20.1 / 2020-08-05

188
* [BUGFIX] SD: Reduce the Consul watch timeout to 2m and adjust the request timeout accordingly. #7724
beorn7's avatar
beorn7 committed
189

beorn7's avatar
beorn7 committed
190
## 2.20.0 / 2020-07-22
191

beorn7's avatar
beorn7 committed
192
193
This release changes WAL compression from opt-in to default. WAL compression will prevent a downgrade to v2.10 or earlier without deleting the WAL. Disable WAL compression explicitly by setting the command line flag `--no-storage.tsdb.wal-compression` if you require downgrading to v2.10 or earlier.

194
* [CHANGE] promtool: Changed rule numbering from 0-based to 1-based when reporting rule errors. #7495
beorn7's avatar
beorn7 committed
195
196
* [CHANGE] Remote read: Added `prometheus_remote_storage_read_queries_total` counter and `prometheus_remote_storage_read_request_duration_seconds` histogram, removed `prometheus_remote_storage_remote_read_queries_total` counter.
* [CHANGE] Remote write: Added buckets for longer durations to `prometheus_remote_storage_sent_batch_duration_seconds` histogram.
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
* [CHANGE] TSDB: WAL compression is enabled by default. #7410
* [FEATURE] PromQL: Added `group()` aggregator. #7480
* [FEATURE] SD: Added Docker Swarm SD. #7420
* [FEATURE] SD: Added DigitalOcean SD. #7407
* [FEATURE] SD: Added Openstack config option to query alternative endpoints. #7494
* [ENHANCEMENT] Configuration: Exit early on invalid config file and signal it with exit code 2. #7399
* [ENHANCEMENT] PromQL: `without` is now a valid metric identifier. #7533
* [ENHANCEMENT] PromQL: Optimized regex label matching for literals within the pattern or as prefix/suffix. #7453 #7503
* [ENHANCEMENT] promtool: Added time range parameters for labels API in promtool. #7463
* [ENHANCEMENT] Remote write: Include samples waiting in channel in pending samples metric. Log number of dropped samples on hard shutdown. #7335
* [ENHANCEMENT] Scrape: Ingest synthetic scrape report metrics atomically with the corresponding scraped metrics. #7562
* [ENHANCEMENT] SD: Reduce timeouts for Openstack SD. #7507
* [ENHANCEMENT] SD: Use 10m timeout for Consul watches. #7423
* [ENHANCEMENT] SD: Added AMI meta label for EC2 SD. #7386
* [ENHANCEMENT] TSDB: Increment WAL corruption metric also on WAL corruption during checkpointing. #7491
* [ENHANCEMENT] TSDB: Improved query performance for high-cardinality labels. #7448
* [ENHANCEMENT] UI: Display dates as well as timestamps in status page. #7544
* [ENHANCEMENT] UI: Improved scrolling when following hash-fragment links. #7456
* [ENHANCEMENT] UI: React UI renders numbers in alerts in a more human-readable way. #7426
* [BUGFIX] API: Fixed error status code in the query API. #7435 
* [BUGFIX] PromQL: Fixed `avg` and `avg_over_time` for NaN, Inf, and float64 overflows. #7346
* [BUGFIX] PromQL: Fixed off-by-one error in `histogram_quantile`. #7393
* [BUGFIX] promtool: Support extended durations in rules unit tests. #6297
* [BUGFIX] Scrape: Fix undercounting for `scrape_samples_post_metric_relabeling` in case of errors. #7342
* [BUGFIX] TSDB: Don't panic on WAL corruptions. #7550
* [BUGFIX] TSDB: Avoid leaving behind empty files in `chunks_head`, causing startup failures. #7573
* [BUGFIX] TSDB: Fixed race between compact (gc, populate) and head append causing unknown symbol error. #7560
* [BUGFIX] TSDB: Fixed unknown symbol error during head compaction. #7526
* [BUGFIX] TSDB: Fixed panic during TSDB metric registration. #7501
* [BUGFIX] TSDB: Fixed `--limit` command line flag in `tsdb` tool. #7430
227

Ganesh Vernekar's avatar
Ganesh Vernekar committed
228
229
## 2.19.3 / 2020-07-24

230
231
* [BUGFIX] TSDB: Don't panic on WAL corruptions. #7550
* [BUGFIX] TSDB: Avoid leaving behind empty files in chunks_head, causing startup failures. #7573
Ganesh Vernekar's avatar
Ganesh Vernekar committed
232

Ganesh Vernekar's avatar
Ganesh Vernekar committed
233
234
## 2.19.2 / 2020-06-26

235
* [BUGFIX] Remote Write: Fix panic when reloading config with modified queue parameters. #7452
Ganesh Vernekar's avatar
Ganesh Vernekar committed
236

Ganesh Vernekar's avatar
Ganesh Vernekar committed
237
238
## 2.19.1 / 2020-06-18

239
* [BUGFIX] TSDB: Fix m-map file truncation leading to unsequential files. #7414
Ganesh Vernekar's avatar
Ganesh Vernekar committed
240

Ganesh Vernekar's avatar
Ganesh Vernekar committed
241
## 2.19.0 / 2020-06-09
Ganesh Vernekar's avatar
Ganesh Vernekar committed
242

243
244
245
246
247
248
249
250
251
252
* [FEATURE] TSDB: Memory-map full chunks of Head (in-memory) block from disk. This reduces memory footprint and makes restarts faster. #6679
* [ENHANCEMENT] Discovery: Added discovery support for Triton global zones. #7250
* [ENHANCEMENT] Increased alert resend delay to be more tolerant towards failures. #7228
* [ENHANCEMENT] Remote Read: Added `prometheus_remote_storage_remote_read_queries_total` counter to count total number of remote read queries. #7328
* [ENHANCEMEMT] Added time range parameters for label names and label values API. #7288
* [ENHANCEMENT] TSDB: Reduced contention in isolation for high load. #7332 
* [BUGFIX] PromQL: Eliminated collision while checking for duplicate labels. #7058 
* [BUGFIX] React UI: Don't null out data when clicking on the current tab. #7243
* [BUGFIX] PromQL: Correctly track number of samples for a query. #7307
* [BUGFIX] PromQL: Return NaN when histogram buckets have 0 observations. #7318
253

254
255
## 2.18.2 / 2020-06-09

256
* [BUGFIX] TSDB: Fix incorrect query results when using Prometheus with remote reads configured #7361
Ganesh Vernekar's avatar
Ganesh Vernekar committed
257

Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
258
259
## 2.18.1 / 2020-05-07

260
* [BUGFIX] TSDB: Fixed snapshot API. #7217
Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
261

262
## 2.18.0 / 2020-05-05
263

264
265
266
267
268
269
270
271
272
* [CHANGE] Federation: Only use local TSDB for federation (ignore remote read). #7096
* [CHANGE] Rules: `rule_evaluations_total` and `rule_evaluation_failures_total` have a `rule_group` label now. #7094
* [FEATURE] Tracing: Added experimental Jaeger support #7148
* [ENHANCEMENT] TSDB: Significantly reduce WAL size kept around after a block cut. #7098
* [ENHANCEMENT] Discovery: Add `architecture` meta label for EC2. #7000
* [BUGFIX] UI: Fixed wrong MinTime reported by /status. #7182
* [BUGFIX] React UI: Fixed multiselect legend on OSX. #6880
* [BUGFIX] Remote Write: Fixed blocked resharding edge case. #7122
* [BUGFIX] Remote Write: Fixed remote write not updating on relabel configs change. #7073
273
274


Julien Pivotto's avatar
Julien Pivotto committed
275
276
## 2.17.2 / 2020-04-20

277
278
279
280
281
* [BUGFIX] Federation: Register federation metrics #7081
* [BUGFIX] PromQL: Fix panic in parser error handling #7132
* [BUGFIX] Rules: Fix reloads hanging when deleting a rule group that is being evaluated #7138
* [BUGFIX] TSDB: Fix a memory leak when prometheus starts with an empty TSDB WAL #7135
* [BUGFIX] TSDB: Make isolation more robust to panics in web handlers #7129 #7136
Julien Pivotto's avatar
Julien Pivotto committed
282

Julien Pivotto's avatar
Julien Pivotto committed
283
284
## 2.17.1 / 2020-03-26

285
* [BUGFIX] TSDB: Fix query performance regression that increased memory and CPU usage #7051
Julien Pivotto's avatar
Julien Pivotto committed
286

Julien Pivotto's avatar
Julien Pivotto committed
287
## 2.17.0 / 2020-03-24
Julien Pivotto's avatar
Julien Pivotto committed
288
289
290
291
292
293

This release implements isolation in TSDB. API queries and recording rules are
guaranteed to only see full scrapes and full recording rules. This comes with a
certain overhead in resource usage. Depending on the situation, there might be
some increase in memory usage, CPU usage, or query latency.

294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
* [FEATURE] TSDB: Support isolation #6841
* [ENHANCEMENT] PromQL: Allow more keywords as metric names #6933
* [ENHANCEMENT] React UI: Add normalization of localhost URLs in targets page #6794
* [ENHANCEMENT] Remote read: Read from remote storage concurrently #6770
* [ENHANCEMENT] Rules: Mark deleted rule series as stale after a reload #6745
* [ENHANCEMENT] Scrape: Log scrape append failures as debug rather than warn #6852
* [ENHANCEMENT] TSDB: Improve query performance for queries that partially hit the head #6676
* [ENHANCEMENT] Consul SD: Expose service health as meta label #5313
* [ENHANCEMENT] EC2 SD: Expose EC2 instance lifecycle as meta label #6914
* [ENHANCEMENT] Kubernetes SD: Expose service type as meta label for K8s service role #6684
* [ENHANCEMENT] Kubernetes SD: Expose label_selector and field_selector #6807
* [ENHANCEMENT] Openstack SD: Expose hypervisor id as meta label #6962
* [BUGFIX] PromQL: Do not escape HTML-like chars in query log #6834 #6795
* [BUGFIX] React UI: Fix data table matrix values #6896
* [BUGFIX] React UI: Fix new targets page not loading when using non-ASCII characters #6892
* [BUGFIX] Remote read: Fix duplication of metrics read from remote storage with external labels #6967 #7018
* [BUGFIX] Remote write: Register WAL watcher and live reader metrics for all remotes, not just the first one #6998
* [BUGFIX] Scrape: Prevent removal of metric names upon relabeling #6891
* [BUGFIX] Scrape: Fix 'superfluous response.WriteHeader call' errors when scrape fails under some circonstances #6986
* [BUGFIX] Scrape: Fix crash when reloads are separated by two scrape intervals #7011
Julien Pivotto's avatar
Julien Pivotto committed
314

Callum Styan's avatar
Callum Styan committed
315
## 2.16.0 / 2020-02-13
316

317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
* [FEATURE] React UI: Support local timezone on /graph #6692
* [FEATURE] PromQL: add absent_over_time query function #6490
* [FEATURE] Adding optional logging of queries to their own file #6520
* [ENHANCEMENT] React UI: Add support for rules page and "Xs ago" duration displays #6503
* [ENHANCEMENT] React UI: alerts page, replace filtering togglers tabs with checkboxes #6543
* [ENHANCEMENT] TSDB: Export metric for WAL write errors #6647
* [ENHANCEMENT] TSDB: Improve query performance for queries that only touch the most recent 2h of data. #6651
* [ENHANCEMENT] PromQL: Refactoring in parser errors to improve error messages #6634
* [ENHANCEMENT] PromQL: Support trailing commas in grouping opts #6480
* [ENHANCEMENT] Scrape: Reduce memory usage on reloads by reusing scrape cache #6670
* [ENHANCEMENT] Scrape: Add metrics to track bytes and entries in the metadata cache #6675
* [ENHANCEMENT] promtool: Add support for line-column numbers for invalid rules output #6533
* [ENHANCEMENT] Avoid restarting rule groups when it is unnecessary #6450
* [BUGFIX] React UI: Send cookies on fetch() on older browsers #6553
* [BUGFIX] React UI: adopt grafana flot fix for stacked graphs #6603
* [BUFGIX] React UI: broken graph page browser history so that back button works as expected #6659
* [BUGFIX] TSDB: ensure compactionsSkipped metric is registered, and log proper error if one is returned from head.Init #6616
* [BUGFIX] TSDB: return an error on ingesting series with duplicate labels #6664
* [BUGFIX] PromQL: Fix unary operator precedence #6579
* [BUGFIX] PromQL: Respect query.timeout even when we reach query.max-concurrency #6712
* [BUGFIX] PromQL: Fix string and parentheses handling in engine, which affected React UI #6612
* [BUGFIX] PromQL: Remove output labels returned by absent() if they are produced by multiple identical label matchers #6493
* [BUGFIX] Scrape: Validate that OpenMetrics input ends with `# EOF` #6505
* [BUGFIX] Remote read: return the correct error if configs can't be marshal'd to JSON #6622
* [BUGFIX] Remote write: Make remote client `Store` use passed context, which can affect shutdown timing #6673
* [BUGFIX] Remote write: Improve sharding calculation in cases where we would always be consistently behind by tracking pendingSamples #6511
* [BUGFIX] Ensure prometheus_rule_group metrics are deleted when a rule group is removed #6693
344

Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
345
346
## 2.15.2 / 2020-01-06

347
348
* [BUGFIX] TSDB: Fixed support for TSDB blocks built with Prometheus before 2.1.0. #6564
* [BUGFIX] TSDB: Fixed block compaction issues on Windows. #6547
Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
349

Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
350
351
## 2.15.1 / 2019-12-25

352
* [BUGFIX] TSDB: Fixed race on concurrent queries against same data. #6512
Bartlomiej Plotka's avatar
Bartlomiej Plotka committed
353

354
355
## 2.15.0 / 2019-12-23

356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
* [CHANGE] Discovery: Removed `prometheus_sd_kubernetes_cache_*` metrics. Additionally `prometheus_sd_kubernetes_workqueue_latency_seconds` and `prometheus_sd_kubernetes_workqueue_work_duration_seconds` metrics now show correct values in seconds. #6393
* [CHANGE] Remote write: Changed `query` label on `prometheus_remote_storage_*` metrics to `remote_name` and `url`. #6043
* [FEATURE] API: Added new endpoint for exposing per metric metadata `/metadata`. #6420 #6442
* [ENHANCEMENT] TSDB: Significantly reduced memory footprint of loaded TSDB blocks. #6418 #6461
* [ENHANCEMENT] TSDB: Significantly optimized what we buffer during compaction which should result in lower memory footprint during compaction. #6422 #6452 #6468 #6475
* [ENHANCEMENT] TSDB: Improve replay latency. #6230
* [ENHANCEMENT] TSDB: WAL size is now used for size based retention calculation. #5886
* [ENHANCEMENT] Remote read: Added query grouping and range hints to the remote read request #6401
* [ENHANCEMENT] Remote write: Added `prometheus_remote_storage_sent_bytes_total` counter per queue. #6344
* [ENHANCEMENT] promql: Improved PromQL parser performance. #6356
* [ENHANCEMENT] React UI: Implemented missing pages like `/targets` #6276, TSDB status page #6281 #6267 and many other fixes and performance improvements.
* [ENHANCEMENT] promql: Prometheus now accepts spaces between time range and square bracket. e.g `[ 5m]` #6065
* [BUGFIX] Config: Fixed alertmanager configuration to not miss targets when configurations are similar. #6455
* [BUGFIX] Remote write: Value of `prometheus_remote_storage_shards_desired` gauge shows raw value of desired shards and it's updated correctly. #6378
* [BUGFIX] Rules: Prometheus now fails the evaluation of rules and alerts where metric results collide with labels specified in `labels` field. #6469
* [BUGFIX] API: Targets Metadata API `/targets/metadata` now accepts empty `match_targets` parameter as in the spec. #6303
372

Chris Marchbanks's avatar
Chris Marchbanks committed
373
## 2.14.0 / 2019-11-11
374

375
376
377
* [SECURITY/BUGFIX] UI: Ensure warnings from the API are escaped. #6279
* [FEATURE] API: `/api/v1/status/runtimeinfo` and `/api/v1/status/buildinfo` endpoints added for use by the React UI. #6243
* [FEATURE] React UI: implement the new experimental React based UI. #5694 and many more
378
379
  * Can be found by under `/new`.
  * Not all pages are implemented yet.
380
381
382
383
384
385
386
387
388
389
* [FEATURE] Status: Cardinality statistics added to the Runtime & Build Information page. #6125
* [ENHANCEMENT/BUGFIX] Remote write: fix delays in remote write after a compaction. #6021
* [ENHANCEMENT] UI: Alerts can be filtered by state. #5758
* [BUGFIX] API: lifecycle endpoints return 403 when not enabled. #6057
* [BUGFIX] Build: Fix Solaris build. #6149
* [BUGFIX] Promtool: Remove false duplicate rule warnings when checking rule files with alerts. #6270
* [BUGFIX] Remote write: restore use of deduplicating logger in remote write. #6113
* [BUGFIX] Remote write: do not reshard when unable to send samples. #6111
* [BUGFIX] Service discovery: errors are no longer logged on context cancellation. #6116, #6133
* [BUGFIX] UI: handle null response from API properly. #6071
390

391
392
## 2.13.1 / 2019-10-16

393
394
395
* [BUGFIX] Fix panic in ARM builds of Prometheus. #6110
* [BUGFIX] promql: fix potential panic in the query logger. #6094
* [BUGFIX] Multiple errors of http: superfluous response.WriteHeader call in the logs. #6145
396

397
## 2.13.0 / 2019-10-04
398

399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
* [SECURITY/BUGFIX] UI: Fix a Stored DOM XSS vulnerability with query history [CVE-2019-10215](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10215). #6098
* [CHANGE] Metrics: renamed prometheus_sd_configs_failed_total to prometheus_sd_failed_configs and changed to Gauge #5254
* [ENHANCEMENT] Include the tsdb tool in builds. #6089
* [ENHANCEMENT] Service discovery: add new node address types for kubernetes. #5902
* [ENHANCEMENT] UI: show warnings if query have returned some warnings. #5964
* [ENHANCEMENT] Remote write: reduce memory usage of the series cache. #5849
* [ENHANCEMENT] Remote read: use remote read streaming to reduce memory usage. #5703
* [ENHANCEMENT] Metrics: added metrics for remote write max/min/desired shards to queue manager. #5787
* [ENHANCEMENT] Promtool: show the warnings during label query. #5924
* [ENHANCEMENT] Promtool: improve error messages when parsing bad rules. #5965
* [ENHANCEMENT] Promtool: more promlint rules. #5515
* [BUGFIX] Promtool: fix recording inconsistency due to duplicate labels. #6026
* [BUGFIX] UI: fixes service-discovery view when accessed from unhealthy targets. #5915
* [BUGFIX] Metrics format: OpenMetrics parser crashes on short input. #5939
* [BUGFIX] UI: avoid truncated Y-axis values. #6014
414

Julius Volz's avatar
Julius Volz committed
415
## 2.12.0 / 2019-08-17
Julius Volz's avatar
Julius Volz committed
416

417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
* [FEATURE] Track currently active PromQL queries in a log file. #5794
* [FEATURE] Enable and provide binaries for `mips64` / `mips64le` architectures. #5792
* [ENHANCEMENT] Improve responsiveness of targets web UI and API endpoint. #5740
* [ENHANCEMENT] Improve remote write desired shards calculation. #5763
* [ENHANCEMENT] Flush TSDB pages more precisely. tsdb#660
* [ENHANCEMENT] Add `prometheus_tsdb_retention_limit_bytes` metric. tsdb#667
* [ENHANCEMENT] Add logging during TSDB WAL replay on startup. tsdb#662
* [ENHANCEMENT] Improve TSDB memory usage. tsdb#653, tsdb#643, tsdb#654, tsdb#642, tsdb#627
* [BUGFIX] Check for duplicate label names in remote read. #5829
* [BUGFIX] Mark deleted rules' series as stale on next evaluation. #5759
* [BUGFIX] Fix JavaScript error when showing warning about out-of-sync server time. #5833
* [BUGFIX] Fix `promtool test rules` panic when providing empty `exp_labels`. #5774
* [BUGFIX] Only check last directory when discovering checkpoint number. #5756
* [BUGFIX] Fix error propagation in WAL watcher helper functions. #5741
* [BUGFIX] Correctly handle empty labels from alert templates. #5845
Julius Volz's avatar
Julius Volz committed
432

Frederic Branczyk's avatar
Frederic Branczyk committed
433
434
## 2.11.2 / 2019-08-14

435
* [BUGFIX/SECURITY] Fix a Stored DOM XSS vulnerability with query history. #5888
Frederic Branczyk's avatar
Frederic Branczyk committed
436

Frederic Branczyk's avatar
Frederic Branczyk committed
437
438
## 2.11.1 / 2019-07-10

439
* [BUGFIX] Fix potential panic when prometheus is watching multiple zookeeper paths. #5749
Frederic Branczyk's avatar
Frederic Branczyk committed
440

Frederic Branczyk's avatar
Frederic Branczyk committed
441
## 2.11.0 / 2019-07-09
Frederic Branczyk's avatar
Frederic Branczyk committed
442

443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
* [CHANGE] Remove `max_retries` from queue_config (it has been unused since rewriting remote-write to utilize the write-ahead-log). #5649
* [CHANGE] The meta file `BlockStats` no longer holds size information. This is now dynamically calculated and kept in memory. It also includes the meta file size which was not included before. tsdb#637
* [CHANGE] Renamed metric from `prometheus_tsdb_wal_reader_corruption_errors` to `prometheus_tsdb_wal_reader_corruption_errors_total`. tsdb#622
* [FEATURE] Add option to use Alertmanager API v2. #5482
* [FEATURE] Added `humanizePercentage` function for templates. #5670
* [FEATURE] Include InitContainers in Kubernetes Service Discovery. #5598
* [FEATURE] Provide option to compress WAL records using Snappy. [#609](https://github.com/prometheus/tsdb/pull/609)
* [ENHANCEMENT] Create new clean segment when starting the WAL. tsdb#608
* [ENHANCEMENT] Reduce allocations in PromQL aggregations. #5641
* [ENHANCEMENT] Add storage warnings to LabelValues and LabelNames API results. #5673
* [ENHANCEMENT] Add `prometheus_http_requests_total` metric. #5640
* [ENHANCEMENT] Enable openbsd/arm build. #5696
* [ENHANCEMENT] Remote-write allocation improvements. #5614
* [ENHANCEMENT] Query performance improvement: Efficient iteration and search in HashForLabels and HashWithoutLabels. #5707
* [ENHANCEMENT] Allow injection of arbitrary headers in promtool. #4389
* [ENHANCEMENT] Allow passing `external_labels` in alert unit tests groups. #5608
* [ENHANCEMENT] Allows globs for rules when unit testing. #5595
* [ENHANCEMENT] Improved postings intersection matching. tsdb#616
* [ENHANCEMENT] Reduced disk usage for WAL for small setups. tsdb#605
* [ENHANCEMENT] Optimize queries using regexp for set lookups. tsdb#602
* [BUGFIX] resolve race condition in maxGauge. #5647
* [BUGFIX] Fix ZooKeeper connection leak. #5675
* [BUGFIX] Improved atomicity of .tmp block replacement during compaction for usual case. tsdb#636
* [BUGFIX] Fix "unknown series references" after clean shutdown. tsdb#623
* [BUGFIX] Re-calculate block size when calling `block.Delete`. tsdb#637
* [BUGFIX] Fix unsafe snapshots with head block. tsdb#641
* [BUGFIX] `prometheus_tsdb_compactions_failed_total` is now incremented on any compaction failure. tsdb#613
Frederic Branczyk's avatar
Frederic Branczyk committed
470

beorn7's avatar
beorn7 committed
471
## 2.10.0 / 2019-05-25
beorn7's avatar
beorn7 committed
472

473
474
475
476
477
478
479
480
* [CHANGE/BUGFIX] API: Encode alert values as string to correctly represent Inf/NaN. #5582
* [FEATURE] Template expansion: Make external labels available as `$externalLabels` in alert and console template expansion. #5463
* [FEATURE] TSDB: Add `prometheus_tsdb_wal_segment_current` metric for the WAL segment index that TSDB is currently writing to. tsdb#601
* [FEATURE] Scrape: Add `scrape_series_added` per-scrape metric. #5546
* [ENHANCEMENT] Discovery/kubernetes: Add labels `__meta_kubernetes_endpoint_node_name` and `__meta_kubernetes_endpoint_hostname`. #5571
* [ENHANCEMENT] Discovery/azure: Add label `__meta_azure_machine_public_ip`. #5475
* [ENHANCEMENT] TSDB: Simplify mergedPostings.Seek, resulting in better performance if there are many posting lists. tsdb#595
* [ENHANCEMENT] Log filesystem type on startup. #5558
beorn7's avatar
beorn7 committed
481
* [ENHANCEMENT] Cmd/promtool: Use POST requests for Query and QueryRange. client_golang#557
482
483
484
485
486
487
488
489
490
491
492
* [ENHANCEMENT] Web: Sort alerts by group name. #5448
* [ENHANCEMENT] Console templates: Add convenience variables `$rawParams`, `$params`, `$path`. #5463
* [BUGFIX] TSDB: Don't panic when running out of disk space and recover nicely from the condition. tsdb#582
* [BUGFIX] TSDB: Correctly handle empty labels. tsdb#594
* [BUGFIX] TSDB: Don't crash on an unknown tombstone reference. tsdb#604
* [BUGFIX] Storage/remote: Remove queue-manager specific metrics if queue no longer exists. #5445 #5485 #5555
* [BUGFIX] PromQL: Correctly display `{__name__="a"}`. #5552
* [BUGFIX] Discovery/kubernetes: Use `service` rather than `ingress` as the name for the service workqueue. #5520
* [BUGFIX] Discovery/azure: Don't panic on a VM with a public IP. #5587
* [BUGFIX] Discovery/triton: Always read HTTP body to completion. #5596
* [BUGFIX] Web: Fixed Content-Type for js and css instead of using `/etc/mime.types`. #5551
beorn7's avatar
beorn7 committed
493

Brian Brazil's avatar
Brian Brazil committed
494
495
## 2.9.2 / 2019-04-24

496
497
498
499
500
* [BUGFIX] Make sure subquery range is taken into account for selection #5467
* [BUGFIX] Exhaust every request body before closing it #5166
* [BUGFIX] Cmd/promtool: return errors from rule evaluations #5483
* [BUGFIX] Remote Storage: string interner should not panic in release #5487
* [BUGFIX] Fix memory allocation regression in mergedPostings.Seek tsdb#586
Brian Brazil's avatar
Brian Brazil committed
501

Brian Brazil's avatar
Brian Brazil committed
502
503
## 2.9.1 / 2019-04-16

504
505
* [BUGFIX] Discovery/kubernetes: fix missing label sanitization #5462
* [BUGFIX] Remote_write: Prevent reshard concurrent with calling stop #5460
Brian Brazil's avatar
Brian Brazil committed
506

Brian Brazil's avatar
Brian Brazil committed
507
508
509
510
511
## 2.9.0 / 2019-04-15

This releases uses Go 1.12, which includes a change in how memory is released
to Linux. This will cause RSS to be reported as higher, however this is harmless
and the memory is available to the kernel when it needs it.
512

513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
* [CHANGE/ENHANCEMENT] Update Consul to support catalog.ServiceMultipleTags. #5151
* [FEATURE] Add honor_timestamps scrape option. #5304
* [ENHANCEMENT] Discovery/kubernetes: add present labels for labels/annotations. #5443
* [ENHANCEMENT] OpenStack SD: Add ProjectID and UserID meta labels. #5431
* [ENHANCEMENT] Add GODEBUG and retention to the runtime page. #5324 #5322
* [ENHANCEMENT] Add support for POSTing to /series endpoint. #5422
* [ENHANCEMENT] Support PUT methods for Lifecycle and Admin APIs. #5376
* [ENHANCEMENT] Scrape: Add global jitter for HA server. #5181
* [ENHANCEMENT] Check for cancellation on every step of a range evaluation. #5131
* [ENHANCEMENT] String interning for labels & values in the remote_write path. #5316
* [ENHANCEMENT] Don't lose the scrape cache on a failed scrape. #5414
* [ENHANCEMENT] Reload cert files from disk automatically. common#173
* [ENHANCEMENT] Use fixed length millisecond timestamp format for logs. common#172
* [ENHANCEMENT] Performance improvements for postings. tsdb#509 tsdb#572
* [BUGFIX] Remote Write: fix checkpoint reading. #5429
* [BUGFIX] Check if label value is valid when unmarshaling external labels from YAML. #5316
* [BUGFIX] Promparse: sort all labels when parsing. #5372
* [BUGFIX] Reload rules: copy state on both name and labels. #5368
* [BUGFIX] Exponentiation operator to drop metric name in result of operation. #5329
* [BUGFIX] Config: resolve more file paths. #5284
* [BUGFIX] Promtool: resolve relative paths in alert test files. #5336
* [BUGFIX] Set TLSHandshakeTimeout in HTTP transport. common#179
* [BUGFIX] Use fsync to be more resilient to machine crashes. tsdb#573 tsdb#578
* [BUGFIX] Keep series that are still in WAL in checkpoints. tsdb#577
* [BUGFIX] Fix output sample values for scalar-to-vector comparison operations. #5454
538

Ganesh Vernekar's avatar
Ganesh Vernekar committed
539
540
## 2.8.1 / 2019-03-28

541
* [BUGFIX] Display the job labels in `/targets` which was removed accidentally. #5406
Ganesh Vernekar's avatar
Ganesh Vernekar committed
542

Ganesh Vernekar's avatar
Ganesh Vernekar committed
543
544
545
## 2.8.0 / 2019-03-12

This release uses Write-Ahead Logging (WAL) for the remote_write API. This currently causes a slight increase in memory usage, which will be addressed in future releases.
Ganesh Vernekar's avatar
Ganesh Vernekar committed
546

547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
* [CHANGE] Default time retention is used only when no size based retention is specified. These are flags where time retention is specified by the flag `--storage.tsdb.retention` and size retention by `--storage.tsdb.retention.size`. #5216
* [CHANGE] `prometheus_tsdb_storage_blocks_bytes_total` is now `prometheus_tsdb_storage_blocks_bytes`. prometheus/tsdb#506
* [FEATURE] [EXPERIMENTAL] Time overlapping blocks are now allowed; vertical compaction and vertical query merge. It is an optional feature which is controlled by the `--storage.tsdb.allow-overlapping-blocks` flag, disabled by default. prometheus/tsdb#370
* [ENHANCEMENT] Use the WAL for remote_write API. #4588
* [ENHANCEMENT] Query performance improvements. prometheus/tsdb#531
* [ENHANCEMENT] UI enhancements with upgrade to Bootstrap 4. #5226
* [ENHANCEMENT] Reduce time that Alertmanagers are in flux when reloaded. #5126
* [ENHANCEMENT] Limit number of metrics displayed on UI to 10000. #5139
* [ENHANCEMENT] (1) Remember All/Unhealthy choice on target-overview when reloading page. (2) Resize text-input area on Graph page on mouseclick. #5201
* [ENHANCEMENT] In `histogram_quantile` merge buckets with equivalent le values. #5158.
* [ENHANCEMENT] Show list of offending labels in the error message in many-to-many scenarios. #5189
* [ENHANCEMENT] Show `Storage Retention` criteria in effect on `/status` page. #5322
* [BUGFIX] Fix sorting of rule groups. #5260
* [BUGFIX] Fix support for password_file and bearer_token_file in Kubernetes SD. #5211
* [BUGFIX] Scrape: catch errors when creating HTTP clients #5182. Adds new metrics:
Ganesh Vernekar's avatar
Ganesh Vernekar committed
562
563
564
565
  * `prometheus_target_scrape_pools_total`
  * `prometheus_target_scrape_pools_failed_total`
  * `prometheus_target_scrape_pool_reloads_total`
  * `prometheus_target_scrape_pool_reloads_failed_total`
566
* [BUGFIX] Fix panic when aggregator param is not a literal. #5290
Ganesh Vernekar's avatar
Ganesh Vernekar committed
567

Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
568
569
## 2.7.2 / 2019-03-02

570
* [BUGFIX] `prometheus_rule_group_last_evaluation_timestamp_seconds` is now a unix timestamp. #5186
Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
571

572
573
574
575
## 2.7.1 / 2019-01-31

This release has a fix for a Stored DOM XSS vulnerability that can be triggered when using the query history functionality. Thanks to Dor Tumarkin from Checkmarx for reporting it.

576
577
578
* [BUGFIX/SECURITY] Fix a Stored DOM XSS vulnerability with query history. #5163
* [BUGFIX] `prometheus_rule_group_last_duration_seconds` now reports seconds instead of nanoseconds. #5153
* [BUGFIX] Make sure the targets are consistently sorted in the targets page. #5161
579

Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
580
## 2.7.0 / 2019-01-28
581

582
We're rolling back the Dockerfile changes introduced in 2.6.0. If you made changes to your docker deployment in 2.6.0, you will need to roll them back. This release also adds experimental support for disk size based retention. To accommodate that we are deprecating the flag `storage.tsdb.retention` in favour of `storage.tsdb.retention.time`. We print a warning if the flag is in use, but it will function without breaking until Prometheus 3.0.
583

584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
* [CHANGE] Rollback Dockerfile to version at 2.5.0. Rollback of the breaking change introduced in 2.6.0. #5122
* [FEATURE] Add subqueries to PromQL. #4831
* [FEATURE] [EXPERIMENTAL] Add support for disk size based retention. Note that we don't consider the WAL size which could be significant and the time based retention policy also applies. #5109 prometheus/tsdb#343
* [FEATURE] Add CORS origin flag. #5011
* [ENHANCEMENT] Consul SD: Add tagged address to the discovery metadata. #5001
* [ENHANCEMENT] Kubernetes SD: Add service external IP and external name to the discovery metadata. #4940
* [ENHANCEMENT] Azure SD: Add support for Managed Identity authentication. #4590
* [ENHANCEMENT] Azure SD: Add tenant and subscription IDs to the discovery metadata. #4969
* [ENHANCEMENT] OpenStack SD: Add support for application credentials based authentication. #4968
* [ENHANCEMENT] Add metric for number of rule groups loaded. #5090
* [BUGFIX] Avoid duplicate tests for alert unit tests. #4964
* [BUGFIX] Don't depend on given order when comparing samples in alert unit testing. #5049
* [BUGFIX] Make sure the retention period doesn't overflow. #5112
* [BUGFIX] Make sure the blocks don't get very large. #5112
* [BUGFIX] Don't generate blocks with no samples. prometheus/tsdb#374
* [BUGFIX] Reintroduce metric for WAL corruptions. prometheus/tsdb#473
600

Simon Pasquier's avatar
Simon Pasquier committed
601
602
## 2.6.1 / 2019-01-15

603
604
605
* [BUGFIX] Azure SD: Fix discovery getting stuck sometimes. #5088
* [BUGFIX] Marathon SD: Use `Tasks.Ports` when `RequirePorts` is `false`. #5026
* [BUGFIX] Promtool: Fix "out-of-order sample" errors when testing rules. #5069
Simon Pasquier's avatar
Simon Pasquier committed
606

Simon Pasquier's avatar
Simon Pasquier committed
607
## 2.6.0 / 2018-12-17
Simon Pasquier's avatar
Simon Pasquier committed
608

609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
* [CHANGE] Remove default flags from the container's entrypoint, run Prometheus from `/etc/prometheus` and symlink the storage directory to `/etc/prometheus/data`. #4976
* [CHANGE] Promtool: Remove the `update` command. #3839
* [FEATURE] Add JSON log format via the `--log.format` flag. #4876
* [FEATURE] API: Add /api/v1/labels endpoint to get all label names. #4835
* [FEATURE] Web: Allow setting the page's title via the `--web.ui-title` flag. #4841
* [ENHANCEMENT] Add `prometheus_tsdb_lowest_timestamp_seconds`, `prometheus_tsdb_head_min_time_seconds` and `prometheus_tsdb_head_max_time_seconds` metrics. #4888
* [ENHANCEMENT] Add `rule_group_last_evaluation_timestamp_seconds` metric. #4852
* [ENHANCEMENT] Add `prometheus_template_text_expansion_failures_total` and `prometheus_template_text_expansions_total` metrics. #4747
* [ENHANCEMENT] Set consistent User-Agent header in outgoing requests. #4891
* [ENHANCEMENT] Azure SD: Error out at load time when authentication parameters are missing. #4907
* [ENHANCEMENT] EC2 SD: Add the machine's private DNS name to the discovery metadata. #4693
* [ENHANCEMENT] EC2 SD: Add the operating system's platform to the discovery metadata. #4663
* [ENHANCEMENT] Kubernetes SD: Add the pod's phase to the discovery metadata. #4824
* [ENHANCEMENT] Kubernetes SD: Log Kubernetes messages. #4931
* [ENHANCEMENT] Promtool: Collect CPU and trace profiles. #4897
* [ENHANCEMENT] Promtool: Support writing output as JSON. #4848
* [ENHANCEMENT] Remote Read: Return available data if remote read fails partially. #4832
* [ENHANCEMENT] Remote Write: Improve queue performance. #4772
* [ENHANCEMENT] Remote Write: Add min_shards parameter to set the minimum number of shards. #4924
* [ENHANCEMENT] TSDB: Improve WAL reading. #4953
* [ENHANCEMENT] TSDB: Memory improvements. #4953
* [ENHANCEMENT] Web: Log stack traces on panic. #4221
* [ENHANCEMENT] Web UI: Add copy to clipboard button for configuration. #4410
* [ENHANCEMENT] Web UI: Support console queries at specific times. #4764
* [ENHANCEMENT] Web UI: group targets by job then instance. #4898 #4806
* [BUGFIX] Deduplicate handler labels for HTTP metrics. #4732
* [BUGFIX] Fix leaked queriers causing shutdowns to hang. #4922
* [BUGFIX] Fix configuration loading panics on nil pointer slice elements. #4942
* [BUGFIX] API: Correctly skip mismatching targets on /api/v1/targets/metadata. #4905
* [BUGFIX] API: Better rounding for incoming query timestamps. #4941
* [BUGFIX] Azure SD: Fix panic. #4867
* [BUGFIX] Console templates: Fix hover when the metric has a null value. #4906
* [BUGFIX] Discovery: Remove all targets when the scrape configuration gets empty. #4819
* [BUGFIX] Marathon SD: Fix leaked connections. #4915
* [BUGFIX] Marathon SD: Use 'hostPort' member of portMapping to construct target endpoints. #4887
* [BUGFIX] PromQL: Fix a goroutine leak in the lexer/parser. #4858
* [BUGFIX] Scrape: Pass through content-type for non-compressed output. #4912
* [BUGFIX] Scrape: Fix deadlock in the scrape's manager. #4894
* [BUGFIX] Scrape: Scrape targets at fixed intervals even after Prometheus restarts. #4926
* [BUGFIX] TSDB: Support restored snapshots including the head properly. #4953
* [BUGFIX] TSDB: Repair WAL when the last record in a segment is torn. #4953
* [BUGFIX] TSDB: Fix unclosed file readers on Windows systems. #4997
* [BUGFIX] Web: Avoid proxy to connect to the local gRPC server. #4572
Simon Pasquier's avatar
Simon Pasquier committed
652

Frederic Branczyk's avatar
Frederic Branczyk committed
653
## 2.5.0 / 2018-11-06
Frederic Branczyk's avatar
Frederic Branczyk committed
654

655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
* [CHANGE] Group targets by scrape config instead of job name. #4806 #4526
* [CHANGE] Marathon SD: Various changes to adapt to Marathon 1.5+. #4499
* [CHANGE] Discovery: Split `prometheus_sd_discovered_targets` metric by scrape and notify (Alertmanager SD) as well as by section in the respective configuration. #4753
* [FEATURE] Add OpenMetrics support for scraping (EXPERIMENTAL). #4700
* [FEATURE] Add unit testing for rules. #4350
* [FEATURE] Make maximum number of samples per query configurable via `--query.max-samples` flag. #4513
* [FEATURE] Make maximum number of concurrent remote reads configurable via `--storage.remote.read-concurrent-limit` flag. #4656
* [ENHANCEMENT] Support s390x platform for Linux. #4605
* [ENHANCEMENT] API: Add `prometheus_api_remote_read_queries` metric tracking currently executed or waiting remote read API requests. #4699
* [ENHANCEMENT] Remote Read: Add `prometheus_remote_storage_remote_read_queries` metric tracking currently in-flight remote read queries. #4677
* [ENHANCEMENT] Remote Read: Reduced memory usage. #4655
* [ENHANCEMENT] Discovery: Add `prometheus_sd_discovered_targets`, `prometheus_sd_received_updates_total`, `prometheus_sd_updates_delayed_total`, and `prometheus_sd_updates_total` metrics for discovery subsystem. #4667
* [ENHANCEMENT] Discovery: Improve performance of previously slow updates of changes of targets. #4526
* [ENHANCEMENT] Kubernetes SD: Add extended metrics. #4458
* [ENHANCEMENT] OpenStack SD: Support discovering instances from all projects. #4682
* [ENHANCEMENT] OpenStack SD: Discover all interfaces. #4649
* [ENHANCEMENT] OpenStack SD: Support `tls_config` for the used HTTP client. #4654
* [ENHANCEMENT] Triton SD: Add ability to filter triton_sd targets by pre-defined groups. #4701
* [ENHANCEMENT] Web UI: Avoid browser spell-checking in expression field. #4728
* [ENHANCEMENT] Web UI: Add scrape duration and last evaluation time in targets and rules pages. #4722
* [ENHANCEMENT] Web UI: Improve rule view by wrapping lines. #4702
* [ENHANCEMENT] Rules: Error out at load time for invalid templates, rather than at evaluation time. #4537
* [ENHANCEMENT] TSDB: Add metrics for WAL operations. #4692
* [BUGFIX] Change max/min over_time to handle NaNs properly. #4386
* [BUGFIX] Check label name for `count_values` PromQL function. #4585
* [BUGFIX] Ensure that vectors and matrices do not contain identical label-sets. #4589
681

682
683
## 2.4.3 / 2018-10-04

684
685
686
* [BUGFIX] Fix panic when using custom EC2 API for SD #4672
* [BUGFIX] Fix panic when Zookeeper SD cannot connect to servers #4669
* [BUGFIX] Make the skip_head an optional parameter for snapshot API #4674
687

Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
688
689
690
691
## 2.4.2 / 2018-09-21

 The last release didn't have bugfix included due to a vendoring error.

692
693
 * [BUGFIX] Handle WAL corruptions properly prometheus/tsdb#389
 * [BUGFIX] Handle WAL migrations correctly on Windows prometheus/tsdb#392
Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
694

Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
695
696
## 2.4.1 / 2018-09-19

697
698
* [ENHANCEMENT] New TSDB metrics prometheus/tsdb#375 prometheus/tsdb#363
* [BUGFIX] Render UI correctly for Windows #4616
Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
699

Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
700
## 2.4.0 / 2018-09-11
701

702
This release includes multiple bugfixes and features. Further, the WAL implementation has been re-written so the storage is not forward compatible. Prometheus 2.3 storage will work on 2.4 but not vice-versa.
703

704
705
706
707
708
* [CHANGE] Reduce remote write default retries #4279
* [CHANGE] Remove /heap endpoint #4460
* [FEATURE] Persist alert 'for' state across restarts #4061
* [FEATURE] Add API providing per target metric metadata #4183
* [FEATURE] Add API providing recording and alerting rules #4318 #4501
709
* [ENHANCEMENT] Brand new WAL implementation for TSDB. Forwards incompatible with previous WAL.
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
* [ENHANCEMENT] Show rule evaluation errors in UI #4457
* [ENHANCEMENT] Throttle resends of alerts to Alertmanager #4538
* [ENHANCEMENT] Send EndsAt along with the alert to Alertmanager #4550
* [ENHANCEMENT] Limit the samples returned by remote read endpoint #4532
* [ENHANCEMENT] Limit the data read in through remote read #4239
* [ENHANCEMENT] Coalesce identical SD configurations #3912
* [ENHANCEMENT] `promtool`: Add new commands for debugging and querying #4247 #4308 #4346 #4454
* [ENHANCEMENT] Update console examples for node_exporter v0.16.0 #4208
* [ENHANCEMENT] Optimize PromQL aggregations #4248
* [ENHANCEMENT] Remote read: Add Offset to hints #4226
* [ENHANCEMENT] `consul_sd`: Add support for ServiceMeta field #4280
* [ENHANCEMENT] `ec2_sd`: Maintain order of subnet_id label #4405
* [ENHANCEMENT] `ec2_sd`: Add support for custom endpoint to support EC2 compliant APIs #4333
* [ENHANCEMENT] `ec2_sd`: Add instance_owner label #4514
* [ENHANCEMENT] `azure_sd`: Add support for VMSS discovery and multiple environments #4202 #4569
* [ENHANCEMENT] `gce_sd`: Add instance_id label #4488
* [ENHANCEMENT] Forbid rule-abiding robots from indexing #4266
* [ENHANCEMENT] Log virtual memory limits on startup #4418
* [BUGFIX] Wait for service discovery to stop before exiting #4508
* [BUGFIX] Render SD configs properly #4338
* [BUGFIX] Only add LookbackDelta to vector selectors #4399
* [BUGFIX] `ec2_sd`: Handle panic-ing nil pointer #4469
* [BUGFIX] `consul_sd`: Stop leaking connections #4443
* [BUGFIX] Use templated labels also to identify alerts #4500
* [BUGFIX] Reduce floating point errors in stddev and related functions #4533
* [BUGFIX] Log errors while encoding responses #4359
736

Brian Brazil's avatar
Brian Brazil committed
737
738
## 2.3.2 / 2018-07-12

739
740
741
742
743
744
745
* [BUGFIX] Fix various tsdb bugs #4369
* [BUGFIX] Reorder startup and shutdown to prevent panics. #4321
* [BUGFIX] Exit with non-zero code on error #4296
* [BUGFIX] discovery/kubernetes/ingress: fix scheme discovery #4329
* [BUGFIX] Fix race in zookeeper sd #4355
* [BUGFIX] Better timeout handling in promql #4291 #4300
* [BUGFIX] Propagate errors when selecting series from the tsdb #4136
Brian Brazil's avatar
Brian Brazil committed
746

Brian Brazil's avatar
Brian Brazil committed
747
748
## 2.3.1 / 2018-06-19

749
750
751
752
753
754
755
* [BUGFIX] Avoid infinite loop on duplicate NaN values. #4275
* [BUGFIX] Fix nil pointer deference when using various API endpoints #4282
* [BUGFIX] config: set target group source index during unmarshaling #4245
* [BUGFIX] discovery/file: fix logging #4178
* [BUGFIX] kubernetes_sd: fix namespace filtering #4285
* [BUGFIX] web: restore old path prefix behavior #4273
* [BUGFIX] web: remove security headers added in 2.3.0 #4259
Brian Brazil's avatar
Brian Brazil committed
756

Fabian Reinartz's avatar
Fabian Reinartz committed
757
## 2.3.0 / 2018-06-05
758

Fabian Reinartz's avatar
Fabian Reinartz committed
759
760
* [CHANGE] `marathon_sd`: use `auth_token` and `auth_token_file` for token-based authentication instead of `bearer_token` and `bearer_token_file` respectively.
* [CHANGE] Metric names for HTTP server metrics changed
Fabian Reinartz's avatar
Fabian Reinartz committed
761
762
763
* [FEATURE] Add query commands to promtool
* [FEATURE] Add security headers to HTTP server responses
* [FEATURE] Pass query hints via remote read API
Fabian Reinartz's avatar
Fabian Reinartz committed
764
* [FEATURE] Basic auth passwords can now be configured via file across all configuration
Ben Kochie's avatar
Ben Kochie committed
765
* [ENHANCEMENT] Optimize PromQL and API serialization for memory usage and allocations
Fabian Reinartz's avatar
Fabian Reinartz committed
766
* [ENHANCEMENT] Limit number of dropped targets in web UI
Fabian Reinartz's avatar
Fabian Reinartz committed
767
768
* [ENHANCEMENT] Consul and EC2 service discovery allow using server-side filtering for performance improvement
* [ENHANCEMENT] Add advanced filtering configuration to EC2 service discovery
769
* [ENHANCEMENT] `marathon_sd`: adds support for basic and bearer authentication, plus all other common HTTP client options (TLS config, proxy URL, etc.)
Fabian Reinartz's avatar
Fabian Reinartz committed
770
771
772
773
774
775
776
777
778
779
780
* [ENHANCEMENT] Provide machine type metadata and labels in GCE service discovery
* [ENHANCEMENT] Add pod controller kind and name to Kubernetes service discovery data
* [ENHANCEMENT] Move TSDB to flock-based log file that works with Docker containers
* [BUGFIX] Properly propagate storage errors in PromQL
* [BUGFIX] Fix path prefix for web pages
* [BUGFIX] Fix goroutine leak in Consul service discovery
* [BUGFIX] Fix races in scrape manager
* [BUGFIX] Fix OOM for very large k in PromQL topk() queries
* [BUGFIX] Make remote write more resilient to unavailable receivers
* [BUGFIX] Make remote write shutdown cleanly
* [BUGFIX] Don't leak files on errors in TSDB's tombstone cleanup
Fabian Reinartz's avatar
Fabian Reinartz committed
781
782
* [BUGFIX] Unary minus expressions now removes the metric name from results
* [BUGFIX] Fix bug that lead to wrong amount of samples considered for time range expressions
783

Fabian Reinartz's avatar
Fabian Reinartz committed
784
785
786
787
## 2.2.1 / 2018-03-13

* [BUGFIX] Fix data loss in TSDB on compaction
* [BUGFIX] Correctly stop timer in remote-write path
Ben Kochie's avatar
Ben Kochie committed
788
* [BUGFIX] Fix deadlock triggered by loading targets page
Fabian Reinartz's avatar
Fabian Reinartz committed
789
790
791
* [BUGFIX] Fix incorrect buffering of samples on range selection queries
* [BUGFIX] Handle large index files on windows properly

Fabian Reinartz's avatar
Fabian Reinartz committed
792
## 2.2.0 / 2018-03-08
793
794

* [CHANGE] Rename file SD mtime metric.
Fabian Reinartz's avatar
Fabian Reinartz committed
795
796
* [CHANGE] Send target update on empty pod IP in Kubernetes SD.
* [FEATURE] Add API endpoint for flags.
Fabian Reinartz's avatar
Fabian Reinartz committed
797
798
799
* [FEATURE] Add API endpoint for dropped targets.
* [FEATURE] Display annotations on alerts page.
* [FEATURE] Add option to skip head data when taking snapshots.
800
801
802
803
804
* [ENHANCEMENT] Federation performance improvement.
* [ENHANCEMENT] Read bearer token file on every scrape.
* [ENHANCEMENT] Improve typeahead on `/graph` page.
* [ENHANCEMENT] Change rule file formatting.
* [ENHANCEMENT] Set consul server default to `localhost:8500`.
Fabian Reinartz's avatar
Fabian Reinartz committed
805
806
* [ENHANCEMENT] Add dropped Alertmanagers to API info endpoint.
* [ENHANCEMENT] Add OS type meta label to Azure SD.
Fabian Reinartz's avatar
Fabian Reinartz committed
807
* [ENHANCEMENT] Validate required fields in SD configuration.
Fabian Reinartz's avatar
Fabian Reinartz committed
808
809
810
* [BUGFIX] Prevent stack overflow on deep recursion in TSDB.
* [BUGFIX] Correctly read offsets in index files that are greater than 4GB.
* [BUGFIX] Fix scraping behavior for empty labels.
811
812
813
814
815
816
817
818
819
820
* [BUGFIX] Drop metric name for bool modifier.
* [BUGFIX] Fix races in discovery.
* [BUGFIX] Fix Kubernetes endpoints SD for empty subsets.
* [BUGFIX] Throttle updates from SD providers, which caused increased CPU usage and allocations.
* [BUGFIX] Fix TSDB block reload issue.
* [BUGFIX] Fix PromQL printing of empty `without()`.
* [BUGFIX] Don't reset FiredAt for inactive alerts.
* [BUGFIX] Fix erroneous file version changes and repair existing data.

## 2.1.0 / 2018-01-19
Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
821
822
823

* [FEATURE] New Service Discovery UI showing labels before and after relabelling.
* [FEATURE] New Admin APIs added to v1 to delete, snapshot and remove tombstones.
Josh Soref's avatar
Josh Soref committed
824
* [ENHANCEMENT] The graph UI autocomplete now includes your previous queries.
Goutham Veeramachaneni's avatar
Goutham Veeramachaneni committed
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
* [ENHANCEMENT] Federation is now much faster for large numbers of series.
* [ENHANCEMENT] Added new metrics to measure rule timings.
* [ENHANCEMENT] Rule evaluation times added to the rules UI.
* [ENHANCEMENT] Added metrics to measure modified time of file SD files.
* [ENHANCEMENT] Kubernetes SD now includes POD UID in discovery metadata.
* [ENHANCEMENT] The Query APIs now return optional stats on query execution times.
* [ENHANCEMENT] The index now no longer has the 4GiB size limit and is also smaller.
* [BUGFIX] Remote read `read_recent` option is now false by default.
* [BUGFIX] Pass the right configuration to each Alertmanager (AM) when using multiple AM configs.
* [BUGFIX] Fix not-matchers not selecting series with labels unset.
* [BUGFIX] tsdb: Fix occasional panic in head block.
* [BUGFIX] tsdb: Close files before deletion to fix retention issues on Windows and NFS.
* [BUGFIX] tsdb: Cleanup and do not retry failing compactions.
* [BUGFIX] tsdb: Close WAL while shutting down.


Brian Brazil's avatar
Brian Brazil committed
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
## 2.0.0 / 2017-11-08

This release includes a completely rewritten storage, huge performance
improvements, but also many backwards incompatible changes. For more
information, read the announcement blog post and migration guide.

https://prometheus.io/blog/2017/11/08/announcing-prometheus-2-0/
https://prometheus.io/docs/prometheus/2.0/migration/

* [CHANGE] Completely rewritten storage layer, with WAL. This is not backwards compatible with 1.x storage, and many flags have changed/disappeared.
* [CHANGE] New staleness behavior. Series now marked stale after target scrapes no longer return them, and soon after targets disappear from service discovery.
* [CHANGE] Rules files use YAML syntax now. Conversion tool added to promtool.
* [CHANGE] Removed `count_scalar`, `drop_common_labels` functions and `keep_common` modifier from PromQL.
* [CHANGE] Rewritten exposition format parser with much higher performance. The Protobuf exposition format is no longer supported.
* [CHANGE] Example console templates updated for new storage and metrics names. Examples other than node exporter and Prometheus removed.
856
* [CHANGE] Admin and lifecycle APIs now disabled by default, can be re-enabled via flags
Brian Brazil's avatar
Brian Brazil committed
857
858
859
860
861
862
863
864
865
* [CHANGE] Flags switched to using Kingpin, all flags are now --flagname rather than -flagname.
* [FEATURE/CHANGE] Remote read can be configured to not read data which is available locally. This is enabled by default.
* [FEATURE] Rules can be grouped now. Rules within a rule group are executed sequentially.
* [FEATURE] Added experimental GRPC apis
* [FEATURE] Add timestamp() function to PromQL.
* [ENHANCEMENT] Remove remote read from the query path if no remote storage is configured.
* [ENHANCEMENT] Bump Consul HTTP client timeout to not match the Consul SD watch timeout.
* [ENHANCEMENT] Go-conntrack added to provide HTTP connection metrics.
* [BUGFIX] Fix connection leak in Consul SD.
Fabian Reinartz's avatar
Fabian Reinartz committed
866

beorn7's avatar
beorn7 committed
867
868
869
## 1.8.2 / 2017-11-04

* [BUGFIX] EC2 service discovery: Do not crash if tags are empty.
Fabian Reinartz's avatar
Fabian Reinartz committed
870

Brian Brazil's avatar
Brian Brazil committed
871
872
## 1.8.1 / 2017-10-19

873
* [BUGFIX] Correctly handle external labels on remote read endpoint
874

beorn7's avatar
beorn7 committed
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
## 1.8.0 / 2017-10-06

* [CHANGE] Rule links link to the _Console_ tab rather than the _Graph_ tab to
  not trigger expensive range queries by default.
* [FEATURE] Ability to act as a remote read endpoint for other Prometheus
  servers.
* [FEATURE] K8s SD: Support discovery of ingresses.
* [FEATURE] Consul SD: Support for node metadata.
* [FEATURE] Openstack SD: Support discovery of hypervisors.
* [FEATURE] Expose current Prometheus config via `/status/config`.
* [FEATURE] Allow to collapse jobs on `/targets` page.
* [FEATURE] Add `/-/healthy` and `/-/ready` endpoints.
* [FEATURE] Add color scheme support to console templates.
* [ENHANCEMENT] Remote storage connections use HTTP keep-alive.
* [ENHANCEMENT] Improved logging about remote storage.
* [ENHANCEMENT] Relaxed URL validation.
* [ENHANCEMENT] Openstack SD: Handle instances without IP.
* [ENHANCEMENT] Make remote storage queue manager configurable.
* [ENHANCEMENT] Validate metrics returned from remote read.
* [ENHANCEMENT] EC2 SD: Set a default region.
* [ENHANCEMENT] Changed help link to `https://prometheus.io/docs`.
* [BUGFIX] Fix floating-point precision issue in `deriv` function.
* [BUGFIX] Fix pprof endpoints when -web.route-prefix or -web.external-url is
  used.
* [BUGFIX] Fix handling of `null` target groups in file-based SD.
* [BUGFIX] Set the sample timestamp in date-related PromQL functions.
* [BUGFIX] Apply path prefix to redirect from deprecated graph URL.
* [BUGFIX] Fixed tests on MS Windows.
* [BUGFIX] Check for invalid UTF-8 in label values after relabeling.

beorn7's avatar
beorn7 committed
905
906
907
908
909
910
911
912
913
914
915
916
917
## 1.7.2 / 2017-09-26

* [BUGFIX] Correctly remove all targets from DNS service discovery if the
  corresponding DNS query succeeds and returns an empty result.
* [BUGFIX] Correctly parse resolution input in expression browser.
* [BUGFIX] Consistently use UTC in the date picker of the expression browser.
* [BUGFIX] Correctly handle multiple ports in Marathon service discovery.
* [BUGFIX] Fix HTML escaping so that HTML templates compile with Go1.9.
* [BUGFIX] Prevent number of remote write shards from going negative.
* [BUGFIX] In the graphs created by the expression browser, render very large
  and small numbers in a readable way.
* [BUGFIX] Fix a rarely occurring iterator issue in varbit encoded chunks.

Frederic Branczyk's avatar
Frederic Branczyk committed
918
919
920
921
## 1.7.1 / 2017-06-12

* [BUGFIX] Fix double prefix redirect.

Frederic Branczyk's avatar
Frederic Branczyk committed
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
## 1.7.0 / 2017-06-06

* [CHANGE] Compress remote storage requests and responses with unframed/raw snappy.
* [CHANGE] Properly ellide secrets in config.
* [FEATURE] Add OpenStack service discovery.
* [FEATURE] Add ability to limit Kubernetes service discovery to certain namespaces.
* [FEATURE] Add metric for discovered number of Alertmanagers.
* [ENHANCEMENT] Print system information (uname) on start up.
* [ENHANCEMENT] Show gaps in graphs on expression browser.
* [ENHANCEMENT] Promtool linter checks counter naming and more reserved labels.
* [BUGFIX] Fix broken Mesos discovery.
* [BUGFIX] Fix redirect when external URL is set.
* [BUGFIX] Fix mutation of active alert elements by notifier.
* [BUGFIX] Fix HTTP error handling for remote write.
* [BUGFIX] Fix builds for Solaris/Illumos.
* [BUGFIX] Fix overflow checking in global config.
* [BUGFIX] Fix log level reporting issue.
* [BUGFIX] Fix ZooKeeper serverset discovery can become out-of-sync.

941
## 1.6.3 / 2017-05-18
Frederic Branczyk's avatar
Frederic Branczyk committed
942

Josh Soref's avatar
Josh Soref committed
943
* [BUGFIX] Fix disappearing Alertmanager targets in Alertmanager discovery.
Frederic Branczyk's avatar
Frederic Branczyk committed
944
945
946
* [BUGFIX] Fix panic with remote_write on ARMv7.
* [BUGFIX] Fix stacked graphs to adapt min/max values.

947
## 1.6.2 / 2017-05-11
Fabian Reinartz's avatar
Fabian Reinartz committed
948
949
950

* [BUGFIX] Fix potential memory leak in Kubernetes service discovery

951
## 1.6.1 / 2017-04-19
Brian Brazil's avatar
Brian Brazil committed
952
953
954

* [BUGFIX] Don't panic if storage has no FPs even after initial wait

955
## 1.6.0 / 2017-04-14
beorn7's avatar
beorn7 committed
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019

* [CHANGE] Replaced the remote write implementations for various backends by a
  generic write interface with example adapter implementation for various
  backends. Note that both the previous and the current remote write
  implementations are **experimental**.
* [FEATURE] New flag `-storage.local.target-heap-size` to tell Prometheus about
  the desired heap size. This deprecates the flags
  `-storage.local.memory-chunks` and `-storage.local.max-chunks-to-persist`,
  which are kept for backward compatibility.
* [FEATURE] Add `check-metrics` to `promtool` to lint metric names.
* [FEATURE] Add Joyent Triton discovery.
* [FEATURE] `X-Prometheus-Scrape-Timeout-Seconds` header in HTTP scrape
  requests.
* [FEATURE] Remote read interface, including example for InfluxDB. **Experimental.**
* [FEATURE] Enable Consul SD to connect via TLS.
* [FEATURE] Marathon SD supports multiple ports.
* [FEATURE] Marathon SD supports bearer token for authentication.
* [FEATURE] Custom timeout for queries.
* [FEATURE] Expose `buildQueryUrl` in `graph.js`.
* [FEATURE] Add `rickshawGraph` property to the graph object in console
  templates.
* [FEATURE] New metrics exported by Prometheus itself:
  * Summary `prometheus_engine_query_duration_seconds`
  * Counter `prometheus_evaluator_iterations_missed_total`
  * Counter `prometheus_evaluator_iterations_total`
  * Gauge `prometheus_local_storage_open_head_chunks`
  * Gauge `prometheus_local_storage_target_heap_size`
* [ENHANCEMENT] Reduce shut-down time by interrupting an ongoing checkpoint
  before starting the final checkpoint.
* [ENHANCEMENT] Auto-tweak times between checkpoints to limit time spent in
  checkpointing to 50%.
* [ENHANCEMENT] Improved crash recovery deals better with certain index
  corruptions.
* [ENHANCEMENT] Graphing deals better with constant time series.
* [ENHANCEMENT] Retry remote writes on recoverable errors.
* [ENHANCEMENT] Evict unused chunk descriptors during crash recovery to limit
  memory usage.
* [ENHANCEMENT] Smoother disk usage during series maintenance.
* [ENHANCEMENT] Targets on targets page sorted by instance within a job.
* [ENHANCEMENT] Sort labels in federation.
* [ENHANCEMENT] Set `GOGC=40` by default, which results in much better memory
  utilization at the price of slightly higher CPU usage. If `GOGC` is set by
  the user, it is still honored as usual.
* [ENHANCEMENT] Close head chunks after being idle for the duration of the
  configured staleness delta. This helps to persist and evict head chunk of
  stale series more quickly.
* [ENHANCEMENT] Stricter checking of relabel config.
* [ENHANCEMENT] Cache busters for static web content.
* [ENHANCEMENT] Send Prometheus-specific user-agent header during scrapes.
* [ENHANCEMENT] Improved performance of series retention cut-off.
* [ENHANCEMENT] Mitigate impact of non-atomic sample ingestion on
  `histogram_quantile` by enforcing buckets to be monotonic.
* [ENHANCEMENT] Released binaries built with Go 1.8.1.
* [BUGFIX] Send `instance=""` with federation if `instance` not set.
* [BUGFIX] Update to new `client_golang` to get rid of unwanted quantile
  metrics in summaries.
* [BUGFIX] Introduce several additional guards against data corruption.
* [BUGFIX] Mark storage dirty and increment
  `prometheus_local_storage_persist_errors_total` on all relevant errors.
* [BUGFIX] Propagate storage errors as 500 in the HTTP API.
* [BUGFIX] Fix int64 overflow in timestamps in the HTTP API.
* [BUGFIX] Fix deadlock in Zookeeper SD.
* [BUGFIX] Fix fuzzy search problems in the web-UI auto-completion.

1020
## 1.5.3 / 2017-05-11
Fabian Reinartz's avatar
Fabian Reinartz committed
1021
1022
1023

* [BUGFIX] Fix potential memory leak in Kubernetes service discovery

1024
## 1.5.2 / 2017-02-10
beorn7's avatar
beorn7 committed
1025
1026
1027
1028
1029
1030
1031

* [BUGFIX] Fix series corruption in a special case of series maintenance where
  the minimum series-file-shrink-ratio kicks in.
* [BUGFIX] Fix two panic conditions both related to processing a series
  scheduled to be quarantined.
* [ENHANCEMENT] Binaries built with Go1.7.5.

1032
## 1.5.1 / 2017-02-07
beorn7's avatar
beorn7 committed
1033
1034
1035
1036
1037
1038

* [BUGFIX] Don't lose fully persisted memory series during checkpointing.
* [BUGFIX] Fix intermittently failing relabeling.
* [BUGFIX] Make `-storage.local.series-file-shrink-ratio` work.
* [BUGFIX] Remove race condition from TestLoop.

1039
## 1.5.0 / 2017-01-23
Frederic Branczyk's avatar
Frederic Branczyk committed
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055

* [CHANGE] Use lexicographic order to sort alerts by name.
* [FEATURE] Add Joyent Triton discovery.
* [FEATURE] Add scrape targets and alertmanager targets API.
* [FEATURE] Add various persistence related metrics.
* [FEATURE] Add various query engine related metrics.
* [FEATURE] Add ability to limit scrape samples, and related metrics.
* [FEATURE] Add labeldrop and labelkeep relabelling actions.
* [FEATURE] Display current working directory on status-page.
* [ENHANCEMENT] Strictly use ServiceAccount for in cluster configuration on Kubernetes.
* [ENHANCEMENT] Various performance and memory-management improvements.
* [BUGFIX] Fix basic auth for alertmanagers configured via flag.
* [BUGFIX] Don't panic on decoding corrupt data.
* [BUGFIX] Ignore dotfiles in data directory.
* [BUGFIX] Abort on intermediate federation errors.

1056
## 1.4.1 / 2016-11-28
Fabian Reinartz's avatar
Fabian Reinartz committed
1057
1058
1059

* [BUGFIX] Fix Consul service discovery

1060
## 1.4.0 / 2016-11-25
Fabian Reinartz's avatar
Fabian Reinartz committed
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072

* [FEATURE] Allow configuring Alertmanagers via service discovery
* [FEATURE] Display used Alertmanagers on runtime page in the UI
* [FEATURE] Support profiles in AWS EC2 service discovery configuration
* [ENHANCEMENT] Remove duplicated logging of Kubernetes client errors
* [ENHANCEMENT] Add metrics about Kubernetes service discovery
* [BUGFIX] Update alert annotations on re-evaluation
* [BUGFIX] Fix export of group modifier in PromQL queries
* [BUGFIX] Remove potential deadlocks in several service discovery implementations
* [BUGFIX] Use proper float64 modulo in PromQL `%` binary operations
* [BUGFIX] Fix crash bug in Kubernetes service discovery

1073
## 1.3.1 / 2016-11-04
beorn7's avatar
beorn7 committed
1074

1075
This bug-fix release pulls in the fixes from the 1.2.3 release.
beorn7's avatar
beorn7 committed
1076
1077
1078
1079
1080
1081

* [BUGFIX] Correctly handle empty Regex entry in relabel config.
* [BUGFIX] MOD (`%`) operator doesn't panic with small floating point numbers.
* [BUGFIX] Updated miekg/dns vendoring to pick up upstream bug fixes.
* [ENHANCEMENT] Improved DNS error reporting.

1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
## 1.2.3 / 2016-11-04

Note that this release is chronologically after 1.3.0.

* [BUGFIX] Correctly handle end time before start time in range queries.
* [BUGFIX] Error on negative `-storage.staleness-delta`
* [BUGFIX] Correctly handle empty Regex entry in relabel config.
* [BUGFIX] MOD (`%`) operator doesn't panic with small floating point numbers.
* [BUGFIX] Updated miekg/dns vendoring to pick up upstream bug fixes.
* [ENHANCEMENT] Improved DNS error reporting.

## 1.3.0 / 2016-11-01
Frederic Branczyk's avatar
Frederic Branczyk committed
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103

This is a breaking change to the Kubernetes service discovery.

* [CHANGE] Rework Kubernetes SD.
* [FEATURE] Add support for interpolating `target_label`.
* [FEATURE] Add GCE metadata as Prometheus meta labels.
* [ENHANCEMENT] Add EC2 SD metrics.
* [ENHANCEMENT] Add Azure SD metrics.
* [ENHANCEMENT] Add fuzzy search to `/graph` textarea.
* [ENHANCEMENT] Always show instance labels on target page.
Fabian Reinartz's avatar
Fabian Reinartz committed
1104
1105
* [BUGFIX] Validate query end time is not before start time.
* [BUGFIX] Error on negative `-storage.staleness-delta`
beorn7's avatar
beorn7 committed
1106

1107
## 1.2.2 / 2016-10-30
beorn7's avatar
beorn7 committed
1108

Frederic Branczyk's avatar
Frederic Branczyk committed
1109
* [BUGFIX] Correctly handle on() in alerts.
beorn7's avatar
beorn7 committed
1110
1111
1112
1113
1114
* [BUGFIX] UI: Deal properly with aborted requests.
* [BUGFIX] UI: Decode URL query parameters properly.
* [BUGFIX] Storage: Deal better with data corruption (non-monotonic timestamps).
* [BUGFIX] Remote storage: Re-add accidentally removed timeout flag.
* [BUGFIX] Updated a number of vendored packages to pick up upstream bug fixes.
Frederic Branczyk's avatar
Frederic Branczyk committed
1115

1116
## 1.2.1 / 2016-10-10
beorn7's avatar
beorn7 committed
1117
1118

* [BUGFIX] Count chunk evictions properly so that the server doesn't
Josh Soref's avatar
Josh Soref committed
1119
  assume it runs out of memory and subsequently throttles ingestion.
beorn7's avatar
beorn7 committed
1120
1121
* [BUGFIX] Use Go1.7.1 for prebuilt binaries to fix issues on MacOS Sierra.

1122
## 1.2.0 / 2016-10-07
beorn7's avatar
beorn7 committed
1123

1124
* [FEATURE] Cleaner encoding of query parameters in `/graph` URLs.
beorn7's avatar
beorn7 committed
1125
1126
* [FEATURE] PromQL: Add `minute()` function.
* [FEATURE] Add GCE service discovery.
1127
* [FEATURE] Allow any valid UTF-8 string as job name.
beorn7's avatar
beorn7 committed
1128
1129
* [FEATURE] Allow disabling local storage.
* [FEATURE] EC2 service discovery: Expose `ec2_instance_state`.
1130
* [ENHANCEMENT] Various performance improvements in local storage.
beorn7's avatar
beorn7 committed
1131
1132
1133
* [BUGFIX] Zookeeper service discovery: Remove deleted nodes.
* [BUGFIX] Zookeeper service discovery: Resync state after Zookeeper failure.
* [BUGFIX] Remove JSON from HTTP Accept header.
1134
1135
* [BUGFIX] Fix flag validation of Alertmanager URL.
* [BUGFIX] Fix race condition on shutdown.
beorn7's avatar
beorn7 committed
1136
1137
1138
* [BUGFIX] Do not fail Consul discovery on Prometheus startup when Consul
  is down.
* [BUGFIX] Handle NaN in `changes()` correctly.
1139
1140
1141
1142
1143
* [CHANGE] **Experimental** remote write path: Remove use of gRPC.
* [CHANGE] **Experimental** remote write path: Configuration via config file
  rather than command line flags.
* [FEATURE] **Experimental** remote write path: Add HTTP basic auth and TLS.
* [FEATURE] **Experimental** remote write path: Support for relabelling.
beorn7's avatar
beorn7 committed
1144

1145
## 1.1.3 / 2016-09-16
beorn7's avatar
beorn7 committed
1146
1147
1148
1149
1150

* [ENHANCEMENT] Use golang-builder base image for tests in CircleCI.
* [ENHANCEMENT] Added unit tests for federation.
* [BUGFIX] Correctly de-dup metric families in federation output.

1151
## 1.1.2 / 2016-09-08
beorn7's avatar
beorn7 committed
1152
1153
1154

* [BUGFIX] Allow label names that coincide with keywords.

1155
## 1.1.1 / 2016-09-07
Fabian Reinartz's avatar
Fabian Reinartz committed
1156
1157
1158
1159

* [BUGFIX] Fix IPv6 escaping in service discovery integrations
* [BUGFIX] Fix default scrape port assignment for IPv6

1160
## 1.1.0 / 2016-09-03
beorn7's avatar
beorn7 committed
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180

* [FEATURE] Add `quantile` and `quantile_over_time`.
* [FEATURE] Add `stddev_over_time` and `stdvar_over_time`.
* [FEATURE] Add various time and date functions.
* [FEATURE] Added `toUpper` and `toLower` formatting to templates.
* [FEATURE] Allow relabeling of alerts.
* [FEATURE] Allow URLs in targets defined via a JSON file.
* [FEATURE] Add idelta function.
* [FEATURE] 'Remove graph' button on the /graph page.
* [FEATURE] Kubernetes SD: Add node name and host IP to pod discovery.
* [FEATURE] New remote storage write path. EXPERIMENTAL!
* [ENHANCEMENT] Improve time-series index lookups.
* [ENHANCEMENT] Forbid invalid relabel configurations.
* [ENHANCEMENT] Improved various tests.
* [ENHANCEMENT] Add crash recovery metric 'started_dirty'.
* [ENHANCEMENT] Fix (and simplify) populating series iterators.
* [ENHANCEMENT] Add job link on target page.
* [ENHANCEMENT] Message on empty Alerts page.
* [ENHANCEMENT] Various internal code refactorings and clean-ups.
* [ENHANCEMENT] Various improvements in the build system.
Ben Kochie's avatar
Ben Kochie committed
1181
* [BUGFIX] Catch errors when unmarshaling delta/doubleDelta encoded chunks.
beorn7's avatar
beorn7 committed
1182
1183
1184
1185
1186
1187
1188
1189
* [BUGFIX] Fix data race in lexer and lexer test.
* [BUGFIX] Trim stray whitespace from bearer token file.
* [BUGFIX] Avoid divide-by-zero panic on query_range?step=0.
* [BUGFIX] Detect invalid rule files at startup.
* [BUGFIX] Fix counter reset treatment in PromQL.
* [BUGFIX] Fix rule HTML escaping issues.
* [BUGFIX] Remove internal labels from alerts sent to AM.

1190
## 1.0.2 / 2016-08-24
beorn7's avatar
beorn7 committed
1191
1192
1193

* [BUGFIX] Clean up old targets after config reload.

1194
## 1.0.1 / 2016-07-21
beorn7's avatar
beorn7 committed
1195
1196
1197
1198
1199

* [BUGFIX] Exit with error on non-flag command-line arguments.
* [BUGFIX] Update example console templates to new HTTP API.
* [BUGFIX] Re-add logging flags.

1200
## 1.0.0 / 2016-07-18
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218

* [CHANGE] Remove deprecated query language keywords
* [CHANGE] Change Kubernetes SD to require specifying Kubernetes role
* [CHANGE] Use service address in Consul SD if available
* [CHANGE] Standardize all Prometheus internal metrics to second units
* [CHANGE] Remove unversioned legacy HTTP API
* [CHANGE] Remove legacy ingestion of JSON metric format
* [CHANGE] Remove deprecated `target_groups` configuration
* [FEATURE] Add binary power operation to PromQL
* [FEATURE] Add `count_values` aggregator
* [FEATURE] Add `-web.route-prefix` flag
* [FEATURE] Allow `on()`, `by()`, `without()` in PromQL with empty label sets
* [ENHANCEMENT] Make `topk/bottomk` query functions aggregators
* [BUGFIX] Fix annotations in alert rule printing
* [BUGFIX] Expand alert templating at evaluation time
* [BUGFIX] Fix edge case handling in crash recovery
* [BUGFIX] Hide testing package flags from help output

1219
## 0.20.0 / 2016-06-15
Fabian Reinartz's avatar
Fabian Reinartz committed
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236

This release contains multiple breaking changes to the configuration schema.

* [FEATURE] Allow configuring multiple Alertmanagers
* [FEATURE] Add server name to TLS configuration
* [FEATURE] Add labels for all node addresses and discover node port if available in Kubernetes SD
* [ENHANCEMENT] More meaningful configuration errors
* [ENHANCEMENT] Round scraping timestamps to milliseconds in web UI
* [ENHANCEMENT] Make number of storage fingerprint locks configurable
* [BUGFIX] Fix date parsing in console template graphs
* [BUGFIX] Fix static console files in Docker images
* [BUGFIX] Fix console JS XHR requests for IE11
* [BUGFIX] Add missing path prefix in new status page
* [CHANGE] Rename `target_groups` to `static_configs` in config files
* [CHANGE] Rename `names` to `files` in file SD configuration
* [CHANGE] Remove kubelet port config option in Kubernetes SD configuration

1237
## 0.19.3 / 2016-06-14
Fabian Reinartz's avatar
Fabian Reinartz committed
1238
1239
1240
1241

* [BUGFIX] Handle Marathon apps with zero ports
* [BUGFIX] Fix startup panic in retrieval layer

1242
## 0.19.2 / 2016-05-29
beorn7's avatar
beorn7 committed
1243
1244
1245
1246
1247

* [BUGFIX] Correctly handle `GROUP_LEFT` and `GROUP_RIGHT` without labels in
  string representation of expressions and in rules.
* [BUGFIX] Use `-web.external-url` for new status endpoints.

1248
## 0.19.1 / 2016-05-25
Fabian Reinartz's avatar
Fabian Reinartz committed
1249
1250
1251
1252

* [BUGFIX] Handle service discovery panic affecting Kubernetes SD
* [BUGFIX] Fix web UI display issue in some browsers

1253
## 0.19.0 / 2016-05-24
Fabian Reinartz's avatar
Fabian Reinartz committed
1254
1255
1256
1257

This version contains a breaking change to the query language. Please read
the documentation on the grouping behavior of vector matching:

1258
https://prometheus.io/docs/querying/operators/#vector-matching
Fabian Reinartz's avatar
Fabian Reinartz committed
1259
1260
1261
1262
1263
1264

* [FEATURE] Add experimental Microsoft Azure service discovery
* [FEATURE] Add `ignoring` modifier for binary operations
* [FEATURE] Add pod discovery to Kubernetes service discovery
* [CHANGE] Vector matching takes grouping labels from one-side
* [ENHANCEMENT] Support time range on /api/v1/series endpoint
Fabian Reinartz's avatar
Fabian Reinartz committed
1265
* [ENHANCEMENT] Partition status page into individual pages
Fabian Reinartz's avatar
Fabian Reinartz committed
1266
1267
* [BUGFIX] Fix issue of hanging target scrapes

1268
## 0.18.0 / 2016-04-18
Fabian Reinartz's avatar
Fabian Reinartz committed
1269
1270

* [BUGFIX] Fix operator precedence in PromQL
Fabian Reinartz's avatar
Fabian Reinartz committed
1271
1272
* [BUGFIX] Never drop still open head chunk
* [BUGFIX] Fix missing 'keep_common' when printing AST node
Fabian Reinartz's avatar
Fabian Reinartz committed
1273
1274
* [CHANGE/BUGFIX] Target identity considers path and parameters additionally to host and port
* [CHANGE] Rename metric `prometheus_local_storage_invalid_preload_requests_total` to `prometheus_local_storage_non_existent_series_matches_total`
1275
* [CHANGE] Support for old alerting rule syntax dropped
Fabian Reinartz's avatar
Fabian Reinartz committed
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
* [FEATURE] Deduplicate targets within the same scrape job
* [FEATURE] Add varbit chunk encoding (higher compression, more CPU usage – disabled by default)
* [FEATURE] Add `holt_winters` query function
* [FEATURE] Add relative complement `unless` operator to PromQL
* [ENHANCEMENT] Quarantine series file if data corruption is encountered (instead of crashing)
* [ENHANCEMENT] Validate Alertmanager URL
* [ENHANCEMENT] Use UTC for build timestamp
* [ENHANCEMENT] Improve index query performance (especially for active time series)
* [ENHANCEMENT] Instrument configuration reload duration
* [ENHANCEMENT] Instrument retrieval layer
* [ENHANCEMENT] Add Go version to `prometheus_build_info` metric

1288
## 0.17.0 / 2016-03-02
Fabian Reinartz's avatar
Fabian Reinartz committed
1289

1290
This version no longer works with Alertmanager 0.0.4 and earlier!
Fabian Reinartz's avatar
Fabian Reinartz committed
1291
The alerting rule syntax has changed as well but the old syntax is supported
1292
up until version 0.18.
Fabian Reinartz's avatar
Fabian Reinartz committed
1293

Tobias Schmidt's avatar
Tobias Schmidt committed
1294
All regular expressions in PromQL are anchored now, matching the behavior of
Fabian Reinartz's avatar
Fabian Reinartz committed
1295
regular expressions in config files.
Tobias Schmidt's avatar
Tobias Schmidt committed
1296

1297
* [CHANGE] Integrate with Alertmanager 0.1.0 and higher
Fabian Reinartz's avatar
Fabian Reinartz committed
1298
1299
1300
1301
* [CHANGE] Degraded storage mode renamed to rushed mode
* [CHANGE] New alerting rule syntax
* [CHANGE] Add label validation on ingestion
* [CHANGE] Regular expression matchers in PromQL are anchored
1302
* [FEATURE] Add `without` aggregation modifier
Fabian Reinartz's avatar
Fabian Reinartz committed
1303
1304
1305
1306
1307
1308
1309
1310
* [FEATURE] Send alert resolved notifications to Alertmanager
* [FEATURE] Allow millisecond precision in configuration file
* [FEATURE] Support AirBnB's Smartstack Nerve for service discovery
* [ENHANCEMENT] Storage switches less often between regular and rushed mode.
* [ENHANCEMENT] Storage switches into rushed mode if there are too many memory chunks.
* [ENHANCEMENT] Added more storage instrumentation
* [ENHANCEMENT] Improved instrumentation of notification handler
* [BUGFIX] Do not count head chunks as chunks waiting for persistence
Fabian Reinartz's avatar
Fabian Reinartz committed
1311
* [BUGFIX] Handle OPTIONS HTTP requests to the API correctly
Fabian Reinartz's avatar
Fabian Reinartz committed
1312
* [BUGFIX] Parsing of ranges in PromQL fixed
Fabian Reinartz's avatar
Fabian Reinartz committed
1313
* [BUGFIX] Correctly validate URL flag parameters
1314
1315
1316
* [BUGFIX] Log argument parse errors
* [BUGFIX] Properly handle creation of target with bad TLS config
* [BUGFIX] Fix of checkpoint timing issue
Fabian Reinartz's avatar
Fabian Reinartz committed
1317

1318
## 0.16.2 / 2016-01-18
Fabian Reinartz's avatar
Fabian Reinartz committed
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348

* [FEATURE] Multiple authentication options for EC2 discovery added
* [FEATURE] Several meta labels for EC2 discovery added
* [FEATURE] Allow full URLs in static target groups (used e.g. by the `blackbox_exporter`)
* [FEATURE] Add Graphite remote-storage integration
* [FEATURE] Create separate Kubernetes targets for services and their endpoints
* [FEATURE] Add `clamp_{min,max}` functions to PromQL
* [FEATURE] Omitted time parameter in API query defaults to now
* [ENHANCEMENT] Less frequent time series file truncation
* [ENHANCEMENT] Instrument number of  manually deleted time series
* [ENHANCEMENT] Ignore lost+found directory during storage version detection
* [CHANGE] Kubernetes `masters` renamed to `api_servers`
* [CHANGE] "Healthy" and "unhealthy" targets are now called "up" and "down" in the web UI
* [CHANGE] Remove undocumented 2nd argument of the `delta` function.
  (This is a BREAKING CHANGE for users of the undocumented 2nd argument.)
* [BUGFIX] Return proper HTTP status codes on API errors
* [BUGFIX] Fix Kubernetes authentication configuration
* [BUGFIX] Fix stripped OFFSET from in rule evaluation and display
* [BUGFIX] Do not crash on failing Consul SD initialization
* [BUGFIX] Revert changes to metric auto-completion
* [BUGFIX] Add config overflow validation for TLS configuration
* [BUGFIX] Skip already watched Zookeeper nodes in serverset SD
* [BUGFIX] Don't federate stale samples
* [BUGFIX] Move NaN to end of result for `topk/bottomk/sort/sort_desc/min/max`
* [BUGFIX] Limit extrapolation of `delta/rate/increase`
* [BUGFIX] Fix unhandled error in rule evaluation

Some changes to the Kubernetes service discovery were integration since
it was released as a beta feature.

1349
## 0.16.1 / 2015-10-16
Fabian Reinartz's avatar
Fabian Reinartz committed
1350
1351
1352
1353
1354
1355

* [FEATURE] Add `irate()` function.
* [ENHANCEMENT] Improved auto-completion in expression browser.
* [CHANGE] Kubernetes SD moves node label to instance label.
* [BUGFIX] Escape regexes in console templates.

1356
## 0.16.0 / 2015-10-09
Julius Volz's avatar
Julius Volz committed
1357
1358
1359
1360
1361
1362
1363

BREAKING CHANGES:

* Release tarballs now contain the built binaries in a nested directory.
* The `hash_mod` relabeling action now uses MD5 hashes instead of FNV hashes to
  achieve a better distribution.
* The DNS-SD meta label `__meta_dns_srv_name` was renamed to `__meta_dns_name`
Julius Volz's avatar
Julius Volz committed
1364
  to reflect support for DNS record types other than `SRV`.
Julius Volz's avatar
Julius Volz committed
1365
1366
1367
* The default full refresh interval for the file-based service discovery has been
  increased from 30 seconds to 5 minutes.
* In relabeling, parts of a source label that weren't matched by
Julius Volz's avatar
Julius Volz committed
1368
1369
  the specified regular expression are no longer included in the replacement
  output.
Julius Volz's avatar
Julius Volz committed
1370
1371
1372
1373
1374
1375
* Queries no longer interpolate between two data points. Instead, the resulting
  value will always be the latest value before the evaluation query timestamp.
* Regular expressions supplied via the configuration are now anchored to match
  full strings instead of substrings.
* Global labels are not appended upon storing time series anymore. Instead,
  they are only appended when communicating with external systems
Julius Volz's avatar
Julius Volz committed
1376
1377
  (Alertmanager, remote storages, federation). They have thus also been renamed
  from `global.labels` to `global.external_labels`.
Julius Volz's avatar
Julius Volz committed
1378
1379
1380
1381
* The names and units of metrics related to remote storage sample appends have
  been changed.
* The experimental support for writing to InfluxDB has been updated to work
  with InfluxDB 0.9.x. 0.8.x versions of InfluxDB are not supported anymore.
Julius Volz's avatar
Julius Volz committed
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
* Escape sequences in double- and single-quoted string literals in rules or query
  expressions are now interpreted like escape sequences in Go string literals
  (https://golang.org/ref/spec#String_literals).

Future breaking changes / deprecated features:

* The `delta()` function had an undocumented optional second boolean argument
  to make it behave like `increase()`. This second argument will be removed in
  the future. Migrate any occurrences of `delta(x, 1)` to use `increase(x)`
  instead.
* Support for filter operators between two scalar values (like `2 > 1`) will be
  removed in the future. These will require a `bool` modifier on the operator,
  e.g.  `2 > bool 1`.
Julius Volz's avatar
Julius Volz committed
1395
1396
1397

All changes:

Julius Volz's avatar
Julius Volz committed
1398
* [CHANGE] Renamed `global.labels` to `global.external_labels`.
Julius Volz's avatar
Julius Volz committed
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
* [CHANGE] Vendoring is now done via govendor instead of godep.
* [CHANGE] Change web UI root page to show the graphing interface instead of
  the server status page.
* [CHANGE] Append global labels only when communicating with external systems
  instead of storing them locally.
* [CHANGE] Change all regexes in the configuration to do full-string matches
  instead of substring matches.
* [CHANGE] Remove interpolation of vector values in queries.
* [CHANGE] For alert `SUMMARY`/`DESCRIPTION` template fields, cast the alert
  value to `float64` to work with common templating functions.
* [CHANGE] In relabeling, don't include unmatched source label parts in the
  replacement.
* [CHANGE] Change default full refresh interval for the file-based service
  discovery from 30 seconds to 5 minutes.
* [CHANGE] Rename the DNS-SD meta label `__meta_dns_srv_name` to
  `__meta_dns_name` to reflect support for other record types than `SRV`.
* [CHANGE] Release tarballs now contain the binaries in a nested directory.
* [CHANGE] Update InfluxDB write support to work with InfluxDB 0.9.x.
Julius Volz's avatar
Julius Volz committed
1417
1418
1419
* [FEATURE] Support full "Go-style" escape sequences in strings and add raw
  string literals.
* [FEATURE] Add EC2 service discovery support.
Julius Volz's avatar
Julius Volz committed
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
* [FEATURE] Allow configuring TLS options in scrape configurations.
* [FEATURE] Add instrumentation around configuration reloads.
* [FEATURE] Add `bool` modifier to comparison operators to enable boolean
  (`0`/`1`) output instead of filtering.
* [FEATURE] In Zookeeper serverset discovery, provide `__meta_serverset_shard`
  label with the serverset shard number.
* [FEATURE] Provide `__meta_consul_service_id` meta label in Consul service
  discovery.
* [FEATURE] Allow scalar expressions in recording rules to enable use cases
  such as building constant metrics.
1430
* [FEATURE] Add `label_replace()` and `vector()` query language functions.
Julius Volz's avatar
Julius Volz committed
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
* [FEATURE] In Consul service discovery, fill in the `__meta_consul_dc`
  datacenter label from the Consul agent when it's not set in the Consul SD
  config.
* [FEATURE] Scrape all services upon empty services list in Consul service
  discovery.
* [FEATURE] Add `labelmap` relabeling action to map a set of input labels to a
  set of output labels using regular expressions.
* [FEATURE] Introduce `__tmp` as a relabeling label prefix that is guaranteed
  to not be used by Prometheus internally.
* [FEATURE] Kubernetes-based service discovery.
* [FEATURE] Marathon-based service discovery.
* [FEATURE] Support multiple series names in console graphs JavaScript library.
* [FEATURE] Allow reloading configuration via web handler at `/-/reload`.
* [FEATURE] Updates to promtool to reflect new Prometheus configuration
  features.
* [FEATURE] Add `proxy_url` parameter to scrape configurations to enable use of
  proxy servers.
* [FEATURE] Add console templates for Prometheus itself.
* [FEATURE] Allow relabeling the protocol scheme of targets.
* [FEATURE] Add `predict_linear()` query language function.
* [FEATURE] Support for authentication using bearer tokens, client certs, and
  CA certs.
* [FEATURE] Implement unary expressions for vector types (`-foo`, `+foo`).
* [FEATURE] Add console templates for the SNMP exporter.
* [FEATURE] Make it possible to relabel target scrape query parameters.
* [FEATURE] Add support for `A` and `AAAA` records in DNS service discovery.
Julius Volz's avatar
Julius Volz committed
1457
1458
1459
* [ENHANCEMENT] Fix several flaky tests.
* [ENHANCEMENT] Switch to common routing package.
* [ENHANCEMENT] Use more resilient metric decoder.
Julius Volz's avatar
Julius Volz committed
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
* [ENHANCEMENT] Update vendored dependencies.
* [ENHANCEMENT] Add compression to more HTTP handlers.
* [ENHANCEMENT] Make -web.external-url flag help string more verbose.
* [ENHANCEMENT] Improve metrics around remote storage queues.
* [ENHANCEMENT] Use Go 1.5.1 instead of Go 1.4.2 in builds.
* [ENHANCEMENT] Update the architecture diagram in the `README.md`.
* [ENHANCEMENT] Time out sample appends in retrieval layer if the storage is
  backlogging.
* [ENHANCEMENT] Make `hash_mod` relabeling action use MD5 instead of FNV to
  enable better hash distribution.
* [ENHANCEMENT] Better tracking of targets between same service discovery
  mechanisms in one scrape configuration.
* [ENHANCEMENT] Handle parser and query evaluation runtime panics more
  gracefully.
* [ENHANCEMENT] Add IDs to H2 tags on status page to allow anchored linking.
Julius Volz's avatar
Julius Volz committed
1475
1476
1477
1478
1479
1480
1481
* [BUGFIX] Fix watching multiple paths with Zookeeper serverset discovery.
* [BUGFIX] Fix high CPU usage on configuration reload.
* [BUGFIX] Fix disappearing `__params` on configuration reload.
* [BUGFIX] Make `labelmap` action available through configuration.
* [BUGFIX] Fix direct access of protobuf fields.
* [BUGFIX] Fix panic on Consul request error.
* [BUGFIX] Redirect of graph endpoint for prefixed setups.
Julius Volz's avatar
Julius Volz committed
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
* [BUGFIX] Fix series file deletion behavior when purging archived series.
* [BUGFIX] Fix error checking and logging around checkpointing.
* [BUGFIX] Fix map initialization in target manager.
* [BUGFIX] Fix draining of file watcher events in file-based service discovery.
* [BUGFIX] Add `POST` handler for `/debug` endpoints to fix CPU profiling.
* [BUGFIX] Fix several flaky tests.
* [BUGFIX] Fix busylooping in case a scrape configuration has no target
  providers defined.
* [BUGFIX] Fix exit behavior of static target provider.
* [BUGFIX] Fix configuration reloading loop upon shutdown.
* [BUGFIX] Add missing check for nil expression in expression parser.
* [BUGFIX] Fix error handling bug in test code.
* [BUGFIX] Fix Consul port meta label.
* [BUGFIX] Fix lexer bug that treated non-Latin Unicode digits as digits.
Julius Volz's avatar
Julius Volz committed
1496
1497
1498
* [CLEANUP] Remove obsolete federation example from console templates.
* [CLEANUP] Remove duplicated Bootstrap JS inclusion on graph page.
* [CLEANUP] Switch to common log package.
Julius Volz's avatar
Julius Volz committed
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
* [CLEANUP] Update build environment scripts and Makefiles to work better with
  native Go build mechanisms and new Go 1.5 experimental vendoring support.
* [CLEANUP] Remove logged notice about 0.14.x configuration file format change.
* [CLEANUP] Move scrape-time metric label modification into SampleAppenders.
* [CLEANUP] Switch from `github.com/client_golang/model` to
  `github.com/common/model` and related type cleanups.
* [CLEANUP] Switch from `github.com/client_golang/extraction` to
  `github.com/common/expfmt` and related type cleanups.
* [CLEANUP] Exit Prometheus when the web server encounters a startup error.
* [CLEANUP] Remove non-functional alert-silencing links on alerting page.
* [CLEANUP] General cleanups to comments and code, derived from `golint`,
  `go vet`, or otherwise.
* [CLEANUP] When entering crash recovery, tell users how to cleanly shut down
  Prometheus.
* [CLEANUP] Remove internal support for multi-statement queries in query engine.
* [CLEANUP] Update AUTHORS.md.
* [CLEANUP] Don't warn/increment metric upon encountering equal timestamps for
  the same series upon append.
* [CLEANUP] Resolve relative paths during configuration loading.

1519
## 0.15.1 / 2015-07-27
Julius Volz's avatar
Julius Volz committed
1520
1521
1522
1523
1524
* [BUGFIX] Fix vector matching behavior when there is a mix of equality and
  non-equality matchers in a vector selector and one matcher matches no series.
* [ENHANCEMENT] Allow overriding `GOARCH` and `GOOS` in Makefile.INCLUDE.
* [ENHANCEMENT] Update vendored dependencies.

1525
## 0.15.0 / 2015-07-21
beorn7's avatar
beorn7 committed
1526
1527
1528

BREAKING CHANGES:

Julius Volz's avatar
Julius Volz committed
1529
1530
1531
* Relative paths for rule files are now evaluated relative to the config file.
* External reachability flags (`-web.*`) consolidated.
* The default storage directory has been changed from `/tmp/metrics`
beorn7's avatar
beorn7 committed
1532
  to `data` in the local directory.
Julius Volz's avatar
Julius Volz committed
1533
* The `rule_checker` tool has been replaced by `promtool` with
beorn7's avatar
beorn7 committed
1534
  different flags and more functionality.
Julius Volz's avatar
Julius Volz committed
1535
* Empty labels are now removed upon ingestion into the
beorn7's avatar
beorn7 committed
1536
1537
1538
  storage. Matching empty labels is now equivalent to matching unset
  labels (`mymetric{label=""}` now matches series that don't have
  `label` set at all).
Julius Volz's avatar