README.md 17.7 KB
Newer Older
beorn7's avatar
beorn7 committed
1
# Prometheus Pushgateway
2

beorn7's avatar
beorn7 committed
3
[![Build Status](https://travis-ci.org/prometheus/pushgateway.svg)][travis]
4
5
6
[![CircleCI](https://circleci.com/gh/prometheus/pushgateway/tree/master.svg?style=shield)][circleci]
[![Docker Repository on Quay](https://quay.io/repository/prometheus/pushgateway/status)][quay]
[![Docker Pulls](https://img.shields.io/docker/pulls/prom/pushgateway.svg?maxAge=604800)][hub]
Bernerd Schaefer's avatar
Bernerd Schaefer committed
7

8
The Prometheus Pushgateway exists to allow ephemeral and batch jobs to
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
9
10
expose their metrics to Prometheus. Since these kinds of jobs may not
exist long enough to be scraped, they can instead push their metrics
11
to a Pushgateway. The Pushgateway then exposes these metrics to
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
12
Prometheus.
Bernerd Schaefer's avatar
Bernerd Schaefer committed
13

beorn7's avatar
beorn7 committed
14
15
16
## Non-goals

The Pushgateway is explicitly not an _aggregator or distributed counter_ but
beorn7's avatar
beorn7 committed
17
18
19
rather a metrics cache. It does not have
[statsd](https://github.com/etsy/statsd)-like semantics. The metrics pushed are
exactly the same as you would present for scraping in a permanently running
beorn7's avatar
beorn7 committed
20
21
22
23
24
25
26
program. If you need distributed counting, you could either use the actual
statsd in combination with the [Prometheus statsd
exporter](https://github.com/prometheus/statsd_exporter), or have a look at
[Weavework's aggregation
gateway](https://github.com/weaveworks/prom-aggregation-gateway). With more
experience gathered, the Prometheus project might one day be able to provide a
native solution, separate from or possibly even as part of the Pushgateway.
Bernerd Schaefer's avatar
Bernerd Schaefer committed
27

28
29
For machine-level metrics, the
[textfile](https://github.com/prometheus/node_exporter/blob/master/README.md#textfile-collector)
beorn7's avatar
beorn7 committed
30
31
32
33
34
35
36
37
collector of the Node exporter is usually more appropriate. The Pushgateway is
intended for service-level metrics.

The Pushgateway is not an _event store_. While you can use Prometheus as a data
source for
[Grafana annotations](http://docs.grafana.org/reference/annotations/), tracking
something like release events has to happen with some event-logging framework.

38
39
40
A while ago, we
[decided to not implement a “timeout” or TTL for pushed metrics](https://github.com/prometheus/pushgateway/issues/19)
because almost all proposed use cases turned out to be anti-patterns we
41
42
strongly discourage. You can follow a more recent discussion on the
[prometheus-developers mailing list](https://groups.google.com/forum/#!topic/prometheus-developers/9IyUxRvhY7w).
43

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
44
45
## Run it

beorn7's avatar
beorn7 committed
46
47
48
49
50
51
Download binary releases for your platform from the
[release page](https://github.com/prometheus/pushgateway/releases) and unpack
the tarball.

If you want to compile yourself from the sources, you need a working Go
setup. Then use the provided Makefile (type `make`).
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
52
53

For the most basic setup, just start the binary. To change the address
James Turnbull's avatar
James Turnbull committed
54
55
to listen on, use the `--web.listen-address` flag (e.g. "0.0.0.0:9091" or ":9091").
By default, Pushgateway does not persist metrics. However, the `--persistence.file` flag
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
56
allows you to specify a file in which the pushed metrics will be
57
persisted (so that they survive restarts of the Pushgateway).
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
58

59
60
61
62
63
64
65
66
67
68
69
70
### Using Docker

You can deploy the Pushgateway using the [prom/pushgateway](https://registry.hub.docker.com/u/prom/pushgateway/) Docker image.

For example:

```bash
docker pull prom/pushgateway

docker run -d -p 9091:9091 prom/pushgateway
```

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
71
## Use it
Bernerd Schaefer's avatar
Bernerd Schaefer committed
72

beorn7's avatar
beorn7 committed
73
74
75
76
77
78
79
### Configure the Pushgateway as a target to scrape

The Pushgateway has to be configured as a target to scrape by Prometheus, using
one of the usual methods. _However, you should always set `honor_labels: true`
in the scrape config_ (see [below](#about-the-job-and-instance-labels) for a
detailed explanation).

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
80
### Libraries
Bernerd Schaefer's avatar
Bernerd Schaefer committed
81

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
82
Prometheus client libraries should have a feature to push the
83
registered metrics to a Pushgateway. Usually, a Prometheus client
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
84
85
86
passively presents metric for scraping by a Prometheus server. A
client library that supports pushing has a push function, which needs
to be called by the client code. It will then actively push the
87
metrics to a Pushgateway, using the API described below.
Bernerd Schaefer's avatar
Bernerd Schaefer committed
88

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
89
### Command line
Bernerd Schaefer's avatar
Bernerd Schaefer committed
90

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
91
92
93
94
Using the Prometheus text protocol, pushing metrics is so easy that no
separate CLI is provided. Simply use a command-line HTTP tool like
`curl`. Your favorite scripting language has most likely some built-in
HTTP capabilities you can leverage here as well.
Bernerd Schaefer's avatar
Bernerd Schaefer committed
95

beorn7's avatar
beorn7 committed
96
97
98
99
*Note that in the text protocol, each line has to end with a line-feed
character (aka 'LF' or '\n'). Ending a line in other ways, e.g. with 'CR' aka
'\r', 'CRLF' aka '\r\n', or just the end of the packet, will result in a
protocol error.*
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
100

beorn7's avatar
beorn7 committed
101
Pushed metrics are managed in groups, identified by a grouping key of any
beorn7's avatar
beorn7 committed
102
103
104
105
106
number of labels, of which the first must be the `job` label. The groups are
easy to inspect via the web interface.

*For implications of special characters in label values see the [URL
section](#url) below.*
beorn7's avatar
beorn7 committed
107

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
108
109
Examples:

Jonas Große Sundrup's avatar
Jonas Große Sundrup committed
110
* Push a single sample into the group identified by `{job="some_job"}`:
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
111

Brian Brazil's avatar
Brian Brazil committed
112
        echo "some_metric 3.14" | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
113

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
114
115
  Since no type information has been provided, `some_metric` will be of type `untyped`.

116
* Push something more complex into the group identified by `{job="some_job",instance="some_instance"}`:
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
117

Brian Brazil's avatar
Brian Brazil committed
118
        cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
119
120
121
122
123
124
125
        # TYPE some_metric counter
        some_metric{label="val1"} 42
        # TYPE another_metric gauge
        # HELP another_metric Just an example.
        another_metric 2398.283
        EOF

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
126
127
128
  Note how type information and help strings are provided. Those lines
  are optional, but strongly encouraged for anything more complex.

beorn7's avatar
beorn7 committed
129
130
* Delete all metrics in the group identified by
  `{job="some_job",instance="some_instance"}`:
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
131

Brian Brazil's avatar
Brian Brazil committed
132
        curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
133

beorn7's avatar
beorn7 committed
134
135
136
137
* Delete all metrics in the group identified by `{job="some_job"}` (note that
  this does not include metrics in the
  `{job="some_job",instance="some_instance"}` group from the previous example,
  even if those metrics have the same job label):
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
138

Brian Brazil's avatar
Brian Brazil committed
139
        curl -X DELETE http://pushgateway.example.org:9091/metrics/job/some_job
140

beorn7's avatar
beorn7 committed
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
### About the job and instance labels

The Prometheus server will attach a `job` label and an `instance` label to each
scraped metric. The value of the `job` label comes from the scrape
configuration. When you configure the Pushgateway as a scrape target for your
Prometheus server, you will probably pick a job name like `pushgateway`. The
value of the `instance` label is automatically set to the host and port of the
target scraped. Hence, all the metrics scraped from the Pushgateway will have
the host and port of the Pushgateway as the `instance` label and a `job` label
like `pushgateway`. The conflict with the `job` and `instance` labels you might
have attached to the metrics pushed to the Pushgateway is solved by renaming
those labels to `exported_job` and `exported_instance`.

However, this behavior is usually undesired when scraping a
Pushgateway. Generally, you would like to retain the `job` and `instance`
labels of the metrics pushed to the Pushgateway. That's why you have set
`honor_labels: true` in the scrape config for the Pushgateway. It enables the
desired behavior. See the
[documentation](https://prometheus.io/docs/operating/configuration/#scrape_config)
for details.

This leaves us with the case where the metrics pushed to the Pushgateway do not
有张纸's avatar
有张纸 committed
163
feature an `instance` label. This case is quite common as the pushed metrics
beorn7's avatar
beorn7 committed
164
165
166
167
168
169
170
171
are often on a service level and therefore not related to a particular
instance. Even with `honor_labels: true`, the Prometheus server will attach an
`instance` label if no `instance` label has been set in the first
place. Therefore, if a metric is pushed to the Pushgateway without an instance
label (and without instance label in the grouping key, see below), the
Pushgateway will export it with an emtpy instance label (`{instance=""}`),
which is equivalent to having no `instance` label at all but prevents the
server from attaching one.
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
172
173
174

### About timestamps

beorn7's avatar
beorn7 committed
175
176
177
178
If you push metrics at time *t*<sub>1</sub>, you might be tempted to believe
that Prometheus will scrape them with that same timestamp
*t*<sub>1</sub>. Instead, what Prometheus attaches as a timestamp is the time
when it scrapes the Pushgateway. Why so?
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
179
180
181
182

In the world view of Prometheus, a metric can be scraped at any
time. A metric that cannot be scraped has basically ceased to
exist. Prometheus is somewhat tolerant, but if it cannot get any
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
183
samples for a metric in 5min, it will behave as if that metric does
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
184
not exist anymore. Preventing that is actually one of the reasons to
185
use a Pushgateway. The Pushgateway will make the metrics of your
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
186
ephemeral job scrapable at any time. Attaching the time of pushing as
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
187
a timestamp would defeat that purpose because 5min after the last
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
188
189
190
191
192
push, your metric will look as stale to Prometheus as if it could not
be scraped at all anymore. (Prometheus knows only one timestamp per
sample, there is no way to distinguish a 'time of pushing' and a 'time
of scraping'.)

beorn7's avatar
beorn7 committed
193
As there aren't any use cases where it would make sense to to attach a
Brian Brazil's avatar
Brian Brazil committed
194
different timestamp, and many users attempting to incorrectly do so (despite no
beorn7's avatar
beorn7 committed
195
196
client library supporting this), the Pushgateway rejects any pushes with
timestamps.
Brian Brazil's avatar
Brian Brazil committed
197
198
199

If you think you need to push a timestamp, please see [When To Use The
Pushgateway](https://prometheus.io/docs/practices/pushing/).
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
200

201
202
203
204
205
In order to make it easier to alert on pushers that have not run recently, the
Pushgateway will add in a metric `push_time_seconds` with the Unix timestamp
of the last `POST`/`PUT` to each group. This will override any pushed metric by
that name.

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
206
207
## API

208
All pushes are done via HTTP. The interface is vaguely REST-like.
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
209
210
211

### URL

Brian Brazil's avatar
Brian Brazil committed
212
The default port the push gateway is listening to is 9091. The path looks like
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
213

214
    /metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}
215

216
`<JOB_NAME>` is used as the value of the `job` label, followed by any
217
218
219
220
221
222
number of other label pairs (which might or might not include an
`instance` label). The label set defined by the URL path is used as a
grouping key. Any of those labels already set in the body of the
request (as regular labels, e.g. `name{job="foo"} 42`)
_will be overwritten to match the labels defined by the URL path!_

223
224
225
If `job` or any label name is suffixed with `@base64`, the following job name
or label value is interpreted as a base64 encoded string according to [RFC
4648, using the URL and filename safe
226
227
alphabet](https://tools.ietf.org/html/rfc4648#section-5). (Padding is
optional.) This is the only way of using job names or label values that contain
228
229
a `/`. For other special characters, the usual URI component encoding works,
too, but the base64 might be more convenient.
230

231
Ideally, client libraries take care of the suffixing and encoding.
232
233
234
235
236
237
238
239

Examples:

* To use the grouping key `job="directory_cleaner",path="/var/tmp"`, the
  following path will _not_ work:

	  /metrics/job/directory_cleaner/path//var/tmp
	  
240
241
  Instead, use the base64 URL-safe encoding for the label value and mark it by
  suffixing the label name with `@base64`:
242
  
243
  	  /metrics/job/directory_cleaner/path@base64/L3Zhci90bXA
244
245
246
247
248
249
	  
  If you are not using a client library that handles the encoding for you, you
  can use encoding tools. For example, there is a command line tool `base64url`
  (Debian package `basez`), which you could combine with `curl` to push from
  the command line in the following way:
  
250
      echo 'some_metric{foo="bar"} 3.14' | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/directory_cleaner/path@base64/$(echo -n '/var/tmp' | base64url)
251
252
253
254
255
256
257
258
  
* The grouping key `job="titan",name="Προμηθεύς"` can be represented
  “traditionally” with URI encoding:
  
      /metrics/job/titan/name/%CE%A0%CF%81%CE%BF%CE%BC%CE%B7%CE%B8%CE%B5%CF%8D%CF%82
	  
  Or you can use the more compact base64 encoding:
  
259
      /metrics/job/titan/name@base64/zqDPgc6_zrzOt864zrXPjc-C
beorn7's avatar
beorn7 committed
260

261
262
263
264
265
266
### `PUT` method

`PUT` is used to push a group of metrics. All metrics with the
grouping key specified in the URL are replaced by the metrics pushed
with `PUT`.

beorn7's avatar
beorn7 committed
267
268
269
The body of the request contains the metrics to push either as delimited binary
protocol buffers or in the simple flat text format (both in version 0.0.4, see
the
270
[data exposition format specification](https://docs.google.com/document/d/1ZjyKiKxZV83VI9ZKAXRGKaUKK2BIWCT7oiGBKDBpjEY/edit?usp=sharing)).
beorn7's avatar
beorn7 committed
271
Discrimination between the two variants is done via the `Content-Type`
beorn7's avatar
beorn7 committed
272
273
274
header. (Use the value `application/vnd.google.protobuf;
proto=io.prometheus.client.MetricFamily; encoding=delimited` for protocol
buffers, otherwise the text format is tried as a fall-back.)
275
276
277

The response code upon success is always 202 (even if the same
grouping key has never been used before, i.e. there is no feedback to
beorn7's avatar
beorn7 committed
278
279
the client if the push has replaced an existing group of metrics or
created a new one).
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
280

281
282
283
284
_If using the protobuf format, do not send duplicate MetricFamily
proto messages (i.e. more than one with the same name) in one push, as
they will overwrite each other._

285
286
A successfully finished request means that the pushed metrics are
queued for an update of the storage. Scraping the push gateway may
287
288
still yield the old results until the queued update is
processed. Neither is there a guarantee that the pushed metrics are
289
290
291
persisted to disk. (A server crash may cause data loss. Or the push
gateway is configured to not persist to disk at all.)

beorn7's avatar
beorn7 committed
292
293
294
295
296
A `PUT` request with an empty body effectively deletes all metrics with the
specified grouping key. However, in contrast to the
[`DELETE` request](#delete-method) described below, it does update the
`push_time_seconds` metrics.

297
### `POST` method
298

299
300
301
`POST` works exactly like the `PUT` method but only metrics with the
same name as the newly pushed metrics are replaced (among those with
the same grouping key).
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
302

beorn7's avatar
beorn7 committed
303
304
305
A `POST` request with an empty body merely updates the `push_time_seconds`
metrics but does not change any of the previously pushed metrics.

Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
306
307
308
### `DELETE` method

`DELETE` is used to delete metrics from the push gateway. The request
309
310
311
312
313
must not contain any content. All metrics with the grouping key
specified in the URL are deleted.

The response code upon success is always 202. The delete
request is merely queued at that moment. There is no guarantee that the
314
315
316
317
318
319
request will actually be executed or that the result will make it to
the persistence layer (e.g. in case of a server crash). However, the
order of `PUT`/`POST` and `DELETE` request is guaranteed, i.e. if you
have successfully sent a `DELETE` request and then send a `PUT`, it is
guaranteed that the `DELETE` will be processed first (and vice versa).

320
321
322
Deleting a grouping key without metrics is a no-op and will not result
in an error.

beorn7's avatar
beorn7 committed
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
## Exposed metrics

The Pushgateway exposes the following metrics via the configured
`--web.telemetry-path` (default: `/metrics`):
- The pushed metrics.
- For earch pushed group, a metric `push_time_seconds` as explained above.
- The usual metrics provided by the [Prometheus Go client library](https://github.com/prometheus/client_golang), i.e.:
  - `process_...`
  - `go_...`
  - `promhttp_metric_handler_requests_...`
- A number of metrics specific to the Pushgateway, as documented by the example scrape below.

```
# HELP pushgateway_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which pushgateway was built.
# TYPE pushgateway_build_info gauge
pushgateway_build_info{branch="master",goversion="go1.10.2",revision="8f88ccb0343fc3382f6b93a9d258797dcb15f770",version="0.5.2"} 1
# HELP pushgateway_http_push_duration_seconds HTTP request duration for pushes to the Pushgateway.
# TYPE pushgateway_http_push_duration_seconds summary
pushgateway_http_push_duration_seconds{method="post",quantile="0.1"} 0.000116755
pushgateway_http_push_duration_seconds{method="post",quantile="0.5"} 0.000192608
pushgateway_http_push_duration_seconds{method="post",quantile="0.9"} 0.000327593
pushgateway_http_push_duration_seconds_sum{method="post"} 0.001622878
pushgateway_http_push_duration_seconds_count{method="post"} 8
# HELP pushgateway_http_push_size_bytes HTTP request size for pushes to the Pushgateway.
# TYPE pushgateway_http_push_size_bytes summary
pushgateway_http_push_size_bytes{method="post",quantile="0.1"} 166
pushgateway_http_push_size_bytes{method="post",quantile="0.5"} 182
pushgateway_http_push_size_bytes{method="post",quantile="0.9"} 196
pushgateway_http_push_size_bytes_sum{method="post"} 1450
pushgateway_http_push_size_bytes_count{method="post"} 8
# HELP pushgateway_http_requests_total Total HTTP requests processed by the Pushgateway, excluding scrapes.
# TYPE pushgateway_http_requests_total counter
pushgateway_http_requests_total{code="200",handler="static",method="get"} 5
pushgateway_http_requests_total{code="200",handler="status",method="get"} 8
pushgateway_http_requests_total{code="202",handler="delete",method="delete"} 1
pushgateway_http_requests_total{code="202",handler="push",method="post"} 6
pushgateway_http_requests_total{code="400",handler="push",method="post"} 2

```
  
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
363
364
## Development

365
366
The normal binary embeds the web files in the `resources` directory.
For development purposes, it is handy to have a running binary use
367
368
369
370
those files directly (so that you can see the effect of changes immediately).
To switch to direct usage, add `-tags dev` to the `flags` entry in
`.promu.yml`, and then `make build`. Switch back to "normal" mode by
reverting the changes to `.promu.yml` and typing `make assets`.
Bjoern Rabenstein's avatar
Bjoern Rabenstein committed
371
372
373
374
375
376
377

##  Contributing

Relevant style guidelines are the [Go Code Review
Comments](https://code.google.com/p/go-wiki/wiki/CodeReviewComments)
and the _Formatting and style_ section of Peter Bourgon's [Go:
Best Practices for Production
378
Environments](http://peter.bourgon.org/go-in-production/#formatting-and-style).
379

380
381
382
[travis]: https://travis-ci.org/prometheus/pushgateway
[hub]: https://hub.docker.com/r/prom/pushgateway/
[circleci]: https://circleci.com/gh/prometheus/pushgateway
beorn7's avatar
beorn7 committed
383
[quay]: https://quay.io/repository/prometheus/pushgateway