Support server-side generated analytics data via POST from middleware

[dch]

Precheck

Prerequisites

• I have searched open and closed issues to make sure that the feature has not yet been requested
• commented Extract visitor information from web server log? #210 but broken out for discussion

Feature request

site visitors who disable javascript, or use adblocker tech, have their visits lost. CNAMEing and
similar tricks are already tracked and disabled by current adblockers, and this is likely to grow.

If we can send analytics data from the server side directly after each request, we get similar
data without needing any client-side tracking.

How would you like it to work

If the POST /api/event format is documented, it should be possible to submit the analytics data
server side directly in the web framework, as pluggable middleware. This is both nice from a
configuration point of view, and also from an analytics view - we won't miss anything, and users
don't need to run the javascripts.

For example, in Elixir there are 2 major frameworks - Plug and Phoenix - which allow stackable
middleware. The analytics request can extract most of the info from haproxy or nginx details,
and the web server framework, so while we get a reduced set of info, we do get 90% of the value.

Ideally the POST ... request would only include the analytics data in the BODY, and use either
HTTP Headers or a query parameter to track an API key, so that these can safely be logged
locally without exposing plausible.io API tokens.

Update: This is now live. You can take a look at our events API https://plausible.io/docs/events-api

[tckb]

@dch Although, not explicitly documented, you can find the event data documented here https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L73 To be clear, this is the data posted to the event controller in plain text.

POST /api/event HTTP/1.1
Host: <plausible_base_url>
Content-Type: text/plain

{
    "name": "pageview",
    "url": "https://domain-added-to-plausible.com/your-page",
    "domain": "domain-added-to-plausible.com",
    "referrer": "https://tldr.net",
    "source": "google",
    "user_agent": "Mozilla",
    "screen_width": 1920
}

Most Appservers/Webservers dump the above information.

NB: You need to pass-along the clients/visitors IP along with your call. use x-forwarded-for header to pass on the IP.

It being a plain rest call, you can write your own middle ware for your favorite framework for posting the data.

Currently, there's no API key that is being passed along from the client side either. Technically, there's also some need of rate-limiting and/or filtering of bad-actors that might be addressed at a later point.

[ukutaht]

Same as #130, I will close that one in favour of this issue since the discussion seems to be here.

Generally this would work and I have planned to support something like this in the future. Two issues come to mind:

• We cannot get the screen width from server-side. Maybe it's better to align with most other tools and detect device type from User-Agent instead of screen width.
• Tons of bot traffic on server logs. We can exclude the ones that identify themselves via the User-Agent, but beyond that we don't do any sort of advanced bot detection. In a normal setup, having to load and run JS filters out a lot of bots

[tckb]

Regarding the bot detection, I have previously worked with yauaa and is pretty great at detecting bots or getting any sense of client based on UA analysis, if you can test it here. I would like to implement this in elixir or probably for Plausible but more on that later.

[dch]

wow just seen that plausible is written in Elixir :-) 🚀 🔥 🍾

[ItalyPaleAle]

I am (potentially) interested in this feature too.

I'd like to add that a REST endpoint for adding data would ideally allow for batching too, so the back-end could have a buffer internally and then submit the requests to Plausible at regular intervals. This would allow for much better performance.

From the standpoint of a REST API, it could be as simple as allowing an array of objects in addition to a single object, also providing the timestamp of the request.

[Tamschi]

I'm interested in this too, but for use from the browser, as a matter of avoiding remote-loaded JS.

I could download the JS snippet and have my build process warn me when that changes, but that seems a bit brittle.

[ukutaht]

You should be able to achieve the same thing with our NPM library: https://github.com/plausible/plausible-tracker

[alicescfernandes]

Hi. I also needed the same feature, and after searching for a while, i came across with this issue as well. I ended up reading through source code that @tckb pointed to, and also this file, and i quickly wrote this to send analytics server-side. It's a bit raw, but maybe you can adapt to what you need

[alicescfernandes]

Hmmm hard to tell, but i'll give it a couple of days until actually see the user-agents

[huksley]

@alicescfernandes Thanks for code snippet, works nice! One note though - do not send screen_width = 0 or it will be treated as a mobile device, not sure if this is your intention though.

[alicescfernandes]

Hi! Thanks for the tip. What happens if i send null?

[Vigasaurus]

It'll be saved as an empty string, which means it basically won't show up into the Size report of the Devices report. That is probably the result you want, it basically indicates the screen size not being known/unclassified.

See: https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L107 -> https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L190

[alicescfernandes]

It'll be saved as an empty string, which means it basically won't show up into the Size report of the Devices report. That is probably the result you want, it basically indicates the screen size not being known/unclassified.

See: https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L107 -> https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L190

It its, thanks for the tip. Also @ukutaht, here's some data that was collected via backend tracking:

Not bots, but someone trying to access known vulnerable routes.

[andygrunwald]

I am also interested in this feature and I am willing to create a middleware for Go.
Is it possible to have one domain but two tracking accounts?
By this, I would be able to check both results (without creating noise in the analytics):

1. Tracking via the standard remote loaded JS
2. Tracking via server-side middleware

[ukutaht]

That would be an interesting comparison to run for sure!

Domains must be globally unique in our database. You can still register 2 sites with slightly different domain fields and send analytics to both. The domain field in our database doesn't actually have to match the domain of your URL.

[andygrunwald]

Thanks @ukutaht.
I will give it a shot.

[metmarkosaric]

You can take a look at our events API now https://plausible.io/docs/events-api

[markcellus]

Thanks! Just started testing out the new Events API for this. Unfortunately, I ran across an issue during testing: #1309.

[luciandavidescu]

@metmarkosaric
(Assuming it might have some use later on), is there a way to also put a user or session id through that API?

[ukutaht]

Something like this? https://www.youtube.com/watch?v=o9QbRwb5aZ4

[luciandavidescu]

Yes, that's where publishers usually look, it's much more helpful, especially when urls do not reflect titles verbatim.

[ukutaht]

OK, we don't have a way to collect, store or report on page titles, sorry

[luciandavidescu]

Well, that's that, nevermind, the fact that it has an api for server-side input is good enough already, so I'll be using it too since there's zero client overhead, hopefully it may become the only tool needed one day.

[ukutaht]

great, enjoy!