feat(analytics): Plausible funnel goals and storage-free attribution #673

Merged

tobru merged 2 commits from feat/plausible-attribution into main 2026-09-18 12:51:37 +00:00

Conversation 1 Commits 2 Files changed 31 +945 -69

aarno.aukia commented 2026-09-15 13:58:32 +00:00

Member

Copy link

Stacked on PR 674 (fix/trial-end-post-tests), which fixes three staff view tests that fail on main since 2026-09-11. Merge 674 first; this PR's diff against main shows its commit until then.

Summary

servala.com and the VSHN landing pages send visitors to the portal with ref and utm_* parameters. The portal lost them at the first redirect: an anonymous deep link such as /provision/postgresql/?ref=codey_ch went to /accounts/login/?next=..., where ref sits inside next. Plausible reads the source from the first pageview's URL only, so these visits counted as Direct.

This PR makes the funnel from landing page to servala.com to portal order traceable, without storing anything on the visitor's device.

Attribution survives redirects

• AttributionRedirectMiddleware copies ref, utm_source, utm_medium, utm_campaign, utm_term, utm_content and gclid onto every same-host redirect of a GET request. It never overwrites a value the target already has, and it leaves external redirects (Keycloak) alone.
• The login page now arrives as /accounts/login/?next=...&ref=codey_ch. Deep link parameters (provider, plan, instances, sla, storage, zone) pass through unchanged.

Plausible goals on confirmed outcomes

Goal	When	Props
UserSignedUp	allauth user_signed_up (first Keycloak login creates the account)
UserLoggedIn	allauth user_logged_in
OrganizationCreated	after the organization is saved	approval: automatic or pending
ServiceInstanceCreated	unchanged	service, plan
ServiceInstancePlanChanged	unchanged	service, plan
CTA Click	click on a.btn (browser)	label, section
Form Started	first focus on a form[data-analytics-form] (browser)	source: Service Order, Organization Create

• Server goals queue in the session and render once, through json_script, on the next full page. A reload or a failed submission never fires them.
• Goals are removed from the queue only by the base template that renders them. Before this, any render with the request consumed them, which is how allauth's "logged in" message would have dropped UserLoggedIn.
• Browser events use delegated document listeners in static/js/analytics.js, so buttons and forms rendered later by htmx are covered. Form Started fires once per form name per page load, because htmx replaces the order form on every plan or zone change.

Attribution into Odoo

• The organization signup form renders ref, utm_source, utm_medium, utm_campaign and the external HTTP referrer as hidden fields.
• The new sale order gets source_id, medium_id and campaign_id looked up by name, created when missing, and otherwise the optional defaults SERVALA_ODOO_UTM_SOURCE_ID, SERVALA_ODOO_UTM_MEDIUM_ID, SERVALA_ODOO_UTM_CAMPAIGN_ID.
• Referrer: <url> and Referred from: <ref> are appended to internal_note, HTML-escaped and capped at 200 characters.
• Organizations that need approval keep the values in pending_billing_data["attribution"] and replay them when staff approve.

Deployment

• Register the goals and properties on the portal.servala.com dashboard. This is done by landingpager's sync_plausible (companion change in vshn/landingpager).
• Optional: set the three SERVALA_ODOO_UTM_*_ID variables. Without them, sale orders without campaign parameters leave the UTM fields empty, as today.

Testing

• src/tests/test_attribution.py: redirect chain for anonymous, single-org and no-org visitors, the middleware's edge cases, referrer filtering (own host, SSO host), hidden fields, Odoo payload with lookup, create, defaults, a failed UTM lookup, and replay on approval.
• src/tests/test_plausible.py: queueing, one-time rendering, escaping, a real password login rendering UserLoggedIn on the landing page, organization creation with both approval modes, and no goal for an invalid form.
• static/js/analytics.js was checked in a browser harness that loads the real file with a recording plausible stub and asserts the exact event list (nav, section and footer buttons, aria-label fallback, 60 character cap, non-button links ignored, late-added button, htmx-style form swap).
• Full suite, on top of PR 674: 2080 passed, 0 failed.
• Not verified: a login through real Keycloak and events arriving in live Plausible.

Design Decisions

• Rebased on PR 674: CI inherited three date-expired test failures from main. "rebase 673 on 674".
• No device or session storage for attribution: attribution travels in URLs and hidden form fields only. Handoff rule: "store nothing on the visitor's device", and session storage of attribution waits for a legal check.
• Middleware instead of per-view changes: every redirect (login, deep link, organization chooser, onboarding) needs the same treatment, and a missed view would silently count as Direct again.
• Carry utm_term, utm_content and gclid on redirects, but only three UTM values into Odoo: Plausible reads all of them as sources; Odoo sale orders only have source, medium and campaign.
• Attribution goes on the sale order, not a CRM lead: the portal creates no leads. The sale order is the Odoo record for a portal signup.
• Pending organizations store attribution in pending_billing_data: the sale order only exists after approval, so the values must wait somewhere server-side. This is database storage of attribution tied to a customer record, flagged for the same legal check.
• allauth signals, not django.contrib.auth: hijack logs in through django.contrib.auth.login, and staff impersonation must not count as a login.
• UserSignedUp doubles as first login: the portal account is created on the first Keycloak login, so a separate first-login goal would always equal it.
• A UTM lookup failure logs a warning and still creates the sale order: campaign data is reporting only.
• The external referrer ignores the portal's own host and the Keycloak host: after SSO the browser reports Keycloak, which says nothing about the campaign.

Spec Diff

Compared with the handoff:

• As planned: goals only on confirmed outcomes, json_script for props, delegated CTA Click and Form Started, hidden form fields for attribution, UTM lookup by name with create and default fallback, escaped Referrer: and Referred from: capped at 200 characters, tests including one-time firing, reload and failed submission.
• Changed: the queue stub in <head> already existed (with plausible.init), so it was kept rather than replaced. Goals render from a session queue instead of a per-view flag, because several can be pending at once (for example UserSignedUp and UserLoggedIn).
• Changed: Odoo attribution lands on the sale order's internal_note rather than a lead description.
• Added: the redirect middleware, which was needed because the login redirect did drop the query string; OrganizationCreated with an approval prop; replay on approval.
• Dropped: Servala Portal Click does not apply inside the portal. An order-submitted goal separate from ServiceInstanceCreated was not added, since instance creation is the confirmed outcome of the order form.
• Known limitation: attribution survives one redirect chain within a visit, not a return visit days later; the referrer is lost across Keycloak login.

🤖 Generated with Claude Code

https://claude.ai/code/session_014S5eLBkR9DQyhYAG11hBZw

> **Stacked on PR 674** (`fix/trial-end-post-tests`), which fixes three staff view tests that fail on `main` since 2026-09-11. Merge 674 first; this PR's diff against `main` shows its commit until then. ## Summary servala.com and the VSHN landing pages send visitors to the portal with `ref` and `utm_*` parameters. The portal lost them at the first redirect: an anonymous deep link such as `/provision/postgresql/?ref=codey_ch` went to `/accounts/login/?next=...`, where `ref` sits inside `next`. Plausible reads the source from the first pageview's URL only, so these visits counted as Direct. This PR makes the funnel from landing page to servala.com to portal order traceable, without storing anything on the visitor's device. ### Attribution survives redirects - `AttributionRedirectMiddleware` copies `ref`, `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, `utm_content` and `gclid` onto every same-host redirect of a GET request. It never overwrites a value the target already has, and it leaves external redirects (Keycloak) alone. - The login page now arrives as `/accounts/login/?next=...&ref=codey_ch`. Deep link parameters (`provider`, `plan`, `instances`, `sla`, `storage`, `zone`) pass through unchanged. ### Plausible goals on confirmed outcomes | Goal | When | Props | |---|---|---| | `UserSignedUp` | allauth `user_signed_up` (first Keycloak login creates the account) | | | `UserLoggedIn` | allauth `user_logged_in` | | | `OrganizationCreated` | after the organization is saved | `approval`: `automatic` or `pending` | | `ServiceInstanceCreated` | unchanged | `service`, `plan` | | `ServiceInstancePlanChanged` | unchanged | `service`, `plan` | | `CTA Click` | click on `a.btn` (browser) | `label`, `section` | | `Form Started` | first focus on a `form[data-analytics-form]` (browser) | `source`: `Service Order`, `Organization Create` | - Server goals queue in the session and render once, through `json_script`, on the next full page. A reload or a failed submission never fires them. - Goals are removed from the queue only by the base template that renders them. Before this, any render with the request consumed them, which is how allauth's "logged in" message would have dropped `UserLoggedIn`. - Browser events use delegated document listeners in `static/js/analytics.js`, so buttons and forms rendered later by htmx are covered. `Form Started` fires once per form name per page load, because htmx replaces the order form on every plan or zone change. ### Attribution into Odoo - The organization signup form renders `ref`, `utm_source`, `utm_medium`, `utm_campaign` and the external HTTP referrer as hidden fields. - The new sale order gets `source_id`, `medium_id` and `campaign_id` looked up by name, created when missing, and otherwise the optional defaults `SERVALA_ODOO_UTM_SOURCE_ID`, `SERVALA_ODOO_UTM_MEDIUM_ID`, `SERVALA_ODOO_UTM_CAMPAIGN_ID`. - `Referrer: <url>` and `Referred from: <ref>` are appended to `internal_note`, HTML-escaped and capped at 200 characters. - Organizations that need approval keep the values in `pending_billing_data["attribution"]` and replay them when staff approve. ## Deployment - Register the goals and properties on the portal.servala.com dashboard. This is done by landingpager's `sync_plausible` (companion change in vshn/landingpager). - Optional: set the three `SERVALA_ODOO_UTM_*_ID` variables. Without them, sale orders without campaign parameters leave the UTM fields empty, as today. ## Testing - `src/tests/test_attribution.py`: redirect chain for anonymous, single-org and no-org visitors, the middleware's edge cases, referrer filtering (own host, SSO host), hidden fields, Odoo payload with lookup, create, defaults, a failed UTM lookup, and replay on approval. - `src/tests/test_plausible.py`: queueing, one-time rendering, escaping, a real password login rendering `UserLoggedIn` on the landing page, organization creation with both approval modes, and no goal for an invalid form. - `static/js/analytics.js` was checked in a browser harness that loads the real file with a recording `plausible` stub and asserts the exact event list (nav, section and footer buttons, aria-label fallback, 60 character cap, non-button links ignored, late-added button, htmx-style form swap). - Full suite, on top of PR 674: 2080 passed, 0 failed. - Not verified: a login through real Keycloak and events arriving in live Plausible. ## Design Decisions - **Rebased on PR 674**: CI inherited three date-expired test failures from `main`. "rebase 673 on 674". - **No device or session storage for attribution**: attribution travels in URLs and hidden form fields only. Handoff rule: "store nothing on the visitor's device", and session storage of attribution waits for a legal check. - **Middleware instead of per-view changes**: every redirect (login, deep link, organization chooser, onboarding) needs the same treatment, and a missed view would silently count as Direct again. - **Carry `utm_term`, `utm_content` and `gclid` on redirects, but only three UTM values into Odoo**: Plausible reads all of them as sources; Odoo sale orders only have source, medium and campaign. - **Attribution goes on the sale order, not a CRM lead**: the portal creates no leads. The sale order is the Odoo record for a portal signup. - **Pending organizations store attribution in `pending_billing_data`**: the sale order only exists after approval, so the values must wait somewhere server-side. This is database storage of attribution tied to a customer record, flagged for the same legal check. - **allauth signals, not `django.contrib.auth`**: hijack logs in through `django.contrib.auth.login`, and staff impersonation must not count as a login. - **`UserSignedUp` doubles as first login**: the portal account is created on the first Keycloak login, so a separate first-login goal would always equal it. - **A UTM lookup failure logs a warning and still creates the sale order**: campaign data is reporting only. - **The external referrer ignores the portal's own host and the Keycloak host**: after SSO the browser reports Keycloak, which says nothing about the campaign. ## Spec Diff Compared with the handoff: - **As planned**: goals only on confirmed outcomes, `json_script` for props, delegated `CTA Click` and `Form Started`, hidden form fields for attribution, UTM lookup by name with create and default fallback, escaped `Referrer:` and `Referred from:` capped at 200 characters, tests including one-time firing, reload and failed submission. - **Changed**: the queue stub in `<head>` already existed (with `plausible.init`), so it was kept rather than replaced. Goals render from a session queue instead of a per-view flag, because several can be pending at once (for example `UserSignedUp` and `UserLoggedIn`). - **Changed**: Odoo attribution lands on the sale order's `internal_note` rather than a lead description. - **Added**: the redirect middleware, which was needed because the login redirect did drop the query string; `OrganizationCreated` with an `approval` prop; replay on approval. - **Dropped**: `Servala Portal Click` does not apply inside the portal. An order-submitted goal separate from `ServiceInstanceCreated` was not added, since instance creation is the confirmed outcome of the order form. - **Known limitation**: attribution survives one redirect chain within a visit, not a return visit days later; the referrer is lost across Keycloak login. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014S5eLBkR9DQyhYAG11hBZw

aarno.aukia added 1 commit 2026-09-15 13:58:32 +00:00

feat(analytics): Plausible funnel goals and storage-free attribution ... 49e8dc5305

Campaign attribution from servala.com and the landing pages was lost at the
first redirect: an anonymous deep link went to the login page with ref and
utm_* buried inside next, so Plausible counted the visit as Direct.

- AttributionRedirectMiddleware copies ref, utm_* and gclid onto same-host
  redirects of GET requests, never overwriting values already on the target.
- Server-confirmed goals queue in the session and render once on the next
  full page: UserSignedUp, UserLoggedIn (allauth signals, so impersonation
  does not count) and OrganizationCreated with an approval prop. Queued goals
  are consumed only by the base template, so allauth's login message and htmx
  partials no longer swallow them.
- static/js/analytics.js sends CTA Click and Form Started through delegated
  listeners; the order and organization signup forms are marked.
- The organization signup form carries ref, utm_* and the external referrer
  as hidden fields. The Odoo sale order gets utm source, medium and campaign
  by name (created if missing, else SERVALA_ODOO_UTM_*_ID) and a Referrer /
  Referred from note. Pending organizations replay it on approval.

Nothing is stored on the visitor's device.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5eLBkR9DQyhYAG11hBZw

aarno.aukia requested review from tobru 2026-09-15 14:17:46 +00:00

aarno.aukia self-assigned this 2026-09-15 14:17:51 +00:00

aarno.aukia force-pushed feat/plausible-attribution from 49e8dc5305 to a00f26c726 2026-09-15 14:28:52 +00:00 Compare

tobru commented 2026-09-16 19:47:57 +00:00

Owner

Copy link

Before we can continue with that:

1. in src/servala/core/odoo.py -> we need to make sure the Odoo user has access to the referenced models
2. we need to set the SERVALA_ODOO_UTM_* variable values in deployment/kustomize/*

Before we can continue with that: 1) in `src/servala/core/odoo.py` -> we need to make sure the Odoo user has access to the referenced models 2) we need to set the `SERVALA_ODOO_UTM_*` variable values in `deployment/kustomize/*`

tobru force-pushed feat/plausible-attribution from a00f26c726 to bd3ce72b86 2026-09-18 12:31:16 +00:00 Compare

tobru added 1 commit 2026-09-18 12:47:56 +00:00

fix(analytics): track only funnel CTAs, match Odoo UTM names case-insensitively ... 7f84c0e790

CTA Click fired on every a.btn, which is ~50 elements across the portal and
most of them are in-app navigation: Back, Edit, Open, View All Instances,
staff tooling. That buries the funnel in UI noise and spends the Plausible
event quota on clicks nobody will report on.

- The listener now matches [data-analytics-cta], carried by the eight steps
  that move a visitor forward: Create Organization, Continue Provisioning,
  Discover Services, Select Service, Create Instance, Pay Invoice, Request
  Limit Increase and Extend Organization. The label lives in the attribute
  because the button text is translated and reading it would split one CTA
  into a prop value per language. test_only_funnel_ctas_are_tracked pins the
  set against the templates.
- Odoo utm.source, utm.medium and utm.campaign are matched with =ilike, so
  Newsletter no longer creates a second record next to newsletter. A record
  that has to be created keeps the spelling it arrived in. The pattern
  escapes %, _ and \, because ILIKE would otherwise read the underscore in
  servala_com as a wildcard and attribute the order to the wrong source.

test_create_view_shows_warning_at_limit asserted the substring "Create
Organization" was absent at the organization limit; the header entry now
carries it as an attribute value, so both that test and its positive
counterpart pin the submit button by id instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tobru merged commit 9dcdf41bb8 into main 2026-09-18 12:51:37 +00:00

tobru deleted branch feat/plausible-attribution 2026-09-18 12:51:37 +00:00

tobru referenced this pull request from a commit 2026-09-18 12:51:37 +00:00

Merge pull request 'feat(analytics): Plausible funnel goals and storage-free attribution' (#673) from feat/plausible-attribution into main

Sign in to join this conversation.

Reviewers

No reviewers

tobru

Labels

Clear labels   

API

Relates to API

  

Billing

Billing and invoicing

  

UI/UX

User interface and user experience

  

dependencies

Dependencies Update

  

bug

Something is not working

  

change

Change an existing feature

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

API

Billing

UI/UX

dependencies

bug

change

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

aarno.aukia

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

servala/servala-portal!673

No description provided.