Product Outcomes and Metrics Workbench

CompassFlow — outcome objective and metric decision canvas

Synthetic example run. All account counts, research findings, and product facts are fictional simulation data from the supplied source packet. New targets, interventions, thresholds, and ownership roles below are proposals. No experiment or customer outcome has been produced by this document.

Recommendation and replacement OKR

Objective: Newly eligible accounts confidently complete a useful, correct workflow and return to complete another without losing access to the manual setup route.

Prioritize repeated verified value across all eligible accounts next quarter. Treat shipping screens, adding an assistant, and building a dashboard as candidate interventions and instrumentation, not objectives or evidence of success. Start by testing clearer required-field guidance against the current manual setup; an AI assistant has no demonstrated incremental benefit in this packet.

Proposed key result: Increase the share of all newly eligible accounts that complete a verified first workflow within 14 days and repeat a verified workflow in days 15–28 from the synthetic baseline

24% to 30%. This is an aspirational planning target, not a commitment or a predicted intervention effect. For another 500-account cohort, 30% would mean 150 qualifying accounts, 30 more than the baseline 120. That calculation explains the proposed materiality; it does not establish feasibility.

Use first-workflow completion to locate early friction and conditional repeat to understand what happens after first value. These are diagnostics of the same objective, not three unrelated goals. Commercial consequences and support cost are unknown; do not claim retention or revenue improvement.

Baseline: three distinct denominators

Source: one mature synthetic cohort of

500 newly eligible accounts, all with a complete 28-day observation window. Exact cohort dates and segment distribution are not supplied. Verified means the authoritative workspace contains the requested record with required fields; an assistant completion message does not count.

Identity

40% × 60% = 24%, since the repeat numerator is a subset of the first-completion numerator in this cohort. The 300 accounts without timely first completion remain in the primary denominator. The 80 first completers without a qualifying repeat remain in the conditional denominator. Do not count a late first workflow alone as repeat, or assume nothing happened among the other 300 accounts beyond the supplied definition.

Four of 10 research sessions contained uncertainty about required setup fields. This is directional qualitative evidence, not a 40% prevalence estimate among all eligible accounts and not proof that extra screens or an assistant fix the problem.

Outcome and Metric Decision Canvas

Objective, behavior, boundaries

Desired behavior: an account supplies the required setup information, creates the requested valid record in the workspace, and later performs another valid workflow. Immediate beneficiaries are newly eligible account users; support may also benefit, but that remains unmeasured. New guidance can add steps for experienced users; an assistant could introduce trust, latency, or correction work. Preserve the manual route and observe whether new friction displaces existing successful behavior.

Why now: the proposed next-quarter goal is currently an output list, while the cohort shows a large gap between eligibility and repeated value. Non-goals: maximize assistant usage, screen views, dashboard visits, raw workflow attempts, or claimed completion messages; infer retention gains; force all accounts through one route.

Foundational pushback: required-field confusion may be one cause, not the dominant bottleneck. Some accounts may lack an immediate task, approval, data, or product fit. Those are hypotheses, not supplied findings. A narrower field-guidance test can distinguish comprehension from broader causes without assuming AI is required.

Outcome chain

The feedback loop changes from checking delivery/adoption to reviewing verified first value, repeated value, and failure reasons. Do not reward support or assistants for creating duplicate records merely to hit a numerator.

Focused metric portfolio

The 50% first-completion aspiration combined with unchanged 60% conditional repeat would mathematically yield 30% primary outcome. That is a planning scenario, not a forecast that both will occur. A higher conditional rate alone can reflect selection of fewer easy accounts and need not improve whole-population value.

Full metric contracts

Shared eligibility and timing contract

• Unit is a unique account, counted once at first eligibility in the cohort. Define the actual eligibility rule and authoritative eligibility timestamp before a pilot; neither is supplied in detail.
• Include every genuinely newly eligible account assigned to an arm, including non-users, failures, abandonments, manual-route users, accounts requiring support, and accounts later disabled. Exclude only predeclared invalid/test/duplicate eligibility records with auditable reasons; never exclude poor outcomes after assignment.
• Proposed day convention: day 1 starts at the eligibility timestamp; the first interval is elapsed time [0, 14 days), and the repeat interval is [14, 28 days). Confirm that this matches source-query boundary semantics before comparing a new query to the 200/120 baseline. Do not silently reinterpret inclusive day labels.
• For the three comparable rates, use accounts with complete 28-day follow-up. First completion can be shown provisionally at day 14, but label that immature view and do not combine it with a different cohort's repeat denominator. Right-censored accounts wait for maturity; report their count.
• Attribute by assignment at eligibility, regardless of later route switching. Separate intervention exposure from assignment. Segment by assigned arm, setup route used, intended workflow type, account readiness/experience, and relevant account size where those fields can be collected appropriately. Segment baselines are unavailable; do not invent differences.
• State evidence must include account ID, eligibility timestamp, requested-workflow identity, required-field validation result, verified completion timestamp, and distinct workflow ID. The exact schema is proposed, not an existing capability claim. Retried submissions and duplicate events do not create additional account successes.
• Proposed accountability: Data/Engineering for extraction and reconciliation, Product for the decision, Design/Support for task quality and friction review. Named owners and actual assignments are unknown and required before live testing.

M1 — whole-eligible first-and-repeat rate, primary

Numerator: mature eligible accounts with at least one verified first workflow during the first interval

and a distinct verified workflow during the repeat interval. Denominator: all mature newly eligible accounts in the assigned cohort. Source: authoritative workspace record/validation state joined to eligibility and workflow timestamps. Unit: account; window: 28 days from eligibility; segmentation: shared contract. Baseline: 120/500 = 24% from the supplied mature cohort, exact dates unknown.

Count gross verified completion evidence at the time it meets the requested-record/required-field rule, matching the source definition. Separately report later invalidation, incorrect records, unauthorized changes, and recovery in the fixed 28-day window. Do not relabel gross baseline as sustained net success: that net baseline is unavailable. A net sustained version would exclude accounts whose qualifying record is invalidated without a valid replacement by the window end, using a predeclared rule. It is a future quality analysis, not another established 24% fact.

Planning target: proposed 30%. Weekly review only checks maturity/data quality/safety; evaluate product impact after the predeclared mature exposure. Gaming risk: duplicate, trivial, or unrequested records and revised eligibility to shrink the denominator. Reconcile intent and state; audit exclusions and retries.

M2 — timely first-workflow completion, leading

Numerator: mature eligible accounts with a verified first workflow in the first 14-day interval. Denominator: the same mature eligible cohort as M1. Source/unit/segments/owners as above. Baseline: 200/500 = 40%. Proposed planning target: 50%; this identifies a potential lever, not permission to scale if repeat or quality fails. Gross/net reporting follows M1. Failure mode: making an invalid easy workflow appear complete or hiding manual-route failures. Review reasons for no first value, not just assistant usage.

M3 — conditional repeat among timely completers, diagnostic

Numerator: the same accounts counted in M1. Denominator: the same accounts counted in M2. A zero denominator produces “not estimable,” not 0% success. Source: verified distinct workspace workflows; unit: account; repeat window: days 15–28 under the shared convention. Baseline: 120/200 = 60%; no separate observed target. Proposed review: explain changes from baseline by first-completer mix and route before declaring improvement or deterioration.

Gross/net treatment follows M1, with denominator membership recorded from the original timely completion definition. Gaming risk: reduce first completions to select only high-readiness repeaters. It is not the primary success metric and cannot override a declining whole-eligible outcome.

G1 — correctness and authority guardrail

Numerator: attempted workflows producing an incorrect requested record, missing required fields despite a completion claim, destructive unintended changes, or unauthorized writes. Denominator: all workflow attempts in assigned accounts over days 1–28; report both absolute affected-account counts and rate by route/arm/workflow type. Source: authoritative validation, accepted user intent, access/audit records, and reviewed task samples. An assistant success message is an observation to check, not completion evidence.

Baseline unknown. Proposed hard floor: zero unauthorized writes or destructive unintended changes; either suspends the intervention. Proposed target for noncritical invalid-record/completion mismatch is no deterioration versus concurrent control, with a predeclared uncertainty method and materiality bound TBD before pilot. Unknown outcomes cannot be counted as successful. Proposed Support/Engineering accountability, actual owner unknown; immediate incident review plus weekly quality review. Record gross errors and recovered errors separately; recovery does not erase the original harm. Gaming risk: fewer logged attempts or excluding failed accounts.

G2 — support burden guardrail

Numerator: total observed support/rescue minutes related to setup/workflow completion in days 1–28 for assigned accounts. Denominator: all mature assigned eligible accounts, including those making no workflow attempt. Unit: minutes per eligible account; companion absolute incident count retained. Source: support records with task classification and documented coverage; segments: arm, route, workflow type, and account experience. Baseline and coverage unknown.

Proposed boundary: no material deterioration versus concurrent control; acceptable minutes increase and uncertainty rule must be agreed before exposure, not chosen after results. Count repeat contacts and recovery effort; do not net them against hypothetical saved labor. Proposed Support/Data ownership, named owner unknown. Review weekly and at mature outcome review. Gaming risk: moving rescue into untracked channels or measuring only successful users. Missing coverage blocks a support-saving claim and scale decision.

G3 — manual-route access and completion guardrail

Availability numerator: scheduled checks where an eligible manual-route test can reach the existing setup and submit permitted required fields; denominator: all scheduled checks during the pilot, including failed checks. Completion numerator: assigned manual-route attempts resulting in verified correct records; denominator: all manual-route attempts, including failed/abandoned attempts, within days 1–28. Source: controlled checks plus workspace state, not page loads. Segment by arm, workflow type, and permission state; unit: check or attempt as explicitly named.

Baseline unknown beyond the supplied fact that the manual route remains available. Proposed floor: no intervention-induced loss of the manual route; any such loss stops enrollment until repaired. Proposed completion boundary: no material deterioration versus control, bound to be fixed before test. Keep manual rescue and route switching visible. Proposed Engineering/Design accountability, actual owner unknown; ongoing checks plus weekly/mature review. Gaming risk: hiding the route or steering users away so only easy attempts remain. Gross attempts and eventual corrected completion both reported without dropping failures.

Experiment and explicit decisions

Hypothesis: Clearer required-field guidance for newly eligible accounts increases verified timely first completion and subsequent repeated value compared with the current manual route, without worsening correctness, support burden, or manual access.

Run a bounded account-level randomized test after eligibility/state joins and quality checks work. A proposed initial variant simplifies required-field explanations; screen count is not the objective. If an AI assistant is later tested, isolate its incremental effect in a separate design with explicit permissions and validation; do not combine all proposed features and attribute the outcome to AI.

Proposed causal success threshold: At least a five percentage-point absolute increase in mature M1 versus concurrent control, with the lower bound of a predeclared confidence interval above zero, plus all guardrails satisfied. Five points is a proposed practical significance threshold, not baseline evidence. The planning aspiration of 30% and this experimental contrast answer different questions.

Minimum credible sample, allocation, interval method, clustering/contamination treatment, expected control rate, and support/error materiality bounds are TBD. Data/Product must set them before enrollment using the baseline and anticipated variation. Until set, the experiment is not release-ready. Do not declare success from an underpowered favorable point estimate or optional early stopping; declare one primary comparison and decision time in advance.

Examine composition before aggregation: more low-readiness accounts can lower conditional repeat while whole-population repeated value rises. Conversely, filtering to high-readiness accounts can lift conditional repeat without creating more total value. Show counts and rates for all arms/segments; missing segment instrumentation is a limitation, not evidence of equality.

Instrumentation gaps and concrete next actions

No assignments, experiment approvals, or outcomes are recorded. The immediate decision is to adopt the outcome objective and authorize measurement/field-guidance preparation; the next gate is a reviewable experiment plan with real owners and boundaries.

Delivery review performed: 40%, 60%, and 24% retain their distinct denominators; mature windows and authoritative records are required; the causal portfolio is focused; proposed targets are labeled; support, error, and manual-route guardrails can veto scale. Intervention effects, guardrail baselines, segment results, sample sufficiency, and retention impact remain unknown.