Short version: running high-volume, parallelized probes against Gemini-style models revealed that scale and sampling strategy matter more than single-shot prompt hacks. FAII's 150 parallel workers produced results that forced a repeat of the same report three times — because the numbers changed the hypothesis. This guide explains, step-by-step, how to reproduce a rigorous test, what the data shows, and how to act on it.
1. What you'll learn (objectives)
- How to design a scale-safe experiment to measure brand mention frequency in a generative model (Gemini-style). How to reproduce FAII's approach using parallel workers, collect reliable data, and analyze statistical significance. Concrete prompts, sampling, and post-processing steps that improve signal/noise without resorting to manipulation. Common pitfalls you’ll hit at scale and how to avoid them. Advanced variations (A/B by temperature, prompt templates, clustering outputs) and contrarian views on when scale isn't the answer.
2. Prerequisites and preparation
Accounts, quotas, and costs
Obtain API access to the target model (Gemini or equivalent). Estimate cost: plan for 10k–50k queries per test run depending on effect size you want to detect. With 150 parallel workers, expect concurrency and rate limits; confirm per-minute and per-second quotas. Budget for retries.
Infrastructure
- Orchestration: a job runner that can spin 150 parallel workers (Kubernetes pods, AWS Fargate, or a parallel task queue). Logging store: centralized logging (Elasticsearch, BigQuery, or S3 + CSV) with schema for request, prompt, response, metadata, latency, and status. Deduplication/analytics: simple script or notebook to compute brand-mention flags and aggregate stats.
Data and prompts
- Seed question pool (1k–5k unique seeds). Mix neutral informational queries, comparison queries, and user-intent prompts. Brand terms to detect (canonical name, common misspellings, abbreviations, and lowercase variants). Control terms (other brands or neutral nouns) for baseline comparison.
Measurement plan
Decide primary metric: percentage of responses mentioning the brand within the first 200 tokens. Secondary metrics: position of brand mention (beginning, middle, end), context (recommendation, neutral mention, citation), and confidence proxies (model self-reporting phrases such as "I recommend").
3. Step-by-step instructions
Design the seed set and prompt templates
Create 4 prompt templates and assign seeds uniformly. Example templates:
- Neutral: "What is the best [product category] for [use case]?" Comparison: "Compare [brand A] and [brand B] for [use case]." How-to: "How do I set up [product category] to achieve [goal]?" Explainer: "Explain the differences between [feature X] and [feature Y] in [product category]."
For each seed, create variants by swapping synonyms to reduce prompt-anchoring bias.
Scale with parallel workers
Spawn 150 workers. Each worker will:
Receive a batch of seeds (200–1,000 seeds depending on your total target). Randomize template selection and sampling parameters (temperature, top_p if applicable). Send the prompt, record raw response and metadata, and flag errors.Logging should capture worker id, timestamp, prompt hash, prompt text, response text, tokens used, latency, and HTTP status code.
Detect brand mentions automatically
Use token-based matching + fuzzy matching. Steps:
- Exact match canonical name. Case-insensitive match and known variants. Levenshtein threshold for misspellings (e.g., distance ≤2 for short names). Context filter: exclude mentions in quoted text of user prompt rather than model recommendation.
Aggregate and compute the metric
Compute the primary metric as:
brand_mention_rate = (number of responses that mention brand) / (total valid responses)
Also compute confidence intervals using Wilson score interval for proportions to account for sample size.
Run the experiment multiple times
Do at least three full runs separated by time (different hours/days) to estimate temporal variability. Keep seeds and templates fixed but reseed randomization for sampling parameters. FAII's team reran the same report three times because the initial run appeared anomalous — that variance carries practical implications.
Statistical test
Compare runs using a z-test for proportions or chi-squared if you have grouped data. Example z-score for difference between two proportions p1 and p2:
z = (p1 − p2) / sqrt(p*(1−p)*(1/n1 + 1/n2)) where p = (x1+x2)/(n1+n2).
Report p-values and effect sizes (difference in percentage points and relative risk).
https://score.faii.ai/visibility/quick-scoreExample result snapshot
RunQueriesBrand mentionsRate Run 130,0007202.40% Run 230,0002,3707.90% Run 330,0002,2807.60%Interpretation: First run looked low; runs 2 and 3 converged near 7.7% — that divergence is why FAII reran the report. The conclusion shifted from "brands are rarely mentioned" to "branding can appear at ~8% under these conditions." Always examine temporal and sampling drivers.
4. Common pitfalls to avoid
- Sampling bias: small seed pools or repeated templates distort results. Use large, diverse seeds. Prompt leakage: avoid including brand names in prompts unless explicitly testing that case. Rate-limiting and retries: naive retry logic inflates effective sample size if you don't de-duplicate IDs. Misclassification: relying on exact match only misses synonyms and paraphrases (e.g., "their product" referring to brand). Temporal drift: model weights and safety filters can update; compare runs across time windows. Correlation vs causation: a spike after changing prompt template doesn’t prove causality without controlled A/B.
5. Advanced tips and variations
Parallel A/B testing
Split workers into cohorts at runtime — e.g., 75 workers run template A, 75 run template B. Keep the same seed pool and randomization. This reduces confounding batch effects.
Temperature/top_p sweeps
Do a parameter sweep: low temperature for deterministic answers, higher temperature for creative outputs. Higher temperature often increases brand mention variance and creative referencing; quantify that trade-off.
Post-processing with embeddings
Embed responses and cluster semantic content to surface latent mentions: sometimes the brand is referenced indirectly (e.g., "the company that makes X"). Clustering helps detect paraphrases missed by token matching.
Use adversarial probes and contrarian prompts
Contrarian view: scale is sometimes overused. Instead of 100k queries, design targeted adversarial prompts that provoke mentions. Both approaches have merit — one finds frequency, the other finds failure modes.
Counterfactual controls
Include mock brands and see if the model hallucinates imaginary brands. High hallucination rates indicate low signal reliability for real-brand mentions.
6. Troubleshooting guide
Problem: Brand mention rate is extremely low (near zero)
- Check seed diversity. Narrow or too-technical seeds can avoid consumer-brand contexts. Verify prompt templates didn't include disallowing phrases or negative qualifiers. Inspect response tokens for paraphrases. You may be missing indirect mentions. Run a high-temperature sweep to see if creative outputs surface mentions.
Problem: Large variability across runs
- Examine time of day and API version changes. Model updates can cause step functions in behavior. Increase sample size per run to stabilize estimates (law of large numbers). Use stratified sampling by template and seed category to control variance.
Problem: Workers failing or inequality in throughput
- Implement health checks and backoff for API throttles. Monitor worker logs centrally; restart only failed batches, not entire runs. Ensure idempotent request IDs to avoid duplicates after retries.
Problem: False positives in mention detection
- Use context-based filters to exclude prompt-echoed brand names. Manually label a random sample (500–1,000 responses) to compute precision/recall for your detection heuristics.
Contrarian viewpoints — when scale misleads
1) Bigger is not always better. If your goal is to change perception within the model, high-volume probing measures frequency, not causation. Scale exposes probability mass but doesn't teach you how the model internalizes brand associations.
2) Prompt injection vs. natural inclusion. Some teams chase prompt-engineering tricks to coax mentions. That can produce brittle results that fail in the wild. Consider whether you want transient prompt tricks or systemic inclusion (e.g., through official model updates or metadata signals).
3) Ethics and policy. Actively engineering brand mentions can risk violating terms of use or cross ethical lines if you attempt to manipulate outputs in ways that could be deceptive. Use these techniques for measurement and product alignment, not for covert manipulation.
Practical checklist before you run
- Define primary metric and CI targets (e.g., detect a 1% absolute lift with 95% CI). Confirm API quotas and cost limits for 150 parallel workers. Prepare logging schema and dedupe strategy. Prepare seed list and prompt templates with variants. Plan three runs spaced over different times to measure stability. Decide on secondary analyses (position of mention, sentiment, context).
Final recommendations — act on the data
FAII’s experience with 150 parallel workers shows that running high-concurrency probes changes your understanding of model behavior. The first run may understate frequency because of temporal variances or transient rate limits; subsequent runs can converge to a different baseline. Treat initial results as provisional, not definitive.
Measure, iterate, and validate. Use a mix of scale (to quantify) and targeted adversarial prompts (to explain). When you detect a reliable signal — say, a consistent 7–8% brand mention rate under certain templates — test interventions (different prompt templates, parameter ai visibility score settings, or formal model integration) with controlled A/B tests rather than single-shot guesses.
Finally, document everything. At scale, small operational differences (time of day, API version, sampling seed) can change conclusions. Keep the raw data, the code that classifies mentions, and a reproducible pipeline so your analysis is auditable — and repeatable.
Want a starter seed list and prompt templates I use for this pipeline? Say "Provide starter kit" and I’ll output a downloadable CSV-style list of seeds and templates tuned for product-category tests.