Add prompt_style A/B experiment to smoke-project workflow by Copilot · Pull Request #37326 · github/gh-aw

Copilot · 2026-06-06T14:43:30Z

This updates smoke-project to run a 3-variant prompt-style experiment (detailed, concise, step_by_step) and measure whether reduced prompt verbosity preserves project-operation reliability. It also wires experiment metadata/assignment into the compiled lock workflow for runtime tracking.

Experiment configuration (frontmatter)
- Added experiments.prompt_style_test with:
  - variants, hypothesis, primary/secondary metrics
  - guardrail thresholds
  - weights, sample target, start date, tags, notify target
Prompt body varianting
- Replaced the single ## Test Requirements block with explicit value-compare conditionals:
  - concise: high-level 7-step operation list
  - step_by_step: terse numbered tool-oriented sequence
  - else: existing detailed baseline preserved
Compiled workflow update
- Regenerated smoke-project.lock.yml from source workflow to include experiment assignment/state plumbing required by runtime.

{{#if experiments.prompt_style_test == "concise" }}
... concise instructions ...
{{else if experiments.prompt_style_test == "step_by_step" }}
... step-by-step instructions ...
{{else}}
... existing detailed baseline ...
{{/if}}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

This PR adds an A/B (3-variant) prompt-style experiment to the smoke-project workflow to compare detailed vs reduced-verbosity prompt variants, and regenerates the compiled lock workflow to plumb experiment assignment/state into runtime execution.

Changes:

Added experiments.prompt_style_test frontmatter (variants, metrics, guardrails, weights, start date, notify issue).
Introduced handlebars conditional prompt bodies for concise and step_by_step, preserving the existing detailed baseline as the default branch.
Regenerated smoke-project.lock.yml to include experiment state restore/pick/upload and downstream wiring.

Show a summary per file

File	Description
pkg/workflow/data/action_pins.json	Updates embedded action pin entries (notably `github/gh-aw-actions/setup`).
pkg/actionpins/data/action_pins.json	Mirrors the embedded pin update for the `actionpins` package.
.github/workflows/smoke-project.md	Adds experiment config and prompt-body variant conditionals.
.github/workflows/smoke-project.lock.yml	Regenerated compiled workflow including experiment plumbing and other runtime/version deltas.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 4/4 changed files
Comments generated: 6

+    "github/gh-aw-actions/setup@v0.76.1": {
      "repo": "github/gh-aw-actions/setup",
-      "version": "v0.78.2",
-      "sha": "268bf92726cb8153337c07166f382ee46e4fd897"
-    },
-    "github/gh-aw/actions/setup-cli@v0.78.2": {
-      "repo": "github/gh-aw/actions/setup-cli",
-      "version": "v0.78.2",
-      "sha": "a7d4043a8224182262356a32931099b76fd332eb"
+      "version": "v0.76.1",
+      "sha": "46d564922b082d0db93244972e8005ea6904ee5f"
    },


+    "github/gh-aw-actions/setup@v0.76.1": {
      "repo": "github/gh-aw-actions/setup",
-      "version": "v0.78.2",
-      "sha": "268bf92726cb8153337c07166f382ee46e4fd897"
-    },
-    "github/gh-aw/actions/setup-cli@v0.78.2": {
-      "repo": "github/gh-aw/actions/setup-cli",
-      "version": "v0.78.2",
-      "sha": "a7d4043a8224182262356a32931099b76fd332eb"
+      "version": "v0.76.1",
+      "sha": "46d564922b082d0db93244972e8005ea6904ee5f"
    },


+# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a2f85a8fdf159792c99dd73d46b646c60877a2885d36170b127e98df8cb2083f","body_hash":"a06527944f758e799e436bef444ec14e9c38546538220b343897f74877fb3c1c","compiler_version":"v0.77.5","strict":true,"agent_id":"copilot"}
+# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_CI_TRIGGER_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GH_AW_OTEL_GRAFANA_AUTHORIZATION","GH_AW_OTEL_GRAFANA_ENDPOINT","GH_AW_OTEL_SENTRY_AUTHORIZATION","GH_AW_OTEL_SENTRY_ENDPOINT","GH_AW_PROJECT_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"v0.77.5","version":"v0.77.5"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.58"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.58"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.58"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.22"},{"image":"ghcr.io/github/github-mcp-server:v1.1.0","digest":"sha256:71b07d9abecb83b4a2595bcd8ccb35f9a0166361a12335f9e16da1ef07172029","pinned_image":"ghcr.io/github/github-mcp-server:v1.1.0@sha256:71b07d9abecb83b4a2595bcd8ccb35f9a0166361a12335f9e16da1ef07172029"},{"image":"node:lts-alpine","digest":"sha256:2bdb65ed1dab192432bc31c95f94155ca5ad7fc1392fb7eb7526ab682fa5bf14","pinned_image":"node:lts-alpine@sha256:2bdb65ed1dab192432bc31c95f94155ca5ad7fc1392fb7eb7526ab682fa5bf14"}]}


      - name: Setup Scripts
        id: setup
-        uses: ./actions/setup
+        uses: github/gh-aw-actions/setup@v0.77.5
        with:


-          cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_3c30d1736c2958aa_EOF'
-          {"add_comment":{"hide_older_comments":true,"max":2},"add_labels":{"allowed":["smoke-project"]},"create_issue":{"close_older_issues":true,"expires":2,"group":true,"labels":["ai-generated","automation","testing"],"max":1},"create_project_status_update":{"github-token":"${GH_AW_SECRET_GH_AW_PROJECT_GITHUB_TOKEN}","max":1,"project":"https://github.com/orgs/github/projects/24068"},"create_pull_request":{"expires":2,"if_no_changes":"warn","labels":["ai-generated"],"max":1,"max_patch_files":100,"max_patch_size":1024,"protect_top_level_dot_folders":true,"protected_files":["package.json","bun.lockb","bunfig.toml","deno.json","deno.jsonc","deno.lock","global.json","NuGet.Config","Directory.Packages.props","mix.exs","mix.lock","go.mod","go.sum","stack.yaml","stack.yaml.lock","pom.xml","build.gradle","build.gradle.kts","settings.gradle","settings.gradle.kts","gradle.properties","package-lock.json","yarn.lock","pnpm-lock.yaml","npm-shrinkwrap.json","requirements.txt","Pipfile","Pipfile.lock","pyproject.toml","setup.py","setup.cfg","Gemfile","Gemfile.lock","uv.lock","CODEOWNERS","DESIGN.md","README.md","CONTRIBUTING.md","CHANGELOG.md","SECURITY.md","CODE_OF_CONDUCT.md","AGENTS.md","CLAUDE.md","GEMINI.md"],"protected_files_policy":"request_review","title_prefix":"[smoke-project] "},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"true"},"remove_labels":{"allowed":["smoke-project"]},"report_incomplete":{},"update_project":{"github-token":"${GH_AW_SECRET_GH_AW_PROJECT_GITHUB_TOKEN}","max":20,"project":"https://github.com/orgs/github/projects/24068","views":[{"name":"Smoke Test Board","layout":"board","filter":"is:open"}]}}
-          GH_AW_SAFE_OUTPUTS_CONFIG_3c30d1736c2958aa_EOF
+          cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << GH_AW_SAFE_OUTPUTS_CONFIG_dd2e3df9be1ec646_EOF


 # Container images used:
-#   - ghcr.io/github/gh-aw-firewall/agent:0.25.65
-#   - ghcr.io/github/gh-aw-firewall/api-proxy:0.25.65
-#   - ghcr.io/github/gh-aw-firewall/squid:0.25.65
-#   - ghcr.io/github/gh-aw-mcpg:v0.3.23@sha256:0dd1bd91a41e24a3ccc31b1ec6cb61d36608997fabf91f2d643b64e3fc33180a
-#   - ghcr.io/github/github-mcp-server:v1.1.2@sha256:30197479d8036c7811892bc07e06f9a05c9ef3cdd79bc59f256d50647f95788c
+#   - ghcr.io/github/gh-aw-firewall/agent:0.25.58
+#   - ghcr.io/github/gh-aw-firewall/api-proxy:0.25.58
+#   - ghcr.io/github/gh-aw-firewall/squid:0.25.58
+#   - ghcr.io/github/gh-aw-mcpg:v0.3.22
+#   - ghcr.io/github/github-mcp-server:v1.1.0@sha256:71b07d9abecb83b4a2595bcd8ccb35f9a0166361a12335f9e16da1ef07172029
 #   - node:lts-alpine@sha256:2bdb65ed1dab192432bc31c95f94155ca5ad7fc1392fb7eb7526ab682fa5bf14


Initial plan

0077536

Copilot AI assigned Copilot and pelikhan Jun 6, 2026

Copilot started work on behalf of pelikhan June 6, 2026 14:51 View session

Copilot AI linked an issue Jun 6, 2026 that may be closed by this pull request

[ab-advisor] Experiment campaign for smoke-project: A/B test prompt_style #37302

Closed

7 tasks

Copilot AI and others added 2 commits June 6, 2026 14:57

Plan: add smoke-project prompt_style experiment

2e480e6

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Add smoke-project prompt_style experiment variants

ec3f913

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add experiment campaign for A/B test prompt style~~ Add prompt_style A/B experiment to smoke-project workflow Jun 6, 2026

Copilot finished work on behalf of pelikhan June 6, 2026 15:05

Copilot AI requested a review from pelikhan June 6, 2026 15:05

pelikhan marked this pull request as ready for review June 6, 2026 15:07

Copilot AI review requested due to automatic review settings June 6, 2026 15:07

pelikhan merged commit 6872532 into main Jun 6, 2026

pelikhan deleted the copilot/experiment-campaign-prompt-style branch June 6, 2026 15:08

Copilot AI reviewed Jun 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prompt_style A/B experiment to smoke-project workflow#37326

Add prompt_style A/B experiment to smoke-project workflow#37326
pelikhan merged 3 commits into
mainfrom
copilot/experiment-campaign-prompt-style

Copilot AI commented Jun 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a2f85a8fdf159792c99dd73d46b646c60877a2885d36170b127e98df8cb2083f","body_hash":"a06527944f758e799e436bef444ec14e9c38546538220b343897f74877fb3c1c","compiler_version":"v0.77.5","strict":true,"agent_id":"copilot"}
		# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_CI_TRIGGER_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GH_AW_OTEL_GRAFANA_AUTHORIZATION","GH_AW_OTEL_GRAFANA_ENDPOINT","GH_AW_OTEL_SENTRY_AUTHORIZATION","GH_AW_OTEL_SENTRY_ENDPOINT","GH_AW_PROJECT_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"v0.77.5","version":"v0.77.5"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.58"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.58"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.58"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.22"},{"image":"ghcr.io/github/github-mcp-server:v1.1.0","digest":"sha256:71b07d9abecb83b4a2595bcd8ccb35f9a0166361a12335f9e16da1ef07172029","pinned_image":"ghcr.io/github/github-mcp-server:v1.1.0@sha256:71b07d9abecb83b4a2595bcd8ccb35f9a0166361a12335f9e16da1ef07172029"},{"image":"node:lts-alpine","digest":"sha256:2bdb65ed1dab192432bc31c95f94155ca5ad7fc1392fb7eb7526ab682fa5bf14","pinned_image":"node:lts-alpine@sha256:2bdb65ed1dab192432bc31c95f94155ca5ad7fc1392fb7eb7526ab682fa5bf14"}]}

Conversation

Copilot AI commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 6, 2026 •

edited

Loading