Content design as prompt engineering for conversational search
I worked behind the scenes so a talented, ambitious designer on my team could thrive in an ambiguous, new space. eBay’s conversational search started as unplanned vision work and grew into one of the company’s most visible AI projects.
Role: Content design manager and project lead
Partners: A senior content designer and a principal content designer, data science, search product management and design, AI innovation team, product marketing, leadership
Time frame: 2025–2026
The challenge: redefining content design
Conversational search brought together two teams that worked very differently. AI Foundations wanted to explore what the technology could do, and the search team wanted to put specific information in front of customers. When the ask came to my team, I staffed it with Barbara, a Berlin-based content designer we had hired in part for her prompt engineering experience. She had been working on a chat-based shopping assistant, and this flow was a logical continuation of the same thinking and partnerships. What changed was where the experience started, in the search bar, and the executive attention on the project.
Most stakeholders saw content design as button copy. Barbara worked as a prompt engineer. She co-wrote the core prompt with data science, applied UX research, and developed the evaluation criteria behind an LLM judge. It was a new way of working for the organization, on a high-stakes, highly visible project. I helped her articulate the process, sell it to leadership, and navigate stakeholder feedback.
Where we started
We began the way most teams do with generative AI. A product requirement became an agent prompt, the prompt produced an output, and we looked at the result. If something didn’t work, we changed the prompt and tried again.
Content design entered the loop at the review step. Take a simple question: “How do I get a refund?” The model’s answer held useful information, but it came back as one large block of text with no clear hierarchy or sequence, and the key information competed with supporting detail. A user had to work to find the answer.
We didn’t rewrite the response ourselves. We changed the instructions: use paragraphs, number the steps, bold what matters most, be more concise, and lead with what the user needs to know. The revised output was much easier to scan and act on, and the supporting detail was still there but easier to find. We also leaned on a few basics, like showing examples, setting boundaries, and keeping terminology consistent. At first, that worked well.
Where it stopped working
Tweaking by example eventually wasn’t enough. A rule that fixed one output could change behavior in situations we hadn’t looked at. Reviewers also disagreed: the same response was too long to one person, too thin to another, and missed the user’s actual need for a third. And a handful of examples couldn’t tell us how the system would behave across different queries, contexts, and edge cases. We needed a way to tell whether a change improved the system, not just one response.
How the process changed
Content design moved in front of the prompt. Content strategy and style guidelines became inputs to prompt engineering, alongside the PRD.
We agreed on what “good” means. The cross-functional team defined shared evaluation criteria instead of choosing one ideal response. Did the response understand the user’s intent and constraints? Did it give grounded, meaningful value? Was it clear and easy to scan? Did it help the user move forward? Safety was a baseline. Examples helped us calibrate the criteria, but the criteria set the bar.
We built an evaluation set that represented the experience. It included common requests, different user contexts, edge cases, and places where the system was known to struggle, so a strong score couldn’t come from easy queries alone.
We separated defining quality from applying it. Human reviewers, including content design, applied the criteria to examples, compared judgments, and worked out where they disagreed. An independent evaluation team then tested the system against the criteria, and an LLM judge applied them at scale. People defined and calibrated the judgment, and the judge applied it.
We turned results into hypotheses. When a dimension scored low, we looked at examples to see why. For clarity, important information was often buried. We formed a hypothesis, such as stronger response hierarchy. Then we changed the prompt, context, or behavior and evaluated again to check that nothing else got worse.
The prompt stayed important, but it became one input to a loop: define the experience, design the behavior, generate, evaluate, learn, and improve.
What I did
Planned the work and articulated swim lanes. I wrote a staffing brief covering what content design meant on this project, what we’d take responsibility for, and the swim lanes among the three content designers. It became a key source of alignment on an ambiguous project.
Gave my direct report air cover. Barbara focused on the prompt and evaluation work, and I handled much of the work around it. I coached her on presenting to each audience, from in-the-weeds product partners to VPs. I also advocated for her with stakeholders, especially when competing narratives ran along reporting lines. Lisa, a principal content designer, helped cover time zones and leadership presentations.
Coached Barbara to demystify her work. At one point, Barbara’s documentation had grown into pages of prompt iterations and notes. That made sense as a private workspace, but expert terminology and a messy archive were hiding her impact from stakeholders. Lisa and I helped her rebuild it around what stakeholders needed: key content strategy principles written for human readers, not an LLM, and a decision log showing the feedback we heard and how it would change the prompt.
Contributed to the evaluation process. Barbara, Lisa, and I co-designed how we evaluated responses: writing criteria, judging outputs against them, and building toward an LLM as judge. Weekly review meetings were meant to bring together content design, product design, data science, product marketing, legal, and product to dig into specific feedback. For partners who came, the impact on the prompt, the outputs, and the judge was immediate.
Kept customers in the room. UX research was an excellent partner, bringing dozens of iterations to customers in weekly sessions. The findings validated some assumptions and challenged others. Customers mostly wanted the agent for the same jobs as our existing features. Some stakeholders wanted to ship what they knew the LLM would be good at, like detailed getting-started guidance for new card collectors. UX research kept us focused on the real needs of customers.
Outcome
The team built the agent in a new way, with content design acting as prompt engineer. The next step, which Barbara is working on, is putting eBay’s other agents onto one framework. For context, eBay’s 2026 Connect presentation reports the shopping agent launched to 3% of US native users and was evaluated through an LLM judge.
Epilogue: Telling the story in public
I pushed Barbara to tell the story of content design as prompt engineering, which was unusual at the time. She gave the talk at Future Product Days in Copenhagen and at a UX conference in Vienna, and presented an internal version at eBay. It showed other content designers a path for the discipline and showed other disciplines what content design brings. It was also a key case study in my case for Barbara’s promotion from senior content designer to staff content designer.