2 posts tagged with "team"

The eval set is a file. It is also, secretly, a theory of what the AI feature is for. The two are not the same thing, and the team that confuses them has built a quality gate whose calibration depends on a single human's working memory. When that human leaves, the file stays and the theory walks out the door.

This is the failure mode you don't see in the org chart. You scoped a prompt engineering role. You hired someone good. They shipped the v1 prompts, looked at the thin benchmark, and rewrote it into something rich — a taxonomy of failure modes, weights per category, a labeling rubric that disambiguates edge cases. The eval set became the contract for "is this model good enough to ship." Six quarters later you discover that the contract is unreadable by anyone except the person who wrote it.

• Listen to Your Customers

  • Continuously communicate with customers
    • Making friends with customers is the simplest strategy: once you become friends, you can easily and casually ask for their feedback. At a certain point, sending a short message is easier than emailing or having meetings.
  • Selectively choose your customers
    • Eliminate customers who do not feel the pain of the problems you solve
    • Remove customers who want features that you have not yet developed

• Execute Quickly

  • Define the right scope
    • Generally, meet the most basic use case, and then wait to see what else people want. You will naturally know what to do next.
  • Cultivate a good intuition for problem-solving
    • Product is a blend of art and science. Overly data-driven products lack soul and surprise. How to practice intuition? Validate ideas without using code as much as possible.
  • Maximize autonomy and promote a culture of prioritization by persuasion. At the same time, allow teams to choose their work based on their emotions, lives, and interests. Drive progress through persuasion and dynamic adjustments.