Week 38: AI product evals, adoption, new jobs and AI 2027
Your weekly roundup of all things product management, technology and work life.
WE’RE BACK! Happy Friday everyone! Welcome back to The Millennial Product Manager, your weekly product manager newsletter.
I'm using this newsletter to share at least three things l've come across this week to help you build better product. Subscribe so you don't miss these when they come out:
On to this week's thoughts and updates:
1. FINALLY hitting a stride
Look gang, starting a new job is just hard. You have to learn about the people you work with, the technology and product you’re working on, the industry you’re working in, the current state of things…it’s not easy. It takes time to get a solid footing.
In my case, the last six months I’ve been getting onto the on-ramp of this new crazy world we find ourselves in with AI. I went from being a Product Director to essentially becoming an IC Technical PM again. And on top of all that, I had to learn about the construction industry. And I’m 41 with a family! My brain isn’t what it used to be lol. Not a small feat, but here we are six months into it and I finally feel like I can hold my own.
If you’re on this journey too, just know that every day and every week things get better and better. You’ll be the expert in the room again before you know it!
Speaking of learning more about building AI products…
2. Stable, clean eval data is crazy important
Evals are super important for judging how your AI-based product works. The underlying model you’re using is trained and tuned by the model provider of course. But your job is essentially to teach the model how to meet the needs of your customer for your specific use cases.
“Evals” or evaluations evaluate how your application performs once you’ve given the underlying model instructions via the prompt and context via selected documents or endpoints like a search endpoint for example. It’s a lot like giving someone a test, and you pass and fail them based on how well they answer the questions on the test.
What I’ve learned lately is that it’s important to have a stable, clean dataset to test against. Why? If we use the test analogy, imagine administering a test but the answers keep changing. It’s going to be hard to truly evaluate that person’s understanding of the material right? Right.
Stable, clean data sets for evals keep your answers consistent and predictable. You know how the answers are shaped so you can properly evaluate the product experience. And, you can change the eval, or test in the example, when you know for sure that the underlying data set has changed.
So, if you’re building a product, make 100% sure you have a stable, clean eval set! One problem solved for your product, on to the next problem: adoption.
3. Adoption still doesn’t solve itself
I’ve had a few meetings in my career now where someone says something like “oh adoption is easy, just slap an in-product notification up there and we’re good! If the product is good users will just show up!”
Nothing could be further from the truth. This ain’t Field of Dreams yo!
AI is amazing, but there is no shortcut to solving adoption problems. YOUR PRODUCT HAS TO HELP DRIVE ADOPTION. That has been true since I started as a product manager 15 years ago, it is true today, and it will be true on into the future until AI become sentient, all-knowing and all-powerful. But by then we’ll be dead so it won’t be your problem anyway! lol
Seriously though, if you need adoption of your product…get it on your roadmap. Dedicate product energy towards it. Otherwise it’s not happening.
4. Recommended reading: AI 2027 (there’s a podcast version!)
Working in the AI field, I have to admit that I am simultaneously amazed and worried by the breakneck speed of progress with AI. It seems like every month we’re hurtling closer and closer to many unknowns related to our world with AI in it.
Which is why I found AI 2027 to a thought-provoking exploration into those unknowns with the lens of only a few years out. It’s hard to say what’s going to happen beyond 2027, so I’m appreciative of how this content was approached.
I realize this has been out for a while now, and in true dad-of-young-kids fashion I *just* got around to reading it lol but if you haven’t yet I’d recommend giving it a read (finally!) And if you’re like me and haven’t been able to sit down and read a book in 6 years, I highly recommend the podcast version of it.
It’s not as rich as the site version of course but I was much more likely to finally read it so there!
That's it for this week!
If you've found this helpful, please consider showing your support by subscribing:
I’ll be back around this time next week with more useful product manager things!