mhoyeJust saw the CEO of GitHub say “we just did an unpublished study with 22 people in it and it’s official, AI is the future” and I had to imagine @grimalkina@mastodon.social suddenly sitting bolt upright, sniffing the air and looking around suspiciously for reasons she couldn’t immediately explain.
Cat Hicks@mhoye@mastodon.social oh no
Cat Hicks@mhoye@mastodon.social this is so funny because when I interviewed ~22 developers I came up with "learning is important and the culture of this field is really suppressing it"
Cat Hicks@mhoye@mastodon.social interesting that Eirini Kalliamvakou is an author on this I will have to go find the paper...? Is there a qualitative interview paper underneath this blogpost??
Cat Hicks@mhoye@mastodon.social she doesn't have anything new listed on her page since 2024 so I'm betting this is something that'll get packaged up later or maybe it's an ongoing project sliced into for a leadership take idk.... Impossible to evaluate this as a qual study without any detail 🤷♀️
I like interview studies plenty but this Microsoft research playbook of "do interviews --> claim generalizable personas --> call it a framework" is really not one I agree with
Cat Hicks@mhoye@mastodon.social fwiw ? I'm a little out of date because busy living lol but qual interview studies from software research are mostly producing commentary on confusion, complex interaction effects, and extremely mixed positions from developers which is about what I'd expect given how I see my community talk about it all
https://arxiv.org/abs/2507.21280
Cat Hicks@mhoye@mastodon.social
"Our findings imply that widespread organizational expectations for rapid productivity gains without sufficient investment in learning support creates a "Productivity Pressure Paradox," undermining the very productivity benefits that motivate adoption."
Hmm almost like I was right about the core psych needs when I interviewed developers BEFORE genAI 😃
Cat Hicks@mhoye@mastodon.social worth noting that the preprint I linked is specifically with developers at a big tech co that's pro AI adoption!! So even within that population finding this
Insecurity Princess 🌈
💖
🔥
@grimalkina@mastodon.social @mhoye@mastodon.social
a lot of big oofs as I'm reading this
Notably, many of the quotes suggest devs are making the same programming mistakes: assessing usefulness for writing new code. What makes software engineering (not programming) hard in the long-term is writing code that is maintainable and building systems that are reliable and resilient.
There's a lot of holes in what they're assessing. The conclusions are sometimes a leap from the evidence they present, as though they're jumping from pulled quotes to match the agenda of the authors.
"Consistent leadership messaging encourages AI exploration. Without it, some developers are hesitant to use AI." — I've never needed to encourage software engineers to spend time learning or exploring a new tool. They choose how to spend their time, and part of that includes learning, but I don't TELL them which IDE or keyboard to use.
This is a dangerous study framing because their conclusion is that not investing in learning undermines productivity benefits, but still no one has clearly let alone robustly demonstrated productivity gains beyond toy problems, let alone reliably and sustainably at scale for maintaining secure and highly available systems in the long-term.
We have dozens of methods for creating measurable productivity gains in software engineering teams for a few months. A pizza party every Friday would create measurable productivity gains. Or giving genuine praise and encouragement once a week. Or raises and bonuses. Or job insecurity — whoops guess that's already on the table as a confounding variable 🙃
[/cranky rant]
Cat Hicks@saraislet@infosec.exchange @mhoye@mastodon.social I don't read this argument as "investing in learning is counter to productivity" but more that "pushing for short term productivity without a supportive learning structure can negate the potential "productivity" benefits," no?
Insecurity Princess 🌈💖🔥@grimalkina@mastodon.social @mhoye@mastodon.social
Oops, I meant to write "their conclusion is that NOT investing in learning undermines productivity benefits".
I've seen no scientifically well supported evidence that LLMs improve productivity outside of limited use cases (which is useful! but the magical thinking about LLMs is unbelievable)
At most organizations I've known, devs have supportive learning structures. They learn languages, tools, infrastructure, new concepts and practices, etc, regularly through their careers.
But usually devs do that because they see clear value to spending time learning a new language or tool, or even developing new tools that make our work more efficient or effective.
Cat Hicks@saraislet@infosec.exchange @mhoye@mastodon.social I understand why you're frustrated with the state of the conversation and evidence. I am too, and I agree that the "productivity" investigations are super lacking. Really, I engage with qualitative interview studies of this sort as interesting artifacts and mostly read the quotes while leaving the researcher frames as you know, their frame, which I like more in some cases than others. In this particular case I thought it was an interesting contrast to the assumed unanimous...
Cat Hicks@saraislet@infosec.exchange @mhoye@mastodon.social ...stance from the CEO quotes, when even similar qualitative work with a sample of very similar people surfaces a lot of conflicting opinions! I read these qual studies less as statements of ground truth and more as investigations into perceptions.
On the learning environments piece though, I do think this can be pretty lacking for a lot of devs and that paying attention to it is helpful. It's a stated value but I think cultures that truly incorporate it are rarer in tech.
Cat Hicks@saraislet@infosec.exchange @mhoye@mastodon.social but I see your point that we should look to the natural behaviors of developers as a signal of what's easily and readily relevant to them though, and totally agree. I mean tool adoption is difficult and I don't like to take it as a proof of efficacy either (people can flock to stuff that is bad for them) but I think we're probably aligned on that.
Insecurity Princess 🌈💖🔥@grimalkina@mastodon.social true facts, good points, thank you 💖
My cynicism gets the better of me sometimes. It's disappointing to see a low degree of strategy in investment choices and more everything-everywhere-all-at-once