There won't be a big WAN Show segment about this or anything. Most of what I have to say, I've already said, and I've done so privately.
To Steve, I expressed my disappointment that he didn't go through proper journalistic practices in creating this piece. He has my email and number (along with numerous other members of our team) and could have asked me for context that may have proven to be valuable (like the fact that we didn't 'sell' the monoblock, but rather auctioned it for charity due to a miscommunication... AND the fact that while we haven't sent payment yet, we have already agreed to compensate Billet Labs for the cost of their prototype). There are other issues, but I've told him that I won't be drawn into a public sniping match over this and that I'll be continuing to move forward in good faith as part of 'Team Media'. When/if he's ready to do so again I'll be ready.
To my team (and my CEO's team, but realistically I was at the helm for all of these errors, so I need to own it), I stressed the importance of diligence in our work because there are so many eyes on us. We are going through some growing pains - we've been very public about them in the interest of transparency - and it's clear we have some work to do on internal processes and communication. We have already been doing a lot of work internally to clean up our processes, but these things take time. Rome wasn't built in a day, but that's no excuse for sloppiness.
Now, for my community, all I can say is the same things I always say. We know that we're not perfect. We wear our imperfection on our sleeves in the interest of ensuring that we stay accountable to you. But it's sad and unfortunate when this transparency gets warped into a bad thing. The Labs team is hard at work hard creating processes and tools to generate data that will benefit all consumers - a work in progress that is very much not done and that we've communicated needs to be treated as such. Do we have notes under some videos? Yes. Is it because we are striving for transparency/improvement? Yeah... What we're doing hasn't been in many years, if ever.. and we would make a much larger correction if the circumstances merited it. Listing the wrong amount of cache on a table for a CPU review is sloppy, but given that our conclusions are drawn based on our testing, not the spec sheet, it doesn't materially change the recommendation. That doesn't mean these things don't matter. We've set KPIs for our writing/labs team around accuracy, and we are continually installing new checks and balances to ensure that things continue to get better. If you haven't seen the improvement, frankly I wonder if you're really looking for it... The thoroughness that we managed on our last handful of GPU videos is getting really incredible given the limited time we have for these embargoes. I'm REALLY excited about what the future will hold.
With all of that said, I still disagree that the Billet Labs video (not the situation with the return, which I've already addressed above) is an 'accuracy' issue. It's more like I just read the room wrong. We COULD have re-tested it with perfect accuracy, but to do so PROPERLY - accounting for which cases it could be installed in (none) and which radiators it would be plumbed with (again... mystery) would have been impossible... and also didn't affect the conclusion of the video... OR SO I THOUGHT...
I wanted to evaluate it as a product, and as a product, IF it could manage to compete with the temperatures of the highest end blocks on the planet, it still wouldn't make sense to buy... so from my point of view, re-testing it and finding out that yes, it did in fact run cooler made no difference to the conclusion, so it didn't really make a difference.
Adam and I were talking about this today. He advocated for re-testing it regardless of how non-viable it was as a product at the time and I think he expressed really well today why it mattered. It was like making a video about a supercar. It doesn't mater if no one watching will buy it. They just wanna see it rip. I missed that, but it wasn't because I didn't care about the consumer.. it was because I was so focused on how this product impacted a potential buyer. Either way, clearly my bad, but my intention was never to harm Billet Labs. I specifically called out their incredible machining skills because I wanted to see them create something with a viable market for it and was hoping others would appreciate the fineness of the craftsmanship even if the product was impractical. I still hope they move forward building something else because they obviously have talent and I've watched countless niche water cooling vendors come and go. It's an astonishingly unforgiving market.
Either way, I'm sorry I got the community's priorities mixed-up on this one, and that we didn't show the Billet in the best light. Our intention wasn't to hurt anyone. We wanted no one to buy it (because it's an egregious waste of money no matter what temps it runs at) and we wanted Billet to make something marketable (so they can, y'know, eat).
With all of this in mind, it saddens me how quickly the pitchforks were raised over this. It also comes across a touch hypocritical when some basic due diligence could have helped clarify much of it. I have a LONG history of meeting issues head on and I've never been afraid to answer questions, which lands me in hot water regularly, but helps keep me in tune with my peers and with the community. The only reason I can think of not to ask me is because my honest response might be inconvenient.
We can test that... with this post. Will the "It was a mistake (a bad one, but a mistake) and they're taking care of it" reality manage to have the same reach? Let's see if anyone actually wants to know what happened. I hope so, but it's been disheartening seeing how many people were willing to jump on us here. Believe it or not, I'm a real person and so is the rest of my team. We are trying our best, and if what we were doing was easy, everyone would do it. Today sucks.
Not sure if this gets seen by anyone from LTT, but I’m a regular consumer of their products (ie., their YT content).
I find that more and more of their videos don’t seem to end with some satisfactory conclusion, and quite a number of it is way too janky when with a little more thought and planning, could have been done and concluded much better.
Take for example the fanless heatsink dipped in ice bath video, or the car radiator water cooling video. It’s always them trying to fix some jank or scramble on camera to fight some spontaneous issues or leaks or whatever. It’s fun, yes, entertaining, yes, but utterly unsatisfactory in the end. Reminds me of a post-nut sad depression wank.
Ultimately this is probably a good thing. From what I've gathered of Linus he is a person that can recognize his mistakes, but only once you get past his stubbornness. His response saying they should've hit him up is kind of classic response. What could you have said really? Like GN is right that like almost every video has a pinned comment or on screen edit of a correction. And you know these companies are complaining. Yet it continues.
Having their audience put them to task will go a long way to making them recognize it is a bigger issue than individual videos. I'm really curious about the idea of Labs being able to actually test things publically in a way consumers haven't been able to. (Give me more of transparency like digital foundry for games and Jerryrig everything for durability.) So they need to pull their shit together if they want to claim to be a data driven lab. Like accuracy is all that matters.
I think the ethical concerns section is overblown. LTT is almost harsh on their sponsors and I got a kick out of Linus starting the last video talking about Framework saying how he doesn't actually use the Framework laptop as his daily driver at the time and complained about it.
For some great irony check out this wan show segment where linus talks about how he doesn't like that a prototype (backpack) of theirs ended up in the hands of the public
https://www.youtube.com/watch?v=EwgZaSYuBLc&t=3209
Time 53:29, in case the timestamped link doesn't work
How Linus publicly responds to these very fairly laid out criticisms will really affect their standing in the tech review space going forward.
Linus generally sucks at taking warranted feedback & criticism, so I can see him crashing and burning super hard in whatever post or podcast comment he makes publicly about this.
This looks like a huge issue as far as moving from a "haha wacky video" tech channel to a "hard data driven testing" tech channel, but also it's not like they haven't done "serious" reviews prior to the Labs stuff in the past so I'm not about to hand wave away their issues as "growing pains" or anything like that; it's just indicative of sloppy workflow and low effort internal culture.
What I thought was interesting about this is that apparently not only me, but tons of other people have all kinda felt like LTT has seemed off the past year or so, and this video kinda explained it all
The one that gets me the most is the Billet Labs copper water cooler... Like holy shit how big of an asshole do you have to be to NOT return to them what you agreed to return, and then to just auction it off
I watched LTT for Top Gear of computer hardware videos. This commentary from GN made me wonder when was the last time they did one. I couldn't remember and unsubscribed.
I'd be interested in videos by Alex, Nicholas and maybe few others. Hope they go their own way some day. Alex videos are probably quite expensive to do so he's probably stuck there :/
LTT needs to make sure quality catches up to quantity wrt their yt videos. Essentially, they need:
More technical experts/researchers to find and filter news/info before it goes into the script.
Quality control after shooting a vid, i.e. they must be willing to reshoot segments with wrong info, not just put a note on the screen or in the yt comments
Both requires a huge investment in manpower and studio space. As it is, they are short staffed and juggling sets for shoots. If Linus wants to go on this path of business growth, he has no choice in this matter. They can't go on like a small garage operation anymore.
Linus has already put out a response and it's not great either. There's some points of contrition, that it was done on his watch so ultimately he's reasonable, but overall he justifies their actions as a company coming into it's stride and being transparent with their mistakes (instead of them just rushing out content without proper vetting)
It's not a great look, but it's one I've come to expect from him given how he's handled previous community backlash with the backpack, screwdriver and other scenarios.
I watched a couple of Linus videos over the years and recently subscribed. I had no idea how many videos they push out a week - that is something they should really adress. Rather make three or four good videos a week than the absolute crap content they partially put out just to get their schedule full.
After this whole thing now I unsubscribed again. You can absolutely fuck up a situation and be good in my book if you own up to it, but Linus response is childish at best and a disaster at worse.
I know these guys are too busy to do it but I would gladly pay a monthly fee for a podcast with Steve (GN), Steve (HUB), Wendell (L1) and Gordon (PCWorld).
It was really obvious to me Linus was full of shit when he pretended to give Linux a try but then defied a scary error message warning him not to continue which he did anyway and it wiped his desktop environment completely.
He then basically implied if you couldn't just randomly copy paste cli commands without understanding them AND ignoring a scary warning message, then Linux just was too sketchy for average users to use. It would be trivial to make that exact argument about windows... Because it's a bullshit meaningless argument
Anyone with even the tiniest experience with a cli, imo, should've known at that point he's full of shit. Fuck their shitty videos.
I watched this yesterday and even I was cringing for both of them because i like LTT but GN's points are valid, but also you can tell this was a very tough video for GN to make.
There's a certain tone or cadence to the way that Steve talks that really makes him sound like an authority on the subjects he's talking about. Like an old school radio reporter from the 30s or something. Not sure exactly how to describe it, but once you hear him say something, you know it's a very serious topic.
Either way, I think GN made some very valid points, but also some much less so. I mostly watch LTT for the entertainment value but I've never taken their reviews particularly seriously. I've always hated how LTT handles charts and data - they just zip through the data so bloody fast. At LTT it is clear that Linus knows his stuff, but it's also clear that he's not trained as a scientist or engineer in the scientific method. Listening to him, yiu can tell there's a lack of knowledge in the methodology of setting up an experiment, but on the other hand if you step back for a second, you realize how ridiculous some of the topics all these reviewers obsess over really are. 5 FPS in a game that's already getting 150+ is utterly pointless in the real world. These are just games, guys. It's not the end of the world here.
I'll continue to watch LTT, mostly for TechLinked and now GameLinked because Riley, James and now Jacok crack me up.
All valid points. But regarding the measurements, neither LTT or GN testing methodology are realy scientific. Those are youtubers with limited understanding of experimental design and analysis. I have never seen them do simple significance tests or try to explain the variability in the data. But I havent watched all the videos so 乁( •_• )ㄏ
I love LTT, but Linus has always been a marketing guy first. The whole purpose of the Tech tip videos was to move product, and with the seventeen ads per video it sorta still is.
Although good points were made, as I've been following this. I think its also suspicious GN is rather new with their ethics pages etc and it seems like it was a planned attack.
LTT does do corrections, but also GN alone has had 3+ in 2023 errors. Now account for the absolute size difference...
I get the whole video, it has a lot of stuff that is bad for LTT, what bothers me is the editing style. It feels more like a hit piece than a piece of criticism with how the cuts are made. If this was to be a video calling out the behaviour and actions of LTT and their channels I think it could have been edited in a way that didn’t seem to have constant cut ins from various videos mocking them. I could see a different cut done by a different editor that took the whole project seriously that would be more effective.