Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won't work as desired.. we see several issues:
Performance
Loading is mostly OK, but sometimes things take forever
We (and you) see many 502 errors, resulting in empty pages etc.
System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
2FA seems to be a problem for many people. It doesn't always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.
Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.
i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.
but please, don't feel forced to overwork yourselves.
i understand you want to do it soon so more people can move from Reddit,
but i wouldn't like that Lemmy software and community developers overwork and feel miserable,
as those things are some of the very motives you escaped from Reddit in first place.
in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.
this applies to all lemmy instances and communities, ofc.
have a nice day you all! ^^
I am very forgiving of the bugs I encounter on Lemmy instances because Lemmy is still growing and it's essentially still in beta. I am totally unforgiving of Reddit crashing virtually every day after almost two decades.
The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.
Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?
Cloud architect here— I’m sure someone’s probably already brought it up, but I’m curious if any cloud native services have been considered to take the place of what I’m sure are wildly expensive server machines. E.g. serve frontends from cloudfront, host the read-side API on Lambda@Edge so you can aggressively and regionally cache API responses, anything other than an SQL for the database — model it in DynamoDB for dirt cheap wicked speed, or Neptune for a graph database that’s more expensive but more featureful. Drop sync jobs for federated connections into SQS, have a lambda process that too, and it will scale as horizontally as you need to clear the queue in reasonable time.
It’s not quite as simple to develop and deploy as docker containers you can throw anywhere, but the massive scale you can achieve with that for fractions of the cost of servers or fargate with that much RAM is pretty great.
Or maybe you already tried/modeled this and discovered it’s terrible for you use case, in which case ignore me ;-)
Thank you for everything! Can we donate to cover the costs? If more people throw in $5, we all will benefit. Can Lemmy server scale up without needing RAM exponentially growing with the user number? I hope the system will get better optimized for bigger user base as the time goes on..
.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that's nothing in comparison.
You guys are doing an amazing work! I'm broke, so here are some coins 🪙🪙🪙🪙 beans 🫘🫘🫘🫘
I feel that lemmy runs smoothly the past few hours with very few hiccups and mostly on the upvoting and commenting side. I encountered no issues yet on the loading of the post.
I can't imagine how hard it is to rua server that has influx of large users. I Thank you for your hard work to run and maintain this instance. Hope it works out well and future will be a smooth sail.
To all the folks that are worried, don't be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: "Facebook wasn't built in a day." In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time... scaling is very difficult, but it's not impossible. The way things are going, Lemmy is going to thrive!
Thank you for the transparency @ruud@lemmy.world
It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!
Thanks for the transparency, and communication. I think it's always better when the userbase is able to understand what is going on rather than being left in the dark wondering what is going wrong. Keep up the good work, but also be sure you guys get enough rest, and take care of yourselves too.
Something changed in the last 24hrs that fixed my account! I couldn't login on desktop at all. Tried Chrome, cleared the cache, tried Firefox, tried an Ubuntu live USB - login never worked. Jerboa could access my account that was already logged in, but trying a new login would result in a HTML error. But today login worked perfectly, zero issues at all.
Amazing work team. I am already seeing improvements. Hope you are not killing yourselves though, I'm sure everyone realises how difficult it is and that it will take time to fix. We're here for the long haul! Thanks again
The good thing about the fediverse is we can sign up on federated servers and still interact with everything on lemmy.world without the errors, then come back to our lemmy.world accounts when everything is smoother
As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.
As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth
@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻
This has been such a refreshing change from reddit! I'm willing to deal with occasional hiccups, and I know it will only get better with time. Keep it up!
I recall that previously we couldn't update jerboa due to it not being compatible with the last server version. Is that updated now, can we update Jerboa?
Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here
The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something
Thanks for your hard work. Noticing janky bits getting ironed out everyday. Has anyone had any luck making Lemmy.world work in Qutebrowser? It just hangs on loading comments/posts..Any work arounds? I'd love that..working quite well in Firefox and mobile though.
System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
That DM sending bug was already fixed in a later rc version. Although in rc.9 (and possibly rc.10) there is a weird bug where you can't sign out. Fun stuff.
Thank you for all of your effort on this. Adjusting to rapid growth is a huge challenge, and we all appreciate the effort you and everyone else is putting in to trying to make this instance run smoothly.
I love the community here and am excited to see it grow. Thanks for giving a report, it awesome how invested y'all are and makes me want to stick with this instance even through the hardships. Keep up the good stuff!
I hope hosting all these massive communities is enough a portfolio to land Ruud a crazy job offer one day that pays him stupid levels of money while allowing him plenty of time off to maintain these massive hobby projects.
I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.
Sometimes when I post a comment (without any image) it takes forever to post and sometimes it keeps spinning and I need to cancel and post again. My internet connection is fine (100 Mbps) and it works flawlessly on other websites. Is it happening due to overloading on the lemmy server?
around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
25 GB of RAM?????
Looks like your instance is the edge case with Lemmy's scalability, when there's really big amount of active users. Looks like some problems won't fade with really high-grade hardware. But I have to appreciate, that's one gargantuan job!
Keep up the great work! We believe in you! Just a reminder to everyone to help sustain lemmy.world by donating in the sidebar! If 1000 people all donated $2/month then that's 24K a year to keep the hardware running! Reddit is what happens when you rely on Corporate Overlords to provide the services you love.
I'm sorry to hear you're using Discord, for me it fits right in with the likes of Reddit, Twitter and Facebook.
Why not use something independant and federated like Matrix or XMPP?
Hi, am I correct that typing on iPad currently doesn’t work? As in the keypad won’t show up. I have to type in google or notepad and paste (the paste popup still works) to lemmy.world lol! Anyone else?
As always, thanks for the hard work everybody puts into this!
Curious about the server info. Is there more info available somewhere else?
Is lemmy.world running on a single server?
What's the CPU and memory capacity? You mentioned a percentage of CPU usage and an amount for ram, so that made me curious about what's available for each.
Does restarting take down all services or is the application clustered or not monolithic somehow?