[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back
[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back
We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn't work.
I'll create an issue for the devs and retry once it's fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..
Thank you for the transparancy!
3I love post like this, another reason of why this instance is so good
3
Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.
Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though
any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?
2Thanks for all your hard work @ruud@lemmy.world. It's not easy!
1Thank you so much for all the hard work here, I know how frustrating attempting an upgrade like this can be.
1As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.
We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.
1Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?
1Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!
Dessalines:
Nutomic:
1Sucks that you couldnt get it working yet, but there are bound to be problems at this scale.
Anyone have any resources for learning what running a lemmy server is like ? Seems pretty interesting
0If you're running a small server it's pretty straightforward and hands-off. It's only when you get to the scale of these larger instances that you run into issues. I just rerun ansible every time there's an update and it otherwise just manages itself.
1
Good luck! May the software gods bless you with a functioning website with minimal bugs!
0- 2
was there a test environment? why rc in production?
0We are the test environment. You get what you pay for.
1
@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?
With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.
0Sucks that lemmy.world won't be working with Jerboa when the reddit apps quit working in a couple days.
0On the plus side, I'm about to open the beta that handles both just fine (plus a lot of features no one else is doing yet). The deadline I for myself is Saturday when the Reddit clients go down.
Unfortunately it was built by someone who lurks and comments, and I realized even though merging feeds from accounts on different servers is going to be awesome, other people like to post and I need to stop getting off track
The stress of the timeline is starting to get to me and I'm terrified no one is going to give it a chance, but I've fixed most of the things that drove me nuts in jerboa... That should count for something, right?
0
