Performance has gotten quite bad over the past few days.
Performance has gotten quite bad over the past few days.
Page load times have been very slow for some communities, especially those hosted on other instances, and especially over the past few days. Not sure if this was related to the maintenance over the weekend. Here's some quick examples from a sample of 3 communities. I'm listing them in the order that I visited them (I'm not sure if images et. al. are cached across instances, but just in case):
- https://pawb.social/c/196@lemmy.blahaj.zone - Load: 6.8s
- https://lemmy.blahaj.zone/c/196 - Load: 655ms
- https://lemmy.world/c/196@lemmy.blahaj.zone - Load: 705ms
- https://lemmy.world/c/technology - Load: 705ms
- https://pawb.social/c/technology@lemmy.world - Load: 17.58ms
- https://yiffit.net/c/technology@lemmy.world - Load: 557ms
- https://yiffit.net/c/memes - Load: 557ms
- https://lemmy.world/c/memes@yiffit.net - Load: 699ms
- https://pawb.social/c/memes@yiffit.net - Load: 587ms
Of these three tests, we performed fine on one, but the other two were markedly slower. Refreshing the home feed (settings: Subscribed, New) has also been very slow (with load times in excess of 5 seconds being very common).
Is anyone else seeing this, or is this a 'Me' problem?
(I swear I don't only complain.) :D
👋 Apologies for the late reply, but we have been investigating and believe that this isn't an isolated case.
It initially appears like our database is being overloaded due to the vast queries that Lemmy runs for generating statistics, so we're trialing a hardware upgrade for a new database server backed by NVMe drives to hopefully improve the throughput.
We're not quite ready to go live with it yet, we're still ensuring everything is in working order and the database replicates successfully from the live copy to the new database reliably.
We're hoping to have this in place before the end of September and we'll post an announcement for maintenance when we go to do the switchover, as the change will also affect furry.engineer and pawb.fun.
3Excellent news! :3
1Great! Thanks for the update. I saw the announcement on Telegram about the database being at max load, figured that was probably related. Really appreciate all the work you do to keep us running!
1