PSA: Please do not upload any pictures until further notice
We have to move to object storage to have efficient image storage. We are currently at 70% of our disk storage.
We are sorry for the short notice and we don't know how long it will take though we believe it will take multiple hours.
Small post mortem :
The migration required us to be down - we thought we could run without images
It took much longer than initially anticipated
It broke at close to 2:30AM - 3 hours into the migration and I couldn't fix it so the server has been down doing nothing for hours now
We're booting back up from the backup made before the migration was attempted, we'll try another strategy.
There might've been some data loss in images, we're looking into it. In the meanwhile, if your profile picture or banner is broken, feel free to re-upload them.
Update :
Hi Beeple!
We're trying this again.
This time, Beehaw should remain available though no pictures will be able to be uploaded. The error will likely be weird because Lemmy will think it is possible but we will block the upload from happening.
We'll take a snapshot of the pictures before the migration in 60 minutes at 21:00 UTC - it should take around one hour, do the migration and testing on our own before shipping on Beehaw.
Once we've resolved all these kinks, Beehaw will momentarily go down and then back up with the migration complete without losing any old pictures.
Didn't go as planned of course, restored from backups, pre migration attempt. Thank you for your patience while we try to get all these moving parts working well together. Sorry for the troubles.
I once caused an AWS outage that impacted 20% of their customers in their largest region. They called my manager to ask why we were performing around 10k writes per second to a bucket. It was fun times
Thanks for the update and hope you have less trouble in the future! Don't worry about the downtime I really appreciate that here it's serving a clear purpose unlike Twitter lol
I appreciate the late night efforts and the clear communication. For me, Beehaw is a positive place I can visit, but there are other things I can do also, and I have no need for many 9s of uptime here. (I'm trying to reduce any pressure you & others might feel - perhaps not communicated it well tho, hence this addition.)
It could, and will. Hopefully they are taking advantage of CDNs for image delivery so they aren't paying high egress costs and can keep it in slow, cheap, storage.
I'm honestly surprised that Lemmy hasn't embraced distributed, community, hosting. Many existing niche communities (outside of Lemmy) operate with the ability for others to run their service to serve up images and media, or to act as workers for computationally expensive operations like compression or encoding (Which will also save you a ton of space). Even gamificating it in the case of e-hentai.
Hard drives (Tapes even more) at home/office are incredibly cheap compared to cloud storage costs (even including networking, server, redundancy...etc hardware costs), but come with reliability concerns, which is where a distributed community becomes critical. Though you'll always have to have them stored somewhere like Backblaze B2, or somewhere slower/cheaper/frozen to ensure safety.
I feel like Lemmy definitely needs to embrace distributed computing in some fashion. I have no interest in hosting my own instance, but I'm not against running a docker image that would offload some of the processing requirements large instances need. It would just need to be relatively straightforward for me to setup
Every technical bump in the road we hit now is one we won't hit/will know how to handle quickly in the future! Thank you for doing what you do for Beehaw!
Yeah, moving to object storage is best to do now. Arguably, we should've done it sooner since the longer we've waited, the more it was gonna catch up to us and cost us in time and money.