Firstly, the maintenance for Pawb.Social services has been completed and we've upgraded Lemmy to 0.19.6-beta.13, and our Mastodon instances to v4.3.0.

Thanks for your patience.

----

and now, onto the "Toastify is awesome" (because it is. Totally.) postmortem:

Short summary: Crash did a stupid with the config and emails weren’t getting sent (oops). Emails should now be working again, and registration shouldn't require admin intervention.

Longer story:

Lemmy has several configuration options that need to be updated to correspond to the specific instance that's running it. For example, here's a snippet of Pawb.Social's config:

{ database: { host: "the.best.db.url.internal" user: "pawbsocial_lemmy" database: "pawbsocial_lemmy" password: "pawbsocial_do_you_really_think_i_would_leak_this" pool_size: 128 } hostname: "pawb.social" pictrs: { url: "http://pawb-social-pictrs:8080/" api_key: "sekrit" } email: { smtp_server: "email-smtp.us-east-2.amazonaws.com:587" smtp_login: "pls_send_emails" smtp_password: "jeff_im_begging_you" smtp_from_address: "Pawb.Social Lemmy <{{domain}}>" tls_type: "starttls" } }

Can you spot the problem? (No, it's not my awful password, or the pleading I'm doing with AWS to send emails so y'all stop having email problems.)

Originally, Lemmy was deployed through docker-compose which uses variable replacement to change something like {{domain}} to pawb.social automatically. Crash swapped away from using docker-compose due to repeated issues with trying to deploy the Lemmy changes and opted for a more bare Docker configuration.

Can you guess what didn’t get changed? …Yeah, the {{domain}} part was never removed and replaced with pawb.social so Lemmy correctly caught the error and threw an exception. The unfortunate part is that the thrown exception caused the whole comment process to close the connection to the web proxy server which returned a 502 Bad Gateway, which gave the UI HTML instead of a JSON response. The response handler in the UI then tried to figure out what was wrong, failed to translate the HTML to JSON, and gave the default error message “Toastify is awesome!” (which it is. Objectively.)

HOPEFULLY. This does fix the issue. I can’t guarantee it, but I’ve been able to replicate it by replying to someone else’s comment (which usually triggers an email notification) and after making the change, it successfully sent off the comment with a happy green confirmation toast.

We've successfully upgraded Lemmy from v0.19.3 to v0.19.5 and have a bunch of changes to share!

Bug Fixes

• Previously users were reporting that the web client kept logging them out. This was suspected to be due to a misconfiguration of the cookie storing the session information and should be fixed.

• Some communities were slow to load. Federation changes in v0.19.4 and v0.19.5 should rectify some federation issues. Additionally, we increased the number of database connections for Lemmy and will gradually increase this if we identify any further slowdowns.

(Longer term, we're planning to upgrade the database hardware which should improve the performance for all services)

Local Only Communities

When creating or editing your communities, you'll now see a "Visibility" option that allows you to choose between "Public" and "Local Only".

The "Public" setting will continue to operate as normal and federate your community to all other instances we federate with.

"Local Only" will disable federation for your community and restrict the community to being viewed by local logged in users only; Logged out users may not be able to see your community until they log in.

Image Proxying

Once we've established that the database is no longer under substantial load and the performance issues previously experienced are fixed, we'll be enabling an image proxying mode which will allow us to locally cache images from remote instances rather than serving them directly.

This will allow us to improve the performance of loading images without applying additional load to other instances, and prevent remote bad actors from trying to scrape IP addresses.

Other changes

There are a handful of other miscellaneous changes that you can view in the full change logs:

• https://join-lemmy.org/news/2024-06-07_-Lemmy_Release_v0.19.4-_Image_Proxying_and_Federation_improvements

• https://join-lemmy.org/news/2024-06-19_-Lemmy_Release_v0.19.5-_A_Few_Bugfixes

We’re aware of an issue affecting our Lemmy service (pawb.social) that seems to be causing users to be randomly logged out. We believe we’ve identified the issue thanks to a bug fix in a newer version of the UI code.

We’re needing to verify there are no other breaking changes for the upgrade process, and expect to perform maintenance this Sunday, Sept 22nd at Noon US Mountain Time (https://everytimezone.com/s/7abd4955). We’ll provide an update tomorrow to confirm the maintenance window.

Thank you to everyone who brought this to our attention and we apologize for the inconvenience.

After reviewing a report regarding the personal account of the admin of tenforward.social, and discussing with other fediverse admins, we're deciding to suspend the admin and limit tenforward.social until this blows over. I'll explain why on both, starting with the admin themselves.

Suspending packetcat@tenforward.social

Rule 1: Threats of Violence

packetcat made several posts[1][2] that were perceived as a threat of violence, metered by it being "theoretical". This is not acceptable behavior for any user, especially an instance administrator, irrespective of the intended target of their annoyance.

This comes across as extremely volatile and dangerous, and on any other platform (except the cesspool that is X / Twatter) would be immediately removed.

Rule 6: No Spam

packetcat appears to have been going on a run of posts using several hashtags excessively in ways we, and other instances, have previously actioned as abuse / spam.

In the first case, the #fediblock hashtag was being used for its intended purpose to inform and make other instances aware of abusive conduct by other users. More specifically, their posts[3][4] were about Eris (originally from disqordia.space)[5] -- None of this is at issue as there's a laundry list of poor and "edgy" conduct that the historical receipts appear to cover, and privately corroborated by other instance admins we're in contact with.

However, that then began to devolve into airing their grievances generally and repeatedly escalating their posting when users began to report them, unaware they were the instance administrator of tenforward.social[6][7][8][9][10].

Later, they made two posts[11][12] with the #Mastodon4Harris hashtag for seemingly no other reason than one of their users said they should[13] and wanting to instigate drama[14]. This is just petty and childish trolling.

Rule 3: Harassment of Other Instances and Users

As tenforward.social received reports from other instances, packetcat would post the report[3][6] and in some cases encourage the administrators of other instances to take action against the reporters[10][15]. In the case of latter request to remove content, the linked post has been deleted so assumed to be a valid request to remove content, but their reaction threatening defederation is unwarranted[15].

While this isn't entirely unusual for an admin to do (we reach out and have other instances reach out to discuss reports), airing it in public is not acceptable, especially given tenforward.social's explicit statements about using the report system and confidentiality of reports[16] and then posting an email chain report from a concerned user[17][18]. Revealing private information publicly, especially like this, is utterly unacceptable and a gross violation of trust.

Limiting tenforward.social

As a result of the actions above, we're locally suspending packetcat@tenforward.social across all of our instances, and we're limiting tenforward.social due to the instability of their admin and egregious breach of trust that makes us concerned with interactions going forward.

We realize there are connections back and forth between our Mastodon instances (furry.engineer and pawb.fun) and tenforward.social, so we don't want to unduly impact our users or theirs based solely on their administrator's conduct by suspending and permanently breaking those connections.

Should there be additional poor conduct or an improvement in the situation, we'll re-review our stance in a few weeks time.

Important Reminder

It should really go without saying, but DO NOT use this as an excuse to harass packetcat or anyone else involved in this situation. Any users found doing so will be suspended without warning per Rule 3. While their conduct is unbecoming, it does not warrant abuse.

Clarification: What is a Limit

A limit means that the instance is effectively hidden from view on furry.engineer and pawb.fun. All of the content still exists and can be found if searching or interacting (like following a user on tenforward.social), but won't appear on the public timelines and won't be recommended to our users.

If someone from tenforward.social tries to mention someone on furry.engineer or pawb.fun, and they aren't being followed by that person, the message may not be visible. Similarly, if someone from tenforward.social tries to follow a furry.engineer, or pawb.fun user, they'll instead send a follow request that you'll need to approve.

References

CW: Threats, Harassment; Some linked posts have content warnings, but have been omitted here for transparency.

[1] @packetcat@tenforward.social (Sept 7, 2024) "I saw some folks talking about baeddel on the TL earlier and my thoughts: If I ever encounter these baedellist fuckers, I'm going to buy an AK-47 and shoot them in the head. That's too quick of a death for them because I'm merciful." Retrieved from https://tenforward.social/@packetcat/113097974087422618. [REMOVED] see note below.

[2] @packetcat@tenforward.social (Sept 7, 2024) ""The AK-47, officially known as the Avtomat Kalashnikova (Russian: Автомат Калашникова, lit. 'Kalashnikov's automatic [rifle]'; also known as the Kalashnikov or just AK), is an assault rifle that is chambered for the 7.62×39mm cartridge. Developed in the Soviet Union by Russian small-arms designer Mikhail Kalashnikov, it is the originating firearm of the Kalashnikov (or "AK") family of rifles. After more than seven decades since its creation, the AK-47 model and its variants remain one of the most popular and widely used firearms in the world." https://en.wikipedia.org/wiki/AK-47 The AK-47 is the people's weapon. And it is the people's justice enacted on these fuckers." Retrieved from https://tenforward.social/@packetcat/113097988673182417. [REMOVED] see note below.

Note: Since the posts linked above have been subsequently deleted, a screenshot has been attached from the furry.engineer admin panel: !

[3] @packetcat@tenforward.social (Sept 7, 2024) "A new (to me) Eris account was just reported to me. Y'all wanna block/suspend this one real quick. eris@yourwalls.today #fediblock" [Extended Thread] Retrieved from https://tenforward.social/@packetcat/113096843977126847.

[4] @packetcat@tenforward.social (Sept 8, 2024) "Racism in the Fediverse: Eris of Disqordia. https://www.youtube.com/watch?v=WUXOUd2AENo. I wanted to put my thoughts on Eris and racism on fediverse in video form so here it is. I want this wide. Boost boost boost. I want this on every fediverse instance's timeline." Retrieved from https://tenforward.social/@packetcat/113102432762834938.

[5] @puf@rage.love (Aug 15, 2022) "if you need some proof that eris is a unpleasant individual --- make your own choices whether this is #fediblock worthy, but note that i know some instances defederated from disqordia due to the below, among other things: this thread is cached correctly here, on the external server, but eris' original toots were lost when the old disqordia instance went down https://cybre.space/@DangerDyke/108595663835969647 (top of thread at https://jubi.life/@sixthhokage95/108592329213498710)" Retrieved from https://rage.love/@puf/108829412633265534.

[6] @packetcat@tenforward.social (Sept 7, 2024) "I just got this report. This is the divide and conquer tactic these people play. @OttawaPlaceAdmin you want to ban whoever this is. Trust me you don't want them in your space. They are accusing me of transphobia. Right. Of course they are. I've seen this play before. It ain't working on anyone anymore Eris. We know your game. Also accusing @so_treu of deadnaming her. Which they haven't done because I know what actually happened." Retrieved from https://tenforward.social/@packetcat/113098074860190153.

[7] @packetcat@tenforward.social (Sept 7, 2024) "don't fuck with me. https://infosec.space/@kkarhan/113098123869087463 https://infosec.space/@kkarhan/113098129378235029 This is what Eris does." Retrieved from https://tenforward.social/@packetcat/113098141033911677.

[8] @packetcat@tenforward.social (Sept 7, 2024) "Telling me to use the report system is hiliarous mate. Yeah mate. I know about the report system. Not like I'm a moderator and admin of a Mastodon instance since 2017. Fucking cunt." Retrieved from https://tenforward.social/@packetcat/113098152858203750.

[9] @packetcat@tenforward.social (Sept 7, 2024) "techhub.social/@Raccoon/113098175071285637 lol. lmao. Now this is some proper entertainment only the fediverse can give me." Retrieved from https://tenforward.social/@packetcat/113098184753861669.

[10] @packetcat@tenforward.social (Sept 7, 2024) "lmao. mastodon.top now eh? @thomas_virtubox hey. ban whoever is the originator of this report. you don't them in your space." Retrieved from https://tenforward.social/@packetcat/113098204368592089.

[11] @packetcat@tenforward.social (Sept 8, 2024) "I'll improve more Americans' lives in 1 hour than Kamala Harris ever will in 4 years. https://subscribe.packetcat.ca" Retrieved from https://tenforward.social/@packetcat/113103985937178864.

[12] @packetcat@tenforward.social (Sept 8, 2024) "I'm not American, or in America but I can help Americans from where I am and I do every single day. https://subscribe.packetcat.ca" Retrieved from https://tenforward.social/@packetcat/113103990459803459.

[13] Inken Paper, @crashglasshouses (Sept 8, 2024) "@packetcat that's gonna require trolling #MastodonForHarris and #Mastodon4Harris more" Retrieved from https://tenforward.social/@crashglasshouses/113103964074902674.

[14] @packetcat@tenforward.social (Sept 8, 2024) "Oh look I already caught a fish. From hachyderm.io. Wow what a surprise. alexhammy@hachyderm.io "Alexander Hamizdat for Harris" told me to "GFY". and immediately deleted it and blocked me. lol I saw it before you delete it lmao." Retrieved from https://tenforward.social/@packetcat/113104003943902913.

[15] @packetcat@tenforward.social (Sept 7, 2024) "kitty.social/notes/9xwbjjklx6v700cq another one for the blocklist. harassment. kitty.social, you have 24 hours to deal with this piece of shit. If you don't, I will terminate federation with you." Retrieved from https://tenforward.social/@packetcat/113098806737556514.

[16] About - Tenforward.Social "Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting an admin by using the report button. All reports will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Admins are obligated to maintain confidentiality with regard to the reporter of an incident." Retrieved from https://tenforward.social/about. Retrieved on Sept 8, 2024.

[17] @packetcat@tenforward.social (Sept 7, 2024) "lmao. Another challenger appears. Email too huh?" [Screenshot includes a report from a concerned user, and their full email address] Retrieved from https://tenforward.social/@packetcat/113098303738022025.

[18] @packetcat@tenforward.social (Sept 7, 2024) "lmao. free entertainment. Who needs Netflix when I have this?" Retrieved from https://tenforward.social/@packetcat/113098352226657071.

Edits

• Added section "Clarification: What is a Limit"
• Updated deleted image proof from https://pawb.social/pictrs/image/7670f660-3174-467a-a9cf-9da5411c994b.png to https://pawb.social/pictrs/image/82e428b8-83e7-4fb6-8fa2-6a2c4ff0fbca.png. The original had the first post still collapsed.
• Added section "Edits"

test

Given the size the community has grown to and concerns around our current approach to maintenance, I’d like to get some feedback on when we should look to schedule maintenance going forward.

There’s two cases covered by the poll, so please choose the all options that fit your preferences.

Case 1: Non-interactive Maintenance This covers any maintenance where we don’t actively need to monitor the upgrade process; Copying data from one location to another, etc. (Usually, this could be left overnight and brought back up in the morning)

Case 2: Interactive Maintenance This covers any maintenance that requires us to perform a series of actions and continually monitor the process to ensure no issues; Upgrading Mastodon or Lemmy, database migration, or upgrading the Kubernetes cluster. (Usually, this would need to occur during the day due to time constraints)

To vote, please find the corresponding pinned comments and upvote them!

If you’ve got any questions, concerns, etc. please leave them below and we’ll get back to them asap!

Due to the recent spam waves affecting the Fediverse, we'd like to open requests for comment on the use of automated moderation tools across Pawb.Social services.

We have a few ideas on what we'd like to do, but want to make sure users would feel comfortable with this before we go ahead with anything.

For each of these, please let us know if you believe each use-case is acceptable or not acceptable in your opinion, and if you feel like sharing additional info, we'd appreciate it.

---

1. Monitoring of Public Streaming Feed

We would like to set up a bot that monitors the public feed (all posts with Public visibility that appears in the Federated timeline) to flag any posts that meet our internally defined heuristic rules.

Flagged posts would be reported per normal from a special system-user account, but reports would not be forwarded to remote instances to avoid false-positives.

These rules would be fixed based on metadata from the posts (account indicators, mentions, links, etc.), but not per-se the content of the posts themselves.

2. Building of a local AI spam-detection model

Taking this a step further, we would like to experiment with using TensorFlow Lite and Google Coral Edge TPUs to make a fully local model, trained on the existing decisions made by our moderation team. To stress, the model would be local only and would not share data with any third party, or service.

This model would analyze the contents of the post for known spam-style content and identifiers, and raise a report to the moderation team where it exceeds a given threshold.

However, we do recognize that this would result in us processing posts from remote instances and users, so we would commit to not using any remote posts for training unless they are identified as spam by our moderators.

3. Use of local posts for non-spam training

If we see support with #2, we'd also like to request permission from users on a voluntary basis to provide as "ham" (or non-spam / known good posts) to the spam-detection model.

While new posts would be run through the model, they would not be used for training unless you give us explicit permission to use them in that manner.

I'm hoping this method will allow users who feel comfortable with this to assist in development of the model, while not compelling anyone to provide permission where they dislike or are uncomfortable with the use of their data for AI training.

4. Temporarily limiting suspected spam accounts

If our heuristics and / or AI detection identify a significant risk or pattern of spammy behavior, we would like to be able to temporarily hide / suppress content from the offending account until a moderator is able to review it. We've also suggested an alternative idea to Glitch-SOC, the fork we run for furry.engineer and pawb.fun, to allow hiding a post until it can be reviewed.

Limiting the account would prevent anyone not following them from seeing posts or mentions by them, until their account restriction is lifted by a moderator.

In a false-positive scenario, an innocent user may not have their posts or replies seen by a user on furry.engineer / pawb.fun until their account restriction is lifted which may break existing conversations or prevent new ones.

---

We'll be leaving this Request for Comment open-ended to allow for evolving opinions over time, but are looking for initial feedback within the next few days for Idea #1, and before the end of the week for ideas #2 through #4.

On Feb 14th we migrated Lemmy from its standalone Docker setup to the same Kubernetes cluster operating furry.engineer and pawb.fun, discussed in https://pawb.social/post/6591445.

As of 5:09 PM MT on Feb 14th, we are still transferring the media to the new storage, which may result in broken images. Please do still reply to this thread if your issue is media related, but please check again after a few hours and edit your comment to say "resolved" if it's rectified by the transfer.

As of 11:02 AM MT on Feb 15th, we have migrated all media and are awaiting the media service coming back online and performing a hash check of all files. Once this is completed, uploads should work per normal.

----

To make it easier for us to go through your issues, please include the following information:

• Time / Date Occurred
• Page URL where you encountered the issue
• What you were trying to do at the time you encountered the issue
• Any other info you think might be important / relevant

tl;dr summary furry.engineer and pawb.fun will be down for several hours this evening (5 PM Mountain Time onward) as we migrate data from the cloud to local storage. We'll post updates via our announcements channel at https://t.me/pawbsocial.

----

In order to reduce costs and expand our storage pool, we'll be migrating data from our existing Cloudflare R2 buckets to local replicated network storage, and from Proxmox-based LXC containers to Kubernetes pods.

Currently, according to Mastodon, we're using about 1 TB of media storage, but according to Cloudflare, we're using near 6 TB. This appears to be due to Cloudflare R2's implementation of the underlying S3 protocol that Mastodon uses for cloud-based media storage, which is preventing Mastodon from properly cleaning up no longer used files.

As part of the move, we'll be creating / using new Docker-based images for Glitch-SOC (the fork of Mastodon we use) and hooking that up to a dedicated set of database nodes and replicated storage through Longhorn. This should allow us to seamlessly move the instances from one Kubernetes node to another for performing routine hardware and system maintenance without taking the instances offline.

We're planning to roll out the changes in several stages:

1. Taking furry.engineer and pawb.fun down for maintenance to prevent additional media being created.

2. Initiating a transfer from R2 to the new local replicated network storage for locally generated user content first, then remote media. (This will happen in parallel to the other stages, so some media may be unavailable until the transfer fully completes).

3. Exporting and re-importing the databases from their LXC containers to the new dedicated database servers.

4. Creating and deploying the new Kubernetes pods, and bringing one of the two instances back online, pointing at the new database and storage.

5. Monitoring for any media-related issues, and bringing the second instance back online.

We'll be beginning the maintenance window at 5 PM Mountain Time (4 PM Pacific Time) and have no ETA at this time. We'll provide updates through our existing Telegram announcements channel at https://t.me/pawbsocial.

During this maintenance window, furry.engineer and pawb.fun will be unavailable until the maintenance concluded. Our Lemmy instance at pawb.social will remain online, though you may experience longer than normal load times due to high network traffic.

----

Finally and most importantly, I want to thank those who have been donating through our Ko-Fi page as this has allowed us to build up a small war chest to make this transfer possible through both new hardware and the inevitable data export fees we'll face bringing content down from Cloudflare R2.

Going forward, we're looking into providing additional fediverse services (such as Pixelfed) and extending our data retention length to allow us to maintain more content for longer, but none of this would be possible if it weren't for your generous donations.

We've updated to Lemmy v0.19.3!

For a full change log, see the updates below:

• v0.19.0
• v0.19.1
• v0.19.2
• v0.19.3

Major changes

Improved Post Ranking

There is a new scaled sort which takes into account the number of active users in a community, and boosts posts from less-active communities to the top. Additionally there is a new controversial sort which brings posts and comments to the top that have similar amounts of upvotes and downvotes. Lemmy’s sorts are detailed here.

Instance Blocks for Users

Users can now block instances. Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesn’t affect users from the blocked instance, their posts and comments can still be seen normally in other communities.

Two-Factor Auth Rework

Previously 2FA was enabled in a single step which made it easy to lock yourself out. This is now fixed by using a two-step process, where the secret is generated first, and then 2FA is enabled by entering a valid 2FA token. It also fixes the problem where 2FA can be disabled without passing any 2FA token. As part of this change, 2FA is disabled for all users. This allows users who are locked out to get into their account again.

New Federation Queue

Outgoing federation actions are processed through a new persistent queue. This means that actions don’t get lost if Lemmy is restarted. It is also much more performant, with separate senders for each target instance. This avoids problems when instances are unreachable. Additionally it supports horizontal scaling across different servers. The endpoint /api/v3/federated_instances contains details about federation state of each remote instance

Remote Follow

Another new feature is support for remote follow. When browsing another instance where you don’t have an account, you can click the subscribe button and enter the domain of your home instance in the popup dialog. It will automatically redirect you to your home instance where it fetches the community and presents a subscribe button. Here is a video showing how it works.

Moderation

Reports are now resolved automatically when the associated post/comment is marked as deleted. This reduces the amount of work for moderators. There is a new log for image uploads which stores uploader. For now it is used to delete all user uploads when an account is purged. Later the list can be used for other purposes and made available through the API.

• Instance: pisskey.io
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Nazi imagery, affiliated with poa.st and other known abusive instances
• Fediseer Action: Censured

• Instance: the.asbestos.cafe
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Homophobia, harassment, trolling, admin involved abuse
• Fediseer Action: Censured

Block has been applied to the entire domain.

Evidence

• https://the.asbestos.cafe/objects/14decba5-b6fb-4c20-b8d7-8cf68374ec58
• https://the.asbestos.cafe/objects/094cfc21-7182-471a-90cc-a493fb126635
• https://the.asbestos.cafe/objects/764af9ee-288c-4151-aa41-b2e295dc95cc
• https://the.asbestos.cafe/objects/3f5960bb-8fc1-41b0-b180-7748cd0e15b3
• https://the.asbestos.cafe/objects/bee81dc2-85ee-45ae-b1b6-39f2d2d4dd6f

• Instance: lab.nyanide.com
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Trolling, harassment, homophobia, nazi imagery, admin / mod engaged abuse
• Fediseer Action: Censured

Block has been applied to the entire domain.

Evidence

! ! !

cross-posted from: https://pawb.social/post/3393854

> - Instance: cunnyborea.space > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery > - Fediseer Action: Censured > > ::: spoiler Evidence > - https://i.imgur.com/HgvMVOh.png > - https://union.place/@inquiline/111247251661793670 > :::

• Instance: cunnyborea.space
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
• Fediseer Action: Censured

Evidence

• https://i.imgur.com/HgvMVOh.png
• https://union.place/@inquiline/111247251661793670

cross-posted from: https://pawb.social/post/3337642

> - Instance: bv.umbrellix.org > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Toxicity, abusive admin, death threats, emotional abuse > - Fediseer Action: Censured > > ::: spoiler Evidence > - (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi > - https://sunbeam.city/@kayttaja@toot.site/111142509643025163 > ::: > > Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.

• Instance: bv.umbrellix.org
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Toxicity, abusive admin, death threats, emotional abuse
• Fediseer Action: Censured

Evidence

• (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi
• https://sunbeam.city/@kayttaja@toot.site/111142509643025163

Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.

Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects.

We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop.

We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up.

We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.

cross-posted from: https://pawb.social/post/3077174

> - Instance: liberdon.com > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Harassment, trolling, misinformation, sexism, homophobia > - Fediseer Action: Censured > > ::: spoiler Evidence > - https://liberdon.com/@ghast > - https://liberdon.com/@ghast/111178732016951128 > - https://liberdon.com/@rimugu/111178647646581863 > - https://liberdon.com/@aristoh4ck8r/111177188127718387 > - https://liberdon.com/@gunkslinger/111175171054509579 > - https://liberdon.com/@TylerAbeoJordan/111174105820298526 > - https://liberdon.com/@TylerAbeoJordan/111174078685171464 > - https://liberdon.com/@TylerAbeoJordan/111173743376687580 > - https://liberdon.com/@medicineman9/111173062104385765 > - https://liberdon.com/@spinneria/111169619274938220 > - https://liberdon.com/@nosat/111165212240469156 > ::: >

One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. No data, except ElasticSearch, was stored on CEPH.

When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable.

Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present.

I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.