I’ve been monitoring the situation and unfortunately there’s not much that can be done from our end. I’ve already tried throwing as much hardware as I can afford at the problem. My understanding is that it should be resolved once they upgrade to 0.19.6 with the new Parallel Federation feature. We’ll be performing the upgrade soon, but I haven’t seen an ETA for them to upgrade yet.

I understand this impacts a large amount of our federated content and apologize for the disruption. Hopefully it can be resolved soon with the new releases.

Community creation is currently disabled. If you have an idea for a new community, feel free to DM me and I can help you get set up.

This issue should be fixed now. Let me know if you continue to see any issues.

For the technical details:

Pictrs is used for managing images in Lemmy. It uses a Sqlite database on the local filesystem which limits its scalability. After a recent upgrade, it does now support migrating this data into Postgresql. I’m hoping to schedule some time soon to perform the update. This will allow multiple instances of Pictrs to be run in parallel providing better uptime and recovery.

What happens here is that the service gets into a bad state and it tries to bring up a replacement. But due to the local DB, the replacement is unable to get a lock on the filesystem and crashes. Once we switch it to use Postgres, this should no longer be an issue.

Feel free to use whichever method you prefer! The only ask is to avoid directly uploading porn to lemmus. This is due to the limited moderation controls currently in-place and preferring to avoid the legal risk.

As for costs, while our bills are a bit high right now almost all of it comes from database and compute. Storage and bandwidth are pretty cheap for us and we aggressively cache everything we can.