Hi all,
The disk on the instance ran out of space today, due to the way Lemmy software caches images from all other instances. That cache had filled up about 60 GB’s of disk, despite us being a small instance with very little local activity.
I had to delete the last 10 days of cached images again, and I plan to delete quite a lot of older cached images as well. The mobile apps seems to not be affected by this (they have a local image cache I believe), but on the web site, this leads to missing thumbnail images.
They are working on a fix for this in the Lemmy software so the disks dont fill up so enormously with cached thumbnails, and as soon as its out, we will install it here.
Hope you guys didnt get too annoyed or sad by the instance being unavailable for a while.
Just keeping me productive is all. Actually, I appreciate that I don’t normally spend nearly as much time in the Fediverse as I did on Reddit. My daily life is improved by it. I guess I’d selfishly rather this remain a small thing, but maybe I ought to take better control of my habits instead.
I feel the same. I use Lemmy a lot since it feels like the early days of the internet, but there is not enough interesting content to stick around for hours. Maybe it will grow but even if it doesn’t, it’s cosy and gives time for other things in life. :)
Thank you for jumping on this. I appreciate your efforts.
Thank you very much!
No I actually thought of you and hoped you weren’t stressing over it. I have an alt mayo account on .world I can use if I absolute must share my opinion on the internet, but honestly .today could be down a week and I’d survive.
I’ve noticed the missing thumbnails since you mentioned it last time but I just click the next post and seems like they come back after a day or so anyway.
They could dither the thumbnails to reduce their size another chunk.
I did stress a bit over it honestly but it was a quick fix at least. I have an idea now how to fix this situation for the future - there is an environment variable that can be set for the pict-rs software that will remove any cached image not viewed after X days. So I will try that one and see if it gets us disk back. :)
Thanks for fixing, I was wondering WTF was going on because this instance hasn’t had any unexpected downtime since I joined!
I don’t know much about how Lemmy works under the hood, I really need to spend some time looking at it, but isn’t it possible to set a cron job to check the available free disk and then do a cache purge of the oldest X days of cached images if needed?
Edit: Since I asked the question I took 90 seconds to look and it appears that it possible to tell Post-Gre via CL to do this.
I have been reading up on pict-rs and it seems that the environment variable
PICTRS__MEDIA__RETENTION__PROXYis the one to set for cleaning up old cached images.It seems they added this environment variable in
pict-rsversion 0.4.2 so I bumped our pict-rs version from 0.4.0 to 0.4.2 and applied the environment variable now.Hopefully this will take care of the disk space growing on its own, but will keep an eye on it. :)
Awesome! Hopefully it works!
I found a problem. It removes Avatar & Banner pictures from your profile!
I noticed this also. Since all pictures are handled by pict rs, deleting anything from disk risks removing any picture users have uploaded also.
Interesting design choice of Lemmy to keep both cached images and user images in the same place with the same names… They all look the same on disk with generated names.
I noticed today that community pics are gone also.
Try to upload the same images again and if that doesn’t work, try to create new images to replace them. I will also try tomorrow to look into this more…
Yeah it was the first downtime we have had since we started! :)
I think it’s an excellent idea to make it purge the oldest X days of cached images. I will look at your link and see what I can do. Thank you!
