I would seriously look at how many files are in each directory on the pic8 server. I just read a line in a page I was reading on a search I did, the person said and this made something in me say this is the correct reason shit happens.
Even though the FS may be able to handle many files, the shell can't. This is wise. Just because you can store the files and retrieve them doesn't mean the system controlling them is not overworked. Likely not a database but a cpu being overloaded searching the database and the ram being constantly full from all the indexes.
I personally would add another drive to the system and then ls the files with dates older than a month or so and put them in the other drive or the slowest one for archiving in batches of 10 - 32K images even the thumbs limited. Heck it's not the size of the files but the number of them per directory that is likely fucking it all up. Even though supposedly limitless the ext4 fs does have to search inodes and index updating constantly.
I imagine those indexes are huge if the directories are not limited to a certain number of files and have the directories and subdirectories named alphanumerically so the system knows that image fucktheworld is in F maybe subdirectory s-w so it's like jump jump search index of that subfolder with 9k images and retrieve wanted files in 12 miliseconds flat not 200 ms since the index in each folder is short as fuck.
Then the system can find the folder effortlessly if to many files are in the bottom folder say the s-w one then have s-t then divide to s,t,u,v,w then even s"whatever symbol sorts first" - sla, so forth and on and on. First folder checked is the one chronologically most recent like a cpu's cache sorts what is kept and what is dumped.
If picman replaced a major drive with a new huge one and put to many files on that new drive then he'd be better of with slower multiple drives if his motherboard cpu setup can handle it then running 5 slower drives for retriving small files like thumbs and images would be far faster than on super fast sata that is limited to the fact that it's using one cable to talk back and forth where the 5 drives are using 5 cables so their combined throughput would be greater and smoother overall since it's not a mob of files but a instead a steady smooth flow of files orderly and then only the motherboard limitations or the networking will be the bottleneck or even PSU if it's running excessive HW for it's original intent.
There is very little interaction with the file system.
Files are only stored on the FS during initial upload. After that, everything is a simple string value in a database. Lookups are basic string comparisons, very little computational effort, even for million of entries (only at 100k)
I do apprecaite the thought. I believe the recent issues with images sometimes not being found is an operation issue on pic8's side, along with some fuckery with an underlying host. Been working on fixing some of these operational issues, but sadly the codebase was written back when I was a much less experienced developer.
(post is archived)