Hello,
I have this weird issue that my Debian 11 will tell me the root folder is full, while I can only find files for half of the accounted space.
df -h reports 56G while the disk analyser (sudo baobab) only finds 28G.
Anyone ever encountered this? I don't have anything mounted twice.... (Not sure what udev is). Also it does not add up to 100%, it should say 7.2G left not 4.1G
OK, one possibility I can think of. At some point, files may have been created where there is currently a mount point which is hiding folders that are still there, on the root partition.
You can remount just the root partition elsewhere by doing something like
mkdir /mnt/rootonly
mount -o bind / /mnt/rootonly
Then use du or similar to see if the numbers more closely resemble the values seen in df. I'm not sure if that graphical tool you used that views the filesystem can see those files hidden this way. So, it's probably worth checking just to rule it out.
Anyway, if you see bigger numbers in /mnt/rootonly, then check the mount points (like /mnt/rootonly/home and /mnt/rootonly/boot/efi). They should be empty, if not those are likely files/folders that are being hidden by the mounts.
When finished you can unmount the bound folder with
This!
Thank you, this allowed me to find the culprit!
It turns out I had an external disk failure some weeks ago, and a cron rsync job was writing in /mnt/thatdrive. When the externaldrive died rsync created a folder /mnt/thatdrive. Now that I replaced the drive, /mnt was disregarded by the disk analyser, but the folder was still there and indeed hidden by the mount... It is just a coincidence that it was half the size of /
This might help in the future in case you setup a remote mount for backups in the future. Look into using systemd's automount feature. If the mount suddenly fails then it will instead create an unwritable directory in its place. This prevents your rsync from erroneously writing data to your root partition instead.
Very weird, I can think of some things I might check:
It is possible that you have files on disk that don't have a filename anymore. This can happen when a file gets deleted while it is still opened by some process. Only the filename is gone then, but the file still exist until that process gets killed. If this were the problem, it would go away if you rebooted, since that kills all processes.
Maybe it is file system corruption. Try running fsck.
Maybe the files are impossible to see for baobab. Like if you had gigs of stuff under (say) /homeon you root fs, then mount another partition as /home over that, those files would be hidden behind the mount point. Try booting into a live usb and checking your disk usage from there, when nothing is mounted except root.
If you have lots and lots of tiny files, that can in theory use up a lot more disk space than the combined size of the files would, because on a lot file systems, small files always use up some minimum amount of space, and each file also has some metadata. This would show up as some discrepancy between du and df output. For me, df --inodes / shows ~300000 used, or about 10% of total. Each file, directory, symlink etc. should require one inode, I think.
I have never heard of baobab, maybe that program is buggy or has some caveats. Does du -shx / give the same results?
I think it’s possible that the filesystem ran out of inodes, so even though there is space on disk, there is no space in the filesystem metadata to store new files.
Now, I don’t know off the top of my head how to check this, but I assume the answer is on the internet somewhere (am on phone and can’t help much more than this, sorry)
The large /var suggests flatpak, and that plays some hardlinking games.
(If you ever need to free up / space, shifting your flatpak usage to a --user repo will help a lot. No there is no handy command for that, it's a matter of adding and deleting one package at a time.)
You might have some files hard-linked across directories, or worse (but less likely), there's a directory hard-link (not supposed to happen) somewhere.
For the uninitiated, a hard-link is when more than one filename points at the same file data on the disk. This is not the same as a symbolic link. Symbolic links are special files that contain a file or directory name and the OS knows to follow them to that destination. (And they can be used to link to directories safely.)
Some programs are not hard-link aware and will count a hard-linked file as many times as it sees it through its different names. Likewise they will count the entire contents of a hard-linked directory through each name.
Programs tend not to be fooled by symlinks because it's more obvious what's going on.
Try running a duplicate file finder. Don't use it to delete anything, but it might help you determine which directories the files are in and maybe why it's like that.
Also back up everything important and arrange for a fsck on next boot. If it's a hard-linked directory fsck might be able to fix it safely, but it might choose the wrong name to be the main one and remove the other, breaking something. Or remove both. Or it's something else entirely, which by "fixing" will stabilise the system but might cause some other form data loss.
That's all unlikely, but it's nice to have that backup just in case.