/dev/sda1 is full, what can I clean?

Got problems with your B2 or B3? Share and get helped!
RichoDemus
Posts: 22
Joined: 27 Mar 2012, 17:39

/dev/sda1 is full, what can I clean?

Post by RichoDemus » 25 Sep 2012, 07:49

Hi, I was attempting to add a cronjob and got a "No space left on device" warning so I ran df and found this:

Code: Select all

user@b3:~/bin$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  9.5G     0 100% /
tmpfs                 252M     0  252M   0% /lib/init/rw
udev                   10M  156K  9.9M   2% /dev
tmpfs                 252M  4.0K  252M   1% /dev/shm
/dev/md0              907G  211G  650G  25% /home
overflow              1.0M   36K  988K   4% /tmp
How do I handle this, ie how do I find what files to delete etc, I'm guessing it's log files?

DanielM
Posts: 637
Joined: 28 Mar 2008, 06:37
Location: Sweden

Re: /dev/sda1 is full, what can I clean?

Post by DanielM » 25 Sep 2012, 09:09

Try this:

Code: Select all

du -h /var/log
Might give some hints.

/Daniel

RandomUsername
Posts: 904
Joined: 09 Oct 2009, 18:49

Re: /dev/sda1 is full, what can I clean?

Post by RandomUsername » 25 Sep 2012, 09:28

Yeah, something's wrong if the / has filled up. It's a good bet something had gone haywire and wrote a massive log file.

RichoDemus
Posts: 22
Joined: 27 Mar 2012, 17:39

Re: /dev/sda1 is full, what can I clean?

Post by RichoDemus » 25 Sep 2012, 09:31

my /var/log is only 13 megabytes, I think the culprit is elsewhere
I did

Code: Select all

du -h / | sort -nr | head -30
but the only large files I found were in /home which is on another disk

DanielM
Posts: 637
Joined: 28 Mar 2008, 06:37
Location: Sweden

Re: /dev/sda1 is full, what can I clean?

Post by DanielM » 25 Sep 2012, 12:17

Sometimes it's easier to take one level at the time. Something like "du -h --max-depth=1 /" first and then continue down to the largest directory.

/Daniel

RichoDemus
Posts: 22
Joined: 27 Mar 2012, 17:39

Re: /dev/sda1 is full, what can I clean?

Post by RichoDemus » 25 Sep 2012, 12:31

I found it!

I have 32k files in /var/spool/postfix/maildrop and it's 7.8 gigabytes


-rwxr--r-- 1 root postdrop 245828 Jul 8 09:40 36B4735D93
-rwxr--r-- 1 root postdrop 245828 Apr 13 17:25 36B6E306C4
-rwxr--r-- 1 root postdrop 245828 May 9 04:50 36B7B1A1CC
-rwxr--r-- 1 root postdrop 245828 Apr 14 21:40 36B9A30820
-rwxr--r-- 1 root postdrop 245828 Apr 15 21:55 36BCD30957
-rwxr--r-- 1 root postdrop 245828 May 31 01:30 36BD83303F
-rwxr--r-- 1 root postdrop 245828 Jun 29 22:35 36BE635412


I'm guessing these are unsent emails? can I view them somehow? are they safe to delete?

RandomUsername
Posts: 904
Joined: 09 Oct 2009, 18:49

Re: /dev/sda1 is full, what can I clean?

Post by RandomUsername » 25 Sep 2012, 12:36

I think you can view them with the "mail" command. Could be errors created by a cron job or something.

DanielM
Posts: 637
Joined: 28 Mar 2008, 06:37
Location: Sweden

Re: /dev/sda1 is full, what can I clean?

Post by DanielM » 25 Sep 2012, 13:24

RichoDemus wrote:I have 32k files in /var/spool/postfix/maildrop and it's 7.8 gigabytes
I think that directory is supposed to be empty, sounds to me like mail that should be delivered somewhere but has stuck. What do you get when running "mailq"?

/Daniel

RichoDemus
Posts: 22
Joined: 27 Mar 2012, 17:39

Re: /dev/sda1 is full, what can I clean?

Post by RichoDemus » 26 Sep 2012, 03:17

running sudo mailq gives me thousands of lines:

Code: Select all

C80B4307E7   245828 Sat Apr 14 14:55:02  root
                                         root

DD4FC303CB   245828 Wed Apr 11 00:05:02  root
                                         root

6B422724A9   245828 Mon May 14 23:05:03  root
                                         root

4C476371AA   245828 Mon Sep 17 21:35:06  root
                                         root

A16BB327C6   245828 Wed May 23 10:20:02  root
                                         root
Is there a way to see the contents of these files? running mail as root simply gives me "no mail for root"

DanielM
Posts: 637
Joined: 28 Mar 2008, 06:37
Location: Sweden

Re: /dev/sda1 is full, what can I clean?

Post by DanielM » 26 Sep 2012, 03:32

RichoDemus wrote:Is there a way to see the contents of these files? running mail as root simply gives me "no mail for root"
Those messages are stuck in the queue. For some reason postfix doesn't know what to do with them.

I don't really know what the default config is, but I've set up an alias so mail destinated for root ends up in my "normal" users mailbox instead. I've just added a line in my /etc/postfix/virtual for this, though I don't know if this is correct for you. Check "man 5 virtual" for a description about what to do.

/Daniel

nobody
Posts: 226
Joined: 10 Mar 2012, 14:46

Re: /dev/sda1 is full, what can I clean?

Post by nobody » 26 Sep 2012, 13:22

Pleas run

Code: Select all

postsuper -d ALL
to clean the messages. They contain only stats and log data.
Then edit

Code: Select all

/etc/aliases
and alter the last line of that file. You need to enter a avlid mail address here, or a username that can receive mail.
Now run

Code: Select all

newaliases

RichoDemus
Posts: 22
Joined: 27 Mar 2012, 17:39

Re: /dev/sda1 is full, what can I clean?

Post by RichoDemus » 27 Sep 2012, 06:46

It seems like postfix had stopped working which is why the queue kept getting bigger. I started it again after deleting the emails and now everything works.

But it seems like the whole email system is pretty messed up at the moment, I tried sending an email to myself:

Code: Select all

root@b3:/# mail -s asdf root
Cc:
test
root@b3:/# mail
No mail for root
root@b3:/#
mail.log says

Code: Select all

Sep 27 12:40:37 b3 postfix/pickup[6357]: B51163022E: uid=0 from=<root>
Sep 27 12:40:37 b3 postfix/cleanup[6518]: B51163022E: message-id=<20120927104037.B51163022E@b3.localdomain>
Sep 27 12:40:37 b3 postfix/qmgr[25517]: B51163022E: from=<root@b3.localdomain>, size=332, nrcpt=1 (queue active)
Sep 27 12:40:37 b3 postfix/local[6520]: B51163022E: to=<richo@b3.localdomain>, orig_to=<root@b3>, relay=local, delay=0.08, delays=0.01/0/0/0.06, dsn=2.0.0, status=sent (delivered to maildir)
Sep 27 12:40:37 b3 postfix/qmgr[25517]: B51163022E: removed
I have never used mail/sendmail/postix before, am I missing something?

Gordon
Posts: 1378
Joined: 10 Aug 2011, 03:18

Re: /dev/sda1 is full, what can I clean?

Post by Gordon » 27 Sep 2012, 11:45

The mail command uses a different concept. The B3 is configured to use the IMAP maildir format for use with Dovecot and places incoming emails in ~/Mail/new. You need an IMAP compatible application to access the email, but of course root is not permitted to log in to Horde (I'm not sure if it's Horde or Dovecot blocking the login).

Good for pointing this out. I had literally tens of thousands of emails from the cron daemon in roots mailbox from the Horde alarm script. I've added an alias for root now that points to user admin so I can actually read those messages and act on them if required.

Thank you very much.

nobody
Posts: 226
Joined: 10 Mar 2012, 14:46

Re: /dev/sda1 is full, what can I clean?

Post by nobody » 27 Sep 2012, 15:16

The whole issue here is that, with maildir, root is not permitted to receive mail at all. That i why the mail is stuck. The postfix installer is quite lear on this topic.

Gordon
Posts: 1378
Joined: 10 Aug 2011, 03:18

Re: /dev/sda1 is full, what can I clean?

Post by Gordon » 28 Sep 2012, 01:51

nobody wrote:The whole issue here is that, with maildir, root is not permitted to receive mail at all. That i why the mail is stuck. The postfix installer is quite lear on this topic.
That doesn't conform with what I'm seeing. The only issue is that there is no commandline routine to read those emails (that I know of). In any case, the command `mail` reads the content of /var/mail/$USER which stays empty if you use maildir. So that's why it insists that there is no mail.

Post Reply