Author Topic: Secondary Node /dev/md0c 102% capacity  (Read 640 times)

Offline AdamMaxwell

  • VIP Member
  • ***
  • Posts: 30
  • Karma: 2
Secondary Node /dev/md0c 102% capacity
« on: June 19, 2009, 02:44:01 AM »
Hello, the secondary node in one of our HA pairs has seemed to have no capacity left on it.

Filesystem  1K-blocks   Used  Avail Capacity  Mounted on
/dev/md0c      152494 152454  -3008   102%    /
devfs               1      1      0   100%    /dev
procfs              4      4      0   100%    /proc
/dev/ad0s1a    231774  64678 148556    30%    /flash

I'm a bit confused as to how this has happened?? Can anyone shed any light on it for me? also what I can do to resolve it?

The primary node is fine, but this secondary one isn't accepting any config changes and isn't showing a "full" ns.conf file.

Thanks.

Adam

Offline AdamMaxwell

  • VIP Member
  • ***
  • Posts: 30
  • Karma: 2
Re: Secondary Node /dev/md0c 102% capacity
« Reply #1 on: June 22, 2009, 12:14:45 AM »
I've discovered the fault. The 80GB hard drive that is usually mounted /var has some bad sectors. I've logged a call with Citrix and the unit is being RMA'd.


Offline jmelika

  • Administrator
  • Hero Member
  • *****
  • Posts: 341
  • Karma: 7
Re: Secondary Node /dev/md0c 102% capacity
« Reply #2 on: June 22, 2009, 10:46:38 PM »
So the bad sectors made it show the wrong capacity?  Interesting

Offline AdamMaxwell

  • VIP Member
  • ***
  • Posts: 30
  • Karma: 2
Re: Secondary Node /dev/md0c 102% capacity
« Reply #3 on: June 23, 2009, 12:35:33 AM »
I think what has happened was because the hard drive failed to mount to /var the logs that the Netscaler generates went onto / (where /dev/md0c mounted to) and filled up all the space available.. not sure how it managed to get to 102% though.

Offline ohmhyde

  • VIP Member
  • ***
  • Posts: 28
  • Karma: 4
Re: Secondary Node /dev/md0c 102% capacity
« Reply #4 on: June 24, 2009, 09:42:40 AM »
Hey!!   I am having the same problem as yours.
remember my problem about Netscaler hard disk crash on this link?
http://www.netscalerkb.com/netscaler_qa/netscaler_hard_drive_error_booting_problem_please_help_thanks-t616.0.html

Now I have got the RMA's unit back,
But there is still same problem!! Arrrgh
It won't accept any configuration, license or any thing!!...
I dump the ns.conf ( lots of configurations that worked with other box before)
after reboot, everything is the same!!  always 192.168.100.1 default factory setting
I go to shell and run df -k command
it show like this
Filesystem     1K-blocks    Used      Avail   Capacity  Mounted on
/dev/md0c      165294     159438    2552    98%       /
/dev/ad0s1a    231774     15774    55460    74%       /flash
procfs              4                  4       0       100%    /proc


//////////////  some messages during boot ////////////////////////////
Configuration file is correct
nslcd: serial at /dev/cuaa1...
nsconf, pid 179...
nslog: Wed Jun 24 10:01:38 GMT 2009: started as process 198, -time 172800
/var/nslog directory does not exist.
Possible reason: /var partition is not mounted.

/dev/md0c on / (ufs, local)
/dev/ad0s1a on /flash (ufs, local)
procfs on /proc (procfs, local)
Check paritions and try again.
nsdiskspace: Wed Jun 24 10:01:38 GMT 2009: started as process 201
nssync: Wed Jun 24 10:01:38 GMT 2009: started as process 199
nsumond: Wed Jun 24 10:01:38 GMT 2009: starting dispatcher daemon nsumond
nsumond: Wed Jun 24 10:01:38 GMT 2009: starting dispatcher daemon nsumond
rm: /var/nslog/nsumond.log: No such file or directory
touch: /var/nslog/nsumond.log: No such file or directory
chmod: /var/nslog/nsumond.log: No such file or directory
Jun 24 10:01:39 <local0.err> ns nsumond[271]: Monitoring daemon started
///////////////////////////////   booting  messages ////////////////////////////////



It seem there is still some Hard drive reading, mounting, partitioning error.
 
/////////////////  license.log   ////////////////////////////////////////////
liccense error
root@ns# ls
auth.log        httperror.log   lpd-errs        ns.log          security
cron            lastlog         maillog         nsvpn.log       snmpd.log
httpaccess.log  license.log     messages        nsvpnd.log
root@ns# nsconmsg -K license.log
GNOSE status:0055
kbdc: TEST_KBD_PORT status:0000
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sc0: fb0, terminal emulator: sc (syscons terminal)
sio0: irq maps: 0x1 0x11 0x1 0x1
sio0 at port 0x3f8-0x3ff irq 4 flags 0x90 on isa0
sio0: type 16550A, console
sio1: irq maps: 0x11 0x19 0x11 0x11
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type
isa_probe_children: probing PnP devices
SMP: enabled INTs: 3, 4, 11, 14, 15, 16, 17, 18, 19, apic_imen: 0x00f037e7
BIOS Geometries:
 0:00f73f20 0..247=248 cylinders, 0..63=64 heads, 1..32=32 sectors
 0 accounted for
Device configuration finished.
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
bpf: lo0 attached
ad0: success setting PIO4 on generic chip
Creating DISK ad0
ar: FreeBSD check1 failed

ad0: <UNIGEN FLASH/30/06/03> ATA-0 disk at ata0-master
ad0: 249MB (510976 sectors), 998 C, 16 H, 32 S, 512 B
ad0: 1 secs/int, 1 depth queue, PIO4
ad0: piomode=4 dmamode=-1 udmamode=-1 cblid=0
Mounting root from ufs:/dev/md0c
md0: invalid primary partition table: no magic
start_init: trying /sbin/init
ad0s1: type 0xa5, start 32, end = 471039, size 471008 : OK
ad0s2: type 0x1, start 471040, end = 487423, size 16384 : OK
ad0s1: type 0xa5, start 32, end = 471039, size 471008 : OK
ad0s2: type 0x1, start 471040, end = 487423, size 16384 : OK
ad0s1: type 0xa5, start 32, end = 471039, size 471008 : OK
ad0s2: type 0x1, start 471040, end = 487423, size 16384 : OK
NetScaler: 1484Mb of memory allocated
PE init: LO0: nic# 5, if softc 0x3c7dcf80, id ffffffff
PE init: LA0: nic# 6, if softc 0x3c7e3f80, id ffffffff
PE init: LA1: nic# 7, if softc 0x3c7eaf80, id ffffffff
PE init: LA2: nic# 8, if softc 0x3c7f1f80, id ffffffff
PE init: LA3: nic# 9, if softc 0x3c7f8f80, id ffffffff
PE bx0: ID30100 1/3, V1 (tag) nic# 0, softc (if:0x3c7b9f80, PE:0x3c7b9f80)
PE bx1: ID30101 1/4, V1 (tag) nic# 1, softc (if:0x3c7c0f80, PE:0x3c7c0f80)
PE bx2: ID20100 1/1, V1 (tag) nic# 2, softc (if:0x3c7c7f80, PE:0x3c7c7f80)
PE bx3: ID20101 1/2, V1 (tag) nic# 3, softc (if:0x3c7cef80, PE:0x3c7cef80)
PE ez0: IDa0100 0/1, V1 (tag) nic# 4, softc (if:0x3c7d5f80, PE:0x3c7d5f80)
PE LO0: IDffffffff 3/1, V1 nic# 5, softc (if:0x3c7dcf80, PE:0x3c7dcf80)
PE LA0: IDffffffff 2/1, V0 (tag) nic# 6, softc (if:0x3c7e3f80, PE:0x3c7e3f80)
PE LA1: IDffffffff 2/2, V0 (tag) nic# 7, softc (if:0x3c7eaf80, PE:0x3c7eaf80)
PE LA2: IDffffffff 2/3, V0 (tag) nic# 8, softc (if:0x3c7f1f80, PE:0x3c7f1f80)
PE LA3: IDffffffff 2/4, V0 (tag) nic# 9, softc (if:0x3c7f8f80, PE:0x3c7f8f80)
///////////////////////    end of license.log  //////////////////////////////////////////////

Don't know why this is happening to this customer all the time.
Arrrgh!!
Other boxes work fine.
Only this one, this case, this is second time RMA now.

Please advise what I should do or what I missed??
Thanks alot everyone.  : )




 
« Last Edit: June 24, 2009, 09:53:48 AM by ohmhyde »