Prolixium dot com: News >> Blog

I think it's broken. Yes, floating point arithmetic seems to be broken on atlantis:

% ping -c4 -n vega                                                                  
PING vega.prolixium.com (10.3.5.103) 56(84) bytes of data.
64 bytes from 10.3.5.103: icmp_seq=1 ttl=64 time=0.000 ms
64 bytes from 10.3.5.103: icmp_seq=2 ttl=64 time=0.000 ms
64 bytes from 10.3.5.103: icmp_seq=3 ttl=64 time=0.000 ms
64 bytes from 10.3.5.103: icmp_seq=4 ttl=64 time=0.000 ms

--- vega.prolixium.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms
% traceroute -n chronos
traceroute to chronos (10.3.6.102), 30 hops max, 40 byte packets
 1  10.3.5.254  0.000 ms  0.000 ms  0.000 ms
 2  10.3.7.1  4.000 ms  4.000 ms  4.000 ms
 3  10.3.6.102  4.000 ms  4.000 ms  4.000 ms

I think it might be a glibc problem, but other boxes aren't affected…

I snapped a few shots of a Wienermobile as I was leaving work, today:

Yeah, this one's license plate is BIG BUN.

For the past four years, I've kept all of my media files (music, etc.) on a single filesystem spread across a varying number of physical volumes, utilizing LVM 2.0 on Linux. This JBOD array resided on atlantis, my old SuperMicro system with dual P3-800 CPUs.

The progression went the following, if I remember correctly:

2x 120GB, 1x 80GB = 320GB
1x 400GB, 2x 120GB, 1x 80GB = 720GB
1x 400GB, 1x 250GB, 2x 120GB = 890GB
1x 750GB, 1x 400GB, 1x 250GB, 1x 120GB = 1520GB

Adding and removing drives was simple, a couple LVM commands and a lengthly resize2fs. It worked well, although there was obviously no redundancy to speak of. I kept backing up a good portion (80%) of the content on CD-Rs, then DVD-Rs, then DVD-R DLs, just in case.

Yesterday, I set off to replace the old array with 2x Western Digital 1TB green disks that were much quieter and supposedly didn't chew up as much power as 4x EIDE disks. I had to recreate the filesystem, since ext3 doesn't support a 2TiB filesystem with a 1KiB block size. I created the new logical volume and ext3 filesystem, and started to copy the data.

Unfortunately, both the 400GB (Seagate) and 250GB (Western Digital) disks decided to die 20% into the copy (I used rsync) process. Great! I started & stopped the rsync, remounted, rebooted, etc. for an hour or two, and realized that things weren't looking good. Each time ext3 encountered an I/O error, it would invalidate the inode and cause it to be inaccessible, so rsync would log an error, and skip it, but not after creating the destination files filled with NULLs.

I tried running fsck.ext3 on the filesystem with -c, enabling the badblocks program. Unfortunately, that process would have taken a good two weeks, judging from the rate of completion.

I started up rsync again, this time in only one directory (should have been a couple hundred GiB) and let it go all night. It finished halfway through today, but by that time, the superblock(s) were corrupt, and the filesystem was toast. I tried freezing the drives, but that just made them click louder.

I picked up _another_ Western Digital 1TB disk today, and decided to run LVM on top of RAID-5, just in case anything like this happens again (well, hoping I'd only lose one drive at once). I am still in the process of copying and validating what made it onto the new filesystem - to the 750GB disk, so I can pull all three 1TB disks into a RAID-5 device, and put LVM on top. I just hope the 750GB holds up for the process …

Moral of the story is: JBOD is bad. Overall, I think I lost 40% of the files, according to a periodic ls -lR I run during system (boot disk) backups.

We all love MTR, the ICMP-based continuous traceroute tool for Unix. Since sometimes ICMP doesn't give you the whole picture when doing a traceroute, I went ahead and added preliminary UDP support:

% mtr -P udp --report dax.prolixium.com
HOST: nat                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 10.83.192.1                   0.0%    10    6.9   7.3   5.9   9.2   1.0
  2. dstswr2-vlan2.rh.pswynj.cv.n  0.0%    10    6.9   9.1   6.2  20.5   4.3
  3. r3-ge1-1.mhe.prnynj.cv.net    0.0%    10   64.6  14.8   8.2  64.6  17.5
  4. rtr4-tg11-3.wan.prnynj.cv.ne  0.0%    10    8.2  10.1   7.7  14.2   2.4
  5. rtr1-tg11-1.in.nwrknjmd.cv.n  0.0%    10   11.5  10.7   9.0  13.1   1.5
  6. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  7. 0.te6-4.tsr1.lga3.us.voxel.n  0.0%    10   10.1  12.8   9.6  18.4   2.7
  8. 0.te1-49.dsr1.lga6.us.voxel.  0.0%    10    9.7  14.0   9.5  35.4   7.8
  9. 0.ge0-1.esr2.b3.lga6.us.voxe  0.0%    10   14.9  15.3   9.5  42.7   9.9
 10. dax.prolixium.com             0.0%    10   10.9  12.8   9.8  22.7   3.9

The patches and more information can be found here. The prox3 patch breaks IPv6 support, mostly because the kernel doesn't deliver the IPv6 header to the application with raw sockets, and mostly I just haven't had time, yet.

I suggest you try out Crack Attack!, a Tetris Attack-like game for Windows and Unix. I just burned 30 minutes, but only got a score of 139.

I took some photos from the balcony of my hotel room. I think the camera was tilted slightly in the last one.

I'll post higher resolutions of them later.

Surprisingly, the Hyatt in Irvine gives you the option of a "Public IP" when first registering for their (not complimentary) Internet access. This apparently not only gives you a public IP (from Time Warner Telecom) but also places your station directly onto an unfiltered/unfirewalled VLAN. I wonder how many Windows boxes are wormed here, daily?

Too bad TWTC's routing picks a fairly suboptimal 6to4 relay:

  3. hagg-01-t3-0-1-0-0.orng.twte  0.0%    10   29.4  27.1  24.6  37.8   4.0
  4. 66.192.251.27                 0.0%    10   26.4  27.3  25.0  36.6   3.3
  5. nyk-bb1-link.telia.net        0.0%    10  101.4 102.3 101.2 104.1   1.1
  6. kbn-bb1-link.telia.net        0.0%    10  207.3 209.3 193.7 232.7  10.1
  7. s-bb1-link.telia.net          0.0%    10  220.0 219.4 204.8 225.1   6.0
  8. s-b1-link.telia.net           0.0%    10  218.5 270.5 209.1 423.1  82.4
  9. 192.88.99.1                  10.0%    10  222.1 226.6 215.3 232.5   5.5

In other news, it's great weather here, and I hope to [re]learn a bit about virtual systems and multicast routing on Juniper firewalls.

Did Cogent depeer with Telia? Signs point to yes. Traceroute from Cogent to Telia:

Tracing the route to 213.248.83.252

  1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 0 msec 0 msec
  2  *  *  * 
  3  *  *  *

Traceroute from Telia to Cogent (Verizon Business as transit?):

Tracing the route to 38.100.128.10

  1 kbn-b1-geth5-0-11.telia.net (213.248.78.21) 0 msec 0 msec 0 msec
  2 kbn-bb2-link.telia.net (80.91.254.26) [AS 1299] 4 msec 0 msec 0 msec
  3 nyk-bb2-pos5-0-0.telia.net (213.248.64.34) [AS 1299] 88 msec 88 msec 84 msec
  4 nyk-b1-link.telia.net (213.248.83.66) [AS 1299] 92 msec 92 msec 96 msec
  5 GigabitEthernet3-0-0.GW18.NYC4.ALTER.NET (157.130.255.205) [AS 701] 88 msec 88 msec 84 msec
  6 0.ge-3-0-0.XL3.NYC4.ALTER.NET (152.63.22.226) [AS 701] 88 msec 92 msec 92 msec
  7 0.ge-6-0-0.BR3.NYC4.ALTER.NET (152.63.18.5) [AS 701] 92 msec 92 msec 92 msec
  8  *  *  * 
  9  *  *  * 
 10  *  *  *

BGPlay confirms that the event happened between 2008-03-13 21:56:11 and 2008-03-13 22:05:41 UTC. Telia is now trying to access Cogent via Verizon Business / UUNet but Cogent either hasn't negotiated transit to Telia yet, or is being nasty about whatever disagreements took place between the two ISPs.

BGPlay screenshot from 2008-03-13 21:56:11 UTC:

Cogent and Telia Connected

BGPlay screenshot from 2008-03-13 22:05:41 UTC:

Cogent and Telia Disconnected

(AS174 is Cogent Communications, AS701 is Verizon Business / UUNET, and AS1299 is TeliaSonora)

So yeah, Cogent sucks. Avoid.