Present Location: News >> Blog


> Concise Year in Review for 2014
Posted by prox, from Seattle, on January 01, 2015 at 22:39 local (server) time

I seem to do these "year in review" things every other year.  I'm primarily doing one this year because the world really needs a hand-written review to stand out in the sea of those horrible computer-generated Facebook ones.  However, unlike previous years, it will be very short.


Space Needle

I moved to Seattle, WA at the end of February.  I don't mind the weather—the rain makes for humid winters and there are frequent instances of a few sunny (but cold!) days; today is one of them.  That being said, it's taken some time for me to adjust to an urban area.

I Changed Employers

Amazon Web Services

I bid adieu to my colleagues at Time Warner Cable and started at Amazon Web Services in February, too.  It's a fun, challenging, and fast-paced environment.  I still feel like a n00b, frequently.


Burntshirt Vineyards

I got engaged!  I'll be married in March of this year.

(more information about the last three events can be found here)


River Liffey

(the River Liffey, near Temple Bar in Dublin)

I traveled 41,888 km (26,028 mi) in 2014 between the following (not including layovers):

The trips to Dublin and Austin were for work but were both pretty fun.  I'll probably be returning to Dublin in the next year or two again but I'm not sure about Austin.


Interstellar makes my year in review because it's one of the best (top three or four) science fiction movies I've seen.  That says a lot.  I only wish I could have seen it in IMAX (the theater schedules were terrible!).

New Christmas Tree

After Christmas for the past 4-5 years I've convinced myself that my artificial tree wouldn't survive another year.  However, the artifial tree I picked up in 2005 lasted a full eight (8) years.  I figured it would not survive a move across the country, so I picked up a new tree, this year.  It's a 7.5 ft. Pre-Lit Fraser Fir.  As I do every year, here's a time lapse of the installation:

(this is the first year I've added music to the time lapse.. hopefully it's not annoying!)


No conclusions.  2014 was what it was.

Comments: 0
> IPv6 on Comcast Business
Posted by prox, from Seattle, on December 06, 2014 at 01:02 local (server) time

Comcast Business decided to give me some IPv6 lovin' a week or two ago on my DOCSIS connection.  Unfortunately, it still seems to be a work in progress.

IPv6 Options

The above is the IPv6 configuration screen on my (well, Comcast Business') NETGEAR CG3000DCR.  I've got a /56 that I suspect was obtained via DHCPv6-PD on the DOCSIS interface.  The LAN interface has a /64 out of that /56 and supports DHCPv6-PD as well.  It's also got RAs enabled with RDNSS:

21:08:16.491207 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::c604:15ff:febe:e634 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120
	hop limit 64, Flags [other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
	  source link-address option (1), length 8 (1): c4:04:15:be:e6:34
	    0x0000:  c404 15be e634
	  prefix info option (3), length 32 (4): 2601:8:a401:4c00::/64, Flags [onlink, auto], valid time 345600s, pref. time 345600s
	    0x0000:  40c0 0005 4600 0005 4600 0000 0000 2601
	    0x0010:  0008 a401 4c00 0000 0000 0000 0000
	  unknown option (24), length 24 (3): 
	    0x0000:  3800 0005 4600 2601 0008 a401 4c00 0000
	    0x0010:  0000 0000 0000
	  rdnss option (25), length 40 (5):  lifetime 60s, addr: 2001:558:feed::1 addr: 2001:558:feed::2
	    0x0000:  0000 0000 003c 2001 0558 feed 0000 0000
	    0x0010:  0000 0000 0001 2001 0558 feed 0000 0000
	    0x0020:  0000 0000 0002

The RDNSS allows IPv6 clients that have not yet embraced DHCPv6 to mostly work without IPv4, which is nice (although I still think it's a hack).

I was puzzled by the other options on the configuration screen, though.  The "Enable Unicast" and "Enable Rapid Commit" initially meant nothing to me and I assumed the "Enable EUI-64 Addressing" toggled the "A" flag in the RA.  I individually toggled all of the options and compared the results, which required a few reboots, unfortunately.  From what I can tell the "Enable Rapid Commit" and "Enable EUI-64 Addressing" options do absolutely nothing to the RA messages.  However, "Enable Unicast" removes all the flags and all but one option in the RA:

20:59:52.178671 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::c604:15ff:febe:e634 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24
     hop limit 64, Flags [none], pref medium, router lifetime 0s, reachable time 0s, retrans time 0s
       source link-address option (1), length 8 (1): c4:04:15:be:e6:34
         0x0000:  c404 15be e634

I'm not sure why that option isn't labeled appropriately or why enabling "unicast" would result in such a configuration.  Silly consumer networking equipment.

Anyway, I threw a host on the link and, of course, started running some traceroutes.  I expected to see the LAN interface of the CG3000DCR as the first hop, but I didn't:

HOST: rpi                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:558:4082:48::1                              0.0%     1   10.8  10.8  10.8  10.8   0.0
  2.|--    0.0%     1    9.0   9.0   9.0   9.0   0.0
  3.|--       0.0%     1    8.7   8.7   8.7   8.7   0.0
  4.|--       0.0%     1   11.8  11.8  11.8  11.8   0.0
  5.|--  0.0%     1   10.9  10.9  10.9  10.9   0.0
  6.|--   0.0%     1   26.2  26.2  26.2  26.2   0.0
  7.|--     0.0%     1   27.0  27.0  27.0  27.0   0.0
  8.|--       0.0%     1   29.3  29.3  29.3  29.3   0.0
  9.|--                        0.0%     1   81.8  81.8  81.8  81.8   0.0
 10.|--                                           0.0%     1   37.3  37.3  37.3  37.3   0.0

Other than the bad PTR on hop #7, the first hop was unexpected.  Is it, instead, the DOCSIS interface address that's being used in the ICMPv6 response?

(rpi:21:45)% mtr -rwc1 2001:558:4082:48::1
Start: Fri Dec  5 21:46:21 2014
HOST: rpi                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:558:4082:48::1  0.0%     1    9.1   9.1   9.1   9.1   0.0

Maybe.  But when I traceroute to my host from the outside, why do I see 2001:558:a2:f3::2, instead?

HOST:                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2a00:86c0:ff0a:2906::1                      0.0%     2    1.0   1.0   1.0   1.0   0.0
  2.|-- 2a00:86c0:26c::9                            0.0%     1    0.3   0.3   0.3   0.3   0.0
  3.|-- 2001:559::ca1                               0.0%     1    0.9   0.9   0.9   0.9   0.0
  4.|--  0.0%     1    1.6   1.6   1.6   1.6   0.0
  5.|-- 2001:558:0:f747::2                          0.0%     1   19.2  19.2  19.2  19.2   0.0
  6.|--    0.0%     1   20.4  20.4  20.4  20.4   0.0
  7.|-- 2001:558:a2:f3::2                           0.0%     1   18.8  18.8  18.8  18.8   0.0
  8.|-- 2601:8:a401:4c00:ba27:ebff:fe8f:7486        0.0%     1   28.4  28.4  28.4  28.4   0.0

This appears to be an anomaly when an ICMPv6-based traceroute is used.  A UDP one works as expected:

(rpi:21:49)% mtr -u -rwc1          
Start: Fri Dec  5 21:49:35 2014
HOST: rpi                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2601:8:a401:4c00:c604:15ff:feee:e634             0.0%     1    0.9   0.9   0.9   0.9   0.0
  2.|-- 2001:558:4082:48::1                              0.0%     1   16.2  16.2  16.2  16.2   0.0
  3.|--    0.0%     1    8.9   8.9   8.9   8.9   0.0
  4.|--       0.0%     1   13.0  13.0  13.0  13.0   0.0
  5.|--       0.0%     1   12.5  12.5  12.5  12.5   0.0
  6.|--  0.0%     1   11.1  11.1  11.1  11.1   0.0
  7.|--   0.0%     1   28.2  28.2  28.2  28.2   0.0
  8.|--     0.0%     1   26.7  26.7  26.7  26.7   0.0
  9.|--       0.0%     1   31.8  31.8  31.8  31.8   0.0
 10.|--                        0.0%     1   38.2  38.2  38.2  38.2   0.0
 11.|-- ???                                             100.0     1    0.0   0.0   0.0   0.0   0.0

2001:558:4082:48::1 is obviously the CMTS.

Now that we've got the initial exploring out of the way, I figured I would try the "static route" feature on the CG3000DCR to see if it worked with IPv6 at all:

IPv6 Static Route Fail

This was a complete fail.  I tried adding a /64 static route to a static host I have on the link with a subnet mask of "64" and then one of "ffff:ffff:ffff:ffff::" but both failed.

As far as the permanency of this rather useless /56 I'm given, a quick look around on the support forums made me conclude that the current prefixes are dynamic and static prefixes will be rolled out in the future.  I suppose that answers the question about reverse DNS delegation, too.  I can't imagine that being a possibility at all without static assignments.  I also sure do hope that they provide an option for delegation instead of just adding individual records like they do for IPv4 (and, poorly, I may add, since it must be done via a phone call).

I guess I'm not ready to dump my tunnels, yet.  I'd like a static assignment and reverse DNS.  Otherwise, the sweetness of native IPv6 still feels like a downgrade.

Comments: 2
> Inappropriate Words
Posted by prox, from Seattle, on November 19, 2014 at 01:00 local (server) time

No, it's not what you think.

I've recently realized I am starting to use a few words inappropriately:

Comments: 0
> DNS Stupidity
Posted by prox, from Seattle, on September 23, 2014 at 01:18 local (server) time

Whatever you want to call it, I'm a firm believer that this should work:

% cat /etc/resolv.conf
% host has address has IPv6 address 2001:db8::1
% host has address has IPv6 address 2001:db8::1

Unfortunately, it seems like most operating systems are starting to break this by not appending the search prefix to short names that contain dots.  Windows and Mac OS X have changed to this broken behavior by default and now require obscure tweaks to get the correct behavior back.

Now, since Linux lately is becoming a wannabe Mac OS X, it apparently now suffers from the same issue:

(destiny:22:08)% host has address
(destiny:22:08)% host nox.ext               
Host nox.ext not found: 3(NXDOMAIN)
(destiny:22:08)% cat /etc/resolv.conf

We can see that the resolver libraries aren't bothering to append to nox.ext in the second query:

22:08:45.103912 IP > 28936+ A? (39)
22:08:45.104274 IP > 28936* 1/7/14 A (507)
22:08:45.104703 IP > 13808+ AAAA? (39)
22:08:45.104970 IP > 13808* 0/1/0 (108)
22:08:45.105245 IP > 55206+ MX? (39)
22:08:45.105450 IP > 55206* 0/1/0 (108)
22:08:46.946125 IP > 38662+ A? nox.ext. (25)
22:08:46.946415 IP > 38662 NXDomain 0/1/0 (100)

However, I've got another machine sitting here that's running the same version of libc-bin (2.19-7) and does work.  I'm failing to identify the delta between these machines.

Is this a bug?  Is this expected behavior?  How do I fix it?  No, the "ndots" parameter for /etc/resolv.conf is irrelevant.

Update: As some folks have pointed out, the reliance of short names even on controlled networks is being discouraged due to the new ICANN TLDs.  I've been aware of the consequences of using short names and the potential for blackholing legitimate external sites using these new TLDs and still believe I should be able to use the legacy resolver behavior.  If there's another way of not having to type a million times per day, though, I'm all ears!

Update 2: After a post I made to debian-user I realized that the ndots parameter is not irrelevant and fixed my issue.  However, changing it shouldn't have fixed the issue because the defaults (I checked the source) are ndots:4.

Update 3: I'm an idiot.  I wasn't using anything other than host(1) to do testing.  host(1) implements its own resolver library and reads /etc/resolv.conf itself.  Apparently something changed in host(1), but didn't impact anything else!  Case closed!

Comments: 3
> systemd-journal
Posted by prox, from Seattle, on August 02, 2014 at 23:14 local (server) time

I've been slowly converting my Debian boxes over to the systemd init system (by installing the systemd-sysv package), recently.  I haven't encountered any problems, yet.

However, it seems that my lab box has some systemd component already burning away at the CPU cores:

systemd fail

The worst part is that I haven't installed systemd-sysv on that box, yet!

Update: It looks like these systemd-journal processes are running under my Linux containers, which are using systemd.  journalctl(1) does not show any errors so I killed both processes (they immediately respawned) and the CPU usage is gone.  Yeah.. we're going to be dealing with issues like this for years.  Sigh.

Comments: 0
> Virtual Routers and LXC
Posted by prox, from Renton, on July 19, 2014 at 20:30 local (server) time

Ever since 2004 or 2005 I've wanted to see real virtual router functionality on Linux.  It would make setting up a lightweight networking lab with Quagga (or BIRD) a pinch and also allow me to leverage multiple Internet connections for VPN isolation, amongst other things.

You might say, "Hey, Linux has that in the form of tables and rules that can be manipulated by ip(8)!"  It does, sort-of.  It's possible to setup another routing table (optionally naming it in /etc/iproute2/rt_tables), add arbitrary routes, then setup rules to always tell the kernel to use a specific routing table for all packets coming from a certain IP address (or many more things, if you use iptables MARK).  This only "sort-of" works because there's no way (from what I can tell) to actually bind interfaces to a particular routing table.  There's also the issue with overlapping IP space—how do I tell ssh(1), for example, to use a particular routing table if there are two interfaces with the same IP address?  The -b argument won't do me much good.  Also, DHCP is problematic because heavy modifications are needed in dhclient-script and they'd be mostly implmentation-specific.

So, although I've used multiple routing tables with rules, they don't really fit my definition of virtual routers (VRF-lite, in Cisco parlance), which I define as a isolated construct has exclusive access to a set of interfaces and their addresses.

Linux Containers

I've been messing with Linux containers (LXC) recently and I think they might provide the exact functionality I've been looking for all these years.  With LXC it's possible to fire up a small instance that has its own routing table, interfaces, and applications.  There's no need to use rules or -b arguments anymore.  DHCP and Quagga don't require any hacks and work the way they should.

Networking with LXC is about what I expect; interfaces are dedicated to the container.  Overlapping IP addresses are certainly possible, if there's a need for that.  Connecting a container to the host or other containers can be achieved using a virtual Ethernet interface with a bridge.  This makes it easy to setup multiple Linux "virtual routers" without ever having to mess with VMs, routing tables or rules.  A good article on various LXC networking modes can be found here.

If you're familiar with Junos, Linux containers are almost analgous to logical systems in this type of role.  Logical routers, unlike VRFs, have their own copy of RPD, which is a daemon that handles all dynamic routing protocols.  If you're using Quagga with containers, the architecture is similar.

A slight drawback I can see with containers as virtual routers is the disk space usage.  Each container has, by default, its filesystem stored in a separate directory in /var/lib/lxc.  There's quite a bit of redundant data if you fire up many containers using the same distribution (e.g., Debian).  I'm sure there is some way to de-duplicate this (which would help with package upgrades, too!) but I haven't really looked into it because storage is so cheap nowadays and most of us have plenty of it.  A fairly fully-featured Debian container I've got is not that large, anyway:

(vega:17:04)# du -hcs /var/lib/lxc/*
4.0K	/var/lib/lxc/lxc-monitord.log
488M	/var/lib/lxc/soran
488M	total

So, in summary, for general-use virtual routers, I think LXC is pretty great.  The best part is that the only thing required to use LXC is a recent kernel with cgroups enabled and mounted properly.

Comments: 0
> Unit and Time Selection Inconstencies
Posted by prox, from Seattle, on June 07, 2014 at 18:31 local (server) time

Unlike most people in the United States, I like the metric system.  I prefer to use it instead of imperial units as much as possible and change the settings on my electronic equipment to display metric units.  Unfortunately, some equipment is fairly inconsistent with its support of the metric system and even 12 vs. 24-hour time.  Here's what I've got:

Carrier Thermostat

This thing tricked me into thinking it supports both 24-hour time and Celsius by showing "options: 24 hours" in the manual.  Unfortunately, it only supports changing the temperature unit.  The time must stay in 12-hour mode.

DSC Power Series Alarm Keypad

I successfully changed this to 24-hour mode.  There appears to be a thermal diode in the unit but there is no way to actually display the current temperature to the user so there is no way to set the scale.

Acura TSX

My car is a bit odd.  The navigation system can successfully be changed to use the metric system but the outside temperature and fuel level appear to be hard-coded to use imperial units.  The clock must stay in 12-hour mode, too.  Sadly, I do believe there is a "switch" for this but it's inaccessible to the user; the Canadian version of the car displays everything in metric.

Whirlpool Oven

This thing lives in the world of Fahrenheit and has a permanent 12-hour clock.

Whirlpool Microwave

There is no way to change the clock to 24-hour mode.

Motorola HD-DVR

I rent this thing from Comcast.  It displays time in 12-hour mode, only.

Brookstone Alarm Clock

I can switch this thing to 24-hour mode and Celsius, since it features a temperature sensor.

Samsung Clock / Radio

No 24-hour mode for this thing, even though the display is digital.

La Crosse Technology Atomic Wall Clock

This also features a local and remote temperature sensor.  I can switch this to Celsius and 24-hour mode easily.

If I were like one of those people who think we should have a law for everything, I'd say "there should be a law requiring all electronics companies to include support for the metric system in their products."  Thankfully, I'm not one of those people, so I think the best thing to do is vote with your wallet.  The next car I buy will support metric units in every feature!

Comments: 0
> Updates!
Posted by prox, from Seattle, on May 09, 2014 at 00:14 local (server) time

It occurred to me that I haven't blogged in quite some time.  So, here you go!


I moved into my townhouse in West Seattle about a month ago.  The house itself is about eight (8) years old so there's not much I'm going to change or replace right off the bat.  My fiancée has one or two things to say about the color schemes in some of the rooms, though, so there will be painting in the future.  It's in the High Point community, which is still expanding rapidly and there are lots of new townhouses and single family houses being built.  Unpacking took awhile but I'm more or less settled, now.  I do need to buy some more furniture in the future since I have roughly 450 more square footage than I did in Charlotte.

I managed to get lucky with the cabling.  All of the rooms in the house are wired for Cat5e and are already punched down in a central Leviton telecom panel in the closet of the master bedroom:

Leviton Telecom Box

There's also a second port in several rooms that is wired for Cat5e but all of those ports are wired together.  I ended up connecting my old Linksys PAP2T to those ports for analog phone compatibility, which I've actually used a bit.  I ended up putting my Juniper EX2200-C and ALIX box in the closet, as well:

Bedroom Closet

I haven't turned up my lab yet, since I was told I shouldn't be putting any loud things in the bedroom closet.


Ah, Comcast.  I started off with xfinity 50/10 HSD and basic cable, which seemed fine until I realized I couldn't get more than one public IPv4 address.  Back in Charlotte I was able to get the CPE limit increased to three (3) so I could have two firewalls as well as a Linux box connected to the Internet.  After a long call with a fairly clueful CSR, he informed me that they no longer are able to increase the CPE limits for residential HSD and I'd have to get business class, which I did.

I got the same 50/10 service from Comcast Business but with a static /29.  They wouldn't let me use my own modem because I was getting a static assignment so they gave me a Comcast-branded NETGEAR CG3000DCR.  I suspect this is because of how they inject the /29 into their network, which is probably via RIP.  The speed is fine and the latency to the CMTS isn't too bad, although it seems to bounce around a bit:

Comcast CMTS

Here's a traceroute showing that nice paid peering between Comcast and Netflix:

Start: Thu May  8 20:34:26 2014
HOST: starfire                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                                          0.0%     8    0.4   0.4   0.4   0.6   0.0
  2.|--                                        0.0%     8    8.7   8.5   7.2  10.0   0.7
  3.|--      0.0%     8    9.7   9.5   7.6  11.1   0.9
  4.|--            0.0%     8    9.5   9.5   8.2  10.6   0.5
  5.|--         0.0%     8    9.7   9.6   8.5  10.9   0.7
  6.|--    0.0%     8   10.3  11.7   9.3  18.2   2.8
  7.|--      0.0%     8   29.4  28.9  26.8  31.8   1.9
  8.|--  0.0%     8   27.0  37.1  26.9 103.8  27.0
  9.|--               0.0%     8   69.0  68.4  61.2  79.2   5.3
 10.|--                                       0.0%     8   69.3  67.8  63.8  70.3   2.2

They also let me set my own PTR records, which is nice, although they offer no self-service way of doing this.  I had to call them up.  Oh.. and I must say there is a night & day difference between residential and commercial phone support.

Unfortunately, while Comcast does support IPv6 on the xfinity side of the house they do not yet support it for their commercial offering, so I'm still tunneling.

That being said, I had a bit of a headache when I tried to cancel the HSD portion of my xfinity residential service.  At first when I canceled the service everything seemed fine.  However, the CableCARD tied to my account (connected to the ATI DCT on my Windows Media Center PC) magically stopped working around the same time.  I had to call them bad and they admitted they "made a mistake" when they were canceling my HSD account.  That's fine.  However, a few days later after I sold my old Motorola SB6141 to someone at work, I come to find out that the cable modem is still tied to my account.  Again I had to call them and tell them to take it off my account.  Surprisingly, my CableCARD and Motorola HD box didn't break this time.


The commute from West Seattle isn't as bad as everyone had warned.  If I leave at around 07:00 I can be in the parking garage in around 18 minutes.  From there it's a short 5-6 minute walk to the office and I can be at my desk by 07:30.  I haven't quite figured out the best time to leave the office but it seems to take me anywhere from 25 to 30 minutes from the parking deck to my garage.  Regardless, nothing as infuriating as I-485 back in Charlotte!

Work is fairly challenging and my first project is almost done.  Amazon sure puts the wireless industry to shame with the sheer number of acronyms, nicknames, and proprietary lingo that is used for various things.  I'm certainly having fun, so far!  Yes, we are still hiring.


The first two warm days were last week and it got up to 29.4°C before I opened up the windows and cooled it down.  I'm going to try the first summer without any air conditioning and then figure out if I need to get central AC or a portable unit for next year.

Anyway, the spring is fairly nice out here!  It's nice and green, now:

Seattle Spring

The pollen is practically nonexistant compared to Charlotte.  In fact, I haven't noticed any of it and, as a result, I've managed to skip my seasonal allergies completely, which is nice.

Other Stuff

I think there's a bug in Mutt 1.5.23.  Ever since I upgraded from 1.5.21 I've noticed doing a full-text message search is horribly slow.  I think it may have to do with Debian bug 745532.

Comments: 0

Previous PageDisplaying page 2 of 118 of 939 results Next Page