Present Location: News >> Blog


> Inappropriate Words
Posted by prox, from Seattle, on November 19, 2014 at 01:00 local (server) time

No, it's not what you think.

I've recently realized I am starting to use a few words inappropriately:

Comments: 0
> DNS Stupidity
Posted by prox, from Seattle, on September 23, 2014 at 01:18 local (server) time

Whatever you want to call it, I'm a firm believer that this should work:

% cat /etc/resolv.conf
% host has address has IPv6 address 2001:db8::1
% host has address has IPv6 address 2001:db8::1

Unfortunately, it seems like most operating systems are starting to break this by not appending the search prefix to short names that contain dots.  Windows and Mac OS X have changed to this broken behavior by default and now require obscure tweaks to get the correct behavior back.

Now, since Linux lately is becoming a wannabe Mac OS X, it apparently now suffers from the same issue:

(destiny:22:08)% host has address
(destiny:22:08)% host nox.ext               
Host nox.ext not found: 3(NXDOMAIN)
(destiny:22:08)% cat /etc/resolv.conf

We can see that the resolver libraries aren't bothering to append to nox.ext in the second query:

22:08:45.103912 IP > 28936+ A? (39)
22:08:45.104274 IP > 28936* 1/7/14 A (507)
22:08:45.104703 IP > 13808+ AAAA? (39)
22:08:45.104970 IP > 13808* 0/1/0 (108)
22:08:45.105245 IP > 55206+ MX? (39)
22:08:45.105450 IP > 55206* 0/1/0 (108)
22:08:46.946125 IP > 38662+ A? nox.ext. (25)
22:08:46.946415 IP > 38662 NXDomain 0/1/0 (100)

However, I've got another machine sitting here that's running the same version of libc-bin (2.19-7) and does work.  I'm failing to identify the delta between these machines.

Is this a bug?  Is this expected behavior?  How do I fix it?  No, the "ndots" parameter for /etc/resolv.conf is irrelevant.

Update: As some folks have pointed out, the reliance of short names even on controlled networks is being discouraged due to the new ICANN TLDs.  I've been aware of the consequences of using short names and the potential for blackholing legitimate external sites using these new TLDs and still believe I should be able to use the legacy resolver behavior.  If there's another way of not having to type a million times per day, though, I'm all ears!

Update 2: After a post I made to debian-user I realized that the ndots parameter is not irrelevant and fixed my issue.  However, changing it shouldn't have fixed the issue because the defaults (I checked the source) are ndots:4.

Update 3: I'm an idiot.  I wasn't using anything other than host(1) to do testing.  host(1) implements its own resolver library and reads /etc/resolv.conf itself.  Apparently something changed in host(1), but didn't impact anything else!  Case closed!

Comments: 3
> systemd-journal
Posted by prox, from Seattle, on August 02, 2014 at 23:14 local (server) time

I've been slowly converting my Debian boxes over to the systemd init system (by installing the systemd-sysv package), recently.  I haven't encountered any problems, yet.

However, it seems that my lab box has some systemd component already burning away at the CPU cores:

systemd fail

The worst part is that I haven't installed systemd-sysv on that box, yet!

Update: It looks like these systemd-journal processes are running under my Linux containers, which are using systemd.  journalctl(1) does not show any errors so I killed both processes (they immediately respawned) and the CPU usage is gone.  Yeah.. we're going to be dealing with issues like this for years.  Sigh.

Comments: 0
> Virtual Routers and LXC
Posted by prox, from Renton, on July 19, 2014 at 20:30 local (server) time

Ever since 2004 or 2005 I've wanted to see real virtual router functionality on Linux.  It would make setting up a lightweight networking lab with Quagga (or BIRD) a pinch and also allow me to leverage multiple Internet connections for VPN isolation, amongst other things.

You might say, "Hey, Linux has that in the form of tables and rules that can be manipulated by ip(8)!"  It does, sort-of.  It's possible to setup another routing table (optionally naming it in /etc/iproute2/rt_tables), add arbitrary routes, then setup rules to always tell the kernel to use a specific routing table for all packets coming from a certain IP address (or many more things, if you use iptables MARK).  This only "sort-of" works because there's no way (from what I can tell) to actually bind interfaces to a particular routing table.  There's also the issue with overlapping IP space—how do I tell ssh(1), for example, to use a particular routing table if there are two interfaces with the same IP address?  The -b argument won't do me much good.  Also, DHCP is problematic because heavy modifications are needed in dhclient-script and they'd be mostly implmentation-specific.

So, although I've used multiple routing tables with rules, they don't really fit my definition of virtual routers (VRF-lite, in Cisco parlance), which I define as a isolated construct has exclusive access to a set of interfaces and their addresses.

Linux Containers

I've been messing with Linux containers (LXC) recently and I think they might provide the exact functionality I've been looking for all these years.  With LXC it's possible to fire up a small instance that has its own routing table, interfaces, and applications.  There's no need to use rules or -b arguments anymore.  DHCP and Quagga don't require any hacks and work the way they should.

Networking with LXC is about what I expect; interfaces are dedicated to the container.  Overlapping IP addresses are certainly possible, if there's a need for that.  Connecting a container to the host or other containers can be achieved using a virtual Ethernet interface with a bridge.  This makes it easy to setup multiple Linux "virtual routers" without ever having to mess with VMs, routing tables or rules.  A good article on various LXC networking modes can be found here.

If you're familiar with Junos, Linux containers are almost analgous to logical systems in this type of role.  Logical routers, unlike VRFs, have their own copy of RPD, which is a daemon that handles all dynamic routing protocols.  If you're using Quagga with containers, the architecture is similar.

A slight drawback I can see with containers as virtual routers is the disk space usage.  Each container has, by default, its filesystem stored in a separate directory in /var/lib/lxc.  There's quite a bit of redundant data if you fire up many containers using the same distribution (e.g., Debian).  I'm sure there is some way to de-duplicate this (which would help with package upgrades, too!) but I haven't really looked into it because storage is so cheap nowadays and most of us have plenty of it.  A fairly fully-featured Debian container I've got is not that large, anyway:

(vega:17:04)# du -hcs /var/lib/lxc/*
4.0K	/var/lib/lxc/lxc-monitord.log
488M	/var/lib/lxc/soran
488M	total

So, in summary, for general-use virtual routers, I think LXC is pretty great.  The best part is that the only thing required to use LXC is a recent kernel with cgroups enabled and mounted properly.

Comments: 0
> Unit and Time Selection Inconstencies
Posted by prox, from Seattle, on June 07, 2014 at 18:31 local (server) time

Unlike most people in the United States, I like the metric system.  I prefer to use it instead of imperial units as much as possible and change the settings on my electronic equipment to display metric units.  Unfortunately, some equipment is fairly inconsistent with its support of the metric system and even 12 vs. 24-hour time.  Here's what I've got:

Carrier Thermostat

This thing tricked me into thinking it supports both 24-hour time and Celsius by showing "options: 24 hours" in the manual.  Unfortunately, it only supports changing the temperature unit.  The time must stay in 12-hour mode.

DSC Power Series Alarm Keypad

I successfully changed this to 24-hour mode.  There appears to be a thermal diode in the unit but there is no way to actually display the current temperature to the user so there is no way to set the scale.

Acura TSX

My car is a bit odd.  The navigation system can successfully be changed to use the metric system but the outside temperature and fuel level appear to be hard-coded to use imperial units.  The clock must stay in 12-hour mode, too.  Sadly, I do believe there is a "switch" for this but it's inaccessible to the user; the Canadian version of the car displays everything in metric.

Whirlpool Oven

This thing lives in the world of Fahrenheit and has a permanent 12-hour clock.

Whirlpool Microwave

There is no way to change the clock to 24-hour mode.

Motorola HD-DVR

I rent this thing from Comcast.  It displays time in 12-hour mode, only.

Brookstone Alarm Clock

I can switch this thing to 24-hour mode and Celsius, since it features a temperature sensor.

Samsung Clock / Radio

No 24-hour mode for this thing, even though the display is digital.

La Crosse Technology Atomic Wall Clock

This also features a local and remote temperature sensor.  I can switch this to Celsius and 24-hour mode easily.

If I were like one of those people who think we should have a law for everything, I'd say "there should be a law requiring all electronics companies to include support for the metric system in their products."  Thankfully, I'm not one of those people, so I think the best thing to do is vote with your wallet.  The next car I buy will support metric units in every feature!

Comments: 0
> Updates!
Posted by prox, from Seattle, on May 09, 2014 at 00:14 local (server) time

It occurred to me that I haven't blogged in quite some time.  So, here you go!


I moved into my townhouse in West Seattle about a month ago.  The house itself is about eight (8) years old so there's not much I'm going to change or replace right off the bat.  My fiancée has one or two things to say about the color schemes in some of the rooms, though, so there will be painting in the future.  It's in the High Point community, which is still expanding rapidly and there are lots of new townhouses and single family houses being built.  Unpacking took awhile but I'm more or less settled, now.  I do need to buy some more furniture in the future since I have roughly 450 more square footage than I did in Charlotte.

I managed to get lucky with the cabling.  All of the rooms in the house are wired for Cat5e and are already punched down in a central Leviton telecom panel in the closet of the master bedroom:

Leviton Telecom Box

There's also a second port in several rooms that is wired for Cat5e but all of those ports are wired together.  I ended up connecting my old Linksys PAP2T to those ports for analog phone compatibility, which I've actually used a bit.  I ended up putting my Juniper EX2200-C and ALIX box in the closet, as well:

Bedroom Closet

I haven't turned up my lab yet, since I was told I shouldn't be putting any loud things in the bedroom closet.


Ah, Comcast.  I started off with xfinity 50/10 HSD and basic cable, which seemed fine until I realized I couldn't get more than one public IPv4 address.  Back in Charlotte I was able to get the CPE limit increased to three (3) so I could have two firewalls as well as a Linux box connected to the Internet.  After a long call with a fairly clueful CSR, he informed me that they no longer are able to increase the CPE limits for residential HSD and I'd have to get business class, which I did.

I got the same 50/10 service from Comcast Business but with a static /29.  They wouldn't let me use my own modem because I was getting a static assignment so they gave me a Comcast-branded NETGEAR CG3000DCR.  I suspect this is because of how they inject the /29 into their network, which is probably via RIP.  The speed is fine and the latency to the CMTS isn't too bad, although it seems to bounce around a bit:

Comcast CMTS

Here's a traceroute showing that nice paid peering between Comcast and Netflix:

Start: Thu May  8 20:34:26 2014
HOST: starfire                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                                          0.0%     8    0.4   0.4   0.4   0.6   0.0
  2.|--                                        0.0%     8    8.7   8.5   7.2  10.0   0.7
  3.|--      0.0%     8    9.7   9.5   7.6  11.1   0.9
  4.|--            0.0%     8    9.5   9.5   8.2  10.6   0.5
  5.|--         0.0%     8    9.7   9.6   8.5  10.9   0.7
  6.|--    0.0%     8   10.3  11.7   9.3  18.2   2.8
  7.|--      0.0%     8   29.4  28.9  26.8  31.8   1.9
  8.|--  0.0%     8   27.0  37.1  26.9 103.8  27.0
  9.|--               0.0%     8   69.0  68.4  61.2  79.2   5.3
 10.|--                                       0.0%     8   69.3  67.8  63.8  70.3   2.2

They also let me set my own PTR records, which is nice, although they offer no self-service way of doing this.  I had to call them up.  Oh.. and I must say there is a night & day difference between residential and commercial phone support.

Unfortunately, while Comcast does support IPv6 on the xfinity side of the house they do not yet support it for their commercial offering, so I'm still tunneling.

That being said, I had a bit of a headache when I tried to cancel the HSD portion of my xfinity residential service.  At first when I canceled the service everything seemed fine.  However, the CableCARD tied to my account (connected to the ATI DCT on my Windows Media Center PC) magically stopped working around the same time.  I had to call them bad and they admitted they "made a mistake" when they were canceling my HSD account.  That's fine.  However, a few days later after I sold my old Motorola SB6141 to someone at work, I come to find out that the cable modem is still tied to my account.  Again I had to call them and tell them to take it off my account.  Surprisingly, my CableCARD and Motorola HD box didn't break this time.


The commute from West Seattle isn't as bad as everyone had warned.  If I leave at around 07:00 I can be in the parking garage in around 18 minutes.  From there it's a short 5-6 minute walk to the office and I can be at my desk by 07:30.  I haven't quite figured out the best time to leave the office but it seems to take me anywhere from 25 to 30 minutes from the parking deck to my garage.  Regardless, nothing as infuriating as I-485 back in Charlotte!

Work is fairly challenging and my first project is almost done.  Amazon sure puts the wireless industry to shame with the sheer number of acronyms, nicknames, and proprietary lingo that is used for various things.  I'm certainly having fun, so far!  Yes, we are still hiring.


The first two warm days were last week and it got up to 29.4°C before I opened up the windows and cooled it down.  I'm going to try the first summer without any air conditioning and then figure out if I need to get central AC or a portable unit for next year.

Anyway, the spring is fairly nice out here!  It's nice and green, now:

Seattle Spring

The pollen is practically nonexistant compared to Charlotte.  In fact, I haven't noticed any of it and, as a result, I've managed to skip my seasonal allergies completely, which is nice.

Other Stuff

I think there's a bug in Mutt 1.5.23.  Ever since I upgraded from 1.5.21 I've noticed doing a full-text message search is horribly slow.  I think it may have to do with Debian bug 745532.

Comments: 0
> Odd NTP Attack
Posted by prox, from Seattle, on March 23, 2014 at 00:19 local (server) time

We've all heard about the recent NTP reflection attacks.  Last night I noticed a higher-than-normal traffic volume on nox, so I checked it out with tcpdump:

Note, the first and second octets have been anonymized to protect the victim.

21:07:07.999600 IP > NTPv3, Client, length 48
21:07:07.999608 IP > NTPv3, Client, length 48
21:07:07.999617 IP > NTPv3, Client, length 48
21:07:07.999625 IP > NTPv3, Client, length 48
21:07:07.999712 IP > NTPv3, Client, length 48
21:07:07.999722 IP > NTPv3, Client, length 48
21:07:07.999730 IP > NTPv3, Client, length 48

Yes, nox is a public NTP server.  It's a member of the NTP Pool Project.  No, it's not susceptible to an NTP reflection attack.  It looks like some poor soul at (looked like a SonicWALL when I poked around) was being attacked and the traffic above was being spoofed with the intention of having my server send back a reply that's much larger than the request.  Here's a decode of one of the packets:

21:07:07.772681 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto UDP (17), length 76) > [udp sum ok] NTPv3, length 48
        Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 10s, precision -19
        Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3604450027.692652940 (2014/03/21 21:07:07)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3604450027.692652940 (2014/03/21 21:07:07)

What's odd about this is the packet above looks like just a normal NTP query.  Unlike most of the NTP reflection attacks that exploit the monlist or similar commands, this wasn't really going to have the desired effect.  And, of course, if you look at the initial (before I blocked that source address with iptables) traffic volume, it certainly did not:


The desired effect, of course, should have been an outbound traffic volume that was greater than the inbound traffic volume, or amplified.  In this case, my server was just sending back a 48 byte packet for every 48 byte packet coming in, albeit apparently slightly ratelimited by the NTP daemon.

Was this a misconfigured DDoS bot?  Did the attacker really not know what he or she was doing or missed DDoS 101?  Or, was this traffic not actually spoofed and was a result of some broken NTP client?  Maybe.

Regardless, if this wasn't a misconfigured NTP client, BCP 38 would have prevented this from happening to begin with.  I don't know where the traffic was originating, but I do know that it was from a network that probably doesn't implement BCP 38.

Anyway, I thought this was a little odd so I figured I would share.

Comments: 0
> It's Time for A Change
Posted by prox, from Seattle, on March 09, 2014 at 13:40 local (server) time

No, I'm not quoting Bill Clinton.  I haven't posted in a few months because it was time for a change and I decided to pull the trigger.  I left Time Warner Cable in Charlotte and moved to Seattle to work for Amazon Web Services.

I'm not going to say much about the job change because it's really not appropriate for a public-facing blog and there's really not much I can say.  I'm happy to be advancing my career and working on a network that either powers 1% of the Internet or provides content to 1/3 of all Internet users (depending on who you ask).

Before I left Charlotte I got engaged to my beautiful fiancée on Valentine's Day.

Devon and Mark

We're set to be married next year on a day that commemorates the ratio between a circle's circumference and diameter.

At the moment I'm living in the Belltown area of Seattle in Washington state.  My ten (10) minute commute is thoroughly spoiling as it'll be a bit longer when I move into my permanent residence in a couple of weeks.

I'm still trying to figure out Seattle.  It's got elements from both uptown Charlotte and Manhattan but with its own twist.  The people who I come in contact with on a daily basis are very friendly, which from what I hear is actually abnormal.  The weather here is routinely overcast and rainy but I've found that weather forecasts are terribly inaccurate due to the surrounding mountains in the region.  Most days that are forecasted as rainy actually end up with an hour or two of sun between the fronts, which is nice.  I was told that only 17% of residences have central air conditioning.  At first, I thought this was a bit odd (coming from the south!) but I suppose it makes sense due to the climate.  I'll find out this summer if my computers will require extra cooling or if they'll be fine sans-AC!

One nice thing is the Comcast XFINITY service in the area provides native IPv6 with prefix delegation.  Finally, no more tunneling!  Although, I do like the permanent nature of a tunneled /56 or /48 as well as the reverse DNS delegation.  Maybe I'll ultimately use a combination of both.

Comments: 1

Previous PageDisplaying page 2 of 118 of 937 results Next Page