Present Location: News >> Blog


> Dead RPis and Interface Naming
Posted by prox, from Seattle, on May 22, 2016 at 22:18 local (server) time

I recently had a case where a few of my [very] remote Raspberry Pis dropped off the network upon reboot (power failure, etc.).  I run the 'testing' release of Raspbian & Debian on my Linux boxes so I do an apt-get dist-upgrade rather often.  The RPis that dropped off the network had indeed been upgraded recently so I had one shipped back to me for investigation.

It turns out that eth0 and wlan0 had been renamed to enx$MAC and looked like this:

Previously wlan0

(I had the RPi plugged into my TV)

I do know about predictable network interface names, which is supposed to prevent interface names changing when hardware is swapped, among other things.  The enx$MAC naming scheme is used for USB devices.  Up until now I've been content to disable this feature by adding net.ifnames=0 to the kernel command-line.  Indeed, I had this flag in /boot/cmdline.txt on the Pis but it just isn't working anymore.

It turns out that this feature is set to be deprecated in Debian 10, along with the automatically-generated /etc/udev/rules.d/70-persistent-net.rules file, which is appended when a new NIC is added to the system.  This is documented in /usr/share/doc/udev/README.Debian.gz, apparently.

Unfortunately for me, the RPis never had 70-persistent-net.rules written and apparently udev 229-5 already ignores net.ifnames=0.  One of my dist-upgrades bumped udev from 229-4 to 229-5.  This is why none of the interface names that were defined in /etc/network/interfaces were found on boot and the RPis booted up with no enabled interfaces.

/usr/share/doc/udev/README.Debian.gz goes on to say that if custom interface names are needed, use 76-netnames.rules with the following content, which looks similar to what was previously written out in 70-persistent-net.rules automatically:

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:50:56:1a:aa:ab", NAME="eth0"

There's no mention of this method of interface naming being deprecated so I'm going to use it for now.

This is just the latest annoyance, for me, when it comes to Linux's interface naming.  Several years back the right way to setup predictable interface names was to use /etc/iftab and ifrename(8)!

What I haven't figured out is how this is going to impact VMs and virtio.  My own VMs with VirtualBox have static MAC addresses assigned to the vbox file so using 76-netnames.rules will work fine.  However, I'm not sure if EC2 and other Xen-based VPS providers always provide a constant MAC address.  One of my EC2 instances has 53-ec2-network-interfaces.rules, which handles some sort of hotplugging.  I'm not sure if that script will stop working as well after some time and udev upgrades but it would seem the solution is to use some ATTR other than 'addresss'.

Comments: 0
> Watch out for Quagga upgrades!
Posted by prox, from Seattle, on May 07, 2016 at 14:00 local (server) time

I upgraded dax from FreeBSD 10.2 to FreeBSD 10.3 this morning.  Since I haven't upgraded ports in awhile I upgraded net/quagga from to 1.0.20160315 and boy that was a mistake!

First off, my IPv6 default route wasn't getting redistributed anymore.  I've got a static IPv6 route in zebra pointing to Internap's upstream router:

ipv6 route ::/0 2001:48c8:1:2::1

It's seen fine by zebra:

S>* ::/0 [1/0] via 2001:48c8:1:2::1, em0

However, bgpd doesn't see it at all:

dax_bgpd# show ipv6 bgp ::/0
% Network not in table

The redistribution statement I've got in bgpd isn't doing any good, apparently:

router bgp 65304
 bgp router-id
 address-family ipv6
 redistribute static route-map IPv6_ADVERTISE_STATICS
ipv6 prefix-list IPv6_DEFAULT seq 5 permit ::/0
route-map IPv6_ADVERTISE_STATICS permit 10
 match ipv6 address prefix-list IPv6_DEFAULT

I managed to get this working without restarting the daemons by doing this a few times in zebra:

no ipv6 route ::/0 2001:48c8:1:2::1
ipv6 route ::/0 2001:48c8:1:2::1

This fixed it once, but it wasn't consistent, so that's a fail.

While I was trying to fix this, I noticed that some of my single-homed VPN tunnels from dax appeared down, but it was because bgpd wasn't consistently advertising my IPv6 prefixes over the EBGP sessions.  I couldn't figure it out but I didn't spend too much time on it.  2001:48c8:1:1ff::3a is an example VPN neighbor that isn't getting all of the IPv6 routes and 2001:48c8:1:105::/64 is a LAN prefix in Seattle, where my desktop sits.

(dax:13:31:EDT)% show ipv6 bgp 2001:48c8:1:105::/64
BGP routing table entry for 2001:48c8:1:105::/64
Paths: (4 available, best #4, table Default-IP-Routing-Table)
  Advertised to non peer-group peers:
  2001:48c8:1:1ff::6 2001:48c8:1:1ff::16 2001:48c8:1:1ff::1e 2001:48c8:1:1ff::32 2001:48c8:1:1ff::3a 2001:48c8:1:1ff::52 2001:48c8:1:1ff::56

bgpd says it's advertising it above just fine to 2001:48c8:1:1ff::3a, but advertising-routes says otherwise.  The remote box, firefly, doesn't see it at all.  I lost the command output when I closed the terminal in a fit of rage.

Anyway, I downgraded to using ports-mgmt/portdowngrade and rebuilt.  All of the problems magically disappeared.  I suppose I'll try to reproduce this on a FreeBSD VM and see what I can figure with some debugging enabled.

I'm thinking of switching to BIRD but I'll have to do it when I've got a bit more time on my hands.

Comments: 0
> Zsh + MPlayer
Posted by prox, from Seattle, on April 12, 2016 at 23:27 local (server) time

From the "tiny blogs" department...

Yep, I'm a Zsh fan.  I've always been one and probably won't change.

One of the nifty features of Zsh is the ability to expand wildcards on the command-line in a particular order.  For me, this often comes in handy when combined with some sort of media player.  For example, what if I wanted to watch videos in a directory with MPlayer and start with the most recently modified ones?  Zsh has that covered:

% mplayer *.mp4(oa)

The (oa) tells Zsh to expand '*' to all files in $CWD but to order the expansion by last modified.  There are a few other sort specifiers, but I find 'a' the most useful.  Using a capital 'O' will reverse the sort order.

Comments: 0
> OpenSSH 7.x and Keys
Posted by prox, from York, on December 26, 2015 at 13:22 local (server) time

OpenSSH 7.0 was released a few months ago and deprecated both the ssh-dss and ssh-rsa keys used for SSHv2 public key authentication.  I haven't found a definitive source stating why these key types were deprecated other than some issue with entropy (doesn't make sense to me because that sounds like a machine-specific problem).  I, unfortunately, still use these keys quite a bit and it's not feasible to completely convert to one of the newer key types.

OpenSSH 7.0 appeared in FreeBSD ports pretty quickly and recently made its way into Debian testing (stretch).

So, what are the newer key types supported?  From what I can tell, it's just ecdsa and ed25519 for SSHv2.  From the OpenSSH 6.2 ssh-keygen(1) manpage:

     -t type
             Specifies the type of key to create.  The possible values are
             ``rsa1'' for protocol version 1 and ``dsa'', ``ecdsa'' or ``rsa''
             for protocol version 2.

From the OpenSSH 7.1 ssh-keygen(1) manpage:

     -t dsa | ecdsa | ed25519 | rsa | rsa1
             Specifies the type of key to create.  The possible values are
             "rsa1" for protocol version 1 and "dsa", "ecdsa", "ed25519", or
             "rsa" for protocol version 2.

Even though dsa and rsa keys are still listed above in the 7.1 man page as being capable of created, they're not accepted by default anymore by ssh or sshd:

debug1: Next authentication method: publickey
debug1: Skipping ssh-dss key /home/prox/.ssh/id_dsa for not in PubkeyAcceptedKeyTypes

(if you're wondering, and I was too, DSS is a document that describes the creation of DSA keys as answered here)

Running ssh -Q key will also dump the list of keys acceptable by ssh:

(nox:11:46:CST)% ssh -Q key

The solution seems obvious, just throw out the old keys and use an ecdsa key, right?  Sure, that'll work for OpenSSH versions that support it.  However, sometimes I have to log into a few legacy boxes that only support RSA and DSA keys (Solaris, IRIX, random network devices, etc. - I've got some old stuff!).

What about keeping around the old keys?  Sure, we can use PubkeyAcceptedKeyTypes in sshd_config and ssh_config like this:

PubkeyAcceptedKeyTypes ssh-dss,ssh-rsa

The only problem is that this option only exists in 7.0 and above.  I use a common ~/.ssh/ssh_config for all of my systems and OpenSSH 6.x barfs on that line.

What's the solution?  Well, one is to not upgrade to OpenSSH 7.0, but that's just delaying the inevitable.  My solution may just be to use two keys, one for modern systems and one for very old systems that don't support ecdsa or ed25519.  Regardless, it's pretty annoying, but security always is, right?

Update 20151229: This page highlights some of these differences and workarounds, too.

Comments: 0
> Latency vs. Lots o' Bandwidth
Posted by prox, from Seattle, on November 22, 2015 at 12:25 local (server) time

Unlike most folks, I value first-hop latency on residential networks over gobs of bandwidth.  Why?  It's likely to have a larger impact to my web browsing experience than bandwidth, in many cases.

I've got 50 Mbps downstream and 10 Mbps upstream at home via Comcast Business, at the moment.  Unfortunately, I pay a bit for it because I have a static IPv4 /29 prefix.  Anyway, the 50 Mbps downstream is good enough for streaming and downloading most content (20 Mbps would work, too, I guess).  The 10 Mbps is mildly sufficient for me but gets a little painful when uploading videos or transferring large files via rsync over SSH.

If I could pay $5 or $10 more per month for 100 Mbps downstream, I probably wouldn't bother.  Excluding video-rich sites, I don't see more than 4-5 Mbps at peak when loading web pages and browsing the web.

However, I'd pay more to reduce the first-hop latency since most web content is coming from CDN nodes that are nearby.  Lower latency impacts the initial stages of TCP significantly and can have a large impact on transferring web content quickly over short-lived connections.

Comcast's got an Akamai CDN node on-net, which is about 10 ms from me:

(valen:12:31)% mtr --report --report-cycles=10 --report-wide 
Start: Tue Nov 17 12:31:34 2015
HOST: valen                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:558:4082:48::1                            10.0%    10    8.3  24.3   8.3 133.6  41.0
  2.|--  0.0%    10   13.6  10.6   7.8  13.6   1.7
  3.|--       0.0%    10   10.0  10.8   9.2  13.8   1.5
  4.|--     60.0%    10   14.2  12.6  11.3  14.2   1.0
  5.|--   0.0%    10   10.3  11.3   9.1  16.6   2.1
  6.|--        0.0%    10    9.7  11.5   9.7  14.4   1.3
  7.|-- 2600:1409:a:1a0::22d9                           0.0%    10    9.6  13.3   9.1  24.9   5.0

While I live in West Seattle, apparently the CMTS I use is down south in Burien, WA, likely due to the geography of the region.  It takes about ~7-8 ms to get to the CMTS.  This is my first hop latency (hop 1 is my Comcast modem since it's in L3 mode for the static IPv4 assignment).  As you can see above, the bulk of my latency to the Akamai cache is due to the first hop.  DOCSIS has a request/grant system (see slide 16 in this PDF) for accessing upstream bandwidth, which adds an extra "round trip" to the communication from my cable modem to the CMTS.  If that were somehow lowered to 1 ms or so (GPON?), I'm sure most of the web would "feel" faster since my latency to the cache would be around 2-3 ms.

As another example, Google doesn't have a cache on Comcast's network but they aren't too far away:

(valen:12:31)% mtr --report --report-cycles=10 --report-wide           
Start: Tue Nov 17 12:32:08 2015
HOST: valen                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:558:4082:48::1                             30.0%    10   27.5  13.4   9.2  27.5   6.5
  2.|--   0.0%    10    8.1   9.8   8.1  11.6   0.8
  3.|--         0.0%    10    9.3   9.2   8.0  10.2   0.6
  4.|--  0.0%    10   11.2  10.3   8.7  14.6   1.6
  5.|--      70.0%    10   13.6  12.5  10.5  13.6   1.6
  6.|-- 2001:558:0:f510::2                              60.0%    10   13.3  12.0   9.8  13.3   1.4
  7.|-- 2001:559::cfa                                    0.0%    10    9.4  11.7   8.6  28.9   6.1
  8.|-- 2001:4860::1:0:9aa5                              0.0%    10    9.7  11.0   8.8  14.0   1.8
  9.|-- 2001:4860::8:0:6999                              0.0%    10   10.2  16.4   9.3  50.7  12.6
 10.|-- 2001:4860::8:0:61e1                              0.0%    10   14.4  17.0  12.7  44.6   9.7
 11.|-- 2001:4860::8:0:924a                              0.0%    10   16.1  17.0  15.0  20.8   1.5
 12.|-- 2001:4860::2:0:90fe                              0.0%    10   16.3  17.6  15.7  22.0   1.8
 13.|-- ???                                             100.0    10    0.0   0.0   0.0   0.0   0.0
 14.|--                              0.0%    10   28.1  18.8  14.9  28.1   4.6

That 15 ms to the Google content node could be reduced to around 7-8 ms if my first-hop latency went away, too!

It just so happens that I graph my first-hop latency to the CMTS using SmokePing.  Here's a graph over the last year:

Comcast first hop latency

It doesn't change too much.  I'm curious what happened in June to drop my latency by what looks to be 1-1.5 ms, but only Comcast would really know.  The heightened latency in January and February were due to a memory leak in my cable modem, I think.

Also, low latency is obviously good for gaming, which doesn't really apply to me, and other interactive applications like SSH.  vimdiff(1), for example, is pretty painful on a high latency connection due to the way if repaints the entire terminal window.

To compare my Comcast connection with other residential technologies, here's what I've seen for first hop latencies:

  1. DOCSIS 3.x: 7-8 ms
  2. Verizon FiOS: 3-4 ms
  3. ADSL: > 20 ms
  4. Dial-up: > 100 ms

Obviously, the lowest latency is going to come from an Ethernet-based service, something like wave G (formerly condointernet).  I'm fairly sure it's limited to apartments and condiminiums in major metropolitan areas like downtown Seattle.

So, in a nutshell, that's why I'm not impressed with 300 Mbps vs. 50 Mbps DOCSIS service, a 50 Mbps Ethernet DIA service might be appealing.

Comments: 0
> ifconfig(8) on Linux
Posted by prox, from Seattle, on September 17, 2015 at 01:17 local (server) time

Fun, it looks like there's a new upstream release for the net-tools package.  The ifconfig(8) output looks a little different, now.  Old:

(destiny:22:10)% ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:1c:c0:b2:8d:bd  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::21c:c0ff:feb2:8dbd/64 Scope:Link
          inet6 addr: 2001:48c8:1:105:21c:c0ff:feb2:8dbd/64 Scope:Global
          RX packets:368571536 errors:0 dropped:0 overruns:0 frame:0
          TX packets:288951501 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:340038876351 (316.6 GiB)  TX bytes:116559393418 (108.5 GiB)
          Interrupt:20 Memory:d3300000-d3320000


(destiny:22:14)% ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet  netmask  broadcast
        inet6 fe80::21c:c0ff:feb2:8dbd  prefixlen 64  scopeid 0x20<link>
        inet6 2001:48c8:1:105:21c:c0ff:feb2:8dbd  prefixlen 64  scopeid 0x0<global>
        ether 00:1c:c0:b2:8d:bd  txqueuelen 1000  (Ethernet)
        RX packets 368903989  bytes 340125988245 (316.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 289091263  bytes 116569592018 (108.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xd3300000-d3320000

I wonder how many screen-scraping tools are now broken..

Now, that all being said, you should really be using ip(8) instead of ifconfig(8).  It's 2015.

Comments: 0
> Android 5.x's DHCPv4 Client
Posted by prox, from Seattle, on July 06, 2015 at 00:54 local (server) time

Among many other bugs, Android 5.x seems to have a DHCPv4 bug that prevents it from getting addresses from some embedded systems.  For me, this includes the Wi-Fi functions on the GoPro HERO3 and Canon EOS 70D.

We all remember DORA from our networking 101 classes, right?

  1. DHCP Discover
  2. DHCP Offer
  3. DHCP Request
  4. DHCP Acknowledge

Unfortunately, with Android 5.x this turns into DORN when talking to some DHCP servers.

That's right, the server sends a NAK for the DHCP request, which usually indicates the requested address is invalid or in use.

I realized this while on my honeymoon earlier this year and was not pleased at all.  I routinely use the Wi-Fi capabilities of my GoPro HERO3 camera with a "selfie stick" to shoot accurate photos.  The HERO3 turns itself into a Wi-Fi access point and allows a single client to connect in order to use the GoPro application, which is used to control & view live video on camera.  Usually this works without a hitch, but not this time.  When connecting to the camera the Wi-Fi status hung in the "Obtaining IP address..." phase.  After debugging a bit I realized that the server was sending NAKs for the DHCP request from my phone and I had upgraded my OnePlus One to CyanogenMod 12.0 (Android 5.0) a few weeks earlier.  Here's what logcat says:

I/dhcpcd  (25750): wlan0: broadcasting for a lease
I/dhcpcd  (25750): wlan0: offered from
W/dhcpcd  (25750): wlan0: NAK: via
I/dhcpcd  (25750): wlan0: broadcasting for a lease
I/dhcpcd  (25750): wlan0: offered from
W/dhcpcd  (25750): wlan0: NAK: via
I/dhcpcd  (25750): wlan0: broadcasting for a lease
I/dhcpcd  (25750): wlan0: offered from
W/dhcpcd  (25750): wlan0: NAK: via
I/dhcpcd  (25750): wlan0: broadcasting for a lease
I/dhcpcd  (25750): wlan0: offered from
W/dhcpcd  (25750): wlan0: NAK: via

(the HERO3 uses 10.5.5/24 with the camera and the clients starting around or so)

The entry that's missing is the DHCP request debug entry, where Android should be sending a message to the DHCP server asking for  This is actually happening and can be seen via a packet capture:

20:48:52.890280 IP (tos 0x0, ttl 64, id 21568, offset 0, flags [none], proto UDP (17), length 330) > BOOTP/DHCP, Request from c0:ee:fb:24:8c:59 (oui Unknown), length 302, xid 0x1d11a2f4, Flags [none]
	  Client-Ethernet-Address c0:ee:fb:24:8c:59 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Client-ID Option 61, length 19: hardware-type 255, fb:24:8c:59:00:01:00:01:1c:c4:48:b1:c0:ee:fb:24:8c:59
	    MSZ Option 57, length 2: 1500
	    Vendor-Class Option 60, length 12: "dhcpcd-5.5.6"
	    Hostname Option 12, length 5: "omega"
	    Parameter-Request Option 55, length 10: 
	      Subnet-Mask, Static-Route, Default-Gateway, Domain-Name-Server
	      Domain-Name, MTU, BR, Lease-Time
	      RN, RB
20:48:52.896211 IP (tos 0x0, ttl 64, id 81, offset 0, flags [none], proto UDP (17), length 326) > BOOTP/DHCP, Reply, length 298, xid 0x1d11a2f4, Flags [none]
	  Client-Ethernet-Address c0:ee:fb:24:8c:59 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4:
	    Subnet-Mask Option 1, length 4:
	    Lease-Time Option 51, length 4: 10368000
	    TTL Option 23, length 1: 64
	    MTU Option 26, length 2: 1500
	    RN Option 58, length 4: 10368000
	    RB Option 59, length 4: 10371600
	    Domain-Name Option 15, length 3: "lan"
	    BR Option 28, length 4:
	    Default-Gateway Option 3, length 4:
20:48:52.897006 IP (tos 0x0, ttl 64, id 41212, offset 0, flags [none], proto UDP (17), length 342) > BOOTP/DHCP, Request from c0:ee:fb:24:8c:59 (oui Unknown), length 314, xid 0x1d11a2f4, Flags [none]
	  Client-Ethernet-Address c0:ee:fb:24:8c:59 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Client-ID Option 61, length 19: hardware-type 255, fb:24:8c:59:00:01:00:01:1c:c4:48:b1:c0:ee:fb:24:8c:59
	    Requested-IP Option 50, length 4:
	    Server-ID Option 54, length 4:
	    MSZ Option 57, length 2: 1500
	    Vendor-Class Option 60, length 12: "dhcpcd-5.5.6"
	    Hostname Option 12, length 5: "omega"
	    Parameter-Request Option 55, length 10: 
	      Subnet-Mask, Static-Route, Default-Gateway, Domain-Name-Server
	      Domain-Name, MTU, BR, Lease-Time
	      RN, RB
20:48:52.902836 IP (tos 0x0, ttl 64, id 82, offset 0, flags [none], proto UDP (17), length 272) > BOOTP/DHCP, Reply, length 244, xid 0x1d11a2f4, Flags [none]
	  Client-Ethernet-Address c0:ee:fb:24:8c:59 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: NACK

Except for some absurdly high lease times, there's nothing invalid I see in the DHCP request above.  So, why is the HERO3 sending a NAK?  Here's what I know:

Since the 70D and HERO3's DHCP servers are closed, I can't look at their debug log to see what's happening.  The only workaround I have so far is to assign a static IPv4 address to my OPO when connecting to the HERO3, which is with a default gateway of  I don't believe the default gateway is used for the application functionality, but provides some fake HTTP responses that cause Android and iOS' Internet connection quality tests to succeed.

I'd submit a bug for this but I'm worried it'll go nowhere or be tagged WONTFIX like this request for DHCPv6 support.

Comments: 0
> Connecting LXC to VirtualBox
Posted by prox, from Seattle, on June 01, 2015 at 23:09 local (server) time

I've often been annoyed at the way LXC does bridging.  Here's a sample common configuration: = veth = lxcbr0 = 00:50:56:1a:ad:dc

This assumes lxcbr0 is a bridge (created by brcrl addbr lxcbr0).  The above configuration will create a pseudo-random interface name that is mapped to the LXC and add it to the bridge, resulting in the following:

(starfire:19:58)# brctl show
bridge name	bridge id		STP enabled	interfaces
lxcbr0		8000.fe3ffd02f7b8	no		vethXHEHXF

This works well most of the time but not if you want to directly connect an LXC to something that has native bridging capabilities like VirtualBox.  Sure, you could create a host-only interface with VirtualBox and add it to lxcbr0 but this is really unneeded (and probably slow, too).

A better way is to dispense with the Linux bridge itself and have VirtualBox bridge directly to the virtual Ethernet.  This doesn't work out of the box because the interface name is pseudo-random.  However, I was looking through the lxc.container.conf(5) man page the other day and found the option.  This allows one to specify the virtual Ethernet interface's name that appears on the outside of the container!

Here's a better sample configuration: = veth = foobar0 = 00:50:56:1a:ac:cc

This creates a foobar0 interface that connects to the LXC, which can be bridged to VirtualBox through its native method:

VBoxManage modifyvm testvm --nic1 bridged --nictype1 82543GC --bridgeadapter1 foobar0


Comments: 0

Previous PageDisplaying page 2 of 119 of 951 results Next Page