Present Location: News >> Blog


> It's Time for A Change
Posted by prox, from Seattle, on March 09, 2014 at 13:40 local (server) time

No, I'm not quoting Bill Clinton.  I haven't posted in a few months because it was time for a change and I decided to pull the trigger.  I left Time Warner Cable in Charlotte and moved to Seattle to work for Amazon Web Services.

I'm not going to say much about the job change because it's really not appropriate for a public-facing blog and there's really not much I can say.  I'm happy to be advancing my career and working on a network that either powers 1% of the Internet or provides content to 1/3 of all Internet users (depending on who you ask).

Before I left Charlotte I got engaged to my beautiful fiancée on Valentine's Day.

Devon and Mark

We're set to be married next year on a day that commemorates the ratio between a circle's circumference and diameter.

At the moment I'm living in the Belltown area of Seattle in Washington state.  My ten (10) minute commute is thoroughly spoiling as it'll be a bit longer when I move into my permanent residence in a couple of weeks.

I'm still trying to figure out Seattle.  It's got elements from both uptown Charlotte and Manhattan but with its own twist.  The people who I come in contact with on a daily basis are very friendly, which from what I hear is actually abnormal.  The weather here is routinely overcast and rainy but I've found that weather forecasts are terribly inaccurate due to the surrounding mountains in the region.  Most days that are forecasted as rainy actually end up with an hour or two of sun between the fronts, which is nice.  I was told that only 17% of residences have central air conditioning.  At first, I thought this was a bit odd (coming from the south!) but I suppose it makes sense due to the climate.  I'll find out this summer if my computers will require extra cooling or if they'll be fine sans-AC!

One nice thing is the Comcast XFINITY service in the area provides native IPv6 with prefix delegation.  Finally, no more tunneling!  Although, I do like the permanent nature of a tunneled /56 or /48 as well as the reverse DNS delegation.  Maybe I'll ultimately use a combination of both.

Comments: 1
> Cisco IOS IPv6 Prefix List Oddness
Posted by prox, from Charlotte, on December 28, 2013 at 15:22 local (server) time

A week or so I configured some IPv6 prefix lists in my networking lab at home on a few Cisco IOS boxes running 15.1(4)M7.  Almost a day after I finalized the configuration, I noticed that the prefix lists appeared to be reordering themselves from time to time.  Specifically, the "show running-config" command would list the entries in a slightly different order, every once and awhile.

Normally, I wouldn't care.  Prefix lists are not evaluated based on order and this was just a lab, anyway.  However, I have RANCID setup to monitor all configuration file changes in my lab environment and I started getting several e-mails per day.  Here's a sample:

Index: configs/defiant
retrieving revision 1.485
diff -u -4 -r1.485 defiant
@@ -340,11 +340,11 @@
   passive-interface Loopback0
  ipv6 prefix-list DEFAULT-ROUTE permit ::/0
- ipv6 prefix-list SPOCK-NETS permit 2001:48C8:1:104::37/128
  ipv6 prefix-list SPOCK-NETS permit 2001:48C8:1:104::38/127
  ipv6 prefix-list SPOCK-NETS permit 2001:48C8:1:13A::/63
+ ipv6 prefix-list SPOCK-NETS permit 2001:48C8:1:104::37/128
  route-map ADV-DEFAULT permit 10
   match ip address DEFAULT-ROUTE

This is happening on two routers running the same exact version of IOS with a similar prefix list.  It seems like the configuration compilation that is executed from the "show running-config" command may be doing something wrong since when I display the prefix lists manually and they are shown in the same order each time:

defiant#show ipv6 prefix-list 
ipv6 prefix-list DEFAULT-ROUTE: 1 entries
   seq 5 permit ::/0
ipv6 prefix-list SPOCK-NETS: 3 entries
   seq 5 permit 2001:48C8:1:104::37/128
   seq 10 permit 2001:48C8:1:104::38/127
   seq 15 permit 2001:48C8:1:13A::/63

I haven't tried messing with different IOS versions, yet.  I may do that later.

Oh, in other news.. among other things, I got a plush NCC-1701 for Christmas:

Plush Enterprise NCC-1701

Yes, it's from ThinkGeek and makes noise.

Comments: 0
> OS X Mavericks 6in4 Tunnel Crash
Posted by prox, from Charlotte, on December 19, 2013 at 16:17 local (server) time

Well, this one is a bummer.  Apparently, Mac OS X Mavericks introduced a bug in either the gif(4) driver or kernel that crashes the whole machine when an IPv6 address is removed from a 6in4 tunnel interface.  The gif(4) driver in the BSD world provides a generic driver to tunnel IP in IP (any version inside any other version).

If you want to try it, do this as root:

/sbin/ifconfig gif0 up
/sbin/ifconfig gif0 tunnel
/sbin/ifconfig gif0 inet6 2001:db8:1:116::2/64
/sbin/route add -inet6 default 2001:db8:1:116::1
/sbin/route delete -inet6 default
/sbin/ifconfig gif0 -tunnel
/sbin/ifconfig gif0 inet6 2001:db8:1:116::2/64 -alias

The last command triggers a kernel panic and reboot after a few seconds.  I discovered this after loading my IPv6 6in4 tunneling script on my old MacBook Air.  The tunnel works fine when it's up.  However, tearing it down seems to be problematic, to say the least.  My script doesn't even get to the point where gif0 is brought down.

The backtrace and some associated information looks like this:

Thu Dec 19 13:37:50 2013
panic(cpu 1 caller 0xffffff80164d143e): Preemption level underflow, possible cause unlocking an unlocked mutex or spinlock
Backtrace (CPU 1), Frame : Return Address
0xffffff80a5e536d0 : 0xffffff8016422f69 
0xffffff80a5e53750 : 0xffffff80164d143e 
0xffffff80a5e53760 : 0xffffff80164d11cf 
0xffffff80a5e53770 : 0xffffff801664ce14 
0xffffff80a5e537b0 : 0xffffff801662bb33 
0xffffff80a5e538d0 : 0xffffff8016730aac 
0xffffff80a5e539a0 : 0xffffff8016714519 
0xffffff80a5e53dc0 : 0xffffff801672b6cc 
0xffffff80a5e53e20 : 0xffffff801672b308 
0xffffff80a5e53e60 : 0xffffff801672afea 
0xffffff80a5e53f20 : 0xffffff801644a15a 
0xffffff80a5e53fb0 : 0xffffff80164d6aa7 

BSD process name corresponding to current thread: kernel_task

Mac OS version:

Kernel version:
Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64
Kernel UUID: [removed]
Kernel slide:     0x0000000016200000
Kernel text base: 0xffffff8016400000
System model name: MacBookAir3,1 (Mac-[removed])

The full report can be found here.  I removed some values in it that I thought might be harmful if they are public (I could be mistaken).  I also filed a bug report with Apple on this: 15701774.  I'm fairly sure this is not hardware-related or related to VirtualBox, Cisco AnyConnect, or TunnelBlick kernel extensions being present since I can crash another Mavericks install running in a VM that has none of them loaded.

Update: I did realize I was not running the latest version of OS X Mavericks, 10.9.1.  I just upgraded but the issue is still present.  Here is a crash report under 10.9.1.

Update: Apple closed my bug report as a duplicate of 14929904.

Comments: 0
> iPad Air, iOS 7, Jailbreaking, etc.
Posted by prox, from Charlotte, on November 18, 2013 at 23:11 local (server) time

Yes, I have a few things to say about this.  It's not a review of the iPad Air but more of a whiny rant (shocker, I know).

I recently picked up an iPad Air MF018LL/A, which is the white cellular model with 128 GiB of storage.  I use AT&T for carrier diversity with the 3 GiB/mo plan since my phone is on T-Mobile's network.  The iPad Air replaced my iPad 3, which when I owned it was jailbroken and still running iOS 6.  I happened to sell it and all of my accessories to a colleague who is now a first-time iPad owner.

Below: iPad 3 on the left, iPad Air on the right.

iPad 3 and iPad Air

First Impression

My first impression with the iPad Air was awful since it failed at the first thing I asked it to do: install an application.  I got the dreaded "could not connect to iTunes store" error whenever I tried to install an application on Wi-Fi or cellular.

Cannot Connect to iTunes

It turns out this is an iOS 7 issue that some folks got after upgrading from iOS 6 and there was a thread on the Apple Support Communities site about it.  I had to ultimately do a factory reset and wipe of the device to fix it.  I didn't lose anything because it was a brand new iPad and I had not loaded anything back onto it.  It's odd that this hasn't been fixed in iOS 7.0.3, which is what's running on my iPad Air.  I also wonder how iOS 7 was loaded onto the iPad.  I'm almost wondering if iOS 6 was initially running on the iPad Air during development and the first few devices were shipped with a version of iOS 7 that was upgraded from an existing iOS 6 installation.  Curious.

Second Impressions

The iPad Air hardware is good.  The CPU seems fast (Google Earth is smooth as butter) and the shell (+ components) is much lighter and thinner than the iPad 3.  I'm not sure what I think about battery life, yet.  So far, it seems to last slightly less than my iPad 3 but my analysis is most unscientific.

The antennas or cellular radio itself must be better in the iPad Air vs. the iPad 3 because I have five "dots" more often than I had five "bars" on the iPad 3.  I suppose the "dots" could be mapped to different ranges of dBm in iOS 7, now, or the addition of more LTE bands is helping, although I didn't think LTE bands themselves overlapped within an area.

I configured the switch on the side of the iPad Air to control roation lock, since I have volume muted almost all of the time, anyway.  I find that, compared to the iPad 3, it's more difficult to slide the button back & forth with my finger.  The button itself appears to be rounded and the one on the iPad 3 had sharper edges that were easier to "grab" with my finger.

I'm disappointed that the fingerprint reader now being used on the iPhone 5s didn't make its way into the iPad Air.  I guess it doesn't matter to me since I probably wouldn't have used it, anyway.

iOS 7

I don't mind the visual changes in iOS 7, to be honest.  I thought the skeuomorphisms probably needed to go, too.  There are a few things that bug me, though.

The control center is an abomination.  Apple made the decision on what toggles should be available to users and didn't leave any room for customization.  It's infuriating.  I would like to swap out the airplane mode for a toggle that turns cellular data on and off.  If I'm connected to Wi-Fi and want to shut off cellular data to save a little bit of battery life, I have to navigate all of the way to settings.

It appears the text size for items on the status bar at the top of the screen has become smaller.  The text size setting doesn't appear to change the font size of them.  What's weird is the items are larger at the lock screen but shrink when the device is unlocked.  Very odd.

The IPv6 support seems better than in previous versions of iOS.  I didn't have a problem loading all web pages over IPv6 on my dual-stack network at home.  In iOS 6, some of those pages would have loaded over IPv4 due to Apple's odd HE implementation.  DHCPv6 appeared to work on one of my test SSIDs but unfortunately the iPad put the IPv4 DNS servers before the IPv6 one.  More testing is required, apparently.

DHCPv6 Fail?

One difference in the IPv6 support between iOS 6 and 7 is the implementation of RFC 4941 when SLAAC is used.  iOS 6 added temporary addresses with random interface identifiers but always kept the link-local address based on EUI-64 as well as one GUA.  iOS 7 appears to randomize the interface identifiers for both the link-local address and all the GUAs.  I really wish it was possible to disable this.


I really miss the jailbroken functionality I had with iOS 6 on my iPad 3.  Lots of people ask me why I bother to still jailbreak iOS devices—my reasons mostly relate to optimization and performance.  I'll detail them all here.

3G Unrestrictor.  This application works around some of the very annoying and pointless restrictions of iOS itself and many applications.  Podcasts, iTunes, and the App Store won't download anything over 100 MB (I assume MiB) over a cellular connection.

Over 100 MiB (or MB?)

There's no way to disable this and it's infuriating.  TWC TV won't stream live TV over cellular.  The list goes on.  These restrictions are stupid because iOS and the various applications are essentially telling the user that they're too stupid and irresponsible to use their cellular data plan.  If I want to burn up my plan downloading podcasts that is my prerogative!  3G Unrestrictor fixes all of these issues and fools applications into thinking the current network connectivity is via Wi-Fi.

SBSettings.  SBSettings provides functionality similar to iOS' control center but is instead customizable and provides tweaks that allow various things on the status bar to be changed.  Toggles include data, OpenVPN (if installed), 3G Unrestrictor, and more.  The status bar can be configured to show the current Wi-Fi SSID, numeric signal strength indicators for both cellular and Wi-Fi, non-scaled battery percentage, and more other options than I can remember at the moment.

FakeClockUp.  This application speeds up transitions and some animations on iOS.  Although these animations take only a second or so every time, this can add up to minutes and hours of wasted time over the life of a device.  This application is needed because iOS doesn't provide an option to disable these transitions.

Command-line Access.  While most non-geeks do not care about such things, command-line access provides a wealth of flexibility.  Among other things, it allows for network-related diagnostics and troubleshooting as well as the ability to directly back up media and application data that is otherwise inaccessible or only provided by other 3rd party applications.

Hopefully, the above will be usable for iPad Air owners once again when the iOS 7 jailbreak is released.  Until then, we're locked into doing things the way Apple wants us to do them, sans-choice.

Comments: 0
> OS X Mavericks and Boot Media
Posted by prox, from Charlotte, on October 22, 2013 at 16:47 local (server) time

For folks out there, like me, who like to do clean installs of OS X and require boot media to do so, Apple has apparently provided a nifty script in the OS X Mavericks installer.  It's one extra step compared to just writing InstallESD.dmg to a USB flash drive, I guess:

(quark:16:29)% cd /Applications/Install\ OS\ X\
(quark:16:29)% sudo ./createinstallmedia --volume /Volumes/Untitled --applicationpath "/Applications/Install OS X" 
Ready to start.
To continue we need to erase the disk at /Volumes/Untitled.
If you wish to continue type (Y) then press return: Y
Erasing Disk: 0%... 10%... 20%...100%...
Copying installer files to disk...
Copy complete.
Making disk bootable...
Copying boot files...
Copy complete.
(quark:16:41)% df -h
Filesystem      Size   Used  Avail Capacity iused   ifree %iused  Mounted on
/dev/disk0s2    39Gi   17Gi   22Gi    44% 4500660 5775196   44%   /
devfs          189Ki  189Ki    0Bi   100%     658       0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%       0       0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%       0       0  100%   /home
/dev/disk1s2   7.3Gi  5.0Gi  2.3Gi    68% 1307226  615788   68%   /Volumes/Install OS X Mavericks

Yep, it's nice and easy.  The installer takes up roughly 5.0 GiB of space, so an 8 GiB (or GB, in this instance) USB stick is probably required.

Comments: 0
> IPv6 Musings
Posted by prox, from Charlotte, on October 20, 2013 at 20:41 local (server) time

Warning: Soapbox.

It's almost the end of 2013.  World IPv6 Launch was over 15 months ago.  However, it appears we still have some organizations and protocols that just aren't getting with the program.  Even more shocking, we still have some that are taking measures to indirectly inhibit IPv6 development.

I've made a short list.

Get With The Program!

MPLS.  draft-george-mpls-ipv6-only-gap goes over many of the problems that need to be overcome to operate an MPLS network that is free of IPv4.  The short story is that it may be awhile before this is possible.  I can see many new ISPs deplying RFC 6598 (possibly erroneously) or RFC 1918 space on their backbones just to meet the IPv4 requirements for MPLS, in the short term, unfortunately.

Twitter.  Twitter owns 2620:fe::/40 but has not announced it into BGP, yet.  The least they could do is provide something like on a proxy server to show their interest in keeping up with technology.

Debian GNU/Linux bug #592539 for isc-dhcp-server.  The isc-dhcp-server package has both IPv4 and IPv6 functionality but the init scripts to start the IPv6 version of the daemon don't exist.  Yes, this means that you can't use DHCPv6 on Debian without writing your own init script or starting it manually.  The bug was opened in August of 2010 and has yet to be resolved.  Get with the program, folks!

Amazon EC2.  This is a sore subject.  The only AWS service that supports IPv6 is ELB, unfortunately.  EC2 doesn't support it in any region or availability zone, at the moment.  This is preventing lots of organizations from correctly deplying IPv6.

Google Compute Engine.  GCE is a very poor competitor to Amazon EC2 for a variety of reasons.  Google could have actually made one feature more attractive than EC2 by offering IPv6 on it.  Unfortunately, they didn't.  Fail.

Skype.  We all know about this one, so let's just move on.

Why do you hate IPv6?

FreeBSD.  A change in the defaults for the IPv6 address selection algorithm in FreeBSD 9.x leaves users with an OS that always prefers IPv4 connectivity when both A and AAAA RRs exist for a particular label.  There are two knobs that can be put in /etc/rc.conf that change this behavior (ipv6_activate_all_interfaces and ip6addrctl_policy).  That being said, versions of FreeBSD prior to 9.x defaulted to always preferring IPv6 over IPv4 when both RRs exist.  It's very backward-thinking of the FreeBSD folks to introduce such a drastic change in the operating system's defaults after supporting IPv6 for so long.  Apparently, FreeBSD 4.0 introduced support for IPv6 back in 2000.

Carrier Grade NAT.  Unlike traditional NAT, CGN is often deployed on residential networks as a second layer of NAT (the customers' home routers being the first layer).  This second layer of NAT breaks end-to-end connectivity even worse than a single layer of NAT, rendering a variety of peer-to-peer applications useless.  Even though the problems with CGN are numerous, it helps ISPs push back the deployment of IPv6 for a little while longer, unfortunately.

Comments: 0
> Junos 12.1X44/45, 13.2X50
Posted by prox, from Pineville, on September 15, 2013 at 10:49 local (server) time

If any of you are thinking of running Junos 12.1X44 or 12.1X45 on the SRX branch series, think again.  My SRX100H at home running 12.1X45-D10 runs out of memory every 3-4 days and reboots:

einstein (ttyu0)

login: Sep 14 16:50:46 init: low_mem_signal_processes: send signal 16 to  routing
Sep 14 18:23:43 init: low_mem_signal_processes: send signal 16 to  routing
init died (signal 4, exit 0)
panic: Going nowhere without my init!
cpuid = 0
KDB: stack backtrace:
0x4af834+0x20 (0x6,0,0x3f7eef10,0x4bec10) ra 0x4af7fc sz 0
0x4af7c0+0x3c (0x6,0,0x3f7eef10,0x4bec10) ra 0x4ae114 sz 32
0x4ae090+0x84 (0x6,0,0x3f7eef10,0x4bec10) ra 0x445094 sz 56
0x445020+0x74 (0x6,0,0x3f7eef10,0x4bec10) ra 0x445110 sz 40
0x445020+0xf0 (0x6,0,0x3f7eef10,0x4bec10) ra 0x44625c sz 40
0x4461d4+0x88 (0x6,0,0x3f7eef10,0x4bec10) ra 0x446978 sz 64
0x446944+0x34 (0x6,0,0x3f7eef10,0x4bec10) ra 0x4b03f4 sz 32
0x4b03b4+0x40 (0x6,0,0x3f7eef10,0x4bec10) ra 0x4b06b8 sz 40
0x4b05d8+0xe0 (0x6,0,0x3f7eef10,0x4bec10) ra 0x4926b4 sz 32
0x492630+0x84 (0x6,0,0x3f7eef10,0x4bec10) ra 0x48d6b4 sz 48
0x48d588+0x12c (0x6,0,0x3f7eef10,0x4bec10) ra 0x4039a4 sz 3512
0x403968+0x3c (0x6,0x4d22b4,0x3f7f0060,0x4bec10) ra 0x403e40 sz 40
0x403d60+0xe0 (0x6,0x4d22b4,0x3f7f0060,0x4bec10) ra 0x3ffeefe0 sz 32
VA 0x3ffdefdc: not in user area or heuristics failed
_start+0xbfeeef00 (0x6,0x4d22b4,0x3f7f0060,0x4bec10) ra 0 sz 0
pid 1, process: init
Uptime: 4d1h54m46s
Cannot dump. No dump device defined.
Automatic reboot in 15 seconds - press a key on the console to abort
cpu_reset: Stopping other CPUs

It doesn't bother me too much except I'm worried that eventually I'm going to start seeing some things in lost+found.

Also, my EX2200-C running Junos 13.2X50-D10.2 dies every week or two, too.  I don't have a persistent console on it, though, so I'm not sure what's happening.  The RE goes unreachable but the PFE still keeps on going (although possibly not learning additional MACs).

While it's neat to run the latest and greatest code, I'm glad I'm not running any of the mentioned X releases on any production gear at work.  Yes, JTAC recommends certain releases for a reason.

Comments: 3
> Fish Oil
Posted by prox, from Charlotte, on September 07, 2013 at 16:27 local (server) time

A few weeks ago I posted a blog entry about cholesterol, specifically the lack of my HDLs.  One of the suggestions I read about to increase HDLs involved taking fish oil.  So, I've been trying this.

I picked up some Barlean's Fish Oil and have been taking two teaspoons of it per week (Monday and Thursday, usually).  There was some question of whether it should be refrigerated so I decided to keep it in chilled.

I was going to give it six (6) months and see if it made any impact on my HDLs.  I figured it wouldn't have any other effect on my health but it looks like it might be giving me more energy while swimming.  Check out this graph of my swim times:


My times have slightly improved.  I feel like I have more energy than I did before taking the fish oil, oddly enough.  However, correlation does not imply causation so I can't be sure that the reason I have more energy is solely because of the fish oil.  However, I will be continuing with it!

Comments: 0

Previous PageDisplaying page 2 of 117 of 930 results Next Page