Andrew Roderos

Tech Geek

Recovering the EdgeRouter Lite

by Leave a Comment

EdgeRouter Lite
Table Of Contents


My original EdgeRouter Lite (ERL) that I started using since March 2016 crapped out last year. Earlier this year, I received a tip about replacing the USB flash drive to recover the router. A few days ago, I tried recovering the EdgeRouter Lite. I am happy to report that the recovery was successful! This post will discuss my experience with the EdgeRouter recovery and what I did to recover it.

Related: My Home Router – EdgeRouter Lite

The Issue

For most of the day, I was connected to the Internet just fine. Then, one minute, I lost Internet connectivity. As any IT professional would do, I started troubleshooting.

I didn’t suspect that it was the router at first since it has been my most stable router since 2000. When I finally isolated the issue with the router, I rebooted it.

After a few minutes of waiting, I still couldn’t ping the gateway. I decided to bust out my Tripp Lite Keyspan and console cable and connected it to the ERL’s console port.

I didn’t see any console messages even after performing a hard reboot. When I rebooted it, I also heard a hissing noise. I assumed that it was dead, so I swapped it out with my spare router. The replacement process was quick and easy since I have a backup config.

Recovery attempt

After many months from when I received the tip, I finally carved out some time to perform the recovery. Surprisingly, when I consoled into the ERL this time around, it was displaying some output. When I did it last year, I verified that my serial adapter and console cable were fine by connecting to my Cisco switch’s console port. I also made sure that I was using the correct baud rate but still wasn’t displaying any output.

Displaying console messages

To display ERL console messages, you will need the following settings if you use a terminal emulator like SecureCRT on a macOS/Windows or PuTTY on Windows.

  • Baud rate: 115200
  • Data bits: 8
  • Parity: None
  • Stop bits: 1
  • Flow control: None

In my case, it was easier to use the existing software in macOS, which is the GNU screen. Before you can use the screen software, you need to determine the device name.

$ ls /dev | grep -m 10 "tty\."
tty.Bluetooth-Incoming-Port
tty.KeySerial1
tty.MALS
tty.NullModem-1
tty.SOC
tty.USA19H142P1.1

As you can see, there are a few of them. Since I’ve been using Tripp Lite for a while, I know it’s the tty.KeySerial1 device.

Once you determine the right device name, you are now ready to connect to the EdgeRouter via console.

$ screen /dev/tty.KeySerial1 115200

When I plugged the power in, it displayed a lot of messages. Eventually, the console messages repeat itself, so the router is stuck in a boot loop. As you can see from the console messages, one of the SQUASHFS error messages indicated the data is probably corrupt. To me, it is a telltale sign that the flash is readable and probably just needs to be reimaged.

ERL console messages
Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
       scanning bus for storage devices...
  Device 0: Vendor:          Prod.: USB DISK 2.0     Rev: PMAP
            Type: Removable Hard Disk
            Capacity: 3824.0 MB = 3.7 GB (7831552 x 512)
 0 
reading vmlinux.64
.............................

5755008 bytes read
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
ELF file is 64 bit
Allocating memory for mapped kernel segment, alignment: 0x400000
Allocated memory for ELF segment: addr: 0x400000, size 0x6863d0
Processing PHDR 0
  Loading 57ba80 bytes at 400000
  Clearing 10a950 bytes at 97ba80
## Loading Linux kernel with entry point: 0x007e5ff0 ...
Bootloader: Done loading app on coremask: 0x3
Linux version 3.10.107-UBNT (root@dc9c2e14f2e4) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Thu Aug 9 06:45:26 UTC 2018
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-3.1.2
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 000000000053b000 @ 0000000000400000 (kernel data and code)
 memory: 0000000000045000 @ 000000000093b000 (usable after init)
 memory: 0000000000107000 @ 0000000000980000 (kernel data and code)
 memory: 0000000007400000 @ 0000000000d00000 (usable)
 memory: 0000000007c00000 @ 0000000008300000 (usable)
 memory: 000000000fc00000 @ 0000000410300000 (usable)
software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff]
Zone ranges:
  DMA32    [mem 0x00400000-0xefffffff]
  Normal   [mem 0xf0000000-0x41fefffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00400000-0x00a86fff]
  node   0: [mem 0x00d00000-0x080fffff]
  node   0: [mem 0x08300000-0x0fefffff]
  node   0: [mem 0x410300000-0x41fefffff]
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.
PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 125878
Kernel command line:  bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 495244k/504084k available (4032k kernel code, 8840k reserved, 1321k data, 276k init, 0k highmem)
Hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
NR_IRQS:511
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
pid_max: default: 32768 minimum: 501
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
SMP: Booting CPU01 (CoreId  1)...
CPU revision is: 000d0601 (Cavium Octeon+)
Brought up 2 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
octeon_pci_console: Console not created.
HugeTLB registered 2 MB page size, pre-allocated 0 pages
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.13 (for 3.10.34)
aufs 3.10.x-20141215
msgmni has been set to 967
io scheduler noop registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON
loop: module loaded
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB 16f0010000000.usbc: Octeon Host Controller
OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1
OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 56
usbcore: registered new interface driver usb-storage
octeon_wdt: Initial granularity 5 Sec
TCP: cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7
Amd/Fujitsu Extended Query Table at 0x0040
  Amd/Fujitsu Extended Query version 1.1.
phys_mapped_flash: Swapping erase regions for top-boot CFI table.
number of CFI chips: 1
3 cmdlinepart partitions found on MTD device phys_mapped_flash
Creating 3 MTD partitions on "phys_mapped_flash":
0x000000000000-0x000000080000 : "boot0"
0x000000080000-0x000000100000 : "boot1"
0x000000100000-0x000000110000 : "eeprom"
Waiting 15sec before mounting root device...
usb 1-1: new high-speed USB device number 2 using OcteonUSB
usb-storage 1-1:1.0: USB Mass Storage device detected
scsi0 : usb-storage 1-1:1.0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
kjournald starting.  Commit interval 3 seconds
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): recovery complete
EXT3-fs (sda2): mounted filesystem with journal data mode
aufs test_add:305:swapper/0[1]: uid/gid/perm /root.loop 0/0/0755, 0/102/0755
VFS: Mounted root (aufs filesystem) on device 0:11.
Freeing unused kernel memory: 276K (ffffffffc053b000 - ffffffffc0580000)
SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0xa2af6f
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
SQUASHFS error: Unable to read fragment cache entry [a2af6f]
SQUASHFS error: Unable to read page, block a2af6f, size befb
Algorithmics/MIPS FPU Emulator v1.5
SQUASHFS error: zlib_inflate error, data probably corrupt
SQUASHFS error: squashfs_read_data failed to read block 0x48d89a
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
SQUASHFS error: Unable to read fragment cache entry [48d89a]
SQUASHFS error: Unable to read page, block 48d89a, size 9b76
/bin/sh: error while loading shared libraries: /lib/mips-linux-gnu/libdl.so.2: cannot read file data: Input/output error
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00


*** NMI Watchdog interrupt on Core 0x01 ***
        $0      0x0000000000000000      at      0x0000000010008ce0
        v0      0xffffffffc0509018      v1      0x0000000000000001
        a0      0xfffffffffffffffd      a1      0x0000000000000000
        a2      0x0000000100000000      a3      0x0000000000000001
        a4      0x0000000000000401      a5      0x00000000000003ff
        a6      0x0000000000000000      a7      0x800000041c034560
        t0      0x0000000010008ce0      t1      0x000000001000001f
        t2      0xffffffffc0093568      t3      0x800000041c0c8000
        s0      0x800000000178c880      s1      0x0000000000000001
        s2      0x00000000ffffff01      s3      0x00000000ffffff02
        s4      0xffffffffc048ab78      s5      0x0000000000000069
        s6      0x0000000000000000      s7      0x800000041ff00000
        t8      0x0000000000000002      t9      0xffffffffc006c130
        k0      0x0000000000000000      k1      0x0000000000000020
        gp      0x800000041c0bc000      sp      0x800000041c0bfb30
        s8      0xffffffffc048b5e0      ra      0xffffffffc0073834
        err_epc 0xffffffffc0073838      epc     0xffffffffc006a6e0
        status  0x0000000010488ce4      cause   0x0000000040808800
        sum0    0x000000f000000000      en0     0x0000000000000000
*** Chip soft reset soon ***

Official EdgeRouter Lite TFTP recovery

In my first attempt at recovering the EdgeRouter Lite, I used Ubiquiti’s official guide found here. That didn’t work, so I tried replacing the flash drive with a 4GB Sandisk Cruzer Blade. It’s not the best fit for the ERL, so I don’t recommend getting that. If you’re looking for a USB flash drive, then I believe the Sandisk Cruzer Fit should fit better.

Unfortunately, it didn’t work as well. I started to wonder if these flash drives were good. I decided to plug both of them into my MacBook Pro. The ERL’s flash drive was recognized just fine. However, my MBP was having trouble detecting the Sandisk flash drive.

I decided to give it another try by using a known good flash drive. However, that didn’t work also. At that point, I was ready to move on.

Unofficial EdgeRouter Lite TFTP recovery

I stumbled upon a 7-year old post on Ubiquiti’s community site that I decided to give it a try. Even though it was an old post, it worked great. The one thing that didn’t work was the link to the EdgeMax rescue kit (emrk) file. With the help of Google, I was able to find a different link (mirror).

Note

This TFTP recovery file is only compatible with ER-Lite and ER-PoE models. Do not attempt with any other EdgeRouter models.

EdgeRouter Lite recovery instructions

The recovery instructions covered here contains some modifications from the original post. I performed all of the steps here under macOS, so you may have to modify depending on your computer’s OS.

1. Download EMRK

This step assumes that you haven’t done so. Use the link found here. If you already downloaded it, then skip to Step 2.

2. Move EMRK file

Once the download is complete, the file must be in the right directory. The macOS’s built-in TFTP software uses the /private/tftpboot.

$ sudo cp Desktop/emrk-0.9c.bin /private/tftpboot

3. Launch TFTP service

The macOS has a built-in TFTP software, and you can launch it by issuing the commands below.

$ sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
$ sudo launchctl start com.apple.tftpd

4. Launch HTTP server

This step is optional. Since I already have the ERL firmware, I decided to use a local HTTP server.

Note

When you launch Python’s HTTP server, make sure you start at the /private/tftpboot directory since that’s where the EMRK file is.

$ cd /private/tftpboot
$ python -m SimpleHTTPServer 80

5. Connect to the ERL’s console port

This step assumes that you haven’t done so. As mentioned here, you can use GNU Screen software since it’s free. Alternatively, you can use Z-Term or SecureCRT for macOS. If you’re already connected, then skip to step 3.

6. Enter bootloader console

If the ERL is powered off, turn it on and press any key once or multiple times when you see this prompt.

Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build Version: e102_003_eace7) (May 27 2019 - 06:35:40)

If you were successful, you should see this prompt.

Octeon ubnt_e100#

7. Connect to the ERL’s eth0 port

Connect your computer’s Ethernet port to ERL’s eth0 port. Once connected, set a static IP address to your computer. For macOS, you will need to go to System Preferences > Network > Belkin USB-C LAN (or equivalent) > Configure IPv4 > Manually. Use the following settings and click Apply:

IP Address: 192.168.1.10
Subnet Mask: 255.255.255.0

Do you find this content useful? If so, consider buying me a coffee! ☕


8. Configure bootloader

To load the EMRK file to memory, you need to configure the settings. Use the following commands, as shown below.

Octeon ubnt_e100# set ipaddr 192.168.1.20
Octeon ubnt_e100# set netmask 255.255.255.0
Octeon ubnt_e100# set serverip 192.168.1.10
Octeon ubnt_e100# set ethact octeth0
Octeon ubnt_e100# set bootfile emrk-0.9c.bin
Octeon ubnt_e100# tftpboot
Using octeth0 device
TFTP from server 192.168.1.10; our IP address is 192.168.1.20
Filename 'emrk-0.9c.bin'.
Load address: 0x9f00000
Loading: #################################################################
         #############################################
done
Bytes transferred = 15665511 (ef0967 hex), 11093 Kbytes/sec

9. Boot the kernel

Once the ERMK file is in memory, we’re now ready to boot ERL using that kernel. Use the command below to load the kernel.

Octeon ubnt_e100# bootoctlinux $loadaddr

Once you hit enter, you will see messages like the one below.

Loading the kernel
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff81100000 (adjusted to: 0x1100000), size 0xe83940
Allocated memory for ELF segment: addr: 0xffffffff81100000, size 0xe83940
Processing PHDR 0
  Loading e23d80 bytes at ffffffff81100000
  Clearing 5fbc0 bytes at ffffffff81f23d80
## Loading Linux kernel with entry point: 0xffffffff81105ca0 ...
Bootloader: Done loading app on coremask: 0x1
Linux version 2.6.32.13-wau (dmbaturin@v-dev) (gcc version 4.3.3 (Cavium Networks Version: 2_0_0 build 95) ) #81 SMP Tue Jul 23 13:51:58 PDT 2013
CVMSEG size: 2 cache lines (256 bytes)
Cavium Networks SDK-2.0
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 000000000067b000 @ 00000000018b5000 (usable after init)
 memory: 0000000006000000 @ 0000000002000000 (usable)
 memory: 0000000007c00000 @ 0000000008200000 (usable)
 memory: 000000000fc00000 @ 0000000410000000 (usable)
Wasting 354200 bytes for tracking 6325 unused pages
Initrd not found or empty - disabling initrd
Zone PFN ranges:
  DMA32    0x000018b5 -> 0x00100000
  Normal   0x00100000 -> 0x0041fc00
Movable zone start PFN for each node
early_node_map[4] active PFN ranges
    0: 0x000018b5 -> 0x00001f30
    0: 0x00002000 -> 0x00008000
    0: 0x00008200 -> 0x0000fe00
    0: 0x00410000 -> 0x0041fc00
Cavium Hotplug: Available coremask 0x2
PERCPU: Embedded 8 pages/cpu @a8000000027b8000 s11648 r0 d21120 u65536
pcpu-alloc: s11648 r0 d21120 u65536 alloc=16*4096
pcpu-alloc: [0] 0 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 63455
Kernel command line:  bootoctlinux $loadaddr console=ttyS0,115200
PID hash table entries: 1024 (order: 1, 8192 bytes)
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Memory: 475308k/489964k available (3808k kernel code, 14444k reserved, 4082k data, 6636k init, 0k highmem)
Hierarchical RCU implementation.
NR_IRQS:152
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
Brought up 1 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 2, 16384 bytes)
IPv4 FIB: Using LC-trie version 0.409
TCP established hash table entries: 8192 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
/proc/octeon_perf: Octeon performace counter interface loaded
octeon_wdt: Initial granularity 5 Sec.
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.11 (for 2.6.32.55)
msgmni has been set to 928
alg: No test for cipher_null (cipher_null-generic)
alg: No test for ecb(cipher_null) (ecb-cipher_null)
alg: No test for digest_null (digest_null-generic)
alg: No test for compress_null (compress_null-generic)
alg: No test for stdrng (krng)
alg: No test for ghash (ghash-generic)
io scheduler noop registered
io scheduler cfq registered (default)
octeon_rng octeon_rng: Octeon Random Number Generator
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0x1180000000800 (irq = 58) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
loop: module loaded
mdio-octeon: probed
mdio-octeon mdio-octeon.0: Version 1.0
octeon-ethernet 2.0
Interface 0 has 3 ports (RGMII)
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB: Detected 1 ports
OcteonUSB OcteonUSB.0: Octeon Host Controller
OcteonUSB OcteonUSB.0: new USB bus registered, assigned bus number 1
OcteonUSB OcteonUSB.0: irq 80, io mem 0x00000000
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 80
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
Probing USB hub...
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
IPv6 over IPv4 tunneling driver
sit0: Disabled Privacy Extensions
ip6tnl0: Disabled Privacy Extensions
NET: Registered protocol family 17
NET: Registered protocol family 15
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
L2 lock: TLB refill 256 bytes
L2 lock: General exception 128 bytes
L2 lock: low-level interrupt 128 bytes
L2 lock: interrupt 640 bytes
L2 lock: memcpy 1152 bytes
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank
 Amd/Fujitsu Extended Query Table at 0x0040
phys_mapped_flash: Swapping erase regions for broken CFI table.
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Freeing unused kernel memory: 6636k freed
Algorithmics/MIPS FPU Emulator v1.5
init started: BusyBox v1.17.1 (Debian 1:1.17.1-8)
starting pid 203, tty '': '/etc/init.d/rcS'
hub 1-0:1.0: activate --> -22
usb 1-1: new high speed USB device using OcteonUSB and address 2
usb 1-1: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
eth0: 1000 Mbps Full duplex, port  0, queue  0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk

Loading EMRK 0.9a
Mounting filesystems
Bringing up eth0

Checking boot partition
Boot partition looks intact
Attempting to mount boot partition
Boot partition successfully mounted
Looking for kernel file
Found a kernel
Checking kernel MD5 sum file
Found kernel MD5 sum file
Checking kernel MD5 sum
Kernel MD5 sum is correct

Checking root partition
Root partition looks intact
Attempting to mount root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with writeback data mode.
Root partition successfully mounted
Looking for system image file
Found a system image file
Checking system image MD5 sum file
Found system image MD5 sum file
Checking system image MD5 sum
sd 0:0:0:0: [sda] Unhandled sense code
sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08
sd 0:0:0:0: [sda] Sense Key : 0x3 [current] 
sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0
sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 df 90 00 00 f0 00
end_request: I/O error, dev sda, sector 581520
sd 0:0:0:0: [sda] Unhandled sense code
sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08
sd 0:0:0:0: [sda] Sense Key : 0x3 [current] 
sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0
sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 e0 40 00 00 08 00
end_request: I/O error, dev sda, sector 581696
ash: 109f4af8c5db580ec13f156a21caddea: unknown operand
System image MD5 sum is not correct! Your image may be corrupted.

**********************************************
Welcome to EdgeMax Rescue Kit!

This tool is distributed under the terms of
GNU General Public License and other licenses

Brought to you by SO3 Group

WARNING: This tool is not developed, officially
supported or endorsed by Ubiquiti Networks!

Using it may lead to destroying your router
configuration or operating system

Ubiquiti Networks support will not help you
with using it or fixing consequences of
using it.

This tool itself is distributed without any
warranty and authors are not liable for
any damage it may cause

By using this tool you agree you are doing
it at your own risk and understand what
you are doing

*********************************************

When you see the following prompts, answer the questions, as shown below.

Note

Since I didn’t opt to connect the ERL to my network, I chose to assign a static IP address. If you want to download ERL’s firmware directly from Ubiquiti, then connect ERL’s eth0 to your network and set it as DHCP

Enter 'Yes' to proceed, 'No' to reboot
yes or no: yes

Do you want to configure network via DHCP?
yes or no: no

Do you want to configure network statically?
yes or no: yes

Enter IPv4 address in CIDR format (e.g. 192.0.2.10/24): 192.168.1.0/24
Enter IPv4 gateway address: 192.168.1.1
Enter DNS server address: 192.168.1.1

EMRK provides some scripts for automated
recovery procedures:

emrk-factory-reset -- reset config to factory default
emrk-remove-user-data -- remove all the user data including
    config and everything
emrk-reinstall -- reinstall EdgeOS from scratch
    (wipes any user data too)

Enter 'reboot' to reboot your router


BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/ash: can't access tty; job control turned off
EMRK>

10. Reinstall ERL’s firmware

There are three options that you can do from the EMRK prompt. I opted for the emrk-reinstall script.

Note

Change the EdgeOS image URL if you want to download directly from Ubiquiti’s site.

EMRK>emrk-reinstall
WARNING: This script will reinstall EdgeOS from scratch
If you have any usable data on your router storage,
it will be irrecoverably destroyed!
Do you want to continue?
yes or no: yes
Unmounting boot partition
Unmounting root partition
Re-creating partition table
Creating boot partition
Formatting boot partition
mkfs.vfat 3.0.9 (31 Jan 2010)
Creating root partition
Formatting root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: http://192.168.1.10/ER-e100.v1.10.11.5274249.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 82.9M  100 82.9M    0     0  3156k      0  0:00:26  0:00:26 --:--:-- 3111k
Unpacking EdgeOS release image
Verifying EdgeOS kernel
Copying EdgeOS kernel to boot partition
Verifying EdgeOS system image
Copying EdgeOS system image to root partition
Copying version file to the root partition
Creating EdgeOS writable data directory
Cleaning up
Installation finished
Please reboot your router

11. Reboot the ERL

Now you are ready to reboot the EdgeRouter Lite. Issue the command reboot, and you should eventually see the familiar prompt.

ERL boot messages
EMRK>reboot
starting pid 295, tty '': '/bin/umount -a -r'
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
Restarting system.

Looking for valid bootloader image....
Jumping to start of image at address 0xbfc80000


U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
       scanning bus for storage devices...
  Device 0: Vendor:          Prod.: USB DISK 2.0     Rev: PMAP
            Type: Removable Hard Disk
            Capacity: 3824.0 MB = 3.7 GB (7831552 x 512)
 0 
reading vmlinux.64
.............................

5787776 bytes read
argv[2]: coremask=0x3
argv[3]: root=/dev/sda2
argv[4]: rootdelay=15
argv[5]: rw
argv[6]: rootsqimg=squashfs.img
argv[7]: rootsqwdir=w
argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
ELF file is 64 bit
Allocating memory for mapped kernel segment, alignment: 0x400000
Allocated memory for ELF segment: addr: 0x400000, size 0x696350
Processing PHDR 0
  Loading 583a80 bytes at 400000
  Clearing 1128d0 bytes at 983a80
## Loading Linux kernel with entry point: 0x007eb7d0 ...
Bootloader: Done loading app on coremask: 0x3
Linux version 3.10.107-UBNT (root@aed5f1a708f2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Fri Feb 21 09:09:08 UTC 2020
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-3.1.2
bootconsole [early0] enabled
CPU revision is: 000d0601 (Cavium Octeon+)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
Determined physical RAM map:
 memory: 0000000000543000 @ 0000000000400000 (kernel data and code)
 memory: 000000000004d000 @ 0000000000943000 (usable after init)
 memory: 0000000000107000 @ 0000000000990000 (kernel data and code)
 memory: 0000000007400000 @ 0000000000d00000 (usable)
 memory: 0000000007c00000 @ 0000000008300000 (usable)
 memory: 000000000fc00000 @ 0000000410300000 (usable)
software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff]
Zone ranges:
  DMA32    [mem 0x00400000-0xefffffff]
  Normal   [mem 0xf0000000-0x41fefffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00400000-0x00a96fff]
  node   0: [mem 0x00d00000-0x080fffff]
  node   0: [mem 0x08300000-0x0fefffff]
  node   0: [mem 0x410300000-0x41fefffff]
Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes.
Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes.
Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes.
PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 125893
Kernel command line:  bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 495244k/504116k available (4057k kernel code, 8872k reserved, 1330k data, 308k init, 0k highmem)
SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
NR_IRQS:511
Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000)
pid_max: default: 32768 minimum: 501
Security Framework initialized
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
SMP: Booting CPU01 (CoreId  1)...
CPU revision is: 000d0601 (Cavium Octeon+)
Brought up 2 CPUs
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource OCTEON_CVMCOUNT
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
octeon_pci_console: Console not created.
HugeTLB registered 2 MB page size, pre-allocated 0 pages
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.13 (for 3.10.34)
aufs 3.10.x-20141215
msgmni has been set to 967
io scheduler noop registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON
loop: module loaded
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
OcteonUSB 16f0010000000.usbc: Octeon Host Controller
OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1
OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
OcteonUSB: Registered HCD for port 0 on irq 56
usbcore: registered new interface driver usb-storage
octeon_wdt: Initial granularity 5 Sec
TCP: cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
Bootbus flash: Setting flash for 4MB flash at 0x1f800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7
Amd/Fujitsu Extended Query Table at 0x0040
  Amd/Fujitsu Extended Query version 1.1.
phys_mapped_flash: Swapping erase regions for top-boot CFI table.
number of CFI chips: 1
3 cmdlinepart partitions found on MTD device phys_mapped_flash
Creating 3 MTD partitions on "phys_mapped_flash":
0x000000000000-0x000000080000 : "boot0"
0x000000080000-0x000000100000 : "boot1"
0x000000100000-0x000000110000 : "eeprom"
Waiting 15sec before mounting root device...
usb 1-1: new high-speed USB device number 2 using OcteonUSB
usb-storage 1-1:1.0: USB Mass Storage device detected
scsi0 : usb-storage 1-1:1.0
scsi 0:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 6
sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] No Caching mode page found
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
kjournald starting.  Commit interval 3 seconds
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): mounted filesystem with journal data mode
VFS: Mounted root (aufs filesystem) on device 0:11.
Freeing unused kernel memory: 308K (ffffffffc0543000 - ffffffffc0590000)
Algorithmics/MIPS FPU Emulator v1.5
INIT: version 2.88 booting
2043+0 records in
2043+0 records out
INIT: Entering runlevel: 2
[....] Starting SSH recovery service in the background
ssh-recovery: starting...
ssh-recovery: if=(all) port=(60257) terminate-timeout=(60)
[ ok ] Starting daemon monitor: monit.
[ ok ] Starting routing daemon: rib nsm ribd.
/opt/vyatta/etc/config.boot.default created
/config/config.boot created
[....] Starting EdgeOS router: migrate rl-systemssh-recovery: enabling link on interfaces...
ssh-recovery: eth0 :: mac=(44:d9:e7:95:a3:88)
ssh-recovery: eth1 :: mac=(44:d9:e7:95:a3:89)
ssh-recovery: eth2 :: mac=(44:d9:e7:95:a3:8a)
ssh-recovery: service started :: pid=(1021)
[ ok igure.
Starting network plug daemon: netplugd.

Welcome to EdgeOS ubnt ttyS0

By logging in, accessing, or using the Ubiquiti product, you
acknowledge that you have read and understood the Ubiquiti
License Agreement (available in the Web UI at, by default,
http://192.168.1.1) and agree to be bound by its terms.

ubnt login:

When you log in, you might see the following message.

ubnt login: ubnt
Password: ubnt
Boot image can be upgraded to version [ e102_003_eace7 ].
Run "add system boot-image" to upgrade boot image.
ubnt@ubnt:~$ add system boot-image
Uboot version [UNKNOWN] is about to be replaced
Warning: Don't turn off the power or reboot during the upgrade!
Are you sure you want to replace old version? (Yes/No) [Yes]: Yes
Preparing to upgrade...Done
Copying upgrade boot image...Done
Checking boot version: Current is UNKNOWN; new is e102_003_eace7 ...Done
Checking upgrade image...Done
Writing image...
Upgrade boot completed
ubnt@ubnt:~$

Final thoughts

There seem to be many ways to perform the EdgeRouter Lite recovery, depending on the issue. As I was writing this, I found this post, and it seems to be the same SQUASHFS issue that I encountered.

Another one I found was this post. According to the author, it is much simpler to use than the TFTP-based method, as demonstrated here. Though, users reported some issues.

If I ever see this again on this same hardware (I have two ERLs), then I will replace the USB flash drive to a Sandisk Cruzer Fit or Kingston Data Traveler and use this guide since I know it worked flawlessly.

You might like to read

How to configure EdgeRouter Lite via CLI – Part 1
How to configure EdgeRouter Lite via CLI – Part 2
Hardening EdgeRouter Lite – Part 1
Hardening EdgeRouter Lite – Part 2

Do you find this content useful? If so, consider buying me a coffee! ☕


Disclosure

AndrewRoderos.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

About Andrew Roderos

I am a network security engineer with a passion for networking and security. Follow me on Twitter, LinkedIn, and Instagram.