
My original EdgeRouter Lite (ERL) that I started using since March 2016 crapped out last year. Earlier this year, I received a tip about replacing the USB flash drive to recover the router. A few days ago, I tried recovering the EdgeRouter Lite. I am happy to report that the recovery was successful! This article will discuss my experience with the EdgeRouter recovery and what I did to recover it.
This post contains affiliate links. If you use these links to buy something I may earn a commission. Full disclosure here.
The Issue
For most of the day, I was connected to the Internet just fine. Then, one minute, I lost Internet connectivity. As any IT professional would do, I started troubleshooting.
I didn’t suspect that it was the router at first since it has been my most stable router since 2000. When I finally isolated the issue with the router, I rebooted it.
Related: My Home Router – EdgeRouter Lite |
After a few minutes of waiting, I still couldn’t ping the gateway. I decided to bust out my Tripp Lite Keyspan and console cable and connected it to the ERL’s console port.
I didn’t see any console messages even after performing a hard reboot. When I rebooted it, I also heard a hissing noise. I assumed that it was dead, so I swapped it out with my spare router. The replacement process was quick and easy since I have a backup config.
Recovery attempt
After many months from when I received the tip, I finally carved out some time to perform the recovery. Surprisingly, when I consoled into the ERL this time around, it was displaying some output. When I did it last year, I verified that my serial adapter and console cable were fine by connecting to my Cisco switch’s console port. I also made sure that I was using the correct baud rate but still wasn’t displaying any output.
Displaying console messages
To display ERL console messages, you will need the following settings if you use a terminal emulator like SecureCRT on a macOS/Windows or PuTTY on Windows.
- Baud rate: 115200
- Data bits: 8
- Parity: None
- Stop bits: 1
- Flow control: None
In my case, it was easier to use the existing software in macOS, which is the GNU screen. Before you can use the screen software, you need to determine the device name.
$ ls /dev | grep -m 10 "tty\." tty.Bluetooth-Incoming-Port tty.KeySerial1 tty.MALS tty.NullModem-1 tty.SOC tty.USA19H142P1.1
As you can see, there are a few of them. Since I’ve been using Tripp Lite for a while, I know it’s the tty.KeySerial1
device.
Once you determine the right device name, you are now ready to connect to the EdgeRouter via console.
$ screen /dev/tty.KeySerial1 115200
When I plugged the power in, it displayed a lot of messages. Eventually, the console messages repeat itself, so the router is stuck in a boot loop. As you can see from the console messages, one of the SQUASHFS
error messages indicated the data is probably corrupt. To me, it is a telltale sign that the flash is readable and probably just needs to be reimaged.
ERL console messages
Looking for valid bootloader image.... Jumping to start of image at address 0xbfc80000 U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22) BIST check passed. UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388 MPR 13-00318-18 Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate) DRAM: 512 MB Clearing DRAM....... done Flash: 4 MB Net: octeth0, octeth1, octeth2 USB: (port 0) scanning bus for devices... 1 USB Devices found scanning bus for storage devices... Device 0: Vendor: Prod.: USB DISK 2.0 Rev: PMAP Type: Removable Hard Disk Capacity: 3824.0 MB = 3.7 GB (7831552 x 512) 0 reading vmlinux.64 ............................. 5755008 bytes read argv[2]: coremask=0x3 argv[3]: root=/dev/sda2 argv[4]: rootdelay=15 argv[5]: rw argv[6]: rootsqimg=squashfs.img argv[7]: rootsqwdir=w argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) ELF file is 64 bit Allocating memory for mapped kernel segment, alignment: 0x400000 Allocated memory for ELF segment: addr: 0x400000, size 0x6863d0 Processing PHDR 0 Loading 57ba80 bytes at 400000 Clearing 10a950 bytes at 97ba80 ## Loading Linux kernel with entry point: 0x007e5ff0 ... Bootloader: Done loading app on coremask: 0x3 Linux version 3.10.107-UBNT (root@dc9c2e14f2e4) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Thu Aug 9 06:45:26 UTC 2018 CVMSEG size: 2 cache lines (256 bytes) Cavium Inc. SDK-3.1.2 bootconsole [early0] enabled CPU revision is: 000d0601 (Cavium Octeon+) Checking for the multiply/shift bug... no. Checking for the daddiu bug... no. Determined physical RAM map: memory: 000000000053b000 @ 0000000000400000 (kernel data and code) memory: 0000000000045000 @ 000000000093b000 (usable after init) memory: 0000000000107000 @ 0000000000980000 (kernel data and code) memory: 0000000007400000 @ 0000000000d00000 (usable) memory: 0000000007c00000 @ 0000000008300000 (usable) memory: 000000000fc00000 @ 0000000410300000 (usable) software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff] Zone ranges: DMA32 [mem 0x00400000-0xefffffff] Normal [mem 0xf0000000-0x41fefffff] Movable zone start for each node Early memory node ranges node 0: [mem 0x00400000-0x00a86fff] node 0: [mem 0x00d00000-0x080fffff] node 0: [mem 0x08300000-0x0fefffff] node 0: [mem 0x410300000-0x41fefffff] Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes. Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes. Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes. PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 125878 Kernel command line: bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200 PID hash table entries: 2048 (order: 2, 16384 bytes) Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) Inode-cache hash table entries: 32768 (order: 6, 262144 bytes) Memory: 495244k/504084k available (4032k kernel code, 8840k reserved, 1321k data, 276k init, 0k highmem) Hierarchical RCU implementation. Additional per-CPU info printed with stalls. NR_IRQS:511 Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000) pid_max: default: 32768 minimum: 501 Security Framework initialized Mount-cache hash table entries: 256 Checking for the daddi bug... no. SMP: Booting CPU01 (CoreId 1)... CPU revision is: 000d0601 (Cavium Octeon+) Brought up 2 CPUs NET: Registered protocol family 16 bio: create slabat 0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switching to clocksource OCTEON_CVMCOUNT NET: Registered protocol family 2 TCP established hash table entries: 4096 (order: 4, 65536 bytes) TCP bind hash table entries: 4096 (order: 4, 65536 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP: reno registered UDP hash table entries: 256 (order: 1, 8192 bytes) UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) NET: Registered protocol family 1 octeon_pci_console: Console not created. HugeTLB registered 2 MB page size, pre-allocated 0 pages squashfs: version 4.0 (2009/01/31) Phillip Lougher Registering unionfs 2.5.13 (for 3.10.34) aufs 3.10.x-20141215 msgmni has been set to 967 io scheduler noop registered io scheduler cfq registered (default) Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled 1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON console [ttyS0] enabled, bootconsole disabled console [ttyS0] enabled, bootconsole disabled 1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON loop: module loaded ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver OcteonUSB 16f0010000000.usbc: Octeon Host Controller OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1 OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000 hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected OcteonUSB: Registered HCD for port 0 on irq 56 usbcore: registered new interface driver usb-storage octeon_wdt: Initial granularity 5 Sec TCP: cubic registered NET: Registered protocol family 17 NET: Registered protocol family 15 Bootbus flash: Setting flash for 4MB flash at 0x1f800000 phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7 Amd/Fujitsu Extended Query Table at 0x0040 Amd/Fujitsu Extended Query version 1.1. phys_mapped_flash: Swapping erase regions for top-boot CFI table. number of CFI chips: 1 3 cmdlinepart partitions found on MTD device phys_mapped_flash Creating 3 MTD partitions on "phys_mapped_flash": 0x000000000000-0x000000080000 : "boot0" 0x000000080000-0x000000100000 : "boot1" 0x000000100000-0x000000110000 : "eeprom" Waiting 15sec before mounting root device... usb 1-1: new high-speed USB device number 2 using OcteonUSB usb-storage 1-1:1.0: USB Mass Storage device detected scsi0 : usb-storage 1-1:1.0 scsi 0:0:0:0: Direct-Access USB DISK 2.0 PMAP PQ: 0 ANSI: 6 sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Attached SCSI removable disk kjournald starting. Commit interval 3 seconds EXT3-fs (sda2): using internal journal EXT3-fs (sda2): recovery complete EXT3-fs (sda2): mounted filesystem with journal data mode aufs test_add:305:swapper/0[1]: uid/gid/perm /root.loop 0/0/0755, 0/102/0755 VFS: Mounted root (aufs filesystem) on device 0:11. Freeing unused kernel memory: 276K (ffffffffc053b000 - ffffffffc0580000) SQUASHFS error: zlib_inflate error, data probably corrupt SQUASHFS error: squashfs_read_data failed to read block 0xa2af6f SQUASHFS error: Unable to read fragment cache entry [a2af6f] SQUASHFS error: Unable to read page, block a2af6f, size befb SQUASHFS error: Unable to read fragment cache entry [a2af6f] SQUASHFS error: Unable to read page, block a2af6f, size befb SQUASHFS error: Unable to read fragment cache entry [a2af6f] SQUASHFS error: Unable to read page, block a2af6f, size befb SQUASHFS error: Unable to read fragment cache entry [a2af6f] SQUASHFS error: Unable to read page, block a2af6f, size befb SQUASHFS error: Unable to read fragment cache entry [a2af6f] SQUASHFS error: Unable to read page, block a2af6f, size befb Algorithmics/MIPS FPU Emulator v1.5 SQUASHFS error: zlib_inflate error, data probably corrupt SQUASHFS error: squashfs_read_data failed to read block 0x48d89a SQUASHFS error: Unable to read fragment cache entry [48d89a] SQUASHFS error: Unable to read page, block 48d89a, size 9b76 SQUASHFS error: Unable to read fragment cache entry [48d89a] SQUASHFS error: Unable to read page, block 48d89a, size 9b76 SQUASHFS error: Unable to read fragment cache entry [48d89a] SQUASHFS error: Unable to read page, block 48d89a, size 9b76 SQUASHFS error: Unable to read fragment cache entry [48d89a] SQUASHFS error: Unable to read page, block 48d89a, size 9b76 /bin/sh: error while loading shared libraries: /lib/mips-linux-gnu/libdl.so.2: cannot read file data: Input/output error Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 *** NMI Watchdog interrupt on Core 0x01 *** $0 0x0000000000000000 at 0x0000000010008ce0 v0 0xffffffffc0509018 v1 0x0000000000000001 a0 0xfffffffffffffffd a1 0x0000000000000000 a2 0x0000000100000000 a3 0x0000000000000001 a4 0x0000000000000401 a5 0x00000000000003ff a6 0x0000000000000000 a7 0x800000041c034560 t0 0x0000000010008ce0 t1 0x000000001000001f t2 0xffffffffc0093568 t3 0x800000041c0c8000 s0 0x800000000178c880 s1 0x0000000000000001 s2 0x00000000ffffff01 s3 0x00000000ffffff02 s4 0xffffffffc048ab78 s5 0x0000000000000069 s6 0x0000000000000000 s7 0x800000041ff00000 t8 0x0000000000000002 t9 0xffffffffc006c130 k0 0x0000000000000000 k1 0x0000000000000020 gp 0x800000041c0bc000 sp 0x800000041c0bfb30 s8 0xffffffffc048b5e0 ra 0xffffffffc0073834 err_epc 0xffffffffc0073838 epc 0xffffffffc006a6e0 status 0x0000000010488ce4 cause 0x0000000040808800 sum0 0x000000f000000000 en0 0x0000000000000000 *** Chip soft reset soon ***
Official EdgeRouter Lite TFTP recovery
In my first attempt at recovering the EdgeRouter Lite, I used Ubiquiti’s official guide found here. That didn’t work, so I tried replacing the flash drive with a 4GB Sandisk Cruzer Blade. It’s not the best fit for the ERL, so I don’t recommend getting that. If you’re looking for a USB flash drive, then I believe the Sandisk Cruzer Fit should fit better.
Unfortunately, it didn’t work as well. I started to wonder if these flash drives were good. I decided to plug both of them into my MacBook Pro. The ERL’s flash drive was recognized just fine. However, my MBP was having trouble detecting the Sandisk flash drive.
I decided to give it another try by using a known good flash drive. However, that didn’t work also. At that point, I was ready to move on.
Unofficial EdgeRouter Lite TFTP recovery
I stumbled upon a 7-year old post on Ubiquiti’s community site that I decided to give it a try. Even though it was an old post, it worked great. The one thing that didn’t work was the link to the EdgeMax rescue kit (emrk) file. With the help of Google, I was able to find a different link (mirror).
Note
This TFTP recovery file is only compatible with ER-Lite and ER-PoE models. Do not attempt with any other EdgeRouter models.
EdgeRouter Lite recovery instructions
The recovery instructions covered here contains some modifications from the original post. I performed all of the steps here under macOS, so you may have to modify depending on your computer’s OS.
1. Download EMRK
This step assumes that you haven’t done so. Use the link in the unofficial ERL TFTP recovery section. If you already downloaded it, then skip to Step 2.
2. Move EMRK file
Once the download is complete, the file must be in the right directory. The macOS’s built-in TFTP software uses the /private/tftpboot
.
$ sudo cp Desktop/emrk-0.9c.bin /private/tftpboot
3. Launch TFTP service
The macOS has a built-in TFTP software, and you can launch it by issuing the commands below.
$ sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist $ sudo launchctl start com.apple.tftpd
4. Launch HTTP server
This step is optional. Since I already have the ERL firmware, I decided to use a local HTTP server.
Note
When you launch Python’s HTTP server, make sure you start at the /private/tftpboot
directory since that’s where the EMRK file is.
$ cd /private/tftpboot $ python -m SimpleHTTPServer 80
5. Connect to the ERL’s console port
This step assumes that you haven’t done so. As mentioned here, you can use GNU Screen software since it’s free. Alternatively, you can use Z-Term or SecureCRT for macOS. If you’re already connected, then go to step 6.
6. Enter bootloader console
If the ERL is powered off, turn it on and press any key once or multiple times when you see this prompt.
Looking for valid bootloader image.... Jumping to start of image at address 0xbfc80000 U-Boot 1.1.1 (UBNT Build Version: e102_003_eace7) (May 27 2019 - 06:35:40)
If you were successful, you should see this prompt.
Octeon ubnt_e100#
7. Connect to the ERL’s eth0 port
Connect your computer’s Ethernet port to ERL’s eth0 port. Once connected, set a static IP address to your computer. For macOS, you will need to go to System Preferences > Network > Belkin USB-C LAN (or equivalent) > Configure IPv4 > Manually. Use the following settings and click Apply:
IP Address: 192.168.1.10
Subnet Mask: 255.255.255.0
8. Configure bootloader
To load the EMRK file to memory, you need to configure the settings. Use the following commands, as shown below.
Octeon ubnt_e100# set ipaddr 192.168.1.20 Octeon ubnt_e100# set netmask 255.255.255.0 Octeon ubnt_e100# set serverip 192.168.1.10 Octeon ubnt_e100# set ethact octeth0 Octeon ubnt_e100# set bootfile emrk-0.9c.bin Octeon ubnt_e100# tftpboot Using octeth0 device TFTP from server 192.168.1.10; our IP address is 192.168.1.20 Filename 'emrk-0.9c.bin'. Load address: 0x9f00000 Loading: ################################################################# ############################################# done Bytes transferred = 15665511 (ef0967 hex), 11093 Kbytes/sec
9. Boot the kernel
Once the ERMK file is in memory, we’re now ready to boot ERL using that kernel. Use the command below to load the kernel.
Octeon ubnt_e100# bootoctlinux $loadaddr
Once you hit enter, you will see messages like the one below.
Loading the kernel
ELF file is 64 bit Allocating memory for ELF segment: addr: 0xffffffff81100000 (adjusted to: 0x1100000), size 0xe83940 Allocated memory for ELF segment: addr: 0xffffffff81100000, size 0xe83940 Processing PHDR 0 Loading e23d80 bytes at ffffffff81100000 Clearing 5fbc0 bytes at ffffffff81f23d80 ## Loading Linux kernel with entry point: 0xffffffff81105ca0 ... Bootloader: Done loading app on coremask: 0x1 Linux version 2.6.32.13-wau (dmbaturin@v-dev) (gcc version 4.3.3 (Cavium Networks Version: 2_0_0 build 95) ) #81 SMP Tue Jul 23 13:51:58 PDT 2013 CVMSEG size: 2 cache lines (256 bytes) Cavium Networks SDK-2.0 bootconsole [early0] enabled CPU revision is: 000d0601 (Cavium Octeon+) Checking for the multiply/shift bug... no. Checking for the daddiu bug... no. Determined physical RAM map: memory: 000000000067b000 @ 00000000018b5000 (usable after init) memory: 0000000006000000 @ 0000000002000000 (usable) memory: 0000000007c00000 @ 0000000008200000 (usable) memory: 000000000fc00000 @ 0000000410000000 (usable) Wasting 354200 bytes for tracking 6325 unused pages Initrd not found or empty - disabling initrd Zone PFN ranges: DMA32 0x000018b5 -> 0x00100000 Normal 0x00100000 -> 0x0041fc00 Movable zone start PFN for each node early_node_map[4] active PFN ranges 0: 0x000018b5 -> 0x00001f30 0: 0x00002000 -> 0x00008000 0: 0x00008200 -> 0x0000fe00 0: 0x00410000 -> 0x0041fc00 Cavium Hotplug: Available coremask 0x2 PERCPU: Embedded 8 pages/cpu @a8000000027b8000 s11648 r0 d21120 u65536 pcpu-alloc: s11648 r0 d21120 u65536 alloc=16*4096 pcpu-alloc: [0] 0 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 63455 Kernel command line: bootoctlinux $loadaddr console=ttyS0,115200 PID hash table entries: 1024 (order: 1, 8192 bytes) Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes. Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes. Memory: 475308k/489964k available (3808k kernel code, 14444k reserved, 4082k data, 6636k init, 0k highmem) Hierarchical RCU implementation. NR_IRQS:152 Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000) Security Framework initialized Mount-cache hash table entries: 256 Checking for the daddi bug... no. Brought up 1 CPUs NET: Registered protocol family 16 bio: create slabat 0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switching to clocksource OCTEON_CVMCOUNT NET: Registered protocol family 2 IP route cache hash table entries: 2048 (order: 2, 16384 bytes) IPv4 FIB: Using LC-trie version 0.409 TCP established hash table entries: 8192 (order: 5, 131072 bytes) TCP bind hash table entries: 8192 (order: 5, 131072 bytes) TCP: Hash tables configured (established 8192 bind 8192) TCP reno registered NET: Registered protocol family 1 /proc/octeon_perf: Octeon performace counter interface loaded octeon_wdt: Initial granularity 5 Sec. squashfs: version 4.0 (2009/01/31) Phillip Lougher Registering unionfs 2.5.11 (for 2.6.32.55) msgmni has been set to 928 alg: No test for cipher_null (cipher_null-generic) alg: No test for ecb(cipher_null) (ecb-cipher_null) alg: No test for digest_null (digest_null-generic) alg: No test for compress_null (compress_null-generic) alg: No test for stdrng (krng) alg: No test for ghash (ghash-generic) io scheduler noop registered io scheduler cfq registered (default) octeon_rng octeon_rng: Octeon Random Number Generator Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled serial8250.0: ttyS0 at MMIO 0x1180000000800 (irq = 58) is a OCTEON console [ttyS0] enabled, bootconsole disabled console [ttyS0] enabled, bootconsole disabled loop: module loaded mdio-octeon: probed mdio-octeon mdio-octeon.0: Version 1.0 octeon-ethernet 2.0 Interface 0 has 3 ports (RGMII) ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver OcteonUSB: Detected 1 ports OcteonUSB OcteonUSB.0: Octeon Host Controller OcteonUSB OcteonUSB.0: new USB bus registered, assigned bus number 1 OcteonUSB OcteonUSB.0: irq 80, io mem 0x00000000 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected OcteonUSB: Registered HCD for port 0 on irq 80 Initializing USB Mass Storage driver... usbcore: registered new interface driver usb-storage USB Mass Storage support registered. usbcore: registered new interface driver libusual Probing USB hub... hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 10 lo: Disabled Privacy Extensions Mobile IPv6 IPv6 over IPv4 tunneling driver sit0: Disabled Privacy Extensions ip6tnl0: Disabled Privacy Extensions NET: Registered protocol family 17 NET: Registered protocol family 15 802.1Q VLAN Support v1.8 Ben Greear All bugs added by David S. Miller L2 lock: TLB refill 256 bytes L2 lock: General exception 128 bytes L2 lock: low-level interrupt 128 bytes L2 lock: interrupt 640 bytes L2 lock: memcpy 1152 bytes Bootbus flash: Setting flash for 4MB flash at 0x1f800000 phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank Amd/Fujitsu Extended Query Table at 0x0040 phys_mapped_flash: Swapping erase regions for broken CFI table. number of CFI chips: 1 cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness. Freeing unused kernel memory: 6636k freed Algorithmics/MIPS FPU Emulator v1.5 init started: BusyBox v1.17.1 (Debian 1:1.17.1-8) starting pid 203, tty '': '/etc/init.d/rcS' hub 1-0:1.0: activate --> -22 usb 1-1: new high speed USB device using OcteonUSB and address 2 usb 1-1: configuration #1 chosen from 1 choice scsi0 : SCSI emulation for USB Mass Storage devices eth0: 1000 Mbps Full duplex, port 0, queue 0 scsi 0:0:0:0: Direct-Access USB DISK 2.0 PMAP PQ: 0 ANSI: 6 sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Attached SCSI removable disk Loading EMRK 0.9a Mounting filesystems Bringing up eth0 Checking boot partition Boot partition looks intact Attempting to mount boot partition Boot partition successfully mounted Looking for kernel file Found a kernel Checking kernel MD5 sum file Found kernel MD5 sum file Checking kernel MD5 sum Kernel MD5 sum is correct Checking root partition Root partition looks intact Attempting to mount root partition kjournald starting. Commit interval 5 seconds EXT3 FS on sda2, internal journal EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with writeback data mode. Root partition successfully mounted Looking for system image file Found a system image file Checking system image MD5 sum file Found system image MD5 sum file Checking system image MD5 sum sd 0:0:0:0: [sda] Unhandled sense code sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08 sd 0:0:0:0: [sda] Sense Key : 0x3 [current] sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0 sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 df 90 00 00 f0 00 end_request: I/O error, dev sda, sector 581520 sd 0:0:0:0: [sda] Unhandled sense code sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08 sd 0:0:0:0: [sda] Sense Key : 0x3 [current] sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0 sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 08 e0 40 00 00 08 00 end_request: I/O error, dev sda, sector 581696 ash: 109f4af8c5db580ec13f156a21caddea: unknown operand System image MD5 sum is not correct! Your image may be corrupted. ********************************************** Welcome to EdgeMax Rescue Kit! This tool is distributed under the terms of GNU General Public License and other licenses Brought to you by SO3 Group WARNING: This tool is not developed, officially supported or endorsed by Ubiquiti Networks! Using it may lead to destroying your router configuration or operating system Ubiquiti Networks support will not help you with using it or fixing consequences of using it. This tool itself is distributed without any warranty and authors are not liable for any damage it may cause By using this tool you agree you are doing it at your own risk and understand what you are doing *********************************************
When you see the following prompts, answer the questions, as shown below.
Note
Since I didn’t opt to connect the ERL to my network, I chose to assign a static IP address. If you want to download ERL’s firmware directly from Ubiquiti, then connect ERL’s eth0 to your network and set it as DHCP
Enter 'Yes' to proceed, 'No' to reboot yes or no: yes Do you want to configure network via DHCP? yes or no: no Do you want to configure network statically? yes or no: yes Enter IPv4 address in CIDR format (e.g. 192.0.2.10/24): 192.168.1.0/24 Enter IPv4 gateway address: 192.168.1.1 Enter DNS server address: 192.168.1.1 EMRK provides some scripts for automated recovery procedures: emrk-factory-reset -- reset config to factory default emrk-remove-user-data -- remove all the user data including config and everything emrk-reinstall -- reinstall EdgeOS from scratch (wipes any user data too) Enter 'reboot' to reboot your router BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash) Enter 'help' for a list of built-in commands. /bin/ash: can't access tty; job control turned off EMRK>
10. Reinstall ERL’s firmware
There are three options that you can do from the EMRK prompt. I opted for the emrk-reinstall
script.
Note
Change the EdgeOS image URL if you want to download directly from Ubiquiti’s site.
EMRK>emrk-reinstall WARNING: This script will reinstall EdgeOS from scratch If you have any usable data on your router storage, it will be irrecoverably destroyed! Do you want to continue? yes or no: yes Unmounting boot partition Unmounting root partition Re-creating partition table Creating boot partition Formatting boot partition mkfs.vfat 3.0.9 (31 Jan 2010) Creating root partition Formatting root partition kjournald starting. Commit interval 5 seconds EXT3 FS on sda2, internal journal EXT3-fs: mounted filesystem with writeback data mode. Enter EdgeOS image url: http://192.168.1.10/ER-e100.v1.10.11.5274249.tar % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 82.9M 100 82.9M 0 0 3156k 0 0:00:26 0:00:26 --:--:-- 3111k Unpacking EdgeOS release image Verifying EdgeOS kernel Copying EdgeOS kernel to boot partition Verifying EdgeOS system image Copying EdgeOS system image to root partition Copying version file to the root partition Creating EdgeOS writable data directory Cleaning up Installation finished Please reboot your router
11. Reboot the ERL
Now you are ready to reboot the EdgeRouter Lite. Issue the command reboot
, and you should eventually see the familiar prompt.
ERL boot messages
EMRK>reboot starting pid 295, tty '': '/bin/umount -a -r' The system is going down NOW! Sent SIGTERM to all processes Sent SIGKILL to all processes Requesting system reboot Restarting system. Looking for valid bootloader image.... Jumping to start of image at address 0xbfc80000 U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22) BIST check passed. UBNT_E100 r1:2, r2:18, f:4/71, serial #: 44D9E795A388 MPR 13-00318-18 Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate) DRAM: 512 MB Clearing DRAM....... done Flash: 4 MB Net: octeth0, octeth1, octeth2 USB: (port 0) scanning bus for devices... 1 USB Devices found scanning bus for storage devices... Device 0: Vendor: Prod.: USB DISK 2.0 Rev: PMAP Type: Removable Hard Disk Capacity: 3824.0 MB = 3.7 GB (7831552 x 512) 0 reading vmlinux.64 ............................. 5787776 bytes read argv[2]: coremask=0x3 argv[3]: root=/dev/sda2 argv[4]: rootdelay=15 argv[5]: rw argv[6]: rootsqimg=squashfs.img argv[7]: rootsqwdir=w argv[8]: mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) ELF file is 64 bit Allocating memory for mapped kernel segment, alignment: 0x400000 Allocated memory for ELF segment: addr: 0x400000, size 0x696350 Processing PHDR 0 Loading 583a80 bytes at 400000 Clearing 1128d0 bytes at 983a80 ## Loading Linux kernel with entry point: 0x007eb7d0 ... Bootloader: Done loading app on coremask: 0x3 Linux version 3.10.107-UBNT (root@aed5f1a708f2) (gcc version 4.7.0 (Cavium Inc. Version: SDK_BUILD build 51) ) #1 SMP Fri Feb 21 09:09:08 UTC 2020 CVMSEG size: 2 cache lines (256 bytes) Cavium Inc. SDK-3.1.2 bootconsole [early0] enabled CPU revision is: 000d0601 (Cavium Octeon+) Checking for the multiply/shift bug... no. Checking for the daddiu bug... no. Determined physical RAM map: memory: 0000000000543000 @ 0000000000400000 (kernel data and code) memory: 000000000004d000 @ 0000000000943000 (usable after init) memory: 0000000000107000 @ 0000000000990000 (kernel data and code) memory: 0000000007400000 @ 0000000000d00000 (usable) memory: 0000000007c00000 @ 0000000008300000 (usable) memory: 000000000fc00000 @ 0000000410300000 (usable) software IO TLB [mem 0x0170d000-0x0174d000] (0MB) mapped at [800000000170d000-800000000174cfff] Zone ranges: DMA32 [mem 0x00400000-0xefffffff] Normal [mem 0xf0000000-0x41fefffff] Movable zone start for each node Early memory node ranges node 0: [mem 0x00400000-0x00a96fff] node 0: [mem 0x00d00000-0x080fffff] node 0: [mem 0x08300000-0x0fefffff] node 0: [mem 0x410300000-0x41fefffff] Primary instruction cache 32kB, virtually tagged, 4 way, 64 sets, linesize 128 bytes. Primary data cache 16kB, 64-way, 2 sets, linesize 128 bytes. Secondary unified cache 128kB, 8-way, 128 sets, linesize 128 bytes. PERCPU: Embedded 10 pages/cpu @800000000178a000 s10880 r8192 d21888 u40960 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 125893 Kernel command line: bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) console=ttyS0,115200 PID hash table entries: 2048 (order: 2, 16384 bytes) Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) Inode-cache hash table entries: 32768 (order: 6, 262144 bytes) Memory: 495244k/504116k available (4057k kernel code, 8872k reserved, 1330k data, 308k init, 0k highmem) SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 Hierarchical RCU implementation. Additional per-CPU info printed with stalls. NR_IRQS:511 Calibrating delay loop (skipped) preset value.. 1000.00 BogoMIPS (lpj=5000000) pid_max: default: 32768 minimum: 501 Security Framework initialized Mount-cache hash table entries: 256 Checking for the daddi bug... no. SMP: Booting CPU01 (CoreId 1)... CPU revision is: 000d0601 (Cavium Octeon+) Brought up 2 CPUs NET: Registered protocol family 16 bio: create slabat 0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switching to clocksource OCTEON_CVMCOUNT NET: Registered protocol family 2 TCP established hash table entries: 4096 (order: 4, 65536 bytes) TCP bind hash table entries: 4096 (order: 4, 65536 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP: reno registered UDP hash table entries: 256 (order: 1, 8192 bytes) UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) NET: Registered protocol family 1 octeon_pci_console: Console not created. HugeTLB registered 2 MB page size, pre-allocated 0 pages squashfs: version 4.0 (2009/01/31) Phillip Lougher Registering unionfs 2.5.13 (for 3.10.34) aufs 3.10.x-20141215 msgmni has been set to 967 io scheduler noop registered io scheduler cfq registered (default) Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled 1180000000800.serial: ttyS0 at MMIO 0x1180000000800 (irq = 34) is a OCTEON console [ttyS0] enabled, bootconsole disabled console [ttyS0] enabled, bootconsole disabled 1180000000c00.serial: ttyS1 at MMIO 0x1180000000c00 (irq = 35) is a OCTEON loop: module loaded ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver OcteonUSB 16f0010000000.usbc: Octeon Host Controller OcteonUSB 16f0010000000.usbc: new USB bus registered, assigned bus number 1 OcteonUSB 16f0010000000.usbc: irq 56, io mem 0x00000000 hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected OcteonUSB: Registered HCD for port 0 on irq 56 usbcore: registered new interface driver usb-storage octeon_wdt: Initial granularity 5 Sec TCP: cubic registered NET: Registered protocol family 17 NET: Registered protocol family 15 Bootbus flash: Setting flash for 4MB flash at 0x1f800000 phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0000a7 Amd/Fujitsu Extended Query Table at 0x0040 Amd/Fujitsu Extended Query version 1.1. phys_mapped_flash: Swapping erase regions for top-boot CFI table. number of CFI chips: 1 3 cmdlinepart partitions found on MTD device phys_mapped_flash Creating 3 MTD partitions on "phys_mapped_flash": 0x000000000000-0x000000080000 : "boot0" 0x000000080000-0x000000100000 : "boot1" 0x000000100000-0x000000110000 : "eeprom" Waiting 15sec before mounting root device... usb 1-1: new high-speed USB device number 2 using OcteonUSB usb-storage 1-1:1.0: USB Mass Storage device detected scsi0 : usb-storage 1-1:1.0 scsi 0:0:0:0: Direct-Access USB DISK 2.0 PMAP PQ: 0 ANSI: 6 sd 0:0:0:0: [sda] 7831552 512-byte logical blocks: (4.00 GB/3.73 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sd 0:0:0:0: [sda] No Caching mode page found sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Attached SCSI removable disk kjournald starting. Commit interval 3 seconds EXT3-fs (sda2): using internal journal EXT3-fs (sda2): mounted filesystem with journal data mode VFS: Mounted root (aufs filesystem) on device 0:11. Freeing unused kernel memory: 308K (ffffffffc0543000 - ffffffffc0590000) Algorithmics/MIPS FPU Emulator v1.5 INIT: version 2.88 booting 2043+0 records in 2043+0 records out INIT: Entering runlevel: 2 [....] Starting SSH recovery service in the background ssh-recovery: starting... ssh-recovery: if=(all) port=(60257) terminate-timeout=(60) [ ok ] Starting daemon monitor: monit. [ ok ] Starting routing daemon: rib nsm ribd. /opt/vyatta/etc/config.boot.default created /config/config.boot created [....] Starting EdgeOS router: migrate rl-systemssh-recovery: enabling link on interfaces... ssh-recovery: eth0 :: mac=(44:d9:e7:95:a3:88) ssh-recovery: eth1 :: mac=(44:d9:e7:95:a3:89) ssh-recovery: eth2 :: mac=(44:d9:e7:95:a3:8a) ssh-recovery: service started :: pid=(1021) [ ok igure. Starting network plug daemon: netplugd. Welcome to EdgeOS ubnt ttyS0 By logging in, accessing, or using the Ubiquiti product, you acknowledge that you have read and understood the Ubiquiti License Agreement (available in the Web UI at, by default, http://192.168.1.1) and agree to be bound by its terms. ubnt login:
When you log in, you might see the following message.
ubnt login: ubnt Password: ubnt Boot image can be upgraded to version [ e102_003_eace7 ]. Run "add system boot-image" to upgrade boot image. ubnt@ubnt:~$ add system boot-image Uboot version [UNKNOWN] is about to be replaced Warning: Don't turn off the power or reboot during the upgrade! Are you sure you want to replace old version? (Yes/No) [Yes]: Yes Preparing to upgrade...Done Copying upgrade boot image...Done Checking boot version: Current is UNKNOWN; new is e102_003_eace7 ...Done Checking upgrade image...Done Writing image... Upgrade boot completed ubnt@ubnt:~$
Final thoughts
There seem to be many ways to perform the EdgeRouter Lite recovery, depending on the issue. As I was writing this, I found this post, and it seems to be the same SQUASHFS
issue that I encountered.
Another one I found was this post. According to the author, it is much simpler to use than the TFTP-based method, as demonstrated here. Though, users reported some issues.
If I ever see this again on this same hardware (I have two ERLs), then I will replace the USB flash drive to a Sandisk Cruzer Fit or Kingston Data Traveler and use this guide since I know it worked flawlessly.
You might like to read
How to configure EdgeRouter Lite via CLI – Part 1
How to configure EdgeRouter Lite via CLI – Part 2
Hardening EdgeRouter Lite – Part 1
Hardening EdgeRouter Lite – Part 2
Disclosure
AndrewRoderos.com is a participant of a few referral programs, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to company websites.