Link is Up Doesn't Mean Connected: Porting Android to a Set-Top Box With No Serial Port

Out of curiosity, I spent some time getting LineageOS 23.2 (Android 16) onto an Amlogic S905X4 Android TV box I had sitting around. There is an officially supported device on the same SoC, so the kernel and most HALs come for free. In theory you swap in this box’s device tree and vendor blobs and you are done.

In practice it took twenty-three build-and-flash rounds. This post is not about the result. It is about the things I believed along the way that turned out to be wrong.

Before we dive in, let’s be clear about the spine of this. Looking back, the biggest time sink was not any single technical problem, rather, it was one specific class of signal: something that looks like success, or looks like evidence, but only proves a small segment of the path. That pattern showed up once in Wi-Fi and once in Ethernet, in almost the same shape. So the sections below are organised by how the signal fails, not by subsystem.

A blind bring-up with no UART

This box has exactly one USB Type-A port, and that port is shared with OTG. The board’s UART is not broken out.

Those two facts together mean one thing: I cannot see a kernel panic backtrace. The messages go to /dev/kmsg, but if the system never lives long enough for adb to come up, they vanish with the power. I tried ramoops, wrote a marker into /dev/kmsg and warm-rebooted, and the marker did not survive. That route was closed.

So the first milestone of the whole port was not Wi-Fi or networking, it was adb itself. Which happened to be one of the broken things, so there is a chicken-and-egg problem to get past first.

The fix came down to one line in the device tree:

&dwc2_a {
    controller-type = <1>;   /* the value that looks most wrong */
};

1 is USB_HOST_ONLY in Amlogic’s usbtype.h. Writing host-only to get device mode makes no sense until you follow the kernel command line: the bootloader passes otg_device=1, and get_otg_mode() turns that combination into USB_DEVICE_ONLY and calls force_disable_xhci_port_a() on the way. Filling in the literally sensible 3 (OTG) instead leaves the port in host mode, and adb never enumerates.

Honestly: I got this wrong once first. My initial attempt changed controller-type and the USB3 PHY’s otg property at the same time, and the USB keyboard plugged into the box stopped working too. Splitting them apart showed that otg = 0 prevents the role-switch interrupt from being registered at all, which kills host and device together.

The debug channel is the first milestone, not a by-product. That sounds obvious written down. I only actually accepted it after a few rounds of working without one.

The source of truth is in the shipped image, not the tree

When you port a device you keep asking the same shape of question: what does the retail firmware actually configure here?

The instinctive move is to go read the source tree. That gets you burned, because the same filename often exists several times in the tree and only one copy ships.

The worst case I hit had one config file in three places: once under the SoC product directory, once under a shared vendor directory, once under a different product line. The SoC product copy looked like the obvious one, and its contents even explained the symptom I was chasing. Editing it changed nothing. Extracting the shipped image showed that the copy which actually lands is the shared vendor one, and that the SoC product copy has an install rule only in the other product line.

I settled into this flow:

# extract the retail firmware's image (EROFS or ext4, both fine)
fsck.erofs --extract=/tmp/vt vendor.img

# read the copy that actually ships
cat /tmp/vt/etc/<config>

# sha256 it back against the tree to find which file to edit
sha256sum /tmp/vt/etc/<config> <tree>/.../<config>

Once the hashes match there is nothing left to guess. vendor_dlkm.img extracts the same way, and its lib/modules/modules.load is the authoritative answer to “which kernel modules does the retail build actually auto-load”. I wasted time on that one: I had reasoned backwards from a single chown line in an init rc to “the retail build probably does not auto-load the Wi-Fi driver”, written it into my notes, then later downgraded it to unverified. Extracting the image took three minutes, and it turned out the original guess happened to be right, plus one extra finding (the glue module is not loaded either). Three hours of reasoning is still reasoning.

One more trap, unglamorous but expensive: my scratchpad had a DTS whose filename ended in _cur, where cur meant current. It was the current of one particular extraction. I diffed against it and concluded the PCIe reset GPIO was still wrong, when the source DTS had been fixed long before. Since then I re-extract before touching anything.

When a tool answers “no” and “I don’t know” the same way

I hit the same shape of bug four times in this port, which earns it its own section.

First time: I wanted to know whether a few .so files were present on the device, so in a non-root adb shell:

for l in libcrypto.so libnl.so libxml2.so; do
    [ -f /vendor/lib/$l ] && echo "HAVE $l" || echo "MISS $l"
done

All three came back MISS. All three are on the device. /vendor/lib is listable, but the files in it cannot be stat’d by the shell user, and [ -f ] is a stat(2): a failed stat and a missing file are the same answer in a shell, and it is completely silent. I concluded I needed to pull three libraries out of the shipped image, and built a whole plan on that false negative.

ls -l is the reliable one, because its error message distinguishes the two cases:

ls: /vendor/lib/libcrypto.so: Permission denied        ← present, just not stat-able
ls: /vendor/lib/nosuchfile.so: No such file or directory

With a useful side effect: the wall of Permission denied that ls /vendor/lib/*.so produces in a non-root shell is itself the complete file listing. The glob is expanded by the shell, which only needs r-x on the directory, and only the stat fails.

Second time: after a flash the device showed up in neither adb devices nor fastboot devices for five straight minutes. I was about to record that the change under test had broken boot, and then:

adb kill-server && adb start-server && adb devices
#  <serial>   device        ← it had been fine the whole time

A wedged adb server and a genuinely absent device report the same empty list. This one is especially good at causing a cascade, because I was in the middle of verifying a fix for a boot failure.

The third and fourth times were filters I added myself. Once I checked whether a module had made it into the generated load list with grep -c "^dhdpci.ko$" and got 0, when the list depmod generates uses bare module names with no .ko suffix, so that pattern can never match. The other was dumber: reading a long stretch of kernel log, I reflexively appended | tail -8, and the lines I needed happened to sit above line 9.

There is a variant that does not answer “no” so much as answer a different question. Checking whether pins had been muxed:

adb shell cat /sys/kernel/debug/pinctrl/*/pinmux-pins
# pin 70 (GPIOZ_0): UNCLAIMED

UNCLAIMED is about whether a driver has claimed ownership, not about what the register currently holds. Pins configured by the bootloader also read UNCLAIMED here. So it can prove “my driver did not get this pin”, and it cannot prove “the hardware is unconfigured”.

The shared shape: the tool uses one answer for “no” and for “I cannot tell”. When the answer is “no” and the consequences are large, ask whether the tool is even able to say “I could not check”. If it cannot, pick one that distinguishes them. And if the filter is one you added yourself, take it off and look at the whole thing first.

Which segment does that success signal cover?

This is the most expensive lesson in the post, and I paid for it twice.

Wi-Fi first. It is a Broadcom PCIe part (BCM43752A2, the dhdpci driver), and the failure comes in six layers where the earlier ones hide the later ones:

Every layer you fix changes the error message, which feels like progress. However, the first five are only blocking the view. Two fingerprints here are worth memorising, because they feel identical to a user (Wi-Fi will not turn on) and have completely different fixes: mWifiHal is null means the framework cannot even find something to talk to, while Failed or timed out awaiting driver ready means the HAL is running and waiting for wlan0 before it times out. The second is the correct behaviour for a missing driver, not a new bug.

Layer six wedges here:

wlc_ver_major 12, wlc_ver_minor 1
dhd_msgbuf_dump_iovar_name: resumed on timeout rxcnt_timeout 1 ioctl_cmd 262 \
    trans_id 65238 state 3 busstate=2 ioctl_received=0
dhd_bus_start failed to sync with dongle
Firmware:                              ← version string is empty

Look at the first line. The firmware version reads back, so the chip is alive, the bus works, PCIe is fine. I took that as “the plumbing is done, what’s left is a configuration problem up the stack”, and went off to suspect a driver/firmware version skew. I reverted the driver to the exact retail version and retested: identical failure, down to the same trans_id. Then I suspected different compile flags, and compared string fingerprints between the two .ko files: identical. Then the DTB’s PCIe node, field by field: identical.

The measurement that actually mattered is one line:

adb shell grep dhdpcie /proc/interrupts
# 70:  0 0 0 0  meson-gpio-irqchip 57 Edge   dhdpcie_host_wake
# 71:  0 0 0 0  GIC-0 175 Level              dhdpcie:0000:01:00.0

Zero counts on all four CPUs. Meanwhile the driver’s own log says intstatus=0x1, so the chip has an interrupt pending. The chip asserts, and the CPU never sees it.

That version string reads back because the driver has a fallback on a few paths: dhd_msgbuf_wait_ioctl_cmplt: reschedule dhd_dpc, cnt=1, intstatus=0x1 it reschedules its own DPC and polls the completion out. An ordinary IOCTL has no such fallback. The success I was reading came from a path with a polling escape hatch, and I treated it as proof that the whole interrupt path was healthy.

So why not switch to MSI? Because MSI is disabled on purpose, at two levels:

// dhdpcie_chip_support_msi(), in bcmdhd
if (... si_chipid(bus->sih) == BCM43752_CHIP_ID ...) return FALSE;
//   comment says: intstatus is not getting cleared from these firmwares

// dhd_config.c, CUSTOMER_HW_AMLOGIC + BCMPCIE
conf->d2h_intr_method = 0;      // force INTx for all PCIe

Broadcom disables it for this chip, and Amlogic forces INTx for the whole PCIe path. So either you fix INTx delivery, or you stop depending on interrupts arriving. bcmdhd ships a polling mode for exactly that case, and it needs both halves to do anything:

# driver Makefile
DHDCFLAGS += -DDHD_READ_INTSTATUS_IN_DPC

# the chip's config file
dhd_poll=1
dhd_watchdog_ms=10

Setting dhd_poll=1 on its own does nothing. It sets bus->poll, but the code that actually schedules the DPC sits inside #ifdef DHD_READ_INTSTATUS_IN_DPC, and that macro is undefined by default. A parameter that is read and then acted on by nothing: a very BSP kind of combination.

After flashing, scans come back with a dozen-odd APs on both bands, and /proc/interrupts still reads zero. That’s what really settles it: Wi-Fi works entirely on polling, the interrupt genuinely never arrives, and there is no room left to argue about the diagnosis. The cost is a DPC every 10 ms, with boot load average spiking to 6 to 8 before it settles.

Then Ethernet taught me the same lesson again, and this time I had the precedent and still walked into it. After adding the external PHY node:

meson8b-dwmac eth0: PHY [0.0:00] driver [RTL8211FD-VD-CG] (irq=73)
meson8b-dwmac eth0: configuring for phy/rgmii link mode
meson8b-dwmac eth0: Link is Up - 1Gbps/Full - flow control rx/tx

That reads like the end of the story. Link is Up, gigabit, full duplex, flow control negotiated. Not one packet was getting in.

The problem is which segment each of those signals covers. MDIO is two dedicated management pins, and the RGMII data path is fourteen others. Laying out what each signal actually proves:

Only the last row has diagnostic value. A timing problem (RGMII delay mistuned) produces bad packets, so errors would climb. errors at zero means not even corrupt frames are arriving, so this is not timing, it is a path that was never connected. It took me two rounds of tuning delays to accept that.

Silent overwrites

Continuing from above. RGMII was not connected, and I had already copied the pinctrl setup straight from the retail DTB. The problem is that the copy was a no-op, with no warning at all.

Linux’s pinctrl core auto-applies exactly one state at probe time, and it finds it by the literal string "default":

/* copied verbatim from the retail DTB */
pinctrl-0 = <&external_eth_rmii_pins>;
pinctrl-names = "external_eth_rmii_pins";   /* not "default", so nobody applies it */

If the name is not default, the driver core leaves it alone, and does not complain. The pins stay in whatever state the bootloader left them, pinmux-pins reports UNCLAIMED, and from the previous section we already know UNCLAIMED tells us nothing about register contents. Two blind spots stacked neatly on top of each other.

Renaming it muxed all fourteen pins. RX was still zero.

The second overwrite is better hidden, and it lives in the driver. Amlogic’s dwmac-meson8b sets the EXT_PHY_MODE field of PRG_ETH0 according to phy-mode, so I changed phy-mode from "rmii" to "rgmii" and confirmed it landed in the DTB. However, the same driver has this further down:

/* aml_cust_setting(): if the DTB carries mc_val, the whole register is overwritten */
if (!of_property_read_u32(np, "mc_val", &mc_val))
        writel(mc_val, priv->ioaddr + PRG_ETH0);

The retail DTB’s mc_val is 0x4BE0C, which decodes to a complete RMII configuration: mode = 4, RGMII TX clock off, RMII inversion on. It runs after the phy-mode path, so my previous round’s fix was quietly undone.

That value had been sitting in my property diff all along, in the column for “present only on the retail side”. I waved it through as a harmless extra and spent two more rounds getting back to it. Deleting it brought the link up: ping -s 1472 -c 50, fifty for fifty, zero errors.

The driver had been telling me the whole time, too:

[2 T226] aml_cust_setting

When that line is present, the overwrite happened; delete mc_val and it disappears. A free confirmation signal, as long as you know what it means.

What both overwrites have in common is that neither reports an error. The only way to see them is to go read the consuming code: who reads this property, when, and what they do with it. I have since revised my “copy the reference DTB” rule to be a good starting point, but only up to the point where I can read the consuming code or measure the hardware. Three of the things I copied on this one were actively wrong: phy-mode, the pinctrl layout, and the PHY’s reset GPIO. The test is not “what does the retail build write”, rather, it is “how does the driver in our tree read this value”.

Making an experiment trustworthy

Every misjudgement above traces back to the same root: I accepted a measurement that was not clean. So let’s finish with how to make the experiment itself hold up.

One variable per build. In one round I added the wifi HAL, put the driver module back into boot-time auto-load, and added three BoardConfig variables, all at once. That round kernel-panicked and the device parked in fastboot. Any of the three could have caused it, I had no way to tell, and those three hours were a write-off. The next round changed the driver version alone and produced a clean conclusion immediately.

The control group showing the same symptom is not evidence. I had accused a driver module of causing a first-boot failure, then found that a build with no firmware for it at all failed the same way. When the symptom appears in the control group, it has nothing to do with your change.

Confirm the experiment actually produced the input you think it did. This is the one I am least proud of. I ran a sweep over RGMII delay combinations and concluded that delay had no effect. However, ping had no route at the time and never emitted a single packet, the TX counter never moved, and I did not notice. An experiment with no input produces no output, so that conclusion had no evidence behind it, and I carried it as established fact for several rounds.

Shorten the loop. Changing the DTB means a rebuild, seventy-five minutes a round. Amlogic’s BSP, though, exposes sysfs for reading and writing MAC and PHY registers directly:

# enable root on an eng build first
adb shell setprop persist.sys.root_access 3

adb shell 'echo r 0 > /sys/class/ethernet/macreg; cat /sys/class/ethernet/macreg'
adb shell 'echo r 0 > /sys/class/ethernet/phyreg; cat /sys/class/ethernet/phyreg'

Seventy-five minutes becomes seconds. Progress after finding that was an order of magnitude faster.

“Boots for fifteen seconds then drops to fastboot” should make you think kernel panic, not bad flash. The panic triggers a watchdog reset, sys.boot_completed is never reached, markBootSuccessful() never runs, the bootloader’s retry counter runs down, and the device parks in fastboot. The surface symptom is nearly identical to a botched flash. Telling them apart is one line:

adb logcat -d -s bootstat | grep "boot reason"
# Canonical boot reason: watchdog      ← panic, not a flash problem

Here is everything that got falsified. This table is the most useful part of the post, because every row is something that felt entirely reasonable at the time:

What I’m building next

After all of the above, the real output is not just a box that boots. Writing this up meant asking, section by section, “so what is the correct test here”, and that list turns out to be the spec for three tools:

1. A probe library. Wrapped measurement helpers whose central rule is to refuse to answer “absent” when they cannot distinguish absent from unreadable; check adb server health before querying it; assert that the input exists before each experiment runs.
2. A post-flash verification suite. A fixed set of probes, run and diffed against the previous build. That turns “did my change help” from a judgement call into a diff.
3. A consumer audit checklist. For every property you set or copy, grep for who consumes it and how, first. Had I been doing this earlier, the five networking rounds would likely have compressed into two.

Automating the loop is deliberately last. The biggest time sink here was not missing automation, it was accepting vague measurements, and speeding up a loop that generates wrong conclusions only generates them faster. Guardrails come before throttle.

If you are starting a similar port, the two highest-value habits are to treat the debug channel as the first milestone rather than a by-product, and to keep one line of notes per build recording what changed and what you expect to see. I skipped the second, and reconstructing which of twenty-three rounds carried which change afterwards cost far more than writing that line would have.