DDT: There are resets and then there are resets...

Our inaugural DDT story comes from an embedded systems engineer at FPGA vendor X, proving that hey, even FPGA manufacturers have their fair share of design woes.

"It was a windy, somewhat cloudy day in San Jose, at a time when embedded processor blocks in FPGAs were just starting to become popular.

I was tinkering with an ML300 board, which had a Virtex-II Pro FPGA device with a built-in PPC405 processor. A customer's design was loaded onto the board, and that displayed very strange memory behavior.

• random read/write errors
• software applications crashed at odd places

Sensing a memory problem, we checked the following:

• the software application C source code... hmm... nothing seemed wrong
• dumping memory at various locations and breakpoints... looked okay
• poring over the Verilog/VHDL... worked well on other designs
• simulations ran fine- probing the board hardware
• power supply... up to spec
• voltage levels on the FPGA's pins... checked out okay

Different boards showed the same symptoms so it wasn't a board-specific problem. Essentially, all the checkboxes were being ticked to no avail. But the symptoms persisted and something was obviously wrong. Frustration set in, especially since the ML300 embedded system had been well-used in the field without similar problems.

Finally someone noticed that the errors happened ONLY when the processor underwent a soft reset (hit the reset button instead of power-cycling the board). Bingo. That pointed to the reset states of the modules in the FPGA, and in particular, the processor. And it took a register-by-register comparison of the processor's state after a hard reset versus that after a soft reset to find the bug.

It was a processor firmware bug--a couple of the cache registers weren't initialized properly. On a hard reset, those registers were set to the right values, but on a soft reset... "

-- "Resetter", design engineer