Troubleshooting an FPGA problem

skrasms · Joined: Feb 21, 2008 Posts: 121 Location: Portland, OR

So this design I've been working on runs about 90% correctly. However, for some reason a small part of it doesn't work. I spent all of today analyzing the problem, and I narrowed it down to this:

The hardware behavior does not match the simulation.

The FPGA in this design is constantly writing data to a bus. For some reason, every other data write is missed on the actual hardware. It just doesn't happen. I have a logic analyzer hooked up to the data/address bus, and half the data is visibly absent.

So then I simulated the project. The design will run on its own without any input. As it turns out, the simulation is perfect. It has all the data writes that I want to see coming out of the part. I built the simulated code and put it on the part to double-check: half the data is gone again. This isn't as simple as unconnected pins, either. Looking at the chip select and clock signals, half the writes are not occurring.

So I'm puzzled at the moment. I've never had hardware not match simulation.

The usual reasons I've heard for simulation/hardware mismatches are:
1) Latches
2) Metastability problems with asynchronous clock domains

Just as far as common problems go, what else could potentially be wrong?

Also, does Xilinx have a simple, free FPGA logic analyzer like Altera's SignalTap? ChipScope doesn't appear to be equivalent.

jksuperstar · Posted: Tue Jun 01, 2010 7:44 am Post subject:

This most likely sounds like a timing problem. Are there any warnings in your timing report? Anything to suggest a path can't run as fast as you request?

Also, I've seen much more rare cases of instability due to power supply ripple...switching supplies can send ripple on the core Vcc, and once you add in IR drop some gates get starved in the core, and won't switch as fast, or at all!

JovianPyx · Posted: Tue Jun 01, 2010 9:23 am Post subject:

Can't offer much sage advice really. jksuperstar points out timing (check the timing report). Can the design be run at a lower clock rate just for testing? Or can it be cut down to the core logic that controls the writes just for testing?

jk also mentions warnings, sometimes I have to think about exactly what warnings are trying to tell me. And some warnings, IMO, should really be errors. There are times that I find thoroughly reading warnings leads to a solution to the problem.

I'm trying to think if I've seen a situation like yours where simulation works but the hardware won't, I can't think of one.

I've never used chipscope (I use the free WebPACK ISE). I've developed other test methods that rely on LEDs and scoping test points sent to board I/O pins.

Is this running on a known working development board? If not, jk's remark about power is poignant. If on a dev board, is there an additional power supply load that the dev board may not be designed for? If on a DIY board, are there physical factors of the board (stray capacitance or inductance) that may be problematic? If a signal is supposed to exit the FPGA and go elsewhere, are the user contraints for the I/O pins correct?

Some of these questions may seem inane, but I can't see the design code nor the physical hardware...

skrasms · Joined: Feb 21, 2008 Posts: 121 Location: Portland, OR

Thank you for the responses!

I realized this morning after I woke up that I had only been running behavioral simulations in ISIM. What about doing a post-route simulation?

Well, it turns out the behavioral and post-route simulations don't match each other! The post-route simulation does what I'm seeing in the hardware... half the writes are missing.

I did some digging around, and I think I may have found the problem. I'll post back here if my findings prove correct.

As far as the other questions... I'm running this design at 25 MHz and the Xilinx compiler says it's good to 49 MHz (so slow because I have some un-pipelined DSP). I still need to learn how to do a real timing analysis in ISE.

I built the PCB that's running on, and it has a ground plane plus a decoupling cap on every VCC pin (in addition to bulk storage ceramics). I do need to check the ripple, though.

ChipScope isn't free? I only have the ISE Webpack. I assumed it would have the same features as Quartus, but I guess that is not the case.

skrasms · Joined: Feb 21, 2008 Posts: 121 Location: Portland, OR

Solved!

It turns out that the Xilinx behavioral compiler implements case statements properly, but the Xilinx synthesizer does not.

When I say "properly," I mean as I've read in papers by Cliff Cummings et al. My understanding is that case statements imply an "if... else if... else if... else if..." structure unless the directive "parallel_case" is added. They imply a priority structure.

However, XST treats all case statements as though they are "if... end, if... end, if... end..." logic all running in parallel. There is no priority, and if two cases are true at the same time the results are unpredictable (instead of one having priority).

A simplified version of my code reads like this:

JovianPyx · Posted: Tue Jun 01, 2010 7:17 pm Post subject:

wow, that's very interesting. I see what you're trying to express...

But honestly, the if then elseif stuff is really more explicit.

That's odd/screwy/wrong that the simulator and bitfile generation actually do two different things.

Thanks, I shall try to remember this.

jksuperstar · Posted: Wed Jun 02, 2010 8:02 am Post subject:

I remembered XST having some configuration that allowed user control of how case statements were treated by default (I think it needed the advanced menus enabled to see/change it) (look for "Case Implementation Style" or something similar in the XST settings).

So, I looked at the Xiling Behavioral Verilog Coding guidlines, and found this:
http://www.xilinx.com/itp/xilinx4/data/docs/xst/verilog3.html

Search for "Case", and you'll find a whole section on priority encoders & such. I'd guess your default setting for this is set to parallel or full-parallel. If you change it back to default, your code might be that much more portable.

There's more examples here:
http://www.xilinx.com/itp/xilinx4/data/docs/xst/hdlcode9.html#1000982

JovianPyx · Posted: Wed Jun 02, 2010 3:50 pm Post subject:

Thank you for that information jksuperstar!

skrasms · Joined: Feb 21, 2008 Posts: 121 Location: Portland, OR

jksuperstar · Posted: Thu Jun 03, 2010 9:30 am Post subject:

Hmmmm, this is interesting. I wouldn't expect XST to synthesize something like this by mistake. Aside from a bug in that particular version of XST, I can only guess at either a corrupt project file, or you used another project/constraint file as an example or starting point to your project, and expanded on it. Many settings in the project file can be set through a constraint file as well.

None the less, you should let Xilinx know about it. They are typically very responsive to bugs, even if you haven't paid for the software (XST is the same in both versions). At one point, they used to have a "Mug for a Bug" promotion. If you found a bug that they could recreate/prove, they'd send you a mug for free. They don't want bugs, and do't dance around the fact if one is found. They just fix it & release another version. Deviating from the Verilog Standard seems like a big bug to me.