PCIe FPGA Data Aquisition Board#
The topic in this notebook is the analysis of a high-speed data acquisition PCIe board based around a large FPGA (Field Programmable Gate Array). A number of high-speed ADCs, each with its own power delivery system, is connected to the FPGA with JESD204B links. Data is pre-processed in the FPGA before being sent across the PCIe bus. The board can only draw power from the PCIe connector (no extra ATX connectors), and the task at hand is to determine how many ADC channels can be added to the board within the power limits of the PCIe specification (75W).
from sysloss.components import *
from sysloss.utils import *
from sysloss.system import System
import pandas as pd
System definition#
Each ADC channel will use the following resources:
Two JESD204B lanes to the FPGA
About 7% of FPGA logic for pre-processing
Four power rails: 1.8V and 3x 1.2V (converted from 12V):
The FPGA has 32 transceivers, 8 are used for PCIe while the rest are dedicated to JESD204 lanes. This sets the upper limit for the number of ADC channels to 12. The FPGA itself has 4 power rails: 1.8V (AUX), 1.2V (AVTT), 0.9V (AVCC) and 0.85V (VCCINT). Except for the high-speed transceivers, there are practically no I/O used, so I/O bank power is left out of the analysis.
Power consumption of the ADC is collected from the data sheet, and FPGA power is estimated using the FPGA power tools. FPGA power consumption is summarized in the table below:
Voltage |
ADC (per channel) Power (W) |
System control & PCIe Power (W) |
Static FPGA power (W) |
|---|---|---|---|
VCCINT |
1.58 |
1.254 |
1.67 |
AVCC |
0.051 |
0.45 |
0.57 |
AVTT |
0.22 |
0.76 |
0.031 |
VAUX |
1.35 |
FPGA and ADC’s will be powered from the 12V input, while board monitoring and test signal generators will be powered from the 3.3V input.
Tip
Use limits on components like Converters (input voltage range, output current) and LinRegs (output voltage and current) to get warnings if component voltages or currents are out of spec.
Buck converter efficiency is defined as interpolation data. The subsystems are defined in functions for easy manipulation of key system parameters and system architecture.
Tip
When a PCB design involves high currents, the PCB trace resistance should be taken into account. sysLoss has a function trace_res() for calculating the PCB trace resistance in the utils package.
eff_2v3 = {"vi":[12], "io":[1e-3, 1, 2, 3, 4, 5], "eff":[[0.45, 0.95, 0.94, 0.925, 0.9, 0.88]]}
eff_1v7 = {"vi":[12], "io":[1e-3, 1, 2, 3, 4, 5], "eff":[[0.33, 0.92, 0.92, 0.91, 0.9, 0.875]]}
eff_0v85 = {"vi":[12], "io":[1, 2, 5, 10, 20, 30], "eff":[[0.48, 0.7, 0.86, 0.9, 0.87, 0.84]]}
eff_1v8 = {"vi":[12], "io":[.1, .25, .5, 1.0, 1.5, 2.0], "eff":[[0.63, 0.82, 0.86, 0.9, 0.9, 0.885]]}
eff_0v9 = {"vi":[12], "io":[.1, .25, .5, 1.0, 1.5, 2.0], "eff":[[0.54, 0.75, 0.82, 0.85, 0.84, 0.83]]}
eff_1v2 = {"vi":[12], "io":[.01, .1, .5, 1.0, 2.0, 4.0], "eff":[[0.72, 0.78, 0.84, 0.88, 0.87, 0.8]]}
def adc_subsystem(sys, channel, src):
# assume a star connection to 12V using 8cm long, 50mils wide traces on a 1oz/ft^2 copper layer (return current on ground plane).
rs = trace_res(w1_mm=50*MILS2MM, w2_mm=50*MILS2MM, l_mm=80, t_mm=1*OZ2MM)
idx = "[{}]".format(channel+1)
sys.add_comp(src, comp=RLoss("ADC"+idx+" PCB trace", rs=rs))
sys.add_comp("ADC"+idx+" PCB trace", comp=Converter("ADC"+idx+" buck 2.3V", vo=2.3, eff=eff_2v3))
sys.add_comp("ADC"+idx+" buck 2.3V", comp=RLoss("ADC"+idx+" ferrit1", rs=0.087))
sys.add_comp("ADC"+idx+" ferrit1", comp=LinReg("ADC"+idx+" LDO 1.8V", vo=1.8, ig=0.5e-6, vdrop=0.15, limits={"vo":[1.8, 1.8]}))
sys.add_comp("ADC"+idx+" LDO 1.8V", comp=ILoad("ADC"+idx+" AVVD18", ii=0.5))
sys.add_comp("ADC"+idx+" PCB trace", comp=Converter("ADC"+idx+" buck 1.7V", vo=1.7, eff=eff_1v7))
sys.add_comp("ADC"+idx+" buck 1.7V", comp=RLoss("ADC"+idx+" ferrit2", rs=0.103))
sys.add_comp("ADC"+idx+" ferrit2", comp=LinReg("ADC"+idx+" LDO[1] 1.2V", vo=1.2, ig=0.23e-6, vdrop=0.15, limits={"vo":[1.2, 1.2]}))
sys.add_comp("ADC"+idx+" LDO[1] 1.2V", comp=ILoad("ADC"+idx+" AVVD12", ii=0.74))
sys.add_comp("ADC"+idx+" ferrit2", comp=LinReg("ADC"+idx+" LDO[2] 1.2V", vo=1.2, ig=0.23e-6, vdrop=0.15, limits={"vo":[1.2, 1.2]}))
sys.add_comp("ADC"+idx+" LDO[2] 1.2V", comp=ILoad("ADC"+idx+" CLKVDD", ii=0.086))
sys.add_comp("ADC"+idx+" ferrit2", comp=LinReg("ADC"+idx+" LDO[3] 1.2V", vo=1.2, ig=0.23e-6, vdrop=0.15, limits={"vo":[1.2, 1.2]}))
sys.add_comp("ADC"+idx+" LDO[3] 1.2V", comp=ILoad("ADC"+idx+" DVDD", ii=1.41))
return sys
def fpga_subsystem(sys, channels, src):
# assume connection to 12V using a 5cm long, 40mils wide trace on a 1oz/ft^2 copper layer (return current on ground plane).
rs = trace_res(w1_mm=40*MILS2MM, w2_mm=40*MILS2MM, l_mm=50, t_mm=1*OZ2MM)
sys.add_comp(src, comp=RLoss("FPGA PCB trace", rs=rs))
sys.add_comp("FPGA PCB trace", comp=Converter("FPGA VCCINT", vo=0.85, eff=eff_0v85, limits={"io":[0.0, 30.0]}))
sys.add_comp("FPGA VCCINT", comp=PLoad("FPGA INT static", pwr=1.67))
sys.add_comp("FPGA VCCINT", comp=PLoad("FPGA INT dynamic", pwr=1.25+channels*1.58))
# VCCAUX
sys.add_comp("FPGA PCB trace", comp=Converter("FPGA VCCAUX", vo=1.8, eff=eff_1v8, limits={"io":[0.0, 2.0]}))
sys.add_comp("FPGA VCCAUX", comp=PLoad("FPGA AUX static", pwr=1.35))
# AVCC
sys.add_comp("FPGA PCB trace", comp=Converter("FPGA AVCC", vo=0.9, eff=eff_0v9, limits={"io":[0.0, 2.0]}))
sys.add_comp("FPGA AVCC", comp=PLoad("FPGA AVCC static", pwr=0.57))
sys.add_comp("FPGA AVCC", comp=PLoad("FPGA AVCC dynamic", pwr=0.45+channels*0.051))
# AVTT
sys.add_comp("FPGA PCB trace", comp=Converter("FPGA AVTT", vo=1.2, eff=eff_1v2, limits={"io":[0.0, 4.0]}))
sys.add_comp("FPGA AVTT", comp=PLoad("FPGA AVTT static", pwr=0.031))
sys.add_comp("FPGA AVTT", comp=PLoad("FPGA AVTT dynamic", pwr=0.76+channels*0.22))
sys.add_comp("FPGA PCB trace", comp=ILoad("Board fan", ii=0.06))
return sys
def PCIe_system(channels):
# power inputs 12V and 3.3V with current limits set to PCIe spec.
sys = System("PCIe FPGA board", source=Source("12V", vo=12.0, limits={"io":[0.0, 5.5]}))
sys.add_source(Source("3.3V", vo=3.3, limits={"io":[0.0, 3.0]}))
# 3.3V subsystem
sys.add_comp("3.3V", comp=Converter("Buck 2.5V", vo=2.5, eff=eff_2v3))
sys.add_comp("Buck 2.5V", comp=PLoad("Board monitor", pwr=0.35))
sys.add_comp("Buck 2.5V", comp=PLoad("Signal generators", pwr=1.55))
# FPGA subsystem
sys = fpga_subsystem(sys, channels, "12V")
# ADC channels
for i in range(channels):
sys = adc_subsystem(sys, i, "12V")
return sys
Analysis#
We start by looking at the power tree and power consumption for a one channel board:
sys = PCIe_system(1)
sys.tree()
PCIe FPGA board
├── 3.3V
│ └── Buck 2.5V
│ ├── Signal generators
│ └── Board monitor
└── 12V
├── ADC[1] PCB trace
│ ├── ADC[1] buck 1.7V
│ │ └── ADC[1] ferrit2
│ │ ├── ADC[1] LDO[3] 1.2V
│ │ │ └── ADC[1] DVDD
│ │ ├── ADC[1] LDO[2] 1.2V
│ │ │ └── ADC[1] CLKVDD
│ │ └── ADC[1] LDO[1] 1.2V
│ │ └── ADC[1] AVVD12
│ └── ADC[1] buck 2.3V
│ └── ADC[1] ferrit1
│ └── ADC[1] LDO 1.8V
│ └── ADC[1] AVVD18
└── FPGA PCB trace
├── Board fan
├── FPGA AVTT
│ ├── FPGA AVTT dynamic
│ └── FPGA AVTT static
├── FPGA AVCC
│ ├── FPGA AVCC dynamic
│ └── FPGA AVCC static
├── FPGA VCCAUX
│ └── FPGA AUX static
└── FPGA VCCINT
├── FPGA INT dynamic
└── FPGA INT static
sys.solve()
| Component | Type | Parent | Domain | Vin (V) | Vout (V) | Iin (A) | Iout (A) | Power (W) | Loss (W) | Efficiency (%) | Warnings | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 3.3V | SOURCE | 3.3V | 3.3 | 3.3 | 0.693784 | 0.693784 | 2.289488 | 0.0 | 100.0 | ||
| 1 | Buck 2.5V | CONVERTER | 3.3V | 3.3V | 3.3 | 2.5 | 0.693784 | 0.76 | 2.289488 | 0.389488 | 82.987988 | |
| 2 | Signal generators | LOAD | Buck 2.5V | 3.3V | 2.5 | 0.0 | 0.62 | 0.0 | 1.55 | 0.0 | 100.0 | |
| 3 | Board monitor | LOAD | Buck 2.5V | 3.3V | 2.5 | 0.0 | 0.14 | 0.0 | 0.35 | 0.0 | 100.0 | |
| 4 | 12V | SOURCE | 12V | 12.0 | 12.0 | 1.309342 | 1.309342 | 15.712108 | 0.0 | 100.0 | ||
| 5 | ADC[1] PCB trace | SLOSS | 12V | 12V | 12.0 | 11.984934 | 0.482757 | 0.482757 | 5.793085 | 0.007273 | 99.87445 | |
| 6 | ADC[1] buck 1.7V | CONVERTER | ADC[1] PCB trace | 12V | 11.984934 | 1.7 | 0.345631 | 2.236001 | 4.142367 | 0.341165 | 91.763999 | |
| 7 | ADC[1] ferrit2 | SLOSS | ADC[1] buck 1.7V | 12V | 1.7 | 1.469692 | 2.236001 | 2.236001 | 3.801201 | 0.514969 | 86.452466 | |
| 8 | ADC[1] LDO[3] 1.2V | LINREG | ADC[1] ferrit2 | 12V | 1.469692 | 1.2 | 1.41 | 1.41 | 2.072266 | 0.380266 | 81.649751 | |
| 9 | ADC[1] DVDD | LOAD | ADC[1] LDO[3] 1.2V | 12V | 1.2 | 0.0 | 1.41 | 0.0 | 1.692 | 0.0 | 100.0 | |
| 10 | ADC[1] LDO[2] 1.2V | LINREG | ADC[1] ferrit2 | 12V | 1.469692 | 1.2 | 0.086 | 0.086 | 0.126394 | 0.023194 | 81.649546 | |
| 11 | ADC[1] CLKVDD | LOAD | ADC[1] LDO[2] 1.2V | 12V | 1.2 | 0.0 | 0.086 | 0.0 | 0.1032 | 0.0 | 100.0 | |
| 12 | ADC[1] LDO[1] 1.2V | LINREG | ADC[1] ferrit2 | 12V | 1.469692 | 1.2 | 0.74 | 0.74 | 1.087572 | 0.199572 | 81.649739 | |
| 13 | ADC[1] AVVD12 | LOAD | ADC[1] LDO[1] 1.2V | 12V | 1.2 | 0.0 | 0.74 | 0.0 | 0.888 | 0.0 | 100.0 | |
| 14 | ADC[1] buck 2.3V | CONVERTER | ADC[1] PCB trace | 12V | 11.984934 | 2.3 | 0.137126 | 0.5 | 1.643446 | 0.493445 | 69.975 | |
| 15 | ADC[1] ferrit1 | SLOSS | ADC[1] buck 2.3V | 12V | 2.3 | 2.2565 | 0.5 | 0.5 | 1.150001 | 0.02175 | 98.108694 | |
| 16 | ADC[1] LDO 1.8V | LINREG | ADC[1] ferrit1 | 12V | 2.2565 | 1.8 | 0.5 | 0.5 | 1.128251 | 0.228251 | 79.769476 | |
| 17 | ADC[1] AVVD18 | LOAD | ADC[1] LDO 1.8V | 12V | 1.8 | 0.0 | 0.5 | 0.0 | 0.9 | 0.0 | 100.0 | |
| 18 | FPGA PCB trace | SLOSS | 12V | 12V | 12.0 | 11.979847 | 0.826586 | 0.826586 | 9.919032 | 0.016658 | 99.832055 | |
| 19 | Board fan | LOAD | FPGA PCB trace | 12V | 11.979847 | 0.0 | 0.06 | 0.0 | 0.718791 | 0.0 | 100.0 | |
| 20 | FPGA AVTT | CONVERTER | FPGA PCB trace | 12V | 11.979847 | 1.2 | 0.097293 | 0.8425 | 1.165552 | 0.154552 | 86.74 | |
| 21 | FPGA AVTT dynamic | LOAD | FPGA AVTT | 12V | 1.2 | 0.0 | 0.816667 | 0.0 | 0.98 | 0.0 | 100.0 | |
| 22 | FPGA AVTT static | LOAD | FPGA AVTT | 12V | 1.2 | 0.0 | 0.025833 | 0.0 | 0.031 | 0.0 | 100.0 | |
| 23 | FPGA AVCC | CONVERTER | FPGA PCB trace | 12V | 11.979847 | 0.9 | 0.105649 | 1.19 | 1.265658 | 0.194658 | 84.62 | |
| 24 | FPGA AVCC dynamic | LOAD | FPGA AVCC | 12V | 0.9 | 0.0 | 0.556667 | 0.0 | 0.501 | 0.0 | 100.0 | |
| 25 | FPGA AVCC static | LOAD | FPGA AVCC | 12V | 0.9 | 0.0 | 0.633333 | 0.0 | 0.57 | 0.0 | 100.0 | |
| 26 | FPGA VCCAUX | CONVERTER | FPGA PCB trace | 12V | 11.979847 | 1.8 | 0.128056 | 0.75 | 1.534091 | 0.184091 | 88.0 | |
| 27 | FPGA AUX static | LOAD | FPGA VCCAUX | 12V | 1.8 | 0.0 | 0.75 | 0.0 | 1.35 | 0.0 | 100.0 | |
| 28 | FPGA VCCINT | CONVERTER | FPGA PCB trace | 12V | 11.979847 | 0.85 | 0.435588 | 5.294118 | 5.218281 | 0.718281 | 86.235294 | |
| 29 | FPGA INT dynamic | LOAD | FPGA VCCINT | 12V | 0.85 | 0.0 | 3.329412 | 0.0 | 2.83 | 0.0 | 100.0 | |
| 30 | FPGA INT static | LOAD | FPGA VCCINT | 12V | 0.85 | 0.0 | 1.964706 | 0.0 | 1.67 | 0.0 | 100.0 | |
| 31 | Subsystem 3.3V | 3.3 | 0.693784 | 2.289488 | 0.389488 | 82.987988 | ||||||
| 32 | Subsystem 12V | 12.0 | 1.309342 | 15.712108 | 3.478126 | 77.863401 | ||||||
| 33 | System total | 18.001596 | 3.867614 | 78.515159 |
Note
When the system has more than one voltage source, a new column Domain appears in the results table. The name of the source (voltage domain) of each component is listed here.
Next, we check power consumption for ADC count between 1 and 12:
res = []
for cnt in range(1,13):
psys = PCIe_system(channels=cnt)
res += [psys.solve(tags={"ADCs":cnt})]
df = pd.concat(res, ignore_index=True)
df[df.Component == "System total"][["Component", "ADCs", "Power (W)", "Loss (W)", "Efficiency (%)", "Warnings"]].style.hide(axis='index')
| Component | ADCs | Power (W) | Loss (W) | Efficiency (%) | Warnings |
|---|---|---|---|---|---|
| System total | 1 | 18.001596 | 3.867614 | 78.515159 | |
| System total | 2 | 25.810332 | 6.242406 | 75.814316 | |
| System total | 3 | 33.583143 | 8.581267 | 74.447696 | |
| System total | 4 | 41.413241 | 10.977422 | 73.492965 | |
| System total | 5 | 49.372805 | 13.503060 | 72.650815 | |
| System total | 6 | 57.358484 | 16.054815 | 72.009694 | |
| System total | 7 | 65.370851 | 18.633197 | 71.496168 | |
| System total | 8 | 73.421068 | 21.249490 | 71.058048 | Yes |
| System total | 9 | 81.506463 | 23.900964 | 70.675989 | Yes |
| System total | 10 | 89.623232 | 26.583817 | 70.338253 | Yes |
| System total | 11 | 97.772038 | 29.298710 | 70.033651 | Yes |
| System total | 12 | 105.953564 | 32.046327 | 69.754366 | Yes |
We see that 7 ADC channels are the most we can power. From 8 channels and up we get warnings. Even though the 8-channel case has a system total power of 73W which is less than 75W PCIe specifications, the current drawn from the 12V supply is too high. Let’s look at the 12V input current (the PCIe specification says max 5.5A):
df[df.Component == "12V"][["Component", "ADCs", "Iout (A)", "Power (W)", "Warnings"]].style.hide(axis='index')
| Component | ADCs | Iout (A) | Power (W) | Warnings |
|---|---|---|---|---|
| 12V | 1 | 1.309342 | 15.712108 | |
| 12V | 2 | 1.960070 | 23.520844 | |
| 12V | 3 | 2.607805 | 31.293655 | |
| 12V | 4 | 3.260313 | 39.123753 | |
| 12V | 5 | 3.923610 | 47.083317 | |
| 12V | 6 | 4.589083 | 55.068996 | |
| 12V | 7 | 5.256780 | 63.081363 | |
| 12V | 8 | 5.927632 | 71.131580 | io |
| 12V | 9 | 6.601415 | 79.216975 | io |
| 12V | 10 | 7.277812 | 87.333744 | io |
| 12V | 11 | 7.956879 | 95.482551 | io |
| 12V | 12 | 8.638673 | 103.664076 | io |
With 8 channels, the 12V current is 5.91A, above the 5.5A limit in the PCIe specification.
System optimization#
We can guess that the marketing department is not going to be happy to sell a 7-channel board - what can we do to get up to 8 channels? Looking at the analysis result above, the 3.3V supply is not fully utilized. So, we can try to power one extra channel from 3.3V. The ADC power circuit provides 2.3V as the highest voltage, so this will work fine. Converter efficiency will be better in fact operating from 3.3V than from 12V. The converter efficiency parameter needs an update for 3.3V input, and the system generating function assigns the first ADC channel to the 3.3V supply.
# expand ADC buck converter parameters for 3.3V input
eff_2v3 = {"vi":[3.3, 12], "io":[1e-3, 1, 2, 3, 4, 5], "eff":[[0.63, 0.955, 0.96, 0.94, 0.92, 0.9],[0.45, 0.95, 0.94, 0.925, 0.9, 0.88]]}
eff_1v7 = {"vi":[3.3, 12], "io":[1e-3, 1, 2, 3, 4, 5], "eff":[[0.59, 0.94, 0.95, 0.92, 0.91, 0.89],[0.33, 0.92, 0.92, 0.91, 0.9, 0.875]]}
# redefine system function to allocate one ADC to 3.3V supply
def PCIe_system(channels):
# power inputs 12V and 3.3V with current limits set to PCIe spec.
sys = System("PCIe FPGA board", source=Source("12V", vo=12.0, limits={"io":[0.0, 5.5]}))
sys.add_source(Source("3.3V", vo=3.3, limits={"io":[0.0, 3.0]}))
# 3.3V subsystem
sys.add_comp("3.3V", comp=Converter("Buck 2.5V", vo=2.5, eff=eff_2v3))
sys.add_comp("Buck 2.5V", comp=PLoad("Board monitor", pwr=0.35))
sys.add_comp("Buck 2.5V", comp=PLoad("Signal generators", pwr=1.55))
# FPGA subsystem
sys = fpga_subsystem(sys, channels, "12V")
# ADC channels
for ch in range(channels):
if ch == 0:
sys = adc_subsystem(sys, ch, "3.3V")
else:
sys = adc_subsystem(sys, ch, "12V")
return sys
psys2 = PCIe_system(8)
psys2.solve()
| Component | Type | Parent | Domain | Vin (V) | Vout (V) | Iin (A) | Iout (A) | Power (W) | Loss (W) | Efficiency (%) | Warnings | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 3.3V | SOURCE | 3.3V | 3.3 | 3.3 | 2.344953 | 2.344953 | 7.738344 | 0.0 | 100.0 | ||
| 1 | ADC[1] PCB trace | SLOSS | 3.3V | 3.3V | 3.3 | 3.247308 | 1.688386 | 1.688386 | 5.571675 | 0.088964 | 98.403286 | |
| 2 | ADC[1] buck 1.7V | CONVERTER | ADC[1] PCB trace | 3.3V | 3.247308 | 1.7 | 1.241431 | 2.236001 | 4.031308 | 0.230107 | 94.291998 | |
| 3 | ADC[1] ferrit2 | SLOSS | ADC[1] buck 1.7V | 3.3V | 1.7 | 1.469692 | 2.236001 | 2.236001 | 3.801201 | 0.514969 | 86.452466 | |
| 4 | ADC[1] LDO[3] 1.2V | LINREG | ADC[1] ferrit2 | 3.3V | 1.469692 | 1.2 | 1.41 | 1.41 | 2.072266 | 0.380266 | 81.649751 | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 120 | FPGA INT dynamic | LOAD | FPGA VCCINT | 12V | 0.85 | 0.0 | 16.341176 | 0.0 | 13.89 | 0.0 | 100.0 | |
| 121 | FPGA INT static | LOAD | FPGA VCCINT | 12V | 0.85 | 0.0 | 1.964706 | 0.0 | 1.67 | 0.0 | 100.0 | |
| 122 | Subsystem 3.3V | 3.3 | 2.344953 | 7.738344 | 2.255145 | 70.857529 | ||||||
| 123 | Subsystem 12V | 12.0 | 5.444726 | 65.336707 | 18.648328 | 71.458114 | ||||||
| 124 | System total | 73.075051 | 20.903473 | 71.394515 |
125 rows × 12 columns
The 8-channel case is now within spec.
Summary#
This notebook demonstrates how to define a complex system by splitting it up in subsystems and defining each subsystem separately. This enables easy exploration of different power architectures. We have also seen how key component parameters can be monitored by setting component limits.