6 The only limiting factor on FIB scale is the amount of memory
7 allocated to each heap the FIB uses, and there are 4:
18 The IPv4 heap is used to allocate the memory needed for the
19 data-structures within which the IPv4 prefixes are stored. Each
20 table, created by the user, i.e. with;
22 .. code-block:: console
26 or the default table, comprises 2 *ip4_fib_t* objects.
27 The 'non-forwarding' *ip4_fib_t* contains all the entries in the table
28 and, the 'forwarding' contains the entries that are matched against in
29 the data-plane. The difference between the two sets are the entries
30 that should not be matched in the data-plane.
31 Each *ip4_fib_t* comprises an mtrie (for fast lookup in the data-plane)
32 and a hash table per-prefix length (for lookup in the control plane).
34 To see the amount of memory consumed by the IPv4 tables use:
36 .. code-block:: console
39 ipv4-VRF:0 mtrie:333056 hash:3523
40 ipv4-VRF:1 mtrie:333056 hash:3523
41 totals: mtrie:666112 hash:7046 all:673158
43 Mtrie Mheap Usage: total: 32.06M, used: 662.44K, free: 31.42M, trimmable: 31.09M
44 free chunks 3 free fastbin blks 0
45 max total allocated 32.06M
48 this output shows two 'empty' (i.e. no added routes) tables. Each
49 mtrie uses about 150k of memory, so each table about 300k. the total
50 heap usage statistics for the IP4 heap are shown at the end.
53 Below the output having added 1M, 2M and 4M routes respectively:
55 .. code-block:: console
58 ipv4-VRF:0 mtrie:335744 hash:4695
59 totals: mtrie:335744 hash:4695 all:340439
61 Mtrie Mheap Usage: total: 1.00G, used: 335.20K, free: 1023.74M, trimmable: 1023.72M
62 free chunks 3 free fastbin blks 0
63 max total allocated 1.00G
66 .. code-block:: console
69 ipv4-VRF:0 mtrie:5414720 hash:41177579
70 totals: mtrie:5414720 hash:41177579 all:46592299
72 Mtrie Mheap Usage: total: 1.00G, used: 46.87M, free: 977.19M, trimmable: 955.93M
73 free chunks 61 free fastbin blks 0
74 max total allocated 1.00G
77 .. code-block:: console
80 ipv4-VRF:0 mtrie:22452608 hash:168544508
81 totals: mtrie:22452608 hash:168544508 all:190997116
83 Mtrie Mheap Usage: total: 1.00G, used: 198.37M, free: 825.69M, trimmable: 748.24M
84 free chunks 219 free fastbin blks 0
85 max total allocated 1.00G
88 VPP was started with a 1G IP4 heap.
93 The IPv6 heap is used to allocate the memory needed for the
94 data-structure within which the IPv6 prefixes are stored. IPv6 also
95 has the concept of forwarding and non-forwarding entries, however for
96 IPv6 all the forwarding entries are stored in a single hash table
97 (same goes for the non-forwarding). The key to the hash table includes
100 To see the amount of memory consumed by the IPv4 tables use:
102 .. code-block:: console
105 IPv6 Non-Forwarding Hash Table:
106 Hash table ip6 FIB non-fwding table
107 7 active elements 7 active buckets
109 0 linear search buckets
110 arena: base 7f2fe28bf000, next 803c0
111 used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
113 IPv6 Forwarding Hash Table:
114 Hash table ip6 FIB fwding table
115 7 active elements 7 active buckets
117 0 linear search buckets
118 arena: base 7f2fe48bf000, next 803c0
119 used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
121 as we scale to 128k IPv6 entries:
123 .. code-block:: console
126 IPv6 Non-Forwarding Hash Table:
127 Hash table ip6 FIB non-fwding table
128 131079 active elements 32773 active buckets
131 0 linear search buckets
132 arena: base 7fed7a514000, next 4805c0
133 used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
135 IPv6 Forwarding Hash Table:
136 Hash table ip6 FIB fwding table
137 131079 active elements 32773 active buckets
140 0 linear search buckets
141 arena: base 7fedba514000, next 4805c0
142 used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
146 .. code-block:: console
149 IPv6 Non-Forwarding Hash Table:
150 Hash table ip6 FIB non-fwding table
151 262151 active elements 65536 active buckets
154 0 linear search buckets
155 arena: base 7fed7a514000, next 880840
156 used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
158 IPv6 Forwarding Hash Table:
159 Hash table ip6 FIB fwding table
160 262151 active elements 65536 active buckets
163 0 linear search buckets
164 arena: base 7fedba514000, next 880840
165 used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
169 .. code-block:: console
172 IPv6 Non-Forwarding Hash Table:
173 Hash table ip6 FIB non-fwding table
174 1048583 active elements 65536 active buckets
176 [len 1] 65533 free elts
177 [len 2] 65531 free elts
179 0 linear search buckets
180 arena: base 7fed7a514000, next 3882740
181 used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
183 IPv6 Forwarding Hash Table:
184 Hash table ip6 FIB fwding table
185 1048583 active elements 65536 active buckets
187 [len 1] 65533 free elts
188 [len 2] 65531 free elts
190 0 linear search buckets
191 arena: base 7fedba514000, next 3882740
192 used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
194 as can be seen from the output the IPv6 heap in this case was scaled
195 to 1GB and 1million prefixes has used 56MB of it.
201 The main heap is used to allocate objects that represent the FIB
202 entries in the control and data plane (see :ref:`controlplane` and
203 :ref:`dataplane`) such as *fib_entry_t* and *load_balance_t*. These come
204 from the main heap because they are not protocol specific
205 (i.e. they are used to represent either IPv4, IPv6 or MPLS
208 With 1M prefixes allocated the memory usage is:
210 .. code-block:: console
216 IPv4 unicast 1 33619968
217 IPv6 unicast 2 118502784
219 IPv4 multicast 1 1175
220 IPv6 multicast 1 525312
222 Name Size in-use /allocated totals
223 Entry 72 1048589/ 1048589 75498408/75498408
224 Entry Source 40 1048589/ 1048589 41943560/41943560
225 Entry Path-Extensions 76 0 / 0 0/0
226 multicast-Entry 192 6 / 6 1152/1152
227 Path-list 40 18 / 18 720/720
228 uRPF-list 16 14 / 14 224/224
229 Path 72 22 / 22 1584/1584
230 Node-list elements 20 1048602/ 1048602 20972040/20972040
231 Node-list heads 8 24 / 24 192/192
235 .. code-block:: console
241 IPv4 unicast 1 33619968
242 IPv6 unicast 2 252743040
244 IPv4 multicast 1 1175
245 IPv6 multicast 1 525312
247 Name Size in-use /allocated totals
248 Entry 72 2097165/ 2097165 150995880/150995880
249 Entry Source 40 2097165/ 2097165 83886600/83886600
250 Entry Path-Extensions 76 0 / 0 0/0
251 multicast-Entry 192 6 / 6 1152/1152
252 Path-list 40 18 / 19 720/760
253 uRPF-list 16 18 / 18 288/288
254 Path 72 22 / 23 1584/1656
255 Node-list elements 20 2097178/ 2097178 41943560/41943560
256 Node-list heads 8 24 / 24 192/192
258 However, the situation is not a simple as that. All of the 1M prefixes
259 added above were reachable via the same next-hop, so the path-list
260 (and path) they use is shared. As prefixes are added that use
261 different (sets of) next-hops, the number of path-lists and paths
262 requires will increase.
268 VPP collects statistics for each route. For each route VPP collects
269 byte and packet counters for packets sent to the prefix (i.e. the
270 route was matched in the data-plane) and packets sent via the prefix (i.e. the
271 matching prefix is reachable through it - like a BGP peer). This
272 requires 4 counters per route in the stats segment.
274 Below shows the size of the stats segment with 1M, 2M and 4M routes.
276 .. code-block:: console
278 total: 1023.99M, used: 127.89M, free: 896.10M, trimmable: 830.94M
279 total: 1023.99M, used: 234.14M, free: 789.85M, trimmable: 668.15M
280 total: 1023.99M, used: 456.83M, free: 567.17M, trimmable: 388.91M
282 VPP was started with a 1G stats heap.