1 .. _binary_api_support:
8 VPP provides a binary API scheme to allow a wide variety of client
9 codes to program data-plane tables. As of this writing, there are
10 hundreds of binary APIs.
12 Messages are defined in \*.api files. Today, there are about 80 api
13 files, with more arriving as folks add programmable features. The API
14 file compiler sources reside in src/tools/vppapigen.
16 From `src/vnet/interface.api
17 <https://docs.fd.io/vpp/18.11/de/d75/interface_8api.html>`_, here's a
18 typical request/response message definition:
20 .. code-block:: console
22 autoreply define sw_interface_set_flags
27 /* 1 = up, 0 = down */
31 To a first approximation, the API compiler renders this definition
33 *vpp/build-root/install-vpp_debug-native/vpp/include/vnet/interface.api.h*
38 /****** Message ID / handler enum ******/
41 vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS, vl_api_sw_interface_set_flags_t_handler)
42 vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, vl_api_sw_interface_set_flags_reply_t_handler)
44 /****** Message names ******/
47 vl_msg_name(vl_api_sw_interface_set_flags_t, 1)
48 vl_msg_name(vl_api_sw_interface_set_flags_reply_t, 1)
50 /****** Message name, crc list ******/
52 #ifdef vl_msg_name_crc_list
53 #define foreach_vl_msg_name_crc_interface \
54 _(VL_API_SW_INTERFACE_SET_FLAGS, sw_interface_set_flags, f890584a) \
55 _(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply, dfbf3afa) \
57 /****** Typedefs *****/
60 #ifndef defined_sw_interface_set_flags
61 #define defined_sw_interface_set_flags
62 typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags {
68 }) vl_api_sw_interface_set_flags_t;
71 #ifndef defined_sw_interface_set_flags_reply
72 #define defined_sw_interface_set_flags_reply
73 typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags_reply {
77 }) vl_api_sw_interface_set_flags_reply_t;
80 #endif /* vl_typedefs */
82 To change the admin state of an interface, a binary api client sends a
83 `vl_api_sw_interface_set_flags_t
84 <https://docs.fd.io/vpp/18.11/dc/da3/structvl__api__sw__interface__set__flags__t.html>`_
85 to VPP, which will respond with a
86 vl_api_sw_interface_set_flags_reply_t message.
88 Multiple layers of software, transport types, and shared libraries
89 implement a variety of features:
91 * API message allocation, tracing, pretty-printing, and replay.
92 * Message transport via global shared memory, pairwise/private shared memory, and sockets.
93 * Barrier synchronization of worker threads across thread-unsafe message handlers.
95 Correctly-coded message handlers know nothing about the transport used
96 to deliver messages to/from VPP. It's reasonably straighforward to use
97 multiple API message transport types simultaneously.
99 For historical reasons, binary api messages are (putatively) sent in
100 network byte order. As of this writing, we're seriously considering
101 whether that choice makes sense.
106 Since binary API messages are always processed in order, we allocate
107 messages using a ring allocator whenever possible. This scheme is
108 extremely fast when compared with a traditional memory allocator, and
109 doesn't cause heap fragmentation. See `src/vlibmemory/memory_shared.c
110 <https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html>`_
111 `vl_msg_api_alloc_internal()
112 <https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html#ac6b6797850e1a53bc68b206e6b8413fb>`_.
114 Regardless of transport, binary api messages always follow a `msgbuf_t <https://docs.fd.io/vpp/18.11/d9/d65/structmsgbuf__.html>`_ header:
118 /** Message header structure */
119 typedef struct msgbuf_
121 svm_queue_t *q; /**< message allocated in this shmem ring */
122 u32 data_len; /**< message length not including header */
123 u32 gc_mark_timestamp; /**< message garbage collector mark TS */
124 u8 data[0]; /**< actual message begins here */
127 This structure makes it easy to trace messages without having to
128 decode them - simply save data_len bytes - and allows
130 <https://docs.fd.io/vpp/18.11/d6/d1b/api__common_8h.html#aff61e777fe5df789121d8e78134867e6>`_
131 to rapidly dispose of message buffers:
136 vl_msg_api_free (void *a)
140 api_main_t *am = &api_main;
142 rv = (msgbuf_t *) (((u8 *) a) - offsetof (msgbuf_t, data));
145 * Here's the beauty of the scheme. Only one proc/thread has
146 * control of a given message buffer. To free a buffer, we just clear the
147 * queue field, and leave. No locks, no hits, no errors...
152 rv->gc_mark_timestamp = 0;
159 Message Tracing and Replay
160 __________________________
162 It's extremely important that VPP can capture and replay sizeable
163 binary API traces. System-level issues involving hundreds of thousands
164 of API transactions can be re-run in a second or less. Partial replay
165 allows one to binary-search for the point where the wheels fall
166 off. One can add scaffolding to the data plane, to trigger when
167 complex conditions obtain.
169 With binary API trace, print, and replay, system-level bug reports of
170 the form "after 300,000 API transactions, the VPP data-plane stopped
171 forwarding traffic, FIX IT!" can be solved offline.
173 More often than not, one discovers that a control-plane client
174 misprograms the data plane after a long time or under complex
175 circumstances. Without direct evidence, "it's a data-plane problem!"
177 See `src/vlibmemory/memory_vlib::c
178 <https://docs.fd.io/vpp/18.11/dd/d3e/vpp__get__metrics_8c.html#a7c3855ed3c45b48ff92a7e881bfede73>`_
179 `vl_msg_api_process_file()
180 <https://docs.fd.io/vpp/18.11/d0/d5b/vlib__api__cli_8c.html#a60194e3e91c0dc6a75906ea06f4ec113>`_,
181 and `src/vlibapi/api_shared.c
182 <https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html>`_. See also
183 the debug CLI command "api trace"
185 Client connection details
186 _________________________
188 Establishing a binary API connection to VPP from a C-language client is easy:
193 connect_to_vpe (char *client_name, int client_message_queue_length)
195 vat_main_t *vam = &vat_main;
196 api_main_t *am = &api_main;
197 if (vl_client_connect_to_vlib ("/vpe-api", client_name,
198 client_message_queue_length) < 0)
200 /* Memorize vpp's binary API message input queue address */
201 vam->vl_input_queue = am->shmem_hdr->vl_input_queue;
202 /* And our client index */
203 vam->my_client_index = am->my_client_index;
207 32 is a typical value for client_message_queue_length. VPP *cannot*
208 block when it needs to send an API message to a binary API client. The
209 VPP-side binary API message handlers are very fast. So, when sending
210 asynchronous messages, make sure to scrape the binary API rx ring with
213 **Binary API message RX pthread**
215 Calling `vl_client_connect_to_vlib
216 <https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a6654b42c91be33bfb6a4b4bfd2327920>`_
217 spins up a binary API message RX pthread:
222 rx_thread_fn (void *arg)
225 memory_client_main_t *mm = &memory_client_main;
226 api_main_t *am = &api_main;
229 q = am->vl_input_queue;
231 /* So we can make the rx thread terminate cleanly */
232 if (setjmp (mm->rx_thread_jmpbuf) == 0)
234 mm->rx_thread_jmpbuf_valid = 1;
236 * Find an unused slot in the per-cpu-mheaps array,
237 * and grab it for this thread. We need to be able to
238 * push/pop the thread heap without affecting other thread(s).
240 if (__os_thread_index == 0)
242 for (i = 0; i < ARRAY_LEN (clib_per_cpu_mheaps); i++)
244 if (clib_per_cpu_mheaps[i] == 0)
246 /* Copy the main thread mheap pointer */
247 clib_per_cpu_mheaps[i] = clib_per_cpu_mheaps[0];
248 __os_thread_index = i;
252 ASSERT (__os_thread_index > 0);
255 vl_msg_api_queue_handler (q);
260 To handle the binary API message queue yourself, use
261 `vl_client_connect_to_vlib_no_rx_pthread
262 <https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_.
264 **Queue non-empty signalling**
266 vl_msg_api_queue_handler(...) uses mutex/condvar signalling to wake
267 up, process VPP -> client traffic, then sleep. VPP supplies a condvar
268 broadcast when the VPP -> client API message queue transitions from
271 VPP checks its own binary API input queue at a very high rate. VPP
272 invokes message handlers in "process" context [aka cooperative
273 multitasking thread context] at a variable rate, depending on
274 data-plane packet processing requirements.
276 Client disconnection details
277 ____________________________
279 To disconnect from VPP, call `vl_client_disconnect_from_vlib
280 <https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a82c9ba6e7ead8362ae2175eefcf2fd12>`_. Please
281 arrange to call this function if the client application terminates
282 abnormally. VPP makes every effort to hold a decent funeral for dead
283 clients, but VPP can't guarantee to free leaked memory in the shared
286 Sending binary API messages to VPP
287 __________________________________
289 The point of the exercise is to send binary API messages to VPP, and
290 to receive replies from VPP. Many VPP binary APIs comprise a client
291 request message, and a simple status reply. For example, to set the
292 admin status of an interface:
296 vl_api_sw_interface_set_flags_t *mp;
297 mp = vl_msg_api_alloc (sizeof (*mp));
298 memset (mp, 0, sizeof (*mp));
299 mp->_vl_msg_id = clib_host_to_net_u16 (VL_API_SW_INTERFACE_SET_FLAGS);
300 mp->client_index = api_main.my_client_index;
301 mp->sw_if_index = clib_host_to_net_u32 (<interface-sw-if-index>);
302 vl_msg_api_send (api_main.shmem_hdr->vl_input_queue, (u8 *)mp);
306 * Use `vl_msg_api_alloc <https://docs.fd.io/vpp/18.11/dc/d5a/memory__shared_8h.html#a109ff1e95ebb2c968d43c100c4a1c55a>`_ to allocate message buffers
307 * Allocated message buffers are not initialized, and must be presumed to contain trash.
308 * Don't forget to set the _vl_msg_id field!
309 * As of this writing, binary API message IDs and data are sent in network byte order
310 * The client-library global data structure `api_main <https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#af58e3e46b569573e9622b826b2f47a22>`_ keeps track of sufficient pointers and handles used to communicate with VPP
312 Receiving binary API messages from VPP
313 ______________________________________
315 Unless you've made other arrangements (see
316 `vl_client_connect_to_vlib_no_rx_pthread
317 <https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_),
318 *messages are received on a separate rx pthread*. Synchronization with
319 the client application main thread is the responsibility of the
322 Set up message handlers about as follows:
326 #define vl_typedefs /* define message structures */
327 #include <vpp/api/vpe_all_api_h.h>
329 /* declare message handlers for each api */
330 #define vl_endianfun /* define message structures */
331 #include <vpp/api/vpe_all_api_h.h>
333 /* instantiate all the print functions we know about */
334 #define vl_print(handle, ...)
336 #include <vpp/api/vpe_all_api_h.h>
338 /* Define a list of all message that the client handles */
339 #define foreach_vpe_api_reply_msg \
340 _(SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply)
341 static clib_error_t *
342 my_api_hookup (vlib_main_t * vm)
344 api_main_t *am = &api_main;
346 vl_msg_api_set_handlers(VL_API_##N, #n, \
347 vl_api_##n##_t_handler, \
349 vl_api_##n##_t_endian, \
350 vl_api_##n##_t_print, \
351 sizeof(vl_api_##n##_t), 1);
357 The key API used to establish message handlers is
358 `vl_msg_api_set_handlers
359 <https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#aa8a8e1f3876ec1a02f283c1862ecdb7a>`_
360 , which sets values in multiple parallel vectors in the `api_main_t
361 <https://docs.fd.io/vpp/18.11/dd/db2/structapi__main__t.html>`_
362 structure. As of this writing: not all vector element values can be
363 set through the API. You'll see sporadic API message registrations
364 followed by minor adjustments of this form:
369 * Thread-safe API messages
371 am->is_mp_safe[VL_API_IP_ADD_DEL_ROUTE] = 1;
372 am->is_mp_safe[VL_API_GET_NODE_GRAPH] = 1;
374 API message numbering in plugins
375 --------------------------------
377 Binary API message numbering in plugins relies on vpp to issue a block
378 of message-ID's for the plugin to use:
382 static clib_error_t *
383 my_init (vlib_main_t * vm)
385 my_main_t *mm = &my_main;
387 name = format (0, "myplugin_%08x%c", api_version, 0);
389 /* Ask for a correctly-sized block of API message decode slots */
390 mm->msg_id_base = vl_msg_api_get_msg_ids
391 ((char *) name, VL_MSG_FIRST_AVAILABLE);
395 Control-plane codes use the vl_client_get_first_plugin_msg_id (...) api
396 to recover the message ID block base:
400 /* Ask the vpp engine for the first assigned message-id */
401 name = format (0, "myplugin_%08x%c", api_version, 0);
402 sm->msg_id_base = vl_client_get_first_plugin_msg_id ((char *) name);
404 It's a fairly common error to forget to add msg_id_base when
405 registering message handlers, or when sending messages. Using macros
406 from .../src/vlibapi/api_helper_macros.h can automate the process, but
407 remember to #define REPLY_MSG_ID_BASE before #including the file:
411 #define REPLY_MSG_ID_BASE mm->msg_id_base
412 #include <vlibapi/api_helper_macros.h>