From: Hanoh Haim Date: Sun, 22 May 2016 12:37:16 +0000 (+0300) Subject: integrate PCAP file tutorials X-Git-Url: https://gerrit.fd.io/r/gitweb?a=commitdiff_plain;h=07cd4bed3b8c3e5027536ee92c0d13365ddd8a33;p=trex.git integrate PCAP file tutorials --- diff --git a/release_notes.asciidoc b/release_notes.asciidoc index 23988280..9555f5fc 100755 --- a/release_notes.asciidoc +++ b/release_notes.asciidoc @@ -23,6 +23,22 @@ ifdef::backend-docbook[] endif::backend-docbook[] +== Release 2.01 == + +* First release of latency/jitter per stream see link:trex_stateless.html#_tutorial_per_stream_latency_jitter_packet_errors[here] +* Ability to send server side pcap file - unlimited pcap file size see link:/cp_stl_docs/_modules/trex_stl_lib/trex_stl_client.html#STLClient.push_remote[here] and link:trex_stateless.html#_pcap_based_traffic_tutorials[PCAP tutorial] +* Significatly improve performance for both Stateful and Stateless - in some cases up to x2 due to Scheduler/CPU utilization rewrite +* CPU utilization estimation is done differently +* Ability to significatly improve performance for Stateless profile with Field engine (up to x5, 22MPPS) see link:trex_stateless.html#_tutorial_field_engine_significantly_improve_performance[here] +* Add documentation index link:index.html[index] +* Fix documentation Table of Content javascript +* Update Stateless presentation link:http://www.slideshare.net/HanochHaim/trex-realistic-traffic-generator-stateless-support[Statelss presenation] +* Stateful Python server API - Add support for iterator of remote file for Stateful GUI + +=== fix issues: === + +* Performance issue, link:http://trex-tgn.cisco.com/youtrack/issue/trex-207[trex-207] +* Sporadic timeout on wait_on_traffic() API see link:http://trex-tgn.cisco.com/youtrack/issue/trex-209[trex-209] == Release 2.00 == @@ -37,7 +53,6 @@ endif::backend-docbook[] image::images/trex_stl_gui.png[title="TRex Stateless GUI",align="left",width=600, link="images/trex_stl_gui.png"] - === fix issues: === * X710/XL710 per stream hardware stats diff --git a/trex_stateless.asciidoc b/trex_stateless.asciidoc index 1f2bd9a1..4481929f 100755 --- a/trex_stateless.asciidoc +++ b/trex_stateless.asciidoc @@ -2301,7 +2301,7 @@ In some situations, it is necessary to split streams into threads in such a way image::images/stl_barrier_03.png[title="Stream Barrier",align="left",width={p_width}, link="images/stl_barrier_03.png"] -==== Tutorial: Pcap file to one stream +==== Tutorial: PCAP file to one stream *Goal*:: Load a stream template packet from a pcap file instead of Scapy. @@ -2336,232 +2336,6 @@ Assumption: The pcap file contains only one packet. If the pcap file contains mo <1> Takes the packet from the pcap file, relative to the directory of the *profile* file location. -==== Tutorial: pcap file conversion to many streams - -*Goal*:: Load a pcap file with a *number* of packets, creating a stream with a burst value of 1 for each packet. The inter-stream gap (ISG) for each stream is equal to the inter-packet gap (IPG). - -*File*:: link:{github_stl_path}/pcap.py[pcap.py] - -[source,python] ----- - def get_streams (self, - ipg_usec = 10.0, <1> - loop_count = 1): <2> - - profile = STLProfile.load_pcap(self.pcap_file, <3> - ipg_usec = ipg_usec, - loop_count = loop_count) ----- -<1> The inter-stream gap in microseconds. -<2> Loop count. -<3> Input pcap file. - -// Please leave this comment - helping rendition. - -image::images/stl_loop_count_01b.png[title="Example of multiple streams",align="left",width="80%", link="images/stl_loop_count_01b.png"] - -// OBSOLETE: image::images/stl_loop_count_01b.png[title="Streams, loop_count",align="left",width={p_width_1a}, link="images/stl_loop_count_01b.png"] - -The figure shows the streams for a pcap file with 3 packets, with a loop configured. - -* Each stream is configured to Burst mode with 1 packet. -* Each stream triggers the next stream. -* The last stream triggers the first with `action_loop=loop_count` if `loop_count` > 1. - -The profile runs on one DP thread because it has a burst with 1 packet. (Split cannot work in this case). - -To run this example, enter: - -[source,bash] ----- -./stl-sim -f stl/pcap.py --yaml ----- - -The following output appears: - -[source,python] ----- -$./stl-sim -f stl/pcap.py --yaml -- name: 1 - next: 2 <1> - stream: - action_count: 0 - enabled: true - flags: 0 - isg: 10.0 - mode: - percentage: 100 - total_pkts: 1 - type: single_burst - packet: - meta: '' - rx_stats: - enabled: false - self_start: true - vm: - instructions: [] - split_by_var: '' -- name: 2 - next: 3 - stream: - action_count: 0 - enabled: true - flags: 0 - isg: 10.0 - mode: - percentage: 100 - total_pkts: 1 - type: single_burst - packet: - meta: '' - rx_stats: - enabled: false - self_start: false - vm: - instructions: [] - split_by_var: '' -- name: 3 - next: 4 - stream: - action_count: 0 - enabled: true - flags: 0 - isg: 10.0 - mode: - percentage: 100 - total_pkts: 1 - type: single_burst - packet: - meta: '' - rx_stats: - enabled: false - self_start: false - vm: - instructions: [] - split_by_var: '' -- name: 4 - next: 5 - stream: - action_count: 0 - enabled: true - flags: 0 - isg: 10.0 - mode: - percentage: 100 - total_pkts: 1 - type: single_burst - packet: - meta: '' - rx_stats: - enabled: false - self_start: false - vm: - instructions: [] - split_by_var: '' -- name: 5 - next: 1 <2> - stream: - action_count: 1 <3> - enabled: true - flags: 0 - isg: 10.0 - mode: - percentage: 100 - total_pkts: 1 - type: single_burst - packet: - meta: '' - rx_stats: - enabled: false - self_start: false <4> - vm: - instructions: [] - split_by_var: '' ----- -<1> Each stream triggers the next stream. -<2> The last stream triggers the first. -<3> The current loop count is given in: `action_count: 1` -<4> `Self_start` is enabled for the first stream, disabled for all other streams. - -==== Tutorial: pcap file to many streams and Field Engine - -The following example loads a pcap file to many streams, and attaches Field Engine program to each stream. For example, the Field Engine can change the `IP.src` of all the streams to a random IP address. - -*File*:: link:{github_stl_path}/pcap_with_vm.py[stl/pcap_with_vm.py] - -[source,python] ----- - - def create_vm (self, ip_src_range, ip_dst_range): - if not ip_src_range and not ip_dst_range: - return None - - # until the feature of offsets will be fixed for PCAP use hard coded offsets - - vm = [] - - if ip_src_range: - vm += [STLVmFlowVar(name="src", - min_value = ip_src_range['start'], - max_value = ip_src_range['end'], - size = 4, op = "inc"), - #STLVmWrFlowVar(fv_name="src",pkt_offset= "IP.src") - STLVmWrFlowVar(fv_name="src",pkt_offset = 26) - ] - - if ip_dst_range: - vm += [STLVmFlowVar(name="dst", - min_value = ip_dst_range['start'], - max_value = ip_dst_range['end'], - size = 4, op = "inc"), - - #STLVmWrFlowVar(fv_name="dst",pkt_offset= "IP.dst") - STLVmWrFlowVar(fv_name="dst",pkt_offset = 30) - ] - - vm += [#STLVmFixIpv4(offset = "IP") - STLVmFixIpv4(offset = 14) - ] - - return vm - - - def get_streams (self, - ipg_usec = 10.0, - loop_count = 5, - ip_src_range = None, - ip_dst_range = {'start' : '10.0.0.1', - 'end': '10.0.0.254'}): - - vm = self.create_vm(ip_src_range, ip_dst_range) <1> - profile = STLProfile.load_pcap(self.pcap_file, - ipg_usec = ipg_usec, - loop_count = loop_count, - vm = vm) <2> - - return profile.get_streams() ----- -<1> Creates Field Engine program. -<2> Applies the Field Engine to all packets -> converts to streams. - -.Output -[format="csv",cols="1^,2^,1^", options="header",width="40%"] -|================= -pkt, IPv4 , flow - 1 , 10.0.0.1, 1 - 2 , 10.0.0.1, 1 - 3 , 10.0.0.1, 1 - 4 , 10.0.0.1, 1 - 5 , 10.0.0.1, 1 - 6 , 10.0.0.1, 1 - 7 , 10.0.0.2, 2 - 8 , 10.0.0.2, 2 - 9 , 10.0.0.2, 2 - 10 , 10.0.0.2,2 - 11 , 10.0.0.2,2 - 12 , 10.0.0.2,2 -|================= - ==== Tutorial: Teredo tunnel (IPv6 over IPv4) @@ -3401,17 +3175,19 @@ Use the following command within the TRex console to run the profile. TRex>start -f stl/hlt/hlt_udp_inc_dec_len_9k.py -m 10mbps -a ---- -=== PCAP Based Traffic -TRex provides a method of using a pre-recorded traffic as a profile template. +=== PCAP Based Traffic Tutorials +==== PCAP Based Traffic + +TRex provides a method of using a pre-recorded traffic as a profile template. There are two main distinct ways of creating a profile or a test based on a PCAP. * Local PCAP push * Server based push -==== Local PCAP push +===== Local PCAP push On this mode, the PCAP file is loaded locally by the Python client, transformed to a list of streams which each one contains a single packet @@ -3424,7 +3200,6 @@ However, due to the overhead of processing and sending a list of streams this method is limited to a file size (on default 1MB) - *Pros:* * supports most CAP file formats @@ -3438,8 +3213,7 @@ sending a list of streams this method is limited to a file size (on default 1MB) * high configuration time due to transmitting the CAP file as streams - -==== Server based push +===== Server based push To provide also a way of injecting a much larger PCAP files, TRex also provides a server based push. @@ -3464,9 +3238,155 @@ setting up the server with the required configuration is much lower. * rate of transmition is usually limited by I/O performance and buffering (HDD) -=== PCAP Based Traffic Tutorials +==== Tutorial: Simple PCAP file - Profile + +*Goal*:: Load a pcap file with a *number* of packets, creating a stream with a burst value of 1 for each packet. The inter-stream gap (ISG) for each stream is equal to the inter-packet gap (IPG). + +*File*:: link:{github_stl_path}/pcap.py[pcap.py] + +[source,python] +---- + def get_streams (self, + ipg_usec = 10.0, <1> + loop_count = 1): <2> + + profile = STLProfile.load_pcap(self.pcap_file, <3> + ipg_usec = ipg_usec, + loop_count = loop_count) +---- +<1> The inter-stream gap in microseconds. +<2> Loop count. +<3> Input pcap file. + +// Please leave this comment - helping rendition. + +image::images/stl_loop_count_01b.png[title="Example of multiple streams",align="left",width="80%", link="images/stl_loop_count_01b.png"] + +// OBSOLETE: image::images/stl_loop_count_01b.png[title="Streams, loop_count",align="left",width={p_width_1a}, link="images/stl_loop_count_01b.png"] + +The figure shows the streams for a pcap file with 3 packets, with a loop configured. + +* Each stream is configured to Burst mode with 1 packet. +* Each stream triggers the next stream. +* The last stream triggers the first with `action_loop=loop_count` if `loop_count` > 1. + +The profile runs on one DP thread because it has a burst with 1 packet. (Split cannot work in this case). + +To run this example, enter: + +[source,bash] +---- +./stl-sim -f stl/pcap.py --yaml +---- + +The following output appears: + +[source,python] +---- +$./stl-sim -f stl/pcap.py --yaml +- name: 1 + next: 2 <1> + stream: + action_count: 0 + enabled: true + flags: 0 + isg: 10.0 + mode: + percentage: 100 + total_pkts: 1 + type: single_burst + packet: + meta: '' + rx_stats: + enabled: false + self_start: true + vm: + instructions: [] + split_by_var: '' +- name: 2 + next: 3 + stream: + action_count: 0 + enabled: true + flags: 0 + isg: 10.0 + mode: + percentage: 100 + total_pkts: 1 + type: single_burst + packet: + meta: '' + rx_stats: + enabled: false + self_start: false + vm: + instructions: [] + split_by_var: '' +- name: 3 + next: 4 + stream: + action_count: 0 + enabled: true + flags: 0 + isg: 10.0 + mode: + percentage: 100 + total_pkts: 1 + type: single_burst + packet: + meta: '' + rx_stats: + enabled: false + self_start: false + vm: + instructions: [] + split_by_var: '' +- name: 4 + next: 5 + stream: + action_count: 0 + enabled: true + flags: 0 + isg: 10.0 + mode: + percentage: 100 + total_pkts: 1 + type: single_burst + packet: + meta: '' + rx_stats: + enabled: false + self_start: false + vm: + instructions: [] + split_by_var: '' +- name: 5 + next: 1 <2> + stream: + action_count: 1 <3> + enabled: true + flags: 0 + isg: 10.0 + mode: + percentage: 100 + total_pkts: 1 + type: single_burst + packet: + meta: '' + rx_stats: + enabled: false + self_start: false <4> + vm: + instructions: [] + split_by_var: '' +---- +<1> Each stream triggers the next stream. +<2> The last stream triggers the first. +<3> The current loop count is given in: `action_count: 1` +<4> `Self_start` is enabled for the first stream, disabled for all other streams. -==== Tutorial: Sending a simple PCAP file (size < 1MB) + +==== Tutorial: Simple PCAP file - API For this case we can use the local push: @@ -3500,7 +3420,7 @@ try: ---- -==== Tutorial: Sending a PCAP file iterating over dest IP (size < 1MB) +==== Tutorial: PCAP file iterating over dest IP For this case we can use the local push: @@ -3538,7 +3458,7 @@ try: ---- -==== Tutorial: Sending a PCAP file with VLAN (size < 1MB) +==== Tutorial: PCAP file with VLAN This is a more intresting case where we can provide the push API a function hook. The hook will be called for each packet that is loaded from the PCAP file. @@ -3593,12 +3513,91 @@ try: ---- -==== Tutorial: Sending a huge PCAP file +==== Tutorial: PCAP file and Field Engine - Profile + +The following example loads a pcap file to many streams, and attaches Field Engine program to each stream. For example, the Field Engine can change the `IP.src` of all the streams to a random IP address. + +*File*:: link:{github_stl_path}/pcap_with_vm.py[stl/pcap_with_vm.py] + +[source,python] +---- + + def create_vm (self, ip_src_range, ip_dst_range): + if not ip_src_range and not ip_dst_range: + return None + + # until the feature of offsets will be fixed for PCAP use hard coded offsets + + vm = [] + + if ip_src_range: + vm += [STLVmFlowVar(name="src", + min_value = ip_src_range['start'], + max_value = ip_src_range['end'], + size = 4, op = "inc"), + #STLVmWrFlowVar(fv_name="src",pkt_offset= "IP.src") + STLVmWrFlowVar(fv_name="src",pkt_offset = 26) + ] + + if ip_dst_range: + vm += [STLVmFlowVar(name="dst", + min_value = ip_dst_range['start'], + max_value = ip_dst_range['end'], + size = 4, op = "inc"), + + #STLVmWrFlowVar(fv_name="dst",pkt_offset= "IP.dst") + STLVmWrFlowVar(fv_name="dst",pkt_offset = 30) + ] + + vm += [#STLVmFixIpv4(offset = "IP") + STLVmFixIpv4(offset = 14) + ] + + return vm + + + def get_streams (self, + ipg_usec = 10.0, + loop_count = 5, + ip_src_range = None, + ip_dst_range = {'start' : '10.0.0.1', + 'end': '10.0.0.254'}): + + vm = self.create_vm(ip_src_range, ip_dst_range) <1> + profile = STLProfile.load_pcap(self.pcap_file, + ipg_usec = ipg_usec, + loop_count = loop_count, + vm = vm) <2> + + return profile.get_streams() +---- +<1> Creates Field Engine program. +<2> Applies the Field Engine to all packets -> converts to streams. + +.Output +[format="csv",cols="1^,2^,1^", options="header",width="40%"] +|================= +pkt, IPv4 , flow + 1 , 10.0.0.1, 1 + 2 , 10.0.0.1, 1 + 3 , 10.0.0.1, 1 + 4 , 10.0.0.1, 1 + 5 , 10.0.0.1, 1 + 6 , 10.0.0.1, 1 + 7 , 10.0.0.2, 2 + 8 , 10.0.0.2, 2 + 9 , 10.0.0.2, 2 + 10 , 10.0.0.2,2 + 11 , 10.0.0.2,2 + 12 , 10.0.0.2,2 +|================= + + +==== Tutorial: Huge server side PCAP file Now we would like to use the remote push API. This will require the file path to be visible to the server. - [source,bash] ---- c = STLClient(server = "localhost") @@ -3632,7 +3631,7 @@ try: ---- -==== Tutorial: Sending a long list of PCAP files of varied sizes +==== Tutorial: A long list of PCAP files of varied sizes This is also a good candidate for the remote push API. The total overhead for sending the PCAP files will be high if the list is long,