F5 Agility Labs - Index

Welcome

Welcome to the F5 Analytics and Visibility Solutions lab at F5 Agility 2018

The content contained here leverages a full DevOps CI/CD pipeline and is sourced from the GitHub repository at https://github.com/f5devcentral/f5-agility-labs-analytics. Bugs and Requests for enhancements can be made by opening an Issue within the repository.

Introduction

In this lab session, you will be introduced to a few different ways to gather, visualize and analyze traffic information available on a BIG-IP platform. It is assumed that you are familiar with the basics of setting up a BIG-IP device for various ADC functions. The lab environment has already been setup with an HA pair of BIG-IP Virtual Editions (VEs) that have been pre-configured for a few web applications. Your task will be to configure the BIG-IPs to generate Analytics data so that you may visualize and analyze this data.

Lab Environment Setup

The following components have been setup with basic configurations for you:

  • 2x F5 BIG-IP VEs running version 13.1.0.5, paired in an Active/Standby HA Cluster
  • 1x Linux LAMP Server running Splunk and a few different web applications
  • 1x Windows jumphost

Accessing the Lab Environment

To access the lab environment, you will require a web browser and Remote Desktop Protocol (RDP) client software. The web browser will be used to access the Lab Training Portal to retrieve the IP address for your Windows jump host that you will RDP into to access the entire lab environment.

Note

All work for this lab will be performed exclusively from the Windows jumphost. No installation or interaction with your local system is required.

  • Connect to the Training Portal (details provided by lab instructor)

  • Retrieve the IP address / hostname of the Windows jumpbox (Win7 Client)

  • Establish an RDP connection to your jumpbox and login with the following credentials:

    • User: external_user
    • Password: F5Agility

    Ignore any warnings about Windows activation

  • Ensure that the Windows jumpbox has the correct time. This is necessary in order for other operations to work correctly. To set the correct time, click on the Date/Time in the bottom right-hand corner of the RDP window, and then click on Change date and time settings

    image1

  • In the Date and Time window:

    1. Click on the Internet Time tab
    2. Click Change settings…
    3. Then click Update now.
    4. Finally, click OK, and OK one more time

    image2

    image3

Lab 1 - Introduction to AVR

The Application Visibility and Reporting (AVR) module provides detailed charts and graphs to give you more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). You can use this module to visualize the traffic being processed by your BIG-IP device, and gain a better understanding of where the traffic is originating from (client IP addresses / subnets as well as geographical regions), the nature and volume of request and response traffic (Total Transactions as well as Average and Max Transactions/sec), the most commonly requested URLs, Server Latency and Page Load times, Virtual Server and Pool member performance, and many more metrics.

This lab will give you an introduction on how to setup the AVR module to generate these charts / reports and how to visualize them on your BIG-IP.

Note

The AVR module is a built-in module that is included at no charge with all BIG-IP licenses for software versions 11.x and higher. However, this module is not enabled by default since it consumes additional resources. Hence you must make the conscious decision to provision the module and enable analytics data collection on the Virtual Servers of interest. In your own production environment, you may want to start with enabling analytics data collection on a handful of virtual servers and observe the impact to system performance before enabling it on a larger number of virtual servers.

Configuring AVR to generate on-box Analytics reports

In this lab, you will first configure an Analytics profile to attach to your existing applications (Virtual Servers). You will then view the analytics graphs and charts on the BIG-IP to gain more insight into the traffic patterns for incoming traffic for your applications.

Note

You will perform all configuration tasks from the Windows jump box

  1. On the Windows Jump box, open the Chrome browser, and then use the bookmarks in the bookmark bar to access BIGIP_1 and BIGIP_2 in separate tabs. If you get a warning about an invalid SSL certificate, ignore the warning, and continue to the login page.
  2. Login to both BIG-IPs with these credentials:
    • username: admin
    • password: Agility2018
Task 1 – Provision the AVR module

  1. Determine which BIG-IP is the standby unit by looking at the status in the top-left corner of each of the BIG-IP GUI windows. Normally BIGIP2 should be the standby, but in some cases, you may find that BIGIP1 is the standby unit.

  2. First, on the standby unit, go to System >> Resource Provisioning

  3. For the Application Visibility and Reporting (AVR) module, check-mark the box under the Provisioning column, and ensure the Provisioning level is set to Nominal.

  4. Click Submit

    image4

    Note

    This procedure will cause services to be restarted on the BIG-IP and may cause interruption to the traffic that is passing through the unit. Hence, it is always recommended to perform this step during a maintenance window, and to start with the Standby unit first.

  5. Once the services have been restarted, click Continue.

  6. Repeat the above steps for the other (Active) BIG-IP unit. Note that when you do so, the Active unit will go into Standby state, and the other unit will take over as Active.

Task 2 – Create a new Analytics profile and attach it to your Virtual Servers

  1. On the Active unit, go to Local Traffic >> Profiles >> Analytics >> HTTP Analytics

    image5

  2. Click Create

  3. Click the Custom checkbox in the top-right

  4. Type in Custom_HTTP_Analytics for the Profile Name

    image6

  5. Under the Associated Virtual Servers section, click Add, and then select all listed Virtual Servers. This will add this profile to all the virtual servers simultaneously without having to individually modify each virtual server.

    image7

    image8

  6. Under the Statistics Gathering Configuration, checkmark the following options:

    • Max TPS and Throughput
    • URLs
    • Countries
    • Client IP Addresses
    • Client Subnets
    • Response Codes
    • User Agents
    • Methods
    • OS and Browsers

    image9

  7. Scroll to the bottom and click Finished

You have now configured your BIG-IP device to collect analytics data and start processing it.

Note

Please note that it typically takes 5-10 minutes for the system to start analyzing the data. Please wait at least 5 minutes before proceeding to the next task.

Viewing the Analytics data

Once we have had some traffic received by the application Virtual Servers and processed by the Analytics profile, we can now go in to view and analyze this data.

  1. In the BIG-IP GUI, go to Statistics >> Analytics >> HTTP >> Overview

    image10

  2. This page shows you details about the traffic received by every Virtual Servers that had the HTTP Analytics profile attached.

  3. You can filter and update the graphs, by selecting one or more Virtual Servers from the filters on the right. Selecting one or more Virtual Servers will update the graphs to show data only for the selected Virtual Servers. You can also choose from other filter options, like Pool Members, URLs, Client IP Addresses, etc. Feel free to explore the various graphs and filter options on your own.

  4. You can also go to the the Statistics >> Analytics >> HTTP >> Custom Page, and customize the various widgets shown.

    image11

  5. You can move the widgets around and re-arrange the page by simply dragging-and-dropping the widgets from the top-left corner of each widget. add more widgets on the page by clicking the Add Widget button at the bottom of the page.

  6. Similarly, you can modify the tables on the right side of the page, and add another table by clicking the Add Widget button below the last table on the right.

    image12

  7. Once you have updated the page to show you the data you want, you can create a report by clicking the Export button at the top-right of the page.

    image13

  8. We encourage you to explore the different widgets and graphs on the page, and change the settings on each to view and analyze the data by various metrics.

Lab 2: Integration with Splunk

Splunk is a 3rd-party Security Information and Event Management (SIEM) solution that is used by a large number of organizations to assimilate information and event logs from a large number of disparate sources, and store and analyze it from a single central location in order to correlate data across all devices in the organization.

In this lab, we will integrate our BIG-IPs to send data into Splunk and use Splunk to visualize and analyze the data from a single centralized location rather than viewing/analyzing it on an individual BIG-IP.

Configuring Splunk to use the F5 Splunk app

In order to get Splunk to process and display Analytics data from your BIG-IPs, you need to configure it to accept this data, parse and process it, and display it in a meaningful way for you to get the most out of it. In order to help with this, F5 has collaborated with Splunk to create a Splunk app that is available as a free add-on to your Splunk deployment. This F5 Analytics Splunk app can be downloaded from the Splunkbase web-site here:

https://apps.splunk.com/apps/id/f5

Note

For your convenience, we have already downloaded this Splunk app onto the Windows jump box, so we can just go ahead and install it within our Splunk instance.

Task 1: Install the F5 Splunk app in Splunk

  1. In the Chrome browser on your Windows jump box, click the bookmark for Splunk to launch the Splunk web UI

  2. On the Splunk Enterprise splash page, if you are prompted for an update, click Skip update

  3. In the Splunk Web GUI, click on the settings button next to Apps (on the left) to Manage Apps

    image14

  4. Click Install app from file

    image15

  5. Click Choose File

  6. In the file browser window, navigate to Desktop > Analytics Lab Files, and choose the f5-networks-analytics-new_100.tgz file and click Open

    image16

  7. Click Upload

    image17

  8. Once the upload is complete, you should see the F5 Networks app listed in the Apps table, with the Status set to Enabled

    image18

  9. Click the Splunk logo in the top-left to go to the start page. You should now see the F5 Networks app listed on the left

    image19

  10. Now click the Settings menu in the top-right, and choose Data inputs

    image20

  11. Click on HTTP Event Collector

    image21

  12. Click on Global Settings in the top-right

    image22

  13. In the Edit Global Settings window:

    • Click on Enabled for All Tokens
    • Ensure that Enable SSL is checked
    • Ensure that HTTP Port Number is set to 8088
    • Click Save

    image23

    Note

    Ensure that all of the above settings are exactly as shown, otherwise no data will show up in Splunk.

  14. Click New Token in the top-right

  15. For the Name, enter F5-Analytics, and then Click Next > at the top

  16. On the Input Settings page, scroll down till you see Default Index, and then click the Create a new index link

    image24

  17. In the New Index window, enter f5-default for the Index Name, and click Save

    image25

  18. In the Select Allowed Indexes table, click f5-default to move it to Selected item(s)

    image26

  19. Click Review at the Top

    image27

  20. Ensure your settings match those shown in the screenshot below, then click Submit

    image28

  21. Once your token has been created, highlight the Token Value for the newly created Token, and copy it to your clipboard (Ctrl-C or Right-click > Copy). We will use this later.

    image29

    Note

    Your token value will be different from the one shown above

  22. Click on the Splunk logo in the top-left to go back to the Splunk start page.

Configuring the BIG-IP to send analytics data to Splunk

F5 has created an iApp that simplifies the process of configuring your BIG-IP to send Analytics data to remote sources (including Splunk and/or BIG-IQ). There is also a deployment guide that walks you through the steps needed to configure the iApp.

Note

The F5 Analytics iApp template itself does not ship with the product, but can be downloaded from the F5 downloads site (https://downloads.f5.com).

For your convenience, we have already downloaded the iApp template on the Windows jump box, so we can just import it into our BIG-IP.

Task 2: Import and configure the F5 Analytics iApp template on the BIG-IP

  1. Open a new tab in your Chrome browser, and click on the bookmark for BIGIP_A to connect to the BIG-IP GUI

  2. Login using the following credentials:

    • Username: admin
    • Password: Agility2018

  3. Go to iApps >> Templates

  4. Click Import in the top-right

    image30

  5. Click Choose File

  6. Navigate to Desktop > Analytics Lab Files, and select the f5.analytics.v3.7.0.tmpl file. Click Open

    image31

  7. Click Upload

  8. Once the file is finished uploading, you should see it listed in the iApp Templates table.

  9. Go to iApps >> Application Services >> Applications

  10. Click Create in the top-right

    image32

  11. For the Name, enter F5-Analytics

  12. From the Template pull-down menu, choose f5.analytics.v3.7.0

    image33

  13. In the template configuration, in the Welcome to the f5 BIG-IP Analytics iApp Template section, change the response for the Do you want to see inline help? Question to No, do not show inline help.

    image34

    Note

    If you are not familiar with what all the different settings refer to, you may want to keep the inline help enabled. For now, we have disabled it just to reduce the amount of additional text on the configuration screen.

  14. Under the Information Sources section, set the following:

    1. Data Format: Splunk
    2. System Statistic: Yes
    3. Module High Speed Logging Streams: Yes
    4. Local System Logging (syslog): Yes
    5. System SNMP Alerts: No
    6. iHealth Snapshot Information: No
    7. Facility Name: F5 Lab
    8. Default Tenant: default
    9. Role Based Access Controls: No

    image35

  15. Under the Analytics System Configuration, enter the following:

    1. IP Address or Hostname: 10.1.20.252
    2. Port: 8088
    3. Protocol: HTTPS
    4. API Key: <paste the Token Value that you copied from Splunk in the previous task>

    Leave other settings at their default values

    image36

  16. Leave all settings under Module Log Stream Capture and Local Logging Capture sections at their default values

  17. Under Application Mapping, leave all settings at their default values, except in the Mapping Table, enter the following:

    1. Order: 10
    2. Type: App Name
    3. From: Virtual Name
    1. Regex: (.*)_HTTP[S]*_VS
    2. Action: Map
    3. AppendPrefix: <leave blank>
    4. DirectMapping: <leave blank>

    image37

  18. Click Finished

Note

It may take up to 10 minutes for the system to start showing data in Splunk.

Notes on the F5/Splunk Integration

While you wait for the data to be generated and for Splunk to gather and analyze the data, here is some additional information to help you understand how the integration between F5 and Splunk works, and what kind of information it can provide.

F5.analytics iApp on BIG-IP

This iApp template is designed to gather a large number of statistics and event information from a variety of different sources, and export the data to different kinds of data collectors / SIEM systems. The sources of information that the iApp gathers include system performance metrics (CPU, memory usage, throughput, connection rates, etc.), tmstats (statistics collected by the Traffic Management Microkernel / TMM regarding the traffic that is being handled/processed by TMM), event logs (from the /var/log directory), SNMP trap-related information, and AVR data (detailed HTTP and TCP stats). Note that the AVR module does NOT need to be enabled in order to export the AVR data via this iApp. Also, the configuration options in the iApp allow the user fine-grained control on what data will be collected and bundled up to be sent to external receivers.

The iApp also provides the ability to customize the output format for different receivers, including F5 BIG-IQ, Splunk, as well as other 3rd-party systems. Furthermore, the iApp provides for options to group together and/or map different pieces of information (Virtual Servers and their associated objects, etc.) into Facilities (e.g. data centers), tenants (for multi-tenant environments), and applications, where a single application could consist of multiple virtual servers (for example, a web application could consist of both, an HTTP and an HTTPS virtual server that serve the same application). This application mapping can also be applied across multiple BIG-IPs so that the same application hosted in different locations can be grouped together under a single application name.

For more details on the iApp, please see the iApp Deployment Guide, which can be found here:

https://www.f5.com/pdf/deployment-guides/f5-analytics-dg.pdf

F5 Networks Splunk App for Splunk

Splunk is a very popular Security Information and Event Management (SIEM) system that has the ability to accept statistics and event data from a large variety of sources, and visualize and display it in a meaningful way to allow an end-user to be able to view events and metrics across multiple devices from a single-pane-of-glass view. Additionally, Splunk allows add-on applications to be integrated into the Splunk deployment in order to allow customized processing and display of data from various sources. The F5 Networks Splunk app is just such an add-on that was created by F5 in partnership with Splunk to allow customized processing of data from F5 BIG-IP devices, and to produce easy-to-use dashboards that analyze and present the data in meaningful charts and graphs.

The data presented in the F5 Networks Splunk app includes a lot of data that cannot be easily visualized on a BIG-IP, such as tmstats information, virtual server and pool member health stats, system performance information, and even syslog event information. Additionally, this app provides the ability to collate and present data across multiple BIG-IP devices, even BIG-IPs in different locations, allowing a user to view all their devices and their data in one single central location, rather than having to view it separately on each individual BIG-IP device.

Configuration options for F5.analytics iApp and the F5 Networks Splunk app

Note that this lab guide walks you through some simple setup options for both, the Splunk app as well as the iApp, in order to help you get up-and-running quickly. However, these configuration options are by no means the only way to configure these. To get a better understanding of all the configuration options we ask that you refer to the F5 analytics iApp deployment guide referenced above, which also has a section on configuring the Splunk app.

Viewing the Analytics Data in Splunk

Task 3: Visualize the analytics data in Splunk

Note

It typically takes up to 10 minutes for the system to start showing data in Splunk after you have configured the BIG-IP to send data to it. Please ensure you have waited at least 10 minutes after completing the previous task before you start viewing the data in Splunk.

  1. In your Chrome browser window, open a new tab, and click on the Splunk bookmark to launch the Splunk Web UI

  2. In Splunk, click on the F5 Networks app on the left to launch the F5 Splunk app

    image38

  3. On the Home tab of the F5 Splunk app, change the Time pull-down to Last 60 minutes

    image39

  4. Note that some of the widgets like Non-Responding Hosts or Expiring SSL Certificates may show No results found. This is because there is nothing to report for these metrics in this lab environment.

  5. Scroll down to view other widgets. You may find that your BIG-IP devices are shown under the Unhealthy Devices and/or some applications are shown in the Unhealthy Applications. Let’s investigate:

    image40

  6. Click on the F5_SJC_Cluster device group under Unhealthy Devices. This will open a new browser tab and take you to the Device Cluster Drilldown dashboard. Here you can see a number of different metrics that contribute to the overall device health score.

    image41

  7. Just under the Overview table showing the scores on different metrics, you can also see a table showing the Device Status, with details on the devices included in the group

    image42

  8. If your devices had any error conditions that generated some Diagnostic information, you could see that in the Diagnostics section.

  9. Now click on the System Performance tab. This will show you details on the CPU and Memory usage of your BIG-IP devices, including a breakdown of processes consuming the most amount of CPU or memory

    image43

  10. Next, click on Interface Status & Statistics. This will show you detailed Interface and VLAN stats. You can change the options in the pull-down menus to view different information.

    image44

  11. Next, click on the Events tab. This tab shows you Syslog events, with a time-chart of when different kinds of events occurred. If you see any colored squares in the bigip.syslog.pool.statechange syslog table, click on those squares. If you then scroll down, you will see the actual syslog messages from whatever was happening at that time

    image45

  12. Feel free to explore the other tabs as well to view additional information

  13. Now, let us look at our applications to view more details about them. In the red menu bar at the top, click on Application > Application Dashboard.

    image46

  14. This will show you a listing of all the applications across all your BIG-IPs, based on the application grouping and mappings that you defined in the f5.analytics iApp on the BIG-IP. In our case, we just have a single application. Click on the F5_Demo_App application name to go into the Application Drilldown dashboard

    image47

  15. The Application Drilldown dashboard shows you a lot of detailed statistics about your application(s). You can view the various metrics for your application(s) on the Overview tab. To get more details, you can click the link for the View in Application Health Dashboard, which will give you even more detailed metrics and charts that are used in calculating the health scores for the various metrics:

    image48

  16. In the Application Health Dashboard that opens up in a new browser tab, you can view the various metrics as well as charts for those metrics that make up the overall Application Health score

  17. Now go back to the browser tab for the Application Drilldown dashboard, and then click on the Application Resources tab. This tab shows you various components that make up your applications, including the facility, virtual servers, pools, pool members, and even iRules. In our case, our F5_Demo application is hosted in a single Facility (F5 Lab), and is made up of 2 Virtual Servers: F5_Demo_HTTP_VS and F5_Demo_HTTPS_VS. Each virtual server has its own pool with their corresponding pool members. You can view details for all these components in the tables below.

  18. Next, click on the Traffic Overview, Latency, & Analytics tab. This tab shows you detailed traffic-related stats, similar to the data available via the AVR charts and reports you saw in Lab 1.

  19. Next, click on the Client Visibility tab. This tab provides a lot of visibility into the traffic between the end-clients and the BIG-IP, including connection stats, throughput information, TCP stats, HTTP information (HTTP requests, HTTP version, HTTP compression info, etc.), SSL information (SSL throughput, SSL protocol info, ciphers, SSL renegotiations, etc.). Similarly, the Server Visibility tab provides similar information for the traffic between the BIG-IPs and the back-end application servers.

  20. The Pool Statistics tab provides details on the various pools and pool members across all the BIG-IPs and each application / virtual server on each BIG-IP.

  21. Feel free to explore other tabs including the System Performance and the Alerts and Logs tabs.

Note

This concludes all the lab steps for the Splunk Integration lab. Feel free to explore other portions of the F5 Splunk app, or try out other settings in the f5.analytics iApp. Note that this lab environment does not include other F5 modules (DNS/GTM, ASM, or APM). However, if you have these other modules enabled on your BIG-IP devices in your own environment, you can view data for these modules as well in the F5 Splunk app.