Crawling Real HTTP User

---

Overview

This documentation page provides an example of how to simulate a real user's browsing behavior. It demonstrates a test that replicates user actions on the Locust documentation site. The test identifies all links in the site's table of contents and then accesses these links in a randomized manner, mimicking a genuine user's experience.

Key Features

• Real User Simulation: This test accesses links with a random delay between 10 to 600 seconds, accurately simulating user load.
• Versatility: A prime example for those wanting to recreate real user actions on different websites.
• Driven by Locust: The power of locust.io backs LoadForge, an esteemed open-source load testing tool.

Test Script

Outlined below is the Python script underpinning this test:

import random
from locust import HttpUser, between, task
from pyquery import PyQuery

class AwesomeUser(HttpUser):
    # Simulated user behavior indicates a longer wait time (from 10 to 600 seconds) 
    # between pages due to the considerable content present on each page.
    wait_time = between(10, 600)

    def on_start(self):
        # Initial delay ensures users don't access the site all at once.
        self.wait()
        # The starting point for all users is assumed to be the index page.
        self.index_page()
        self.urls_on_current_page = self.toc_urls

    @task(10)
    def index_page(self):
        r = self.client.get("/")
        pq = PyQuery(r.content)
        link_elements = pq(".toctree-wrapper a.internal")
        self.toc_urls = [l.attrib["href"] for l in link_elements]

    @task(50)
    def load_page(self):
        url = random.choice(self.toc_urls)
        r = self.client.get(url)
        pq = PyQuery(r.content)
        link_elements = pq("a.internal")
        self.urls_on_current_page = [l.attrib["href"] for l in link_elements]

    @task(30)
    def load_sub_page(self):
        url = random.choice(self.urls_on_current_page)
        r = self.client.get(url)

Conclusion

LoadForge equips both novices and experts in load testing with an array of tools and examples, optimizing the efficacy of your tests. The real HTTP user crawling instance is a testament to the myriad ways you can gauge and hone the performance of your online platforms.