Testing Whether AWS WAF Can Detect and Restrict Bot Access from Scrapling

Are you familiar with Scrapling, a Python web scraping library? It is a full-featured scraping library that combines adaptive element tracking, fast parsing, and a choice of multiple fetchers in a single framework.

On the other hand, it also includes anti-bot evasion features such as TLS fingerprint spoofing and automatic Cloudflare Turnstile bypass, and if misused, it poses a challenge that conventional WAFs struggle to handle.

In this article, I actually tested how far AWS WAF's multi-layer defense can handle Scrapling's anti-bot evasion features. To sum up, I confirmed that a multi-layer defense including Bot Control Targeted's TGT_VolumetricIpTokenAbsent works effectively. I will introduce the specific configuration and points to keep in mind for production operation.

What is Scrapling

• Official repository: https://github.com/D4Vinci/Scrapling

It is an adaptive web scraping framework that automatically detects changes in a website's structure and relocates elements.

Main fetchers

# Send a request imitating Chrome's TLS fingerprint
FetcherSession(impersonate='chrome')

# Bypass Cloudflare Turnstile with Camoufox
StealthyFetcher.fetch('https://example.com', headless=True, solve_cloudflare=True)

Multi-layer defense design

We counter the spoofing capabilities of each Scrapling fetcher with AWS WAF's three-layer structure.

• IP rate rule — stops simple high-volume access
• JA4 fingerprint rate rule — stops attacks that use a fixed tool
• Bot Control Targeted — stops advanced attacks, including those that spoof JA4

Why JA4 instead of JA3

impersonate='chrome' faithfully reproduces the ClientHello permutation (a specification that randomizes the order of TLS extensions) introduced in Chrome 110 and later. Because JA3 is a hash that includes the order of extensions, it yields a different value on every request. This is not a bug in Scrapling; it is the result of accurately imitating the behavior of real Chrome.

Checking actual WAF logs, consecutive access using impersonate='chrome' yields a different JA3 value every time, but JA4 is the same value in every request.

# impersonate='chrome' で連続アクセスした際の WAF ログ（抜粋）
JA3=ba507dfd...  JA4=t13d1516h2_8daaf6152...
JA3=1355ef97...  JA4=t13d1516h2_8daaf6152...
JA3=8897b176...  JA4=t13d1516h2_8daaf6152...
JA3=c567ed94...  JA4=t13d1516h2_8daaf6152...

JA3 differs every time, but JA4 is fixed at t13d1516h2_8daaf6152... for all requests.

Detection matrix by user level

We organize how each layer functions depending on the user's skill level (accuracy of spoofing).

Test environment

[Scrapling（公式 Docker イメージ）]
        ↓
[CloudFront]
        ↓ WAF アタッチ
[Lambda Function URL]（WAF SDK 埋め込み済み HTML を返す）

The Lambda is a simple function that just returns test HTML. The WAF SDK (challenge.js) is embedded in <head>.

import os

# Set WAF_SDK_URL to the Application Integration URL (without the trailing /challenge.js)
# Example: https://xxxx.edge.sdk.awswaf.com/xxxx/yyyy
WAF_SDK_URL = os.environ.get('WAF_SDK_URL', '')
SDK_SCRIPT = f'<script src="{WAF_SDK_URL}/challenge.js" defer></script>' if WAF_SDK_URL else ''

HTML = f"""<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Scrapling WAF Test</title>
  {SDK_SCRIPT}
</head>
<body>
  <h1>Scrapling WAF Test Page</h1>
</body>
</html>"""

def handler(event, context):
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "text/html; charset=utf-8"},
        "body": HTML,
    }

The WAF configuration (three layers of rules) is defined in CDK as follows.

// WAF Web ACL（3層の多層防御）
const webAcl = new wafv2.CfnWebACL(this, 'WebAcl', {
  scope: 'CLOUDFRONT',
  defaultAction: { allow: {} },
  rules: [
    // Layer 1: IP レートルール
    {
      name: 'IpRateLimit',
      priority: 1,
      action: { block: {} },
      statement: {
        rateBasedStatement: {
          limit: 100,           // 本番では用途に応じて調整
          evaluationWindowSec: 300,
          aggregateKeyType: 'IP',
        },
      },
      // ...
    },
    // Layer 2: JA4 フィンガープリントレートルール
    {
      name: 'Ja4RateLimit',
      priority: 2,
      action: { block: {} },
      statement: {
        rateBasedStatement: {
          limit: 100,
          evaluationWindowSec: 300,
          aggregateKeyType: 'CUSTOM_KEYS',
          customKeys: [
            { ja4Fingerprint: { fallbackBehavior: 'NO_MATCH' } },
          ],
        },
      },
      // ...
    },
    // Layer 3: Bot Control Targeted（まず Count モードで観測）
    {
      name: 'BotControlTargeted',
      priority: 3,
      overrideAction: { count: {} },  // Count モードで全ルールをラベル付与のみに
      statement: {
        managedRuleGroupStatement: {
          vendorName: 'AWS',
          name: 'AWSManagedRulesBotControlRuleSet',
          managedRuleGroupConfigs: [{
            awsManagedRulesBotControlRuleSet: {
              inspectionLevel: 'TARGETED',
              enableMachineLearning: true,
            },
          }],
        },
      },
      // ...
    },
    // Layer 3 後段: ラベルマッチルールで特定ラベルのみ Block/Challenge
    // 例: TGT_TokenAbsent ラベルが付いたリクエストを Challenge
    // {
    //   name: 'BlockTokenAbsent',
    //   priority: 4,
    //   action: { challenge: {} },
    //   statement: {
    //     labelMatchStatement: {
    //       scope: 'LABEL',
    //       key: 'awswaf:managed:aws:bot-control:TGT_TokenAbsent',
    //     },
    //   },
    // },
  ],
  // ...
});

Test code

We run each scenario using Scrapling's official Docker image.

from scrapling.fetchers import Fetcher, StealthyFetcher

TARGET = "https://xxxx.cloudfront.net"

# WAF block detection (202: Challenge, 403: Block, 429: Rate limit)
def is_blocked(page):
    # The WAF CAPTCHA/Challenge response HTML contains JS that sets aws-waf-token
    # It is not included in the SDK script tag of a normal page, so the presence of this string can be used to detect blocking
    if page.status in (202, 403, 429):
        return True
    body = str(page.body) if page.body else ''
    return 'aws-waf-token' in body or ('awswaf' in body and 'sdk.awswaf.com' not in body)

# Scenario 1: plain HTTP request
page = Fetcher.get(TARGET)
print(f"S1: status={page.status}, blocked={is_blocked(page)}")

# Scenario 2: StealthyFetcher (Camoufox)
page = StealthyFetcher.fetch(TARGET, headless=True, network_idle=True, timeout=15000)
print(f"S2: status={page.status}, blocked={is_blocked(page)}")

# Scenario 3: IP rate loop (confirm Block when the threshold is exceeded)
for i in range(15):
    page = Fetcher.get(TARGET)
    print(f"S3-{i+1}: status={page.status}")
    if is_blocked(page):
        print(f"  -> Blocked at request {i+1}")
        break

# Scenario 4: consecutive access with a fixed JA4 (confirm Block when the threshold is exceeded)
for i in range(15):
    page = Fetcher.get(TARGET, impersonate='chrome')
    print(f"S4-{i+1}: status={page.status}")
    if is_blocked(page):
        print(f"  -> Blocked at request {i+1}")
        break

# Scenario 5: randomly change impersonate (evade JA4 rate -> detected by Bot Control)
for i, imp in enumerate(['chrome', 'firefox', 'chrome120', 'firefox135', 'safari'] * 6):
    page = Fetcher.get(TARGET, impersonate=imp)
    print(f"S5-{i+1}: status={page.status} impersonate={imp}")

docker run --rm \
  -v $(pwd)/scripts:/scripts \
  -e PYTHONPATH=/app \
  --entrypoint python3 \
  pyd4vinci/scrapling \
  /scripts/test_fetcher.py

Test results

Results by scenario

Key point ①: Effectiveness of the JA4 rate rule

When accessing continuously with impersonate='chrome' fixed, the JA4 was the same value, so the requests were 403 Blocked once the threshold was exceeded.

[S4-1]  status=200
[S4-2]  status=200
  ...
[S4-11] status=403 -> Blocked  ← JA4 レートルール発動

Key point ②: Bot Control complements even with JA4 rotation

When impersonate is randomly changed among chrome → firefox → safari, JA4 changes, so the JA4 rate rule is partially evaded. However, Bot Control Targeted applied the TGT_VolumetricIpTokenAbsent label to every request, and switching to Block mode would block all of them.

🔑 This is the core of multi-layer defense: Even if JA4 rate limiting is slipped past, Bot Control does not overlook the IP-level anomaly (VolumetricIpTokenAbsent) that "requests without a WAF token are coming from this IP at an abnormal frequency." Even if the JA4 rate rule (Layer 2) is evaded, Bot Control Targeted (Layer 3) complements it—this is where the true value of multi-layer defense appears.

[S5-1] status=403 impersonate=chrome  -> Blocked（シナリオ4で同一 JA4 の閾値を超過済み）
[S5-2] status=200 impersonate=firefox  ← JA4 レートは通過
[S5-3] status=200 impersonate=chrome120 ← JA4 レートは通過

WAF log (CloudWatch Logs Insights):

action=BLOCK  rule=Ja4RateLimit          ← chrome の JA4 はブロック
action=ALLOW  rule=Default_Action  labels=[token:absent, TGT_VolumetricIpTokenAbsent]
action=ALLOW  rule=Default_Action  labels=[token:absent, TGT_VolumetricIpTokenAbsent]

The action is ALLOW because Bot Control is in Count mode, but the labels are applied accurately, and switching to Block mode would block all requests.

Key point ③: StealthyFetcher (Camoufox) also fails to obtain a token

Camoufox spoofs internal browser signals such as Canvas/WebGL and navigator, but it could not get past the silent challenge of the WAF SDK (challenge.js) and failed to obtain an aws-waf-token at all. Although the JS execution environment is in place, the WAF continuously checks the consistency of multiple internal browser APIs, so inconsistencies peculiar to headless browsers (timing characteristics, event behavior, etc.) were probably detected. The token:absent label continued to be applied, and token:accepted was never reached.

Limitations of the test environment and notes for production

• Same-IP limitation: In this test, the bots and the legitimate browser were run from the same IP, so the legitimate browser was also affected by the IP rate. In a production environment (different IPs), legitimate browsers are given token:accepted and no false positives occur.
• Residential proxies: If a proxy service such as BrightData switches the source IP on every request, neither IpRateLimit nor TGT_VolumetricIpTokenAbsent (the per-IP threshold) will work. In this case, all the WAF can rely on is signals (quality) such as TGT_SignalAutomatedBrowser.
• WAF SDK integration: Simply adding challenge.js to the HTML is not sufficient. In an SPA (React/Vue, etc.), you need an implementation that attaches aws-waf-token to all requests using fetch/XHR interceptors. If this is not handled, legitimate users can get stuck in an endless CAPTCHA.

Recommended workflow for production operation

Bot Control Targeted incurs additional charges, so rather than enabling it suddenly in Block mode, we recommend introducing it gradually in the following steps.

Phase 0: Initial assessment (low cost, no Bot Control needed)

• Check for abnormal User-Agents (default UAs such as curl and python-requests)
• Check the number of abnormal requests by JA4 fingerprint in the WAF logs
• Check CloudFront logs for patterns where subresources are not fetched

Using the WAF console's AI traffic analysis dashboard, you can visually grasp access patterns and request volumes from AI bots and crawlers, which can serve as material for judgment before introducing Bot Control.

https://dev.classmethod.jp/articles/aws-waf-ai-activity-dashboard/

SELECT c_ip, COUNT(*) as cnt
FROM cloudfront_logs
WHERE cs_uri_stem NOT LIKE '%.css'
  AND cs_uri_stem NOT LIKE '%.js'
  AND cs_uri_stem NOT LIKE '%.png'
  AND cs_uri_stem NOT LIKE '%.jpg'
GROUP BY c_ip
HAVING COUNT(*) > 10
ORDER BY cnt DESC

Phase 1: Observe in Count mode

• Enable Bot Control Targeted in Count mode
• Record occurrences of TGT_TokenAbsent / TGT_VolumetricIpTokenAbsent

fields @timestamp, httpRequest.clientIp, action
| filter labels.0.name like /bot-control/
| stats count() by labels.0.name
| sort count desc

Phase 2: Switch to Block/Challenge

• Change rules you are confident about to Block/Challenge

Phase 2.5: Design exception rules (safelist)

• Place an Allow rule before the Bot Control rules
• The principle is to "prepare relief rules first, then Block" rather than "Block first, then rescue"

Points to keep in mind when using Scrapling

Illegitimate crawling imposes direct costs on site operators. WAF Bot Control Targeted costs $10 per month (per WebACL) plus request charges ($10 per million requests); large volumes of access not only drive up defense costs, but strengthened countermeasures can also degrade the experience of legitimate users.

Scrapling itself is a high-performance crawler and can be used widely, including for monitoring your own services, test automation, and research and analysis of public data. When using it against external services, we recommend observing the following:

• Check robots.txt (it can be obeyed automatically with the robots_txt_obey option; see the official documentation for details)
• Check the site's terms of use
• Space out requests and avoid excessive parallel execution
• Do not retry immediately when 429 or 5XX is returned (429 is an explicit rate-limit signal; 5XX may indicate a server-side problem)
• Retrieve only the minimum necessary data

When developing an AI crawler

The basic principle is not to rely on Scrapling's anti-bot evasion features, but to obtain data by means that the target site permits. If the target site supports WBA (Web Bot Authentication), we recommend obtaining permission as a legitimate bot through the standard authentication method. WBA is a cryptographic signature based on HTTP Message Signatures (RFC 9421); AWS WAF added support in November 2025, and Cloudflare and others already support it.

https://dev.classmethod.jp/articles/aws-waf-web-bot-auth-wba-support/

Summary

In this article, we tested AWS WAF's multi-layer defense against each Scrapling fetcher. The results are as follows.

This test confirmed that AWS WAF's multi-layer defense works effectively even against Scrapling's advanced anti-bot evasion features. In particular, Bot Control Targeted's TGT_VolumetricIpTokenAbsent was able to detect traffic even when the JA4 rate rule was evaded.

If you are suffering bot damage from advanced crawlers like Scrapling, please consider tuning AWS WAF. In particular, Bot Control Targeted is a powerful option that can detect, at the JS execution environment level, even tools that make full use of TLS fingerprint spoofing.

References

• Scrapling official repository
• AWS WAF Bot Control
• AWS WAF pricing
• AWS WAF Web Bot Auth support
• Detect and block advanced bot traffic（AWS Security Blog）