GHSA-23j4-mw76-5v7h: Scrapy Scrapy Allows Redirect Following Vulnerability

MODERATE
Share
CVSS 6.5
CWE-552
Published
Advisory Summary

GHSA-23j4-mw76-5v7h is a MODERATE (CVSS 6.5) scrapy allows redirect following vulnerability (CWE-552: Files or Directories Accessible to External Parties) affecting Scrapy libraries in the PyPI ecosystem. The issue involves scrapy allows redirect following in protocols other than http. Remediation is available by upgrading affected packages to version 2.11.2.

GHSA-23j4-mw76-5v7h at a Glance

CVE Identifier N/A
GHSA Identifier GHSA-23j4-mw76-5v7h
Severity MODERATE
CVSS v3.1 Base Score 6.5
CWE Classification CWE-552: Files or Directories Accessible to External Parties
Affected Product Scrapy
Package Ecosystem PyPI
Published Date
Last Updated

Security Impact

CVSS: 6.5

The security impact metrics below are parsed directly from the CVSS v3.1 vector string (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N).

Attack Vector Network (AV:N)
Attack Complexity Low (AC:L)
Privileges Required Low (PR:L)
User Interaction None (UI:N)
Scope Unchanged (S:U)
Confidentiality High (C:H)
Integrity None (I:N)
Availability None (A:N)

How to Fix GHSA-23j4-mw76-5v7h

To remediate this vulnerability, upgrade all affected Scrapy dependencies to a patched release. Both direct and transitive dependencies should be checked, as frameworks or security libraries may pull in vulnerable versions transitively.

Scrapy fixed: 2.11.2 
pip install Scrapy>=2.11.2
Verify Direct and Transitive Dependencies

Run the following command in your project directory to verify that all resolved dependencies use the patched release: 

pip show Scrapy

How to Check if You Are Affected

To determine whether your Python environment is affected by GHSA-23j4-mw76-5v7h, inspect your installed packages using pip:

pip show Scrapy

Analyze the terminal output to locate any version of Scrapy in your dependency tree. If any resolved version matches the vulnerable ranges listed in the table below (for example, versions earlier than 2.11.2), your application is affected.

Note: The pip show Scrapy command checks currently installed packages in your active Python environment.

Technical Details

Vulnerability Mechanism & Technical Analysis

This vulnerability is classified under CWE-552: Files or Directories Accessible to External Parties, where the product makes files or directories accessible to unauthorized actors, even though they should not be. In Scrapy, software state validation or input boundaries are circumvented during processing. Review the official source advisory statement below for specific affected components, classes, and reproduction details.

Source Advisory Statement

### Impact Scrapy was following redirects regardless of the URL protocol, so redirects were working for `data://`, `file://`, `ftp://`, `s3://`, and any other scheme defined in the `DOWNLOAD_HANDLERS` setting. However, HTTP redirects should only work between URLs that use the `http://` or `https://` schemes. A malicious actor, given write access to the start requests (e.g. ability to define `start_urls`) of a spider and read access to the spider output, could exploit this vulnerability to: - Redirect to any local file using the `file://` scheme to read its contents. - Redirect to an `ftp://` URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project. - Redirect to any `s3://` URL to read its content using the S3 credentials configured in the spider or project. For `file://` and `s3://`, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data. ### Patches Upgrade to Scrapy 2.11.2. ### Workarounds Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended. ### References This security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457.

GHSA-23j4-mw76-5v7h Affected Scrapy Versions

1 component(s)
Fixed Versions:
2.11.2
Product / Package Ecosystem Affected Versions Fixed Version
Scrapy
pypi/Scrapy
PyPI
< 2.11.2
2.11.2

References

GitHub Advisory / Wiki ADVISORY
https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h
Scrapy Source Repository REPOSITORY
https://github.com/scrapy/scrapy/issues/457
Security Fix Commit FIX COMMIT
https://github.com/scrapy/scrapy/commit/36287cb665ab4b0c65fd53181c9a0ef04990ada6
Scrapy Source Repository REPOSITORY
https://github.com/scrapy/scrapy

Frequently Asked Questions

What is GHSA-23j4-mw76-5v7h?
GHSA-23j4-mw76-5v7h is a security vulnerability in Scrapy involving Scrapy allows redirect following in protocols other than HTTP under CWE-552: Files or Directories Accessible to External Parties.
Which Scrapy versions are affected?
Affected versions include releases prior to 2.11.2 across affected packages: Scrapy.
What version fixes GHSA-23j4-mw76-5v7h?
Upgrading to 2.11.2 (or the latest patched release for your respective branch) resolves the vulnerability.
Is Scrapy affected?
Yes, Scrapy is affected in releases prior to 2.11.2. Upgrade your dependency to 2.11.2 or later.
What is the CVSS score of GHSA-23j4-mw76-5v7h?
GHSA-23j4-mw76-5v7h has a CVSS v3.1 base score of 6.5 (MODERATE) with vector string CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N.
What CWE is associated with GHSA-23j4-mw76-5v7h?
GHSA-23j4-mw76-5v7h is classified under CWE-552: Files or Directories Accessible to External Parties (The product makes files or directories accessible to unauthorized actors, even though they should not be.).
How can I check PyPI dependencies for GHSA-23j4-mw76-5v7h?
Run the dependency tree command: 'pip show Scrapy'. This command lists direct and transitive dependencies in your build configuration so you can verify if a vulnerable version is resolved.
Machine-readable OSV JSON OSV v1.5 Schema
Normalized schema output
View Raw JSON 
{
  "schema_version" : "1.4.0",
  "id" : "GHSA-23j4-mw76-5v7h",
  "modified" : "2024-05-14T20:14:50Z",
  "published" : "2024-05-14T20:14:49Z",
  "aliases" : [ ],
  "summary" : "Scrapy allows redirect following in protocols other than HTTP",
  "details" : "### Impact\n\nScrapy was following redirects regardless of the URL protocol, so redirects were working for `data://`, `file://`, `ftp://`, `s3://`, and any other scheme defined in the `DOWNLOAD_HANDLERS` setting.\n\nHowever, HTTP redirects should only work between URLs that use the `http://` or `https://` schemes.\n\nA malicious actor, given write access to the start requests (e.g. ability to define `start_urls`) of a spider and read access to the spider output, could exploit this vulnerability to:\n- Redirect to any local file using the `file://` scheme to read its contents.\n- Redirect to an `ftp://` URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project.\n- Redirect to any `s3://` URL to read its content using the S3 credentials configured in the spider or project.\n\nFor `file://` and `s3://`, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data.\n\n### Patches\n\nUpgrade to Scrapy 2.11.2.\n\n### Workarounds\n\nReplace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.\n\n### References\n\nThis security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457.\n",
  "severity" : [ {
    "type" : "CVSS_V3",
    "score" : "CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N"
  } ],
  "affected" : [ {
    "package" : {
      "ecosystem" : "PyPI",
      "name" : "Scrapy"
    },
    "ranges" : [ {
      "type" : "ECOSYSTEM",
      "events" : [ {
        "introduced" : "0"
      }, {
        "fixed" : "2.11.2"
      } ]
    } ]
  } ],
  "references" : [ {
    "type" : "WEB",
    "url" : "https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/scrapy/scrapy/issues/457"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/scrapy/scrapy/commit/36287cb665ab4b0c65fd53181c9a0ef04990ada6"
  }, {
    "type" : "PACKAGE",
    "url" : "https://github.com/scrapy/scrapy"
  } ],
  "database_specific" : {
    "cwe_ids" : [ "CWE-552" ],
    "severity" : "MODERATE",
    "github_reviewed" : true,
    "github_reviewed_at" : "2024-05-14T20:14:49Z",
    "nvd_published_at" : null
  }
}