Vulnerabilities in pkg:pypi/MechanicalSoup

PACKAGE
Advisory Identifier: GHSA-x456-3ccm-m6j4
Share
Advisory Summary

CVE-2023-34457 is a HIGH (CVSS 5.9) mechanicalsoup vulnerable to malicious web vulnerability (CWE-20: Improper Input Validation) affecting MechanicalSoup libraries in the PyPI ecosystem. The issue involves mechanicalsoup vulnerable to malicious web server reading arbitrary files on client using file input inside html form. Remediation is available by upgrading affected packages to version 1.3.0.

1 Known Advisory Record(s)
Package Vulnerability Intelligence

pkg:pypi/MechanicalSoup

1 Known Advisory Record(s)

Vulnerability advisories affecting package pkg:pypi/MechanicalSoup. Select an advisory below to view technical details or fixes.

Advisory ID CVE / NVD Severity Summary
GHSA-x456-3ccm-m6j4 CVE-2023-34457 HIGH MechanicalSoup vulnerable to malicious web server reading arbitrary files on client using file input inside HTML form

CVE-2023-34457 at a Glance

CVE Identifier CVE-2023-34457
GHSA Identifier GHSA-x456-3ccm-m6j4
Severity HIGH
CVSS v3.1 Base Score 5.9
CWE Classification CWE-20: Improper Input Validation
Affected Product MechanicalSoup
Package Ecosystem PyPI
Published Date
Last Updated

Security Impact

CVSS: 5.9

The security impact metrics below are parsed directly from the CVSS v3.1 vector string (CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N).

Attack Vector Network (AV:N)
Attack Complexity High (AC:H)
Privileges Required None (PR:N)
User Interaction None (UI:N)
Scope Unchanged (S:U)
Confidentiality High (C:H)
Integrity None (I:N)
Availability None (A:N)

How to Fix CVE-2023-34457

To remediate this vulnerability, upgrade all affected MechanicalSoup dependencies to a patched release. Both direct and transitive dependencies should be checked, as frameworks or security libraries may pull in vulnerable versions transitively.

MechanicalSoup fixed: 1.3.0 
pip install MechanicalSoup>=1.3.0
Verify Direct and Transitive Dependencies

Run the following command in your project directory to verify that all resolved dependencies use the patched release: 

pip show MechanicalSoup

How to Check if You Are Affected

To determine whether your Python environment is affected by CVE-2023-34457, inspect your installed packages using pip:

pip show MechanicalSoup

Analyze the terminal output to locate any version of MechanicalSoup in your dependency tree. If any resolved version matches the vulnerable ranges listed in the table below (for example, versions earlier than 1.3.0), your application is affected.

Note: The pip show MechanicalSoup command checks currently installed packages in your active Python environment.

Technical Details

Vulnerability Mechanism & Technical Analysis

This vulnerability is classified under CWE-20: Improper Input Validation, where the product receives input or data, but it does not validate or incorrectly validates that the input has the properties that are required to process the data safely and correctly. In MechanicalSoup, software state validation or input boundaries are circumvented during processing. Review the official source advisory statement below for specific affected components, classes, and reproduction details.

Source Advisory Statement

### Summary A malicious web server can read arbitrary files on the client using a `<input type="file" ...>` inside HTML form. ### Details This affects the extremely common pattern of form submission: ```python b = mechanicalsoup.StatefulBrowser() b.select_form(...) b.submit_selected() ``` The problem is with the code in `browser.Browser.get_request_kwargs`: ```python if tag.get("type", "").lower() == "file" and multipart: filepath = value if filepath != "" and isinstance(filepath, str): content = open(filepath, "rb") else: content = "" filename = os.path.basename(filepath) # If value is the empty string, we still pass it # for consistency with browsers (see # https://github.com/MechanicalSoup/MechanicalSoup/issues/250). files[name] = (filename, content) ``` The file path is taken from the bs4 tag "value" attribute. However, this path will default to whatever the server sends. So if a malicious web server were to send something like: ```html <html><body> <form method="post" enctype="multipart/form-data"> <input type="text" name="greeting" value="hello" /> <input type="file" name="evil" value="/home/user/.ssh/id_rsa" /> </form> </body></html> ``` then upon `.submit_selected()` the mechanicalsoup browser will happily send over the contents of your SSH private key. ### PoC ```python import attr import mechanicalsoup import requests class NevermindError(Exception): pass @attr.s class FakeSession: session = attr.ib() headers = property(lambda self: self.session.headers) def request(self, *args, **kwargs): print("requested", args, kwargs) raise NevermindError # don't actually send request def demonstrate(inputs=None): b = mechanicalsoup.StatefulBrowser(FakeSession(requests.Session())) b.open_fake_page("""\ <html><body> <form method="post" enctype="multipart/form-data"> <input type="text" name="greeting" value="hello" /> <input type="file" name="evil" value="/etc/passwd" /> <input type="file" name="second" /> </form> </body></html> """, url="http://127.0.0.1:9/") b.select_form() if inputs is not None: b.form.set_input(inputs) try: b.submit_selected() except NevermindError: pass # %% # unpatched demonstrate() # OUTPUT: requested () {'method': 'post', 'url': 'http://127.0.0.1:9/', 'files': {'evil': ('passwd', <_io.BufferedReader name='/etc/passwd'>), 'second': ('', '')}, 'headers': {'Referer': 'http://127.0.0.1:9/'}, 'data': [('greeting', 'hello')]} # %% # with the patch, this now works. users MUST open the file manually and # use browser.set_input() using the file object. demonstrate({"greeting": "hiya", "evil": open("/etc/hostname", "rb").name, "second": open("/dev/null", "rb")}) # OUTPUT: requested () {'method': 'post', 'url': 'http://127.0.0.1:9/', 'files': {'evil': ('hostname', <_io.BufferedReader name='/etc/hostname'>), 'second': ('null', <_io.BufferedReader name='/dev/null'>)}, 'headers': {'Referer': 'http://127.0.0.1:9/'}, 'data': [('greeting', 'hiya')]} # %% # with the patch, this raises a ValueError with a helpful string demonstrate({"evil": "/etc/hostname"}) # %% # with the patch, we silently send no file if a malicious server tries the attack: demonstrate() ``` ### Suggested patch ```diff diff --git a/mechanicalsoup/browser.py b/mechanicalsoup/browser.py index 285f8bb..68bc65e 100644 --- a/mechanicalsoup/browser.py +++ b/mechanicalsoup/browser.py @@ -1,7 +1,8 @@ +import io import os import tempfile import urllib import weakref import webbrowser import bs4 @@ -227,15 +228,21 @@ class Browser: value = tag.get("value", "") # If the enctype is not multipart, the filename is put in # the form as a text input and the file is not sent. if tag.get("type", "").lower() == "file" and multipart: filepath = value if filepath != "" and isinstance(filepath, str): - content = open(filepath, "rb") + content = getattr(tag, "_mechanicalsoup_file", None) + if content is False: + raise ValueError( + """From v1.3.0 onwards, you must pass an open file object directly, for example using `form.set_input({"name": open("/path/to/filename", "rb")})`. This change is to mitigate a security vulnerability where a malicious web server could read arbitrary files from the client.""" + ) + elif not isinstance(content, io.IOBase): + content = "" else: content = "" filename = os.path.basename(filepath) # If value is the empty string, we still pass it # for consistency with browsers (see # https://github.com/MechanicalSoup/MechanicalSoup/issues/250). files[name] = (filename, content) diff --git a/mechanicalsoup/form.py b/mechanicalsoup/form.py index a67195c..82f6015 100644 --- a/mechanicalsoup/form.py +++ b/mechanicalsoup/form.py @@ -1,8 +1,9 @@ import copy +import io import warnings from bs4 import BeautifulSoup from .utils import LinkNotFoundError @@ -64,15 +65,24 @@ class Form: give it the value ``password``. """ for (name, value) in data.items(): i = self.form.find("input", {"name": name}) if not i: raise InvalidFormMethod("No input field named " + name) - i["value"] = value + + if isinstance(value, io.IOBase): + # Store the actual file object for <input type="file"> + i._mechanicalsoup_file = value + i["value"] = value.name + else: + # We set `_mechanicalsoup_file` to `False` so that we can + # check for deprecated use of the API. + i._mechanicalsoup_file = False + i["value"] = value def uncheck_all(self, name): """Remove the *checked*-attribute of all input elements with a *name*-attribute given by ``name``. """ for option in self.form.find_all("input", {"name": name}): if "checked" in option.attrs: @@ -257,20 +267,20 @@ class Form: .. code-block:: python form.set("login", username) form.set("password", password) form.set("eula-checkbox", True) Example: uploading a file through a ``<input type="file" - name="tagname">`` field (provide the path to the local file, + name="tagname">`` field (provide an open file object, and its content will be uploaded): .. code-block:: python - form.set("tagname", path_to_local_file) + form.set("tagname", open(path_to_local_file, "rb")) """ for func in ("checkbox", "radio", "input", "textarea", "select"): try: getattr(self, "set_" + func)({name: value}) return except InvalidFormMethod: ``` ### Impact All users of MechanicalSoup's form submission are affected, unless they took very specific (and manual) steps to reset HTML form field values.

CVE-2023-34457 Affected MechanicalSoup Versions

1 component(s)
Fixed Versions:
1.3.0
Product / Package Ecosystem Affected Versions Fixed Version
MechanicalSoup
pypi/MechanicalSoup
PyPI
≥ 0.2.0
< 1.3.0
1.3.0

References

GitHub Advisory / Wiki ADVISORY
https://github.com/MechanicalSoup/MechanicalSoup/security/advisories/GHSA-x456-3ccm-m6j4
NVD — CVE-2023-34457 NVD
https://nvd.nist.gov/vuln/detail/CVE-2023-34457
Security Fix Commit FIX COMMIT
https://github.com/MechanicalSoup/MechanicalSoup/commit/d57c4a269bba3b9a0c5bfa20292955b849006d9e
MechanicalSoup Source Repository REPOSITORY
https://github.com/MechanicalSoup/MechanicalSoup
MechanicalSoup Release RELEASE
https://github.com/MechanicalSoup/MechanicalSoup/releases/tag/v1.3.0
MechanicalSoup Source Repository REPOSITORY
https://github.com/pypa/advisory-database/tree/main/vulns/mechanicalsoup/PYSEC-2023-108.yaml

Frequently Asked Questions

What is CVE-2023-34457?
CVE-2023-34457 is a security vulnerability in MechanicalSoup involving MechanicalSoup vulnerable to malicious web server reading arbitrary files on client using file input inside HTML form under CWE-20: Improper Input Validation.
Which MechanicalSoup versions are affected?
Affected versions include releases prior to 1.3.0 across affected packages: MechanicalSoup.
What version fixes CVE-2023-34457?
Upgrading to 1.3.0 (or the latest patched release for your respective branch) resolves the vulnerability.
Is MechanicalSoup affected?
Yes, MechanicalSoup is affected in releases prior to 1.3.0. Upgrade your dependency to 1.3.0 or later.
What is the CVSS score of CVE-2023-34457?
CVE-2023-34457 has a CVSS v3.1 base score of 5.9 (HIGH) with vector string CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N.
What CWE is associated with CVE-2023-34457?
CVE-2023-34457 is classified under CWE-20: Improper Input Validation (The product receives input or data, but it does not validate or incorrectly validates that the input has the properties that are required to process the data safely and correctly.).
How can I check PyPI dependencies for CVE-2023-34457?
Run the dependency tree command: 'pip show MechanicalSoup'. This command lists direct and transitive dependencies in your build configuration so you can verify if a vulnerable version is resolved.
Machine-readable OSV JSON OSV v1.5 Schema
Normalized schema output
View Raw JSON 
{
  "schema_version" : "1.4.0",
  "id" : "GHSA-x456-3ccm-m6j4",
  "modified" : "2024-10-01T19:29:06Z",
  "published" : "2023-07-05T21:35:54Z",
  "aliases" : [ "CVE-2023-34457" ],
  "summary" : "MechanicalSoup vulnerable to malicious web server reading arbitrary files on client using file input inside HTML form",
  "details" : "### Summary\nA malicious web server can read arbitrary files on the client using a `<input type=\"file\" ...>` inside HTML form.\n\n### Details\nThis affects the extremely common pattern of form submission:\n\n```python\nb = mechanicalsoup.StatefulBrowser()\nb.select_form(...)\nb.submit_selected()\n```\n\nThe problem is with the code in `browser.Browser.get_request_kwargs`:\n\n```python\n    if tag.get(\"type\", \"\").lower() == \"file\" and multipart:\n        filepath = value\n        if filepath != \"\" and isinstance(filepath, str):\n            content = open(filepath, \"rb\")\n        else:\n            content = \"\"\n        filename = os.path.basename(filepath)\n        # If value is the empty string, we still pass it\n        # for consistency with browsers (see\n        # https://github.com/MechanicalSoup/MechanicalSoup/issues/250).\n        files[name] = (filename, content)\n```\n\nThe file path is taken from the bs4 tag \"value\" attribute. However, this path will default to whatever the server sends. So if a malicious web server were to send something like:\n\n```html\n<html><body>\n  <form method=\"post\" enctype=\"multipart/form-data\">\n    <input type=\"text\" name=\"greeting\" value=\"hello\" />\n    <input type=\"file\" name=\"evil\" value=\"/home/user/.ssh/id_rsa\" />\n  </form>\n</body></html>\n```\n\nthen upon `.submit_selected()` the mechanicalsoup browser will happily send over the contents of your SSH private key.\n\n### PoC\n\n```python\nimport attr\nimport mechanicalsoup\nimport requests\n\n\nclass NevermindError(Exception):\n    pass\n\n\n@attr.s\nclass FakeSession:\n    session = attr.ib()\n\n    headers = property(lambda self: self.session.headers)\n\n    def request(self, *args, **kwargs):\n        print(\"requested\", args, kwargs)\n        raise NevermindError  # don't actually send request\n\n\ndef demonstrate(inputs=None):\n    b = mechanicalsoup.StatefulBrowser(FakeSession(requests.Session()))\n    b.open_fake_page(\"\"\"\\\n<html><body>\n<form method=\"post\" enctype=\"multipart/form-data\">\n<input type=\"text\" name=\"greeting\" value=\"hello\" />\n<input type=\"file\" name=\"evil\" value=\"/etc/passwd\" />\n<input type=\"file\" name=\"second\" />\n</form>\n</body></html>\n\"\"\", url=\"http://127.0.0.1:9/\")\n    b.select_form()\n    if inputs is not None:\n        b.form.set_input(inputs)\n    try:\n        b.submit_selected()\n    except NevermindError:\n        pass\n\n# %%\n\n# unpatched\ndemonstrate()\n# OUTPUT: requested () {'method': 'post', 'url': 'http://127.0.0.1:9/', 'files': {'evil': ('passwd', <_io.BufferedReader name='/etc/passwd'>), 'second': ('', '')}, 'headers': {'Referer': 'http://127.0.0.1:9/'}, 'data': [('greeting', 'hello')]}\n\n# %%\n\n# with the patch, this now works. users MUST open the file manually and\n# use browser.set_input() using the file object.\ndemonstrate({\"greeting\": \"hiya\", \"evil\": open(\"/etc/hostname\", \"rb\").name, \"second\": open(\"/dev/null\", \"rb\")})\n# OUTPUT: requested () {'method': 'post', 'url': 'http://127.0.0.1:9/', 'files': {'evil': ('hostname', <_io.BufferedReader name='/etc/hostname'>), 'second': ('null', <_io.BufferedReader name='/dev/null'>)}, 'headers': {'Referer': 'http://127.0.0.1:9/'}, 'data': [('greeting', 'hiya')]}\n\n# %%\n\n# with the patch, this raises a ValueError with a helpful string\ndemonstrate({\"evil\": \"/etc/hostname\"})\n\n# %%\n\n# with the patch, we silently send no file if a malicious server tries the attack:\ndemonstrate()\n```\n\n### Suggested patch\n\n```diff\ndiff --git a/mechanicalsoup/browser.py b/mechanicalsoup/browser.py\nindex 285f8bb..68bc65e 100644\n--- a/mechanicalsoup/browser.py\n+++ b/mechanicalsoup/browser.py\n@@ -1,7 +1,8 @@\n+import io\n import os\n import tempfile\n import urllib\n import weakref\n import webbrowser\n \n import bs4\n@@ -227,15 +228,21 @@ class Browser:\n                     value = tag.get(\"value\", \"\")\n \n                 # If the enctype is not multipart, the filename is put in\n                 # the form as a text input and the file is not sent.\n                 if tag.get(\"type\", \"\").lower() == \"file\" and multipart:\n                     filepath = value\n                     if filepath != \"\" and isinstance(filepath, str):\n-                        content = open(filepath, \"rb\")\n+                        content = getattr(tag, \"_mechanicalsoup_file\", None)\n+                        if content is False:\n+                            raise ValueError(\n+                                \"\"\"From v1.3.0 onwards, you must pass an open file object directly, for example using `form.set_input({\"name\": open(\"/path/to/filename\", \"rb\")})`. This change is to mitigate a security vulnerability where a malicious web server could read arbitrary files from the client.\"\"\"\n+                            )\n+                        elif not isinstance(content, io.IOBase):\n+                            content = \"\"\n                     else:\n                         content = \"\"\n                     filename = os.path.basename(filepath)\n                     # If value is the empty string, we still pass it\n                     # for consistency with browsers (see\n                     # https://github.com/MechanicalSoup/MechanicalSoup/issues/250).\n                     files[name] = (filename, content)\ndiff --git a/mechanicalsoup/form.py b/mechanicalsoup/form.py\nindex a67195c..82f6015 100644\n--- a/mechanicalsoup/form.py\n+++ b/mechanicalsoup/form.py\n@@ -1,8 +1,9 @@\n import copy\n+import io\n import warnings\n \n from bs4 import BeautifulSoup\n \n from .utils import LinkNotFoundError\n \n \n@@ -64,15 +65,24 @@ class Form:\n         give it the value ``password``.\n         \"\"\"\n \n         for (name, value) in data.items():\n             i = self.form.find(\"input\", {\"name\": name})\n             if not i:\n                 raise InvalidFormMethod(\"No input field named \" + name)\n-            i[\"value\"] = value\n+\n+            if isinstance(value, io.IOBase):\n+                # Store the actual file object for <input type=\"file\">\n+                i._mechanicalsoup_file = value\n+                i[\"value\"] = value.name\n+            else:\n+                # We set `_mechanicalsoup_file` to `False` so that we can\n+                # check for deprecated use of the API.\n+                i._mechanicalsoup_file = False\n+                i[\"value\"] = value\n \n     def uncheck_all(self, name):\n         \"\"\"Remove the *checked*-attribute of all input elements with\n         a *name*-attribute given by ``name``.\n         \"\"\"\n         for option in self.form.find_all(\"input\", {\"name\": name}):\n             if \"checked\" in option.attrs:\n@@ -257,20 +267,20 @@ class Form:\n         .. code-block:: python\n \n             form.set(\"login\", username)\n             form.set(\"password\", password)\n             form.set(\"eula-checkbox\", True)\n \n         Example: uploading a file through a ``<input type=\"file\"\n-        name=\"tagname\">`` field (provide the path to the local file,\n+        name=\"tagname\">`` field (provide an open file object,\n         and its content will be uploaded):\n \n         .. code-block:: python\n \n-            form.set(\"tagname\", path_to_local_file)\n+            form.set(\"tagname\", open(path_to_local_file, \"rb\"))\n \n         \"\"\"\n         for func in (\"checkbox\", \"radio\", \"input\", \"textarea\", \"select\"):\n             try:\n                 getattr(self, \"set_\" + func)({name: value})\n                 return\n             except InvalidFormMethod:\n```\n\n### Impact\n\nAll users of MechanicalSoup's form submission are affected, unless they took very specific (and manual) steps to reset HTML form field values.",
  "severity" : [ {
    "type" : "CVSS_V3",
    "score" : "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N"
  }, {
    "type" : "CVSS_V4",
    "score" : "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N"
  } ],
  "affected" : [ {
    "package" : {
      "ecosystem" : "PyPI",
      "name" : "MechanicalSoup"
    },
    "ranges" : [ {
      "type" : "ECOSYSTEM",
      "events" : [ {
        "introduced" : "0.2.0"
      }, {
        "fixed" : "1.3.0"
      } ]
    } ]
  } ],
  "references" : [ {
    "type" : "WEB",
    "url" : "https://github.com/MechanicalSoup/MechanicalSoup/security/advisories/GHSA-x456-3ccm-m6j4"
  }, {
    "type" : "ADVISORY",
    "url" : "https://nvd.nist.gov/vuln/detail/CVE-2023-34457"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/MechanicalSoup/MechanicalSoup/commit/d57c4a269bba3b9a0c5bfa20292955b849006d9e"
  }, {
    "type" : "PACKAGE",
    "url" : "https://github.com/MechanicalSoup/MechanicalSoup"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/MechanicalSoup/MechanicalSoup/releases/tag/v1.3.0"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/pypa/advisory-database/tree/main/vulns/mechanicalsoup/PYSEC-2023-108.yaml"
  } ],
  "database_specific" : {
    "cwe_ids" : [ "CWE-20" ],
    "severity" : "HIGH",
    "github_reviewed" : true,
    "github_reviewed_at" : "2023-07-05T21:35:54Z",
    "nvd_published_at" : "2023-07-05T20:15:10Z"
  }
}