CVE-2021-43854: Nltk Inefficient Regular Expression Complexity Vulnerability

{
  "schema_version" : "1.4.0",
  "id" : "GHSA-f8m6-h2c7-8h9x",
  "modified" : "2024-09-26T14:17:15Z",
  "published" : "2022-01-06T17:38:45Z",
  "aliases" : [ "CVE-2021-43854" ],
  "summary" : "Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)",
  "details" : "### Impact\nThe vulnerability is present in [`PunktSentenceTokenizer`](https://www.nltk.org/api/nltk.tokenize.punkt.html#nltk.tokenize.punkt.PunktSentenceTokenizer), [`sent_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize)  and [`word_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize). Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack. \nIn short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:\n```python\nfrom nltk.tokenize import word_tokenize\n\nn = 8\nfor length in [10**i for i in range(2, n)]:\n    # Prepare a malicious input\n    text = \"a\" * length\n    start_t = time.time()\n    # Call `word_tokenize` and naively measure the execution time\n    word_tokenize(text)\n    print(f\"A length of {length:<{n}} takes {time.time() - start_t:.4f}s\")\n```\nWhich gave the following output during testing:\n```python\nA length of 100      takes 0.0060s\nA length of 1000     takes 0.0060s\nA length of 10000    takes 0.6320s\nA length of 100000   takes 56.3322s\n...\n```\nI canceled the execution of the program after running it for several hours.\n\nIf your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.\n\n### Patches\nThe problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:\n```python\nA length of 100      takes 0.0070s\nA length of 1000     takes 0.0010s\nA length of 10000    takes 0.0060s\nA length of 100000   takes 0.0400s\nA length of 1000000  takes 0.3520s\nA length of 10000000 takes 3.4641s\n```\nThis output shows a linear relationship in execution time versus input length, which is desirable for regular expressions.\nWe recommend updating to NLTK 3.6.6+ if possible.\n\n### Workarounds\nThe execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.\n\n### References\n* The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866\n* The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869\n* The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341\n* Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in [github.com/nltk/nltk](https://github.com/nltk/nltk)\n* Email us at [nltk.team@gmail.com](mailto:nltk.team@gmail.com)\n",
  "severity" : [ {
    "type" : "CVSS_V3",
    "score" : "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H"
  }, {
    "type" : "CVSS_V4",
    "score" : "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N"
  } ],
  "affected" : [ {
    "package" : {
      "ecosystem" : "PyPI",
      "name" : "nltk"
    },
    "ranges" : [ {
      "type" : "ECOSYSTEM",
      "events" : [ {
        "introduced" : "0"
      }, {
        "fixed" : "3.6.6"
      } ]
    } ]
  } ],
  "references" : [ {
    "type" : "WEB",
    "url" : "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
  }, {
    "type" : "ADVISORY",
    "url" : "https://nvd.nist.gov/vuln/detail/CVE-2021-43854"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/nltk/nltk/issues/2866"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/nltk/nltk/pull/2869"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
  }, {
    "type" : "PACKAGE",
    "url" : "https://github.com/nltk/nltk"
  }, {
    "type" : "WEB",
    "url" : "https://github.com/pypa/advisory-database/tree/main/vulns/nltk/PYSEC-2021-859.yaml"
  } ],
  "database_specific" : {
    "cwe_ids" : [ "CWE-400" ],
    "severity" : "HIGH",
    "github_reviewed" : true,
    "github_reviewed_at" : "2022-01-05T17:40:20Z",
    "nvd_published_at" : "2021-12-23T18:15:00Z"
  }
}