Investigate Definition and Notice Discrepancies in Integration Tests

[qtomlinson]

After updating ScanCode to the latest version (see PR clearlydefined/crawler#650), our integration tests are showing discrepancies when comparing definitions and notices between dev and prod deployment. These differences need to be investigated to determine whether they are valid improvements introduced by the update or potential issues.

If any of the changes are legitimate enhancements, we should update the relevant test fixtures to reflect them.

For reference, the integration test log is available here

[qtomlinson]

@jamesiri @jeffmendoza As per our dev sync meeting minutes
@yashkohli88 is working on this.

[yashkohli88]

1. pypi/pypi/-/platformdirs/4.2.0 :

Description:

• [Improvement] There is an update in discovered score, hence this test case failed

+ actual - expected

  {
    declared: 'MIT',
    score: {
      consistency: 15,
      declared: 30,
+     discovered: 1,
-     discovered: 0,
      spdx: 15,
      texts: 15,
+     total: 76
-     total: 75
    },
    toolScore: {
      consistency: 15,
      declared: 30,
+     discovered: 1,
-     discovered: 0,
      spdx: 15,
      texts: 15,
+     total: 76
-     total: 75
    }
  }

• [Improvement] A new attribution is detected in the newer version, hence the notice difference is observed. The new notice seems more appropriate

+ actual - expected

{
    content: '** platformdirs; version 4.2.0 -- \n' +
+     'Copyright (c) 2010-202x The platformdirs developers\n' +
      '\n' +
      'MIT License\n' +
      '\n' +
      'Copyright (c) <year> <copyright holders>\n' +
      '\n' +
...
      warnings: {
+       noCopyright: [],
-       noCopyright: [
-         'pypi/pypi/-/platformdirs/4.2.0'
-       ],
        noDefinition: [],
        noLicense: []
      }
    }
  }

+ expected - actual

{
      -  "content": "** platformdirs; version 4.2.0 -- \nCopyright (c) 2010-202x The platformdirs developers\n\nMIT License\n\nCopyright (c) <year> <copyright holders>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
      +  "content": "** platformdirs; version 4.2.0 -- \n\nMIT License\n\nCopyright (c) <year> <copyright holders>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
         "summary": {
           "total": 1
           "warnings": {
      -      "noCopyright": []
      +      "noCopyright": [
      +        "pypi/pypi/-/platformdirs/4.2.0"
      +      ]
             "noDefinition": []
             "noLicense": []
           }
         }

[yashkohli88]

2) conda/conda-forge/linux-aarch64/numpy/1.16.6-py36hdc1b780_0

Description:

Calling fetch. URL: https://dev-api.clearlydefined.io/definitions/conda/conda-forge/linux-aarch64/numpy/1.16.6-py36hdc1b780_0?force=true, Options: {}
Calling fetch. URL: https://api.clearlydefined.io/definitions/conda/conda-forge/linux-aarch64/numpy/1.16.6-py36hdc1b780_0, Options: {}

• [Improvement] Some weird attribution information is expected with true false value

expected: {"path":"lib/python3.6/site-packages/numpy/core/tests/test_half.py","attributions":["(c), True, True, False, False","(c), False, False, False, True","(c), False, False, True, False"],"hashes":{"sha1":"cff768a134d3fc5cd733dd5a409818dde226d865","sha256":"1e7963ed3e3e900293fb680f6bbfaf36b6ce389eaa7d9a1aab48d6e026c2b51b"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/core/tests/test_half.py","hashes":{"sha1":"cff768a134d3fc5cd733dd5a409818dde226d865","sha256":"1e7963ed3e3e900293fb680f6bbfaf36b6ce389eaa7d9a1aab48d6e026c2b51b"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/distutils/cpuinfo.py","license":"BSD-3-Clause","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"d4175e6d6804f6ef973f4a655d2e10499d239033","sha256":"00726e41e83bf3f3f911178ed642ddf8c02886f07e19f57ccee461ec5f2196d2"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/distutils/cpuinfo.py","license":"BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0)","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"d4175e6d6804f6ef973f4a655d2e10499d239033","sha256":"00726e41e83bf3f3f911178ed642ddf8c02886f07e19f57ccee461ec5f2196d2"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/distutils/system_info.py","license":"BSD-3-Clause","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"6d9312bc92cffc776562f68cff7f633ed16535f6","sha256":"0ddbee32e46a4d6732fefa56ac13454fc2441e77b9e23568b7c79e9adbaf5afd"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/distutils/system_info.py","license":"BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0)","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"6d9312bc92cffc776562f68cff7f633ed16535f6","sha256":"0ddbee32e46a4d6732fefa56ac13454fc2441e77b9e23568b7c79e9adbaf5afd"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/distutils/__pycache__/cpuinfo.cpython-36.pyc","license":"BSD-3-Clause","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"81395ee6ca9caf0af52fc86019d98876920f4497","sha256":"48a28a5ebc93b9177a704abdbb9568bcf4c54869675b625c56e518d520714b8a"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/distutils/__pycache__/cpuinfo.cpython-36.pyc","license":"BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0)","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"81395ee6ca9caf0af52fc86019d98876920f4497","sha256":"48a28a5ebc93b9177a704abdbb9568bcf4c54869675b625c56e518d520714b8a"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc","license":"BSD-3-Clause","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"ac8f2a6ea0edc691f5dc1a569a4fd73fbe9d7fb7","sha256":"6026086fd92cd938cb4a3336ca195c381e4b2815f33c5775b7d1e3a9c13d06a0"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc","license":"BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0)","attributions":["Copyright 2002 Pearu Peterson"],"hashes":{"sha1":"ac8f2a6ea0edc691f5dc1a569a4fd73fbe9d7fb7","sha256":"6026086fd92cd938cb4a3336ca195c381e4b2815f33c5775b7d1e3a9c13d06a0"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/f2py/f2py2e.py","license":"BSD-3-Clause AND LicenseRef-scancode-unknown-license-reference","attributions":["Copyright 1999-2011 Pearu Peterson","Copyright 1999 - 2011 Pearu Peterson"],"hashes":{"sha1":"a6c6f2bbc8cd3bed85610cf122cd6264c949dae3","sha256":"c3dcd2246ded9c23323ab81926a8598845280279c8ee853ad64619cefb0b75fa"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/f2py/f2py2e.py","license":"BSD-3-Clause AND (BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0))","attributions":["Copyright 1999-2011 Pearu Peterson","Copyright 1999 - 2011 Pearu Peterson"],"hashes":{"sha1":"a6c6f2bbc8cd3bed85610cf122cd6264c949dae3","sha256":"c3dcd2246ded9c23323ab81926a8598845280279c8ee853ad64619cefb0b75fa"}}

• [Degradation ] Wrong detection in new version for licenses - Apache and Python license. Also both BSD-2 clause and BSD-3 clause reported.

expected: {"path":"lib/python3.6/site-packages/numpy/f2py/__pycache__/f2py2e.cpython-36.pyc","license":"BSD-3-Clause AND LicenseRef-scancode-unknown-license-reference","attributions":["Copyright 1999-2011 Pearu Peterson","Copyright 1999 - 2011 Pearu Peterson"],"hashes":{"sha1":"13f8ab8f760195b5599f66f4be8c8381f68ecad8","sha256":"50297551bfc28e1e9d91879accc23544a05b2446f2f121ee32dc30acc87a8fa0"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/f2py/__pycache__/f2py2e.cpython-36.pyc","license":"BSD-3-Clause AND (BSD-3-Clause AND BSD-2-Clause AND (BSD-3-Clause AND Python-2.0 AND Apache-2.0))","attributions":["Copyright 1999-2011 Pearu Peterson","Copyright 1999 - 2011 Pearu Peterson"],"hashes":{"sha1":"13f8ab8f760195b5599f66f4be8c8381f68ecad8","sha256":"50297551bfc28e1e9d91879accc23544a05b2446f2f121ee32dc30acc87a8fa0"}}

• [Improvement] Extra attribution Multiply a Chebyshev found in expected section. However both the attribution seems wrong.

expected: {"path":"lib/python3.6/site-packages/numpy/polynomial/chebyshev.py","attributions":["(c) Covert Chebyshev","(c) Multiply a Chebyshev"],"hashes":{"sha1":"72ab35fa036cad544b0e22320fb5e5b5ffc477fe","sha256":"efd85c7dda941e002d0687cde196474e438937361e275d8ab028199b8ce783a2"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/polynomial/chebyshev.py","attributions":["(c) Covert Chebyshev"],"hashes":{"sha1":"72ab35fa036cad544b0e22320fb5e5b5ffc477fe","sha256":"efd85c7dda941e002d0687cde196474e438937361e275d8ab028199b8ce783a2"}}

• [Improvement] Wrong attribution removed in new version.

expected: {"path":"lib/python3.6/site-packages/numpy/polynomial/hermite.py","attributions":["(c) Multiply a Hermite"],"hashes":{"sha1":"f660bb56c01afc26905419839e73d6b2ae53f899","sha256":"d781484d818c0d0cd92ec008bba4f021e4bf121e0eddd4e2cd71d19952dd7402"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/polynomial/hermite.py","hashes":{"sha1":"f660bb56c01afc26905419839e73d6b2ae53f899","sha256":"d781484d818c0d0cd92ec008bba4f021e4bf121e0eddd4e2cd71d19952dd7402"}}

• [Improvement] Wrong attribution removed in new version.

expected: {"path":"lib/python3.6/site-packages/numpy/polynomial/hermite_e.py","attributions":["(c) Multiply a Hermite"],"hashes":{"sha1":"bc3ccb79998e46b5250ded3432fe0af692420121","sha256":"36e5e6f25ad1e6cf32962fdb51abaf814b17d29d3233e130b6f363eacace6ed5"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/polynomial/hermite_e.py","hashes":{"sha1":"bc3ccb79998e46b5250ded3432fe0af692420121","sha256":"36e5e6f25ad1e6cf32962fdb51abaf814b17d29d3233e130b6f363eacace6ed5"}}

• [Improvement] Wrong attribution removed in new version.

expected: {"path":"lib/python3.6/site-packages/numpy/polynomial/laguerre.py","attributions":["(c) Multiply a Laguerre"],"hashes":{"sha1":"87042e0d4b3f901ad3c36c0cdaf60e0a6302893e","sha256":"a3766f8148bd8afa561aa3c893a84200672cf0ff669630313f9e285e941f6223"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/polynomial/laguerre.py","hashes":{"sha1":"87042e0d4b3f901ad3c36c0cdaf60e0a6302893e","sha256":"a3766f8148bd8afa561aa3c893a84200672cf0ff669630313f9e285e941f6223"}}

• [Improvement] Wrong attribution removed in new version.

expected: {"path":"lib/python3.6/site-packages/numpy/polynomial/legendre.py","attributions":["(c) Multiply a Legendre"],"hashes":{"sha1":"268166c5c4f9f5a4cf34cd0d7f200555af64385c","sha256":"52f84e9dee92caf04a8f90c8606c9f0fd6eefd170ca800d8e5ecdb56484147cb"}}
actual:   {"path":"lib/python3.6/site-packages/numpy/polynomial/legendre.py","hashes":{"sha1":"268166c5c4f9f5a4cf34cd0d7f200555af64385c","sha256":"52f84e9dee92caf04a8f90c8606c9f0fd6eefd170ca800d8e5ecdb56484147cb"}}

• [Improvement] Wrong attribution removed in new version in notice test case.

+ actual - expected

{
   content: '** linux-aarch64/numpy; version 1.16.6-py36hdc1b780_0 -- [http://numpy.org/\n](http://numpy.org//n)' +
     '(c) Covert Chebyshev\n' +
-     '(c) Multiply a Hermite\n' +
-     '(c) Multiply a Laguerre\n' +
-     '(c) Multiply a Legendre\n' +
-     '(c) Multiply a Chebyshev\n' +
     'Copyright Absoft Corporation\n' +
-     '(c), True, True, False, False\n' +
     'Copyright 2000 Pearu Peterson\n' +
     'Copyright 2002 Pearu Peterson\n' +
-     '(c), False, False, False, True\n' +
-     '(c), False, False, True, False\n' +
     'Copyright 2006 by Michael Spencer\n' +
     'Copyright 1999,2000 Pearu Peterson\n' +
     'Copyright 1999-2004 Pearu Peterson\n' +
     'Copyright 1999-2011 Pearu Peterson\n' +
     'Copyright 2001-2005 Pearu Peterson\n' +

+ expected - actual

       {
      -  "content": "** linux-aarch64/numpy; version 1.16.6-py36hdc1b780_0 -- [http://numpy.org/\n(c)](http://numpy.org//n(c)) Covert Chebyshev\nCopyright Absoft Corporation\nCopyright 2000 Pearu Peterson\nCopyright 2002 Pearu Peterson\nCopyright 2006 by Michael Spencer\nCopyright 1999,2000 Pearu Peterson\nCopyright 1999-2004 Pearu Peterson\nCopyright 1999-2011 Pearu Peterson\nCopyright 2001-2005 Pearu Peterson\nCopyright 1999 - 2011 Pearu Peterson\nCopyright (c) 2005-2019, NumPy Developers\ncopyright 2010 David Wolever <david@wolever.net>\nCopyright Absoft Corporation 1994-2002 Absoft Pro FORTRAN\nCopyright (c) 2003-2005, Jean-Sebastien Roy (js@jeannot.org)\nCopyright Absoft Corporation 1994-1998 mV2 Cray Research, Inc.\nCopyright 1999, 2000, 2001 Regents of the University of California\n\nCopyright (c) <year> <owner> . All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n   1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n   2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\n   3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
      +  "content": "** linux-aarch64/numpy; version 1.16.6-py36hdc1b780_0 -- [http://numpy.org/\n(c)](http://numpy.org//n(c)) Covert Chebyshev\n(c) Multiply a Hermite\n(c) Multiply a Laguerre\n(c) Multiply a Legendre\n(c) Multiply a Chebyshev\nCopyright Absoft Corporation\n(c), True, True, False, False\nCopyright 2000 Pearu Peterson\nCopyright 2002 Pearu Peterson\n(c), False, False, False, True\n(c), False, False, True, False\nCopyright 2006 by Michael Spencer\nCopyright 1999,2000 Pearu Peterson\nCopyright 1999-2004 Pearu Peterson\nCopyright 1999-2011 Pearu Peterson\nCopyright 2001-2005 Pearu Peterson\nCopyright 1999 - 2011 Pearu Peterson\nCopyright (c) 2005-2019, NumPy Developers\ncopyright 2010 David Wolever <david@wolever.net>\nCopyright Absoft Corporation 1994-2002 Absoft Pro FORTRAN\nCopyright (c) 2003-2005, Jean-Sebastien Roy (js@jeannot.org)\nCopyright Absoft Corporation 1994-1998 mV2 Cray Research, Inc.\nCopyright 1999, 2000, 2001 Regents of the University of California\n\nCopyright (c) <year> <owner> . All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n   1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n   2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\n   3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
         "summary": {
           "total": 1
           "warnings": {
             "noCopyright": []

[yashkohli88]

3. pypi/pypi/-/sdbus/0.12.0

Description:

Calling fetch. URL: https://dev-api.clearlydefined.io/definitions/pypi/pypi/-/sdbus/0.12.0?force=true, Options: {}
Calling fetch. URL: https://api.clearlydefined.io/definitions/pypi/pypi/-/sdbus/0.12.0, Options: {}

• [No changes] The sequence for the license changed. Python-2.0 and LGPL-2.1-or-later interchanged.

expected: {"path":"sdbus-0.12.0/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (LGPL-2.1-or-later AND Python-2.0 AND LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}
actual:   {"path":"sdbus-0.12.0/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (Python-2.0 AND LGPL-2.1-only AND LGPL-2.1-or-later AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}

• [No changes] The sequence for the license changed. Python-2.0 and LGPL-2.1-or-later interchanged.

expected: {"path":"sdbus-0.12.0/README.md","license":"LGPL-2.1-or-later AND Python-2.0 AND LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-1.0-or-later","hashes":{"sha1":"4d37c82e9b3469c0fa4403e8a458769bb3af0356","sha256":"28fb01f8b137a0b8a1e67574461017739f3db6136e84bf03cc294418f6930056"}}
actual:   {"path":"sdbus-0.12.0/README.md","license":"Python-2.0 AND LGPL-2.1-only AND LGPL-2.1-or-later AND LGPL-2.0-or-later AND GPL-1.0-or-later","hashes":{"sha1":"4d37c82e9b3469c0fa4403e8a458769bb3af0356","sha256":"28fb01f8b137a0b8a1e67574461017739f3db6136e84bf03cc294418f6930056"}}

• [No changes] The sequence for the license changed. Python-2.0 and LGPL-2.1-or-later interchanged.

expected: {"path":"sdbus-0.12.0/sdbus.egg-info/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (LGPL-2.1-or-later AND Python-2.0 AND LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}
actual:   {"path":"sdbus-0.12.0/sdbus.egg-info/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (Python-2.0 AND LGPL-2.1-only AND LGPL-2.1-or-later AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}

[yashkohli88]

3. nuget/nuget/-/NuGet.Protocol/6.7.1

Description:

Calling fetch. URL: https://dev-api.clearlydefined.io/definitions/nuget/nuget/-/NuGet.Protocol/6.7.1?force=true, Options: {}
Calling fetch. URL: https://api.clearlydefined.io/definitions/nuget/nuget/-/NuGet.Protocol/6.7.1, Options: {}

• [No changes] The sequence for the license changed. ECL-2.0 and Apache-2.0 interchanged.

expected: {"path":"clearlydefined/downloaded/LICENSE","license":"Apache-2.0 AND (ECL-2.0 AND Apache-2.0)","hashes":{"sha1":"39abfb1ed211fe9a9fdef056126133c7eb444f0d","sha256":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"},"token":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"}
actual:   {"path":"clearlydefined/downloaded/LICENSE","license":"(ECL-2.0 AND Apache-2.0) AND Apache-2.0","hashes":{"sha1":"39abfb1ed211fe9a9fdef056126133c7eb444f0d","sha256":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"},"token":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"}

[yashkohli88]

4. deb/debian/-/mini-httpd/1.30-13_armhf

Description:

Calling fetch. URL: https://dev-api.clearlydefined.io/definitions/deb/debian/-/mini-httpd/1.30-13_armhf?force=true, Options: {}
Calling fetch. URL: https://api.clearlydefined.io/definitions/deb/debian/-/mini-httpd/1.30-13_armhf, Options: {}

• [Degradation] The expected result is more clean. The new results contains redundant BSD-2-clause in separate expression. Also there is an extra LicenseRef-scancode-unknown-license-reference present in the new result.

expected: {"path":"data/usr/share/doc/mini-httpd/copyright","license":"BSD-2-Clause AND LicenseRef-scancode-public-domain AND LicenseRef-scancode-warranty-disclaimer","attributions":["Copyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>","Copyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose","Copyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>"],"hashes":{"sha1":"295f4509539e8e9880244d9a3c81c71b585cba3b","sha256":"74beb76024fb9775fd2c79fc7611205e0416941da7f5917e6292ad6fce7bb7a4"}}
actual:   {"path":"data/usr/share/doc/mini-httpd/copyright","license":"BSD-2-Clause AND (BSD-2-Clause AND LicenseRef-scancode-unknown-license-reference AND LicenseRef-scancode-public-domain) AND LicenseRef-scancode-warranty-disclaimer","attributions":["Copyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>","Copyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose","Copyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>"],"hashes":{"sha1":"295f4509539e8e9880244d9a3c81c71b585cba3b","sha256":"74beb76024fb9775fd2c79fc7611205e0416941da7f5917e6292ad6fce7bb7a4"}}

[yashkohli88]

5. debsrc/debian/-/mini-httpd/1.30-13

Description:

Calling fetch. URL: https://dev-api.clearlydefined.io/definitions/debsrc/debian/-/mini-httpd/1.30-13?force=true, Options: {}
Calling fetch. URL: https://api.clearlydefined.io/definitions/debsrc/debian/-/mini-httpd/1.30-13, Options: {}

• [Degradation] The expected result is more clean. The new results contains redundant BSD-2-clause in separate expression. Also there is an extra 'LicenseRef-scancode-unknown-license-reference' present in the new result.

expected: {"path":"debian/copyright","license":"BSD-2-Clause AND LicenseRef-scancode-public-domain AND LicenseRef-scancode-warranty-disclaimer","attributions":["Copyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>","Copyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose","Copyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>"],"hashes":{"sha1":"295f4509539e8e9880244d9a3c81c71b585cba3b","sha256":"74beb76024fb9775fd2c79fc7611205e0416941da7f5917e6292ad6fce7bb7a4"}}
actual:   {"path":"debian/copyright","license":"BSD-2-Clause AND (BSD-2-Clause AND LicenseRef-scancode-unknown-license-reference AND LicenseRef-scancode-public-domain) AND LicenseRef-scancode-warranty-disclaimer","attributions":["Copyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>","Copyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose","Copyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>"],"hashes":{"sha1":"295f4509539e8e9880244d9a3c81c71b585cba3b","sha256":"74beb76024fb9775fd2c79fc7611205e0416941da7f5917e6292ad6fce7bb7a4"}}

• [No change] Attribution info is different. Expected is Jef Poskanzer and actual is Jef Poskanzer jef at mail.acme.com

expected: {"path":"mini_httpd-1.30/mini_httpd.8","license":"BSD-2-Clause","attributions":["Copyright co 1999,2000 by Jef Poskanzer <jef at mail.acme.com>"],"hashes":{"sha1":"e60236922c047568aff16323a2864ee9e5113223","sha256":"df7a9ca348a6c278805bb0271ba4b3e02a55dc6b10f6dcb5bab1fc9ee3b33be6"}}
actual:   {"path":"mini_httpd-1.30/mini_httpd.8","license":"BSD-2-Clause","attributions":["Copyright co 1999,2000 by Jef Poskanzer jef at mail.acme.com"],"hashes":{"sha1":"e60236922c047568aff16323a2864ee9e5113223","sha256":"df7a9ca348a6c278805bb0271ba4b3e02a55dc6b10f6dcb5bab1fc9ee3b33be6"}}

• [No change] _ Notice test case failed as Attribution info is slightly different. Expected is Jef Poskanzer and actual is Jef Poskanzer jef at mail.acme.com_

+ actual - expected

  {
    content: '** mini-httpd; version 1.30-13 -- \n' +
      '(c) Regents of Univ. of California\n' +
      'Copyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>\n' +
      'Copyright (c) 1995 by Jef Poskanzer <jef@mail.acme.com>\n' +
      'Copyright (c) 1995,2000 by Jef Poskanzer <jef@mail.acme.com>\n' +
      'Copyright (c) 1999,2000 by Jef Poskanzer <jef@mail.acme.com>\n' +
+     'Copyright co 1999,2000 by Jef Poskanzer jef at mail.acme.com\n' +
      'Copyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose\n' +
-     'Copyright co 1999,2000 by Jef Poskanzer <jef at mail.acme.com>\n' +
      'Copyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>\n' +
      'Copyright Freely Redistributable Packager Bennett Todd <bet@mordor.net> Group Networking/Daemons\n' +
      '\n' +
      'Copyright (c) <year> <owner> . All rights reserved.\n' +
      '\n' +

+ expected - actual

 {
      -  "content": "** mini-httpd; version 1.30-13 -- \n(c) Regents of Univ. of California\nCopyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1995 by Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1995,2000 by Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1999,2000 by Jef Poskanzer <jef@mail.acme.com>\nCopyright co 1999,2000 by Jef Poskanzer jef at mail.acme.com\nCopyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose\nCopyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>\nCopyright Freely Redistributable Packager Bennett Todd <bet@mordor.net> Group Networking/Daemons\n\nCopyright (c) <year> <owner> . All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n   1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n   2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
      +  "content": "** mini-httpd; version 1.30-13 -- \n(c) Regents of Univ. of California\nCopyright 1999-2000 Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1995 by Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1995,2000 by Jef Poskanzer <jef@mail.acme.com>\nCopyright (c) 1999,2000 by Jef Poskanzer <jef@mail.acme.com>\nCopyright 2006-2015 Marvin Stark <marv@der-marv.de> 2015 Jose\nCopyright co 1999,2000 by Jef Poskanzer <jef at mail.acme.com>\nCopyright 1993-1994 Rob McCool <robm@stanford.edu> 1997 Jef Poskanzer <jef@mail.acme.com>\nCopyright Freely Redistributable Packager Bennett Todd <bet@mordor.net> Group Networking/Daemons\n\nCopyright (c) <year> <owner> . All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n   1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n   2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
         "summary": {
           "total": 1
           "warnings": {
             "noCopyright": []

[yashkohli88]

Summary of Changes

🛑 Degradations

1. conda/conda-forge/linux-aarch64/numpy/1.16.6-py36hdc1b780_0

• Incorrect License Detection:
  Multiple files (cpuinfo.py, system_info.py, some files in __pycache__ folder , and f2py2e.py) now incorrectly list both BSD-2 and BSD-3 clause licenses, as well as Apache and Python licenses. Expected: only BSD-3-Clause or correct values.

2. deb/debian/-/mini-httpd/1.30-13_armhf, debsrc/debian/-/mini-httpd/1.30-13

• License Expression Regression:
  The copyright/license:
  • contains redundant BSD-2-Clause expressions
  • includes an unexpected license reference

---

🚀 Improvements

1. pypi/pypi/-/platformdirs/4.2.0

• Score Correction:
  • discovered score increased from 0 to 1 (total now 76, was 75)
• Attribution/Notice Improvement:
  • New, more appropriate notice detected
• Notice Content:
  • Updated MIT License text and improved formatting
• Copyright
  • noCopyright warning removed

2. conda/conda-forge/linux-aarch64/numpy/1.16.6-py36hdc1b780_0

• Notice/Attribution Cleanup:
  • Extra attributions like (c) Multiply a Chebyshev, etc., removed from polynomial files
  • True/false value attributions removed

---

🟢 No Changes

1. pypi/pypi/-/sdbus/0.12.0

• License Order Change Only:
  License sequence swapped; content unchanged.

2. nuget/nuget/-/NuGet.Protocol/6.7.1

• License Order Change Only:
  ECL-2.0 and Apache-2.0 license order swapped; no other difference.

3. debsrc/debian/-/mini-httpd/1.30-13

• Attribution Minor Wording Change:
  Slight change in how Jef Poskanzer's name/email is presented.
• Notice Case:
  Minor notice wording/order difference; not legally significant.

---

[qtomlinson]

@jamesiri @jeffmendoza @jeffwilcox Both improvements and degradations are observed in the integration test results.

[jamesiri]

@jamesiri @jeffmendoza @jeffwilcox Both improvements and degradations are observed in the integration test results.

since I am new here I am going to ask a dumb question, is someone already working on these fixes or is it something I should start hitting with a hammer? I just want to make sure I understand the next steps and who has ownership over them.

[qtomlinson]

@jamesiri Yash and I will collaborate to update the tests for the no-change cases and the fixtures for the improved cases. For the degradations, we need to work with ScanCode to address these issues.
@pombredanne Since the upgrade of ScanCode, both improvements and degradations have been observed. I wanted to touch base regarding the reported degradations. Are these known issues?

[capfei]

3. pypi/pypi/-/sdbus/0.12.0

Description:

Calling fetch. URL: dev-api.clearlydefined.io/definitions/pypi/pypi/-/sdbus/0.12.0?force=true, Options: {} Calling fetch. URL: api.clearlydefined.io/definitions/pypi/pypi/-/sdbus/0.12.0, Options: {}

• [No changes] The sequence for the license changed. Python-2.0 and LGPL-2.1-or-later interchanged.

expected: {"path":"sdbus-0.12.0/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (LGPL-2.1-or-later AND Python-2.0 AND LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}
actual:   {"path":"sdbus-0.12.0/PKG-INFO","license":"LGPL-2.1-or-later AND LGPL-2.0-or-later AND (Python-2.0 AND LGPL-2.1-only AND LGPL-2.1-or-later AND LGPL-2.0-or-later AND GPL-1.0-or-later)","hashes":{"sha1":"2c1f5ef84a2c0c7eb8a9497d2b0c82c7f47fff69","sha256":"537199af659200d01fb5a089d73a1d507303177a31a5ad4d25b193e383e9b989"}}

I understand the other licenses but is it only adding Python-2.0 because Python is used throughout the file? It doesn't actually mention the Python license anywhere that I could see.

[capfei]

3. nuget/nuget/-/NuGet.Protocol/6.7.1

Description:

Calling fetch. URL: dev-api.clearlydefined.io/definitions/nuget/nuget/-/NuGet.Protocol/6.7.1?force=true, Options: {} Calling fetch. URL: api.clearlydefined.io/definitions/nuget/nuget/-/NuGet.Protocol/6.7.1, Options: {}

• [No changes] The sequence for the license changed. ECL-2.0 and Apache-2.0 interchanged.

expected: {"path":"clearlydefined/downloaded/LICENSE","license":"Apache-2.0 AND (ECL-2.0 AND Apache-2.0)","hashes":{"sha1":"39abfb1ed211fe9a9fdef056126133c7eb444f0d","sha256":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"},"token":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"}
actual:   {"path":"clearlydefined/downloaded/LICENSE","license":"(ECL-2.0 AND Apache-2.0) AND Apache-2.0","hashes":{"sha1":"39abfb1ed211fe9a9fdef056126133c7eb444f0d","sha256":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"},"token":"821602941cab0a35e9f461326976dede9239410a988bdcfc88348cc3f57bc6a5"}

I don't agree with the expected result. When I look at https://api.clearlydefined.io/definitions/nuget/nuget/-/NuGet.Protocol/6.7.1, I see

{
"path": "clearlydefined/downloaded/LICENSE",
"license": "Apache-2.0",
"hashes": {
"sha1": "6215cb16583ad28f22e1a7c905ee5bbee1044635",
"sha256": "e38b0854f4b591f3297571e73c665bd4266e8c8f9c55fb580942e51a4da619a8"
},
"token": "e38b0854f4b591f3297571e73c665bd4266e8c8f9c55fb580942e51a4da619a8"
},

Apache-2.0 should be the only license listed there. ECL is close but there is a whole paragraph added in clause 3. The downloaded license file is 100% Apache and even lists the SPDX identifier in it. https://licenses.nuget.org/Apache-2.0

[qtomlinson]

3. nuget/nuget/-/NuGet.Protocol/6.7.1

Apache-2.0 should be the only license listed there. ECL is close but there is a whole paragraph added in clause 3. The downloaded license file is 100% Apache and even lists the SPDX identifier in it. https://licenses.nuget.org/Apache-2.0

@capfei Thank you for your insight! Yes. The expected should be Apache-2.0, which ScanCode v30 detected correctly. Our baseline (expected result) was updated due to ScanCode v32 upgrade last year. See observation about "clearlydefined/downloaded/LICENSE" in my previous comment. During our current ScanCode minor upgrade, there does not appear to be significant changes for this case, as compared to the current v32 result deployed in production. That is why this case is marked as "no change".