Case-Sensitive IRI as spdxId

[AlexanderDenkBMW]

Hi,

using the tools-java in their 2.0.5 version I got an issue with duplicate spdxIds:

Analysis exception processing SPDX file: Object URI already exists.

Looking at the spdxIds I've notices that they are only identical if they are threaded case-insensitive. By specification the path of an URL should be case-sensitive.
This makes also sense as in fact the file exists twice in the Linux kernel includes the file twice with different content:

Simplified Examples:

https://spdx.example.com/doc-/packages/usr_lib_modules_kernel_net_netfilter_xt_hl_ko_a
https://spdx.example.com/doc-/packages/usr_lib_modules_kernel_net_netfilter_xt_HL_ko_a

In my eyes this is not an error but intended usage.

[goneall]

@AlexanderDenkBMW Agree that IRIs are case sensitive and should be treated as such by the tools.

There is a complication in fixing this issue in that the same ID structure is used for license IDs which are case insensitive. We need to be careful to still issue an error if the ID is a license ID.

[AlexanderDenkBMW]

Hi @goneall , any update on this? :)

[goneall]

I can take a look next week, been traveling the last couple weeks

[AlexanderDenkBMW]

Thanks for the update :)

[goneall]

@AlexanderDenkBMW - can you provide an example SPDX file and an example which duplicates the error? I just check the core libraries where I thought this error would occur and it does distinguish case sensitive IRI's.

[AlexanderDenkBMW]

On which URIs did you check?
For our example we have two packages with the following SPDX IDs:

• hl (lowercase): https://domain.net/tenant/project-build-hash/rootfs-file/usr_lib_modules_6_6_44-ti-g8d9e2fcdd160-dirty_kernel_net_netfilter_xt_hl_ko
• HL(uppercase): https://domain.net/tenant/project-build-hash/rootfs-file/usr_lib_modules_6_6_44-ti-g8d9e2fcdd160-dirty_kernel_net_netfilter_xt_HL_ko

[goneall]

On which URIs did you check?

I wrote a unit test for the library which manages the SPDX IDs: spdx/Spdx-Java-Library#424

The unit tests passed, so I suspect the error is generated by a higher level utility or library.

@AlexanderDenkBMW - if you could let me know what utility or command duplicates the problem, I can track down where the error occurs. e.g. was it a verify, translate, or compare command in the SPDX tools utility?

[MQueiros]

It was java -jar tools-java-2.0.6-jar-with-dependencies.jar Verify $spdx

[goneall]

It was java -jar tools-java-2.0.6-jar-with-dependencies.jar Verify $spdx

Thanks @MQueiros

[goneall]

I'm able to duplicate the problem and I tracked down the source of the issue.

In the SPDX Java Library, the InMemSpdxStore treats the IDs as case insensitive for all IDs.

This is likely to support license reference IDs where were allowed to be case insensitive in SPDX 2.X.

Note that this class supports both the SPDX 2.X and SPDX 3.X formats.

To fix this will involve some redesign to move the case insensitive logic to handle license references as case insensitive and the remaining IDs as case sensitive.

@AlexanderDenkBMW - let me know how urgent this fix is - I'm working on a release right now, to fix it in this release would delay the release by a day or two. It may be a few weeks before we do another release.

[goneall]

I did a bit more research and confirmed that anything involving license refs or SPDX license IDs expects case insensitive IDs.

There are a few design alternatives I can think of:

1. move the case sensitive logic to the classes that deal with license IDs and make the entire model store case sensitive to Object URIs
2. add a new create method that takes a case sensitive ID parameter which store the mapping from case insensitive to case sensitive IDs - this would only be used for license IDs. This could either be a new method (e.g. createCaseInsensitiveId(TypedValue typedValue, String caseSensitiveId) or an optional parameter to the existing create method
3. Check for license ID patterns in the ObjectURI and handle the case sensitive / insensitive mapping similar to alternative 2 above

Alternative 1 is the cleanest, but it won't work for the LicenseExpressionParser since it is static and the solution would require storing a mapping from the case sensitive to the case insensitive IDs.

Alternative 2 would be a change to the public interface for model stores and require changes to several model store implementations - a bit change, could be considered breaking.

Alternative 3 feels like a hack, but would probably be the least impactful implementation.

@bact @pmonks - Any thoughts? Might be worth discussing on one of the implementers calls.

[goneall]

@AlexanderDenkBMW - let me know how urgent this fix is - I'm working on a release right now, to fix it in this release would delay the release by a day or two. It may be a few weeks before we do another release.

Since this is going to require some additional work, I'm going to go ahead and release the current code and take care of this issue in an upcoming release