pam-pkcs11: authenticate through pam_pkcs11 with pamtester

[yosuke-wolfssl]

Problem

The pam-pkcs11 job passed in every cell without ever running pam_pkcs11.so:

• su testuser ran as root, so pam_rootok accepted it before the module was asked. pam_pkcs11 was also only sufficient in common-auth, with pam_unix behind it.
• The setup could not have authenticated anyway. The config named a softhsm module block that does not exist, the token held no private key, the user cert was self-signed instead of issued by the CA, and no mapper read the mapping file.
• The force-fail cell exported WOLFPROV_FORCE_FAIL=1 before softhsm2-util --init-token, which failed during setup. The shared check counted that as "failed as expected".

Fix (.github/scripts/pam-pkcs11-test.sh)

• Setup runs without force fail in a mktemp workspace. It creates a CA and a user cert it issues, and puts the key and cert on a SoftHSM token under one ID.
• pam_pkcs11.conf is written by the script: a softhsm module block, cert_policy = ca,signature, and the cn mapper.
• Authentication goes through a dedicated pam_pkcs11_test service (pam_pkcs11, then pam_deny) via pamtester, so the result does not depend on running as root.
• pam-pkcs11.yml gates on the script's exit status and no longer calls check-workflow-result.sh.

Case	Required result
Valid token, correct PIN	Checking signature, then successfully authenticated
Wrong PIN	Error 2320
Cert from an unknown CA	Error 2328: Certificate signature invalid
Key does not match the cert	Error 2342: Verifying signature failed
Force-fail: valid token, correct PIN	succeeds without force fail, then fails with Error 2320 with it

Under force fail, SoftHSM's PIN hashing fails first (EVP_DigestInit failed) and pam_pkcs11 reports it as a wrong PIN. Running the same PIN without force fail first rules out a real wrong PIN.

Verification

• Local, in wolfprovider-test-deps:bookworm with this repo's master replace-default packages: all 4 cells pass (non-FIPS and FIPS, normal and force-fail).
• Negative controls fail as intended: a self-signed user cert, a key that does not match the cert, and the old force-fail scoping.
• A debug wolfProvider trace (arm64, non-FIPS) shows both RSA verifies in wolfProvider: the CA check via wp_rsa_digest_verify_final and the signature check via wp_rsa_verify, which returns 0 with the wrong key. SoftHSM's SHA-256 and AES go through wolfProvider too.

Not in this PR

SoftHSM 2.6.1 signs with OpenSSL 3.0's legacy RSA_private_encrypt/RSA_sign, so the token's private-key operation never reaches wolfProvider. In FIPS cells it runs outside the FIPS boundary. Same kind of gap as OpenSSH's; it goes in the audit doc's replace-default FIPS item.

[Copilot]

Copilot review overview

🟢 Approval recommended

The authentication coverage is sound; only the workflow documentation needs minor synchronization.

Review effort: Balanced
Findings: 1

Open (1)

• Document self-checking behavior of OSP force-fail workflow · New

What changed in this PR

Reworks PAM PKCS#11 CI to perform genuine certificate-based authentication and meaningful failure checks.

Changes:

• Creates valid, unknown-CA, and mismatched-key SoftHSM tokens.
• Authenticates through a dedicated pamtester service.
• Handles force-fail validation within the test script.

File	Description
.github/​workflows/​pam-pkcs11.yml	Uses the script’s result directly.
.github/​scripts/​pam-pkcs11-test.sh	Implements complete positive and negative authentication tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yosuke-wolfssl]

Jenkins retest this please