Add sloth support

[Gedochao]

Adds Sloth support behind --sloth / //> using sloth
https://github.com/VirtusLab/sloth

Scala 3, up till 3.7.x series, has been using the (now terminally deprecated) sun.misc.Unsafe API. This in turn results in some ugly warnings on JDK 24+ (and likely errors in the future).

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ 
(file:(...)/maven2/org/scala-lang/scala3-library_3/3.7.4/scala3-library_3-3.7.4.jar)
WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

Now, while Scala 3.8+ uses a new implementation, and Scala 3.3 allows for the new implementation on JDK 9+ behind the -Yfuture-lazy-vals compiler flag, the problem is not entirely solved. Even if your app is on the latest Scala, it likely still depends on libraries built with Scala 3.3 (or some other <3.8 version). That in turn brings the problem back. And since it is not widespread to suffix Scala 3 libraries with the minor, as we did in Scala 2 days, it is not entirely easy to spot which libraries will pose a problem.

Sloth is a bytecode patcher, which is capable of patching all past lazy val implementations (including the pre-3.3 one) up to the current one. Kudos to @lbialy for developing it.

Behind the --sloth flag, this PR allows to post-process and patch the bytecode of your app/artifacts to silently get rid of the problem.
The --sloth-agent flag runs Sloth as a Java agent instead, allowing it to i.e. run in the JVM your tests live.
More details on how it works and what it does at https://github.com/VirtusLab/sloth

This feature is brought to the following sub-commands:

• run
• compile
• doc
• fix
• package
• publish
• repl (including the REPL artifacts themselves for Scala <3.8)
• run
• test

Checklist

• tested the solution locally and it works
• ran the code formatter (scala-cli fmt .)
• ran scalafix (./mill -i __.fix)
• ran reference docs auto-generation (./mill -i 'generate-reference-doc[]'.run)

How much have your relied on LLM-based tools in this contribution?

extensively, Cursor + Claude

How was the solution tested?

./mill -i integration.test.jvm '*sun.misc.Unsafe*'

[Gedochao]

@lbialy how close are we to publishing a stable version of lazyvalgrade to Maven Central?

[SethTisue]

Why is it called "lazyvalgrade", what does "grade" mean here?

[Gedochao]

@SethTisue I believe that's a pun on "lazy val upgrade".
We've already been discussing releasing it under a different name, coming very soon 😅

[Gedochao]

https://central.sonatype.com/search?q=org.virtuslab+sloth
Sloth artifacts are up!

[tgodzik]

Do we now have adopt a sloth in a zoo? And have sloth plushies?

[Gedochao]

We should. 🦥

[warcholjakub]

Those are the issues I managed to spot, while experimenting with it in the terminal. I'll look at the code next.

[warcholjakub]

I'm wondering about one thing. Let's assume this hypothetical scenario:

1. Person 1 makes a library, and signs it.
2. Person 2 wants to use this library in his app. They also use sloth.
3. Sloth modifies the library, hence the hashes don't match up.
4. Person 2 runs his app, and verification fails.

Tried to reproduce it, and it indeed failed, but not sure how realistic and common this problem would be. Also I don't really see any straightforward solution. But yeah - just something to think about.

EDIT: --sloth-agent seems to avoid this issue

[warcholjakub]

Final comments

[lbialy]

@warcholjakub Can you elaborate about what do you mean by sloth modifying the dependency library? In what circumstances the verification you mentioned happens? What exactly fails verification?

[warcholjakub]

@warcholjakub Can you elaborate about what do you mean by sloth modifying the dependency library? In what circumstances the verification you mentioned happens? What exactly fails verification?

I mean something like this:

jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> cat SignedLib.scala
//> using scala 3.3.8
package signedlib
object SignedLib:
  lazy val message = "hello from signed library"
jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> cat Main.scala 
import signedlib.SignedLib
@main def main(): Unit =
  println(SignedLib.message)
jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> ./mill -i scala --power package SignedLib.scala --library  -o "signed.jar"
jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> jarsigner -keystore "key.p12" -storepass changeit "signed.jar" test
jar signed.
jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> ./mill -i scala --power run Main.scala --sloth --classpath signed.jar
[...]
Exception in thread "main" java.lang.SecurityException: SHA-384 digest error for Greetings$.class
        at java.base/sun.security.util.ManifestEntryVerifier.verify(ManifestEntryVerifier.java:254)
        at java.base/java.util.jar.JarVerifier.processEntry(JarVerifier.java:248)
        at java.base/java.util.jar.JarVerifier.update(JarVerifier.java:235)
        at java.base/java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:453)
        at java.base/jdk.internal.loader.Resource.getBytes(Resource.java:106)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.getBytes(URLClassPath.java:745)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:773)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:691)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:620)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:578)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490)
        at Main$package$.run(Main.scala:9)
        at run.main(Main.scala:7)

As I said, I'm not sure whether it's a realistic scenario - just smth I noticed.

EDIT: Modified the scenario a bit.

[lbialy]

I did some research and you're right that the problem exists but fortunately:

a) signing is now quite rare

b) the whole signing mechanic is contingent on the presence of

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA
META-INF/*.EC
META-INF/SIG-*

files.

@Gedochao would it be possible to include a step where deps are copied to .scala-build, have these files stripped and then used as the runtime/package classpath when --sloth is specified?

[warcholjakub]

Last two comments 🦥

[warcholjakub]

TLDR: tests seem to not execute properly with --sloth

jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> cat /tmp/sloth_test.scala 
//> using scala 3.3.8
//> using dep org.scalameta::munit::1.3.4

class MainTest extends munit.FunSuite:
  test("must execute"):
    println("TEST_BODY_EXECUTED")
    assertEquals(1, 1)

No sloth:

jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> ./mill -i scala test /tmp/sloth_test.scala
[...]
TEST_BODY_EXECUTED
Test run MainTest started
MainTest: finished 0.011s
Test run MainTest finished: 0 failed, 0 ignored, 1 total 0.015s

Sloth:

jwarchol@vl-d-1635 ~/D/G/scala-cli (feature/lazyvalgrade)> ./mill -i scala test --sloth /tmp/sloth_test.scala
[...]
924] scala
924/924, SUCCESS] /Users/jwarchol/Documents/GitHub/scala-cli/millw scala test --sloth /tmp/sloth_test.scala 4s