module: allow type stripping in node_modules

[anonrig]

This removes the restriction that prevented type stripping from working on files inside node_modules folders, allowing packages to ship TypeScript sources directly instead of being forced to ship JavaScript.

Motivation

When working on a monorepo, without this particular change, every package needs to ship JavaScript (or both JavaScript and TypeScript) for its files to be includable from another package, or the contents of the package need to be manually copied into a non-node_modules directory before they can be loaded. Neither of these workarounds is ideal: they add a build step (or a copy step) for code that Node.js is otherwise perfectly capable of running directly, and they defeat the purpose of built-in type stripping for internal workspace dependencies.

With this change, a workspace package can simply point its exports field at its .ts sources and be consumed by sibling packages with zero build steps.

Changes

• lib: remove the isUnderNodeModules() check from stripTypeScriptModuleTypes(), and remove the now-unthrowable ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING error.
• doc: move ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING to the legacy error codes section; rewrite the "Type stripping in dependencies" section to document the new behavior, recommend wiring .ts files through "exports", and point to the module compile cache for amortizing the stripping cost.
• test: flip the existing node_modules failure expectations to success (.ts, .mts, .cts, import and require() paths, including require(esm)), and add a new fixture covering a "type": "module" package that exposes .ts/.mts files via "exports" subpaths.

Notes

• Performance concerns about stripping dependencies at startup can be mitigated with the module compile cache, which already caches stripped output keyed by filename.
• Since this removes an error that previously prevented these files from loading, it is a behavior change to a stable feature; flagging for the loaders/typescript teams to confirm the desired semverness.

[nodejs-github-bot]

Review requested:

• @nodejs/loaders
• @nodejs/typescript

[marco-ippolito]

I want to make sure the @nodejs/typescript is aligned with this change.
There have been multiple discussions about this and they have always blocked.
I'm afraid things have not changed from their side

[ljharb]

Indeed; unless this change is somehow restricted to local symlinked private:true packages, I don't think it's going to be viable to ever allow this - publishing raw TS should never be a thing (while TS syntax continues to evolve).

[anonrig]

unless this change is somehow restricted to local symlinked private:true packages,

is this the only way of landing this change? if yes, happy to change the PR.

[targos]

The current implementation already works with symlinked packages

[ljharb]

If that's the case, then wouldn't that mean it already works with monorepos?

[targos]

Yes, it does. We depend on it at work.

[marco-ippolito]

I confirm it already works for monorepos that use symlinks.
If the node_modules path is just a symlink for the source somewhere else its resolved correctly.

[andrewbranch]

There’s a lot of discussion at #58429.

I think the remaining “it doesn’t work with monorepos” feedback is specifically around pnpm deploy, which copies a workspace package’s contents and dependencies (both workspace links and normal registry dependencies) into a single directory without symlinks so it can be easily copied into a Docker image context or something similar.

[marco-ippolito]

@nodejs/typescript What if we put it behind a compile flag? If someone wants it so bad they can compile Node from source with their custom config

[DanielRosenwasser]

I don't think anyone wants this enough that they're unwilling to use a more convenient solution, but are willing to compile Node.js themselves.

[ljharb]

Then that sounds like an issue with pnpm, specifically, and not an issue with node's implementation - perhaps that's where efforts should be focused?

[aduh95]

Also, it's already behind a runtime flag, so I very much doubt someone would prefer re-compiling Node.js rather than use the runtime flag

[anonrig]

Then that sounds like an issue with pnpm, specifically, and not an issue with node's implementation - perhaps that's where efforts should be focused?

While gate keeping an important feature, we can not just say "an issue with pnpm". We're the gatekeepers and if we are gatekeeping we have the responsibility to consider and handle every edge case.

[marco-ippolito]

Also, it's already behind a runtime flag, so I very much doubt someone would prefer re-compiling Node.js rather than use the runtime flag

its not, currently there is no way to disable this behavior

[aduh95]

Also, it's already behind a runtime flag, so I very much doubt someone would prefer re-compiling Node.js rather than use the runtime flag

its not, currently there is no way to disable this behavior

What about #58429 (comment)?

[marco-ippolito]

Also, it's already behind a runtime flag, so I very much doubt someone would prefer re-compiling Node.js rather than use the runtime flag

its not, currently there is no way to disable this behavior

What about #58429 (comment)?

that's a crazy workaround, I wouldnt say it's a "runtime flag" it's using a loader encoded in base64:

import { registerHooks, stripTypeScriptTypes } from 'node:module'
function load(u, c, n) {
  const r = n(u, { ...c, format: 'module-typescript' })
  const out = stripTypeScriptTypes(r.source.toString())
  return { ...r, format: 'module', source: out }
}
registerHooks({ load })

[aduh95]

Calling registering a loader a "crazy workaround" is a tad too far, wouldn't you agree? Call me crazy, but I refuse to believe someone would prefer compiling a different version of Node.js rather than use that workaround

[marco-ippolito]

Calling registering a loader a "crazy workaround" is a tad too far, wouldn't you agree? Call me crazy, but I refuse to believe someone would prefer compiling a different version of Node.js rather than use that workaround

It's still a workaround, since it's sole purpose is to bypass the current behavior. The crazy part is to inline it in base64. I doubt someone is running it in production. But I imagine there are several companies running their custom built node that might want this.
Also I didn't call you crazy 😝 but the workaround is.
Btw I figured out why amaro stopped stripping node_modules when used as a loader. I'll fix it

[andrewbranch]

Then that sounds like an issue with pnpm, specifically, and not an issue with node's implementation - perhaps that's where efforts should be focused?

I don’t know if this workflow is specific to pnpm—it’s neither a package manager bug nor a Node.js bug, but I think it’s an unfortunate bad interaction in a pretty compelling use case. When you have a monorepo with a bunch of libraries consumed by an app, you need a way to package that app up for deployment. The way its dependencies are laid out in the monorepo doesn’t make sense for moving it to a server or packaging it into a CLI or baking it into a container image—they might be hoisted a few directories up, interleaved with devDependencies and dependencies needed only by other apps, and full of symlinks that might even resolve to a global store somewhere in your home directory. When I’ve worked on codebases like this at Microsoft, we would handle this by publishing all the libraries to a private internal package feed, copying the app into a Docker context by itself, and npm installing its dependencies from the private package feed, which would also proxy public registry packages. This is overkill for a lot of projects—pnpm deploy replicates the directory structure we achieved in that workflow in a single command.

I think it would be great if we could make this scenario work, but the line we won’t move on is preventing package authors from publishing TypeScript-only packages to the npm registry, for reasons that have been reiterated many times in earlier discussions. What if Node.js allowed loading TS files from packages with "private": true, regardless of realpath location? That would provide a sufficient safeguard against TS-only spreading across the npm registry while unblocking the package-to-deploy workflow from taking advantage of type stripping in workspace packages.

[mcollina]

Having a runtime flag would help with this. I don’t we can ship being on by default

[lemire]

unless this change is somehow restricted to local symlinked private:true packages,

is this the only way of landing this change? if yes, happy to change the PR.

The main objection raised so far is preventing the publication of raw TypeScript to the npm registry. Packages marked as private are explicitly not intended for publishing, so why were they restricted in the first place?

[marco-ippolito]

Previous attempt to enable it for private packages:
#55385
I think private only + runtime flag should be safe enough

[ShogunPanda]

I'd go for a runtime flag (for both public and private) disabled by default. And we also explicitly document that it is discouraged and likely never turned on by default.

[ljharb]

I think it'd be better for the runtime flag to only ever work for private.

This isn't a "gatekeeping" thing - that implies the harm being held back is artificial - this is literally that it would be a catastrophe for the ecosystem if publishing untranspiled TS became common, and no feature or DX whatsoever is worth that outcome.

[ShogunPanda]

I think it'd be better for the runtime flag to only ever work for private.

This isn't a "gatekeeping" thing - that implies the harm being held back is artificial - this is literally that it would be a catastrophe for the ecosystem if publishing untranspiled TS became common, and no feature or DX whatsoever is worth that outcome.

I'm neutral on this. Allowing public packages while this flag stays off by default is something we can always revisit later.

[jakebailey]

More context, because this comes up repeatedly:

• Support type stripping in node_modules #57215 (comment)
• Transpile TypeScript code inside node_modules. #58429 (comment)

I don't see any way a runtime flag gets added and it not eventually become defaulted on, leading to the trouble we've been trying to avoid.

[ljharb]

The point is that anything that enables publishing untranspiled TS must never be revisited. It's a nonstarter, and bringing it up ever again is always a waste of time. (until such time as the TS team decides to never again make a breaking syntax change, perhaps)

[marco-ippolito]

I don't see any way a runtime flag gets added and it not eventually become defaulted on, leading to the trouble we've been trying to avoid.

If we make the flag work for private only packages, even if it was defaulted on (and I dont see this happening) it wont cause a catastrophe.

[ljharb]

@marco-ippolito i mostly agree - but i can see a world where people just set node_modules packages to private: true in a postinstall script so they auto-strip, and that would lead to the same bad outcome.

[marco-ippolito]

@marco-ippolito i mostly agree - but i can see a world where people just set node_modules packages to private: true in a postinstall script so they auto-strip, and that would lead to the same bad outcome.

Post install scripts are being disabled by default in npm 12 and other package managers, it requires enough configuration and machinery that is easier to just use a loader (and its still behind a flag so user must enable it)

[arcanis]

I agree on that, I don't think postinstalls would be viable.

With that said, how do you plan to detect whether something is private? Node.js doesn't currently know what a package is, and the closest heuristic (find the closest package.json) can be abused by adding a private package.json file in a subfolder.

[marco-ippolito]

I agree on that, I don't think postinstalls would be viable.

With that said, how do you plan to detect whether something is private? Node.js doesn't currently know what a package is, and the closest heuristic (find the closest package.json) can be abused by adding a private package.json file in a subfolder.

The only way is with the closest package.json. I dont think your scenario is entirely viable as the entry point for the package still needs to be a js since it finds the package.json in the root (that needs to be private otherwise). I'll try to write a test to cover it.
But still it's still behind a flag so even if someone abuses it, its not the general population. As @aduh95 there is already a oneliner workaround that works for every node_modules package and still it's not being abused, so I'm less concerned about this

[ljharb]

node definitely knows what a package is - only a package's package.json's exports field is considered (not just the closest package.json). It can use the same logic to look there for private: true.

[arcanis]

That's only because you're going through a bare import, and Node.js is able to treat the resolved folder as the "package" root.

If you do a relative import I think the heuristic is just the classic directory backtracking, and it's trivial to cloak the "true" package.json behind one in a subfolder. All you need is an entry point doing a relative import.

Now, it is a little contrived, but it's simple enough that I could imagine it being shared around.

[marco-ippolito]

I created #63869 as a followup. @anonrig I hope you don't mind

[aduh95]

Then that sounds like an issue with pnpm, specifically, and not an issue with node's implementation - perhaps that's where efforts should be focused?

While gate keeping an important feature, we can not just say "an issue with pnpm". We're the gatekeepers and if we are gatekeeping we have the responsibility to consider and handle every edge case.

You want to consume your local deps from source (i.e. with type stripping) ? Use symlinks. You want to consume your local deps from a copy inside node_modules to be closer to the deploy env? Type-strip them when copying, that'll save you from type stripping static files at runtime (arguably that should be done transparently by pnpm, if that's their default behavior it makes no sense to impose that budren on the end user). You still want type-stripping your local copy inside node_modules? Use a loader.
How are we gatekeeping exactly? What use case did we not consider?

[ljharb]

@arcanis if it's a relative import (not counting reaching into node_modules, since that string should never appear in a specifier) it's not reaching into a package, so type-strippability should be the same as the file you're pulling from.

[Jamesernator]

The current implementation already works with symlinked packages

That this works for symlinks, is as far I can tell completely undocumented and seems unintentional based on the docs. Though it's good to know there's a way that does already work today.

Yes, it does. We depend on it at work.

Given people are already using this, perhaps this should actually be officially supported and guaranteed not to break in future. This would also encourage tooling to implement proper support as well (like from a quick testing, TS in VSCode doesn't seem to correctly suggest .ts paths from such symlinked packages, instead suggesting the .js equivalent which fails at runtime even with allowImportingTsExtensions).

---

Type-strip them when copying, that'll save you from type stripping static files at runtime (arguably that should be done transparently by pnpm, if that's their default behavior it makes no sense to impose that budren on the end user).

The problem is .js will often be stale, like if you want to run a build step after every change then it's already possible to do this, but for certain types of projects (especially local scripts) this isn't helpful compared to node simply allowing type stripping automatically.

Like what tends to happen is: make a change, change doesn't work, get confused why change doesn't work, realize .js is stale, run build. And this happens a lot, it's honestly just annoying friction especially when sources are just local build scripts that aren't intended to be published at all.

[aduh95]

The problem is .js will often be stale

But since we're dealing with a copy, .ts files will also be stale, no? So my point is that you might as well type-stripe them.

[Jamesernator]

But since we're dealing with a copy, .ts files will also be stale, no? So my point is that you might as well type-stripe them.

I agree that this is arguably a pnpm problem, however if Node doesn't have an officially supported approach for .ts in node_modules then it's unclear that pnpm is "wrong" to do this.

The solution here might just be a doc update, and some regression tests, to ensure that .ts resolution is officially supported in symlinked node_modules, then pnpm can be encouraged to use the same behavior as npm for file: modules.

---

Also playing around with pnpm some more slightly, the workspace: protocol does cause symlinks but file: protocol doesn't.

This is a wider problem with Node package managers, in that there is no common way to deal with workspaces, with every single package manager having it's own bespoke way of dealing with workspaces that aren't cross compatible.

[aduh95]

Also playing around with pnpm some more slightly, the workspace: protocol does cause symlinks but file: protocol doesn't.

That's indeed interesting, did you check what's the behavior wrt staleness with each protocol?

[Jamesernator]

That's indeed interesting, did you check what's the behavior wrt staleness with each protocol?

Yes, basically if you use "package-name": "file:./packages/package-name/" then it'll just copy the whole package into the node_modules/.pnpm dir then symlink that, however if you use "package-name": "workspace:@jamesernator/package-name@*" it'll search for an find and symlink ./packages/package-name/ directly).

It's actually not that surprising that workspace: works really, as that's the pnpm blessed mechanism for workspaces, just a shame this isn't portable amongst package managers.

---

I also just saw that package maps were merged, so this throws another wrench into the idea of just officially supporting the behavior of symlinks causing this behavior to work.

Ideally any mechanism we have for supporting .ts in node_modules would work have some way to work in package maps as well, so that if a package manager generates a package map we don't lose the ability by using the package map directly.

This definitely makes me think the correct approach here is simply to look at private in package.json as proposed in #63869

[aduh95]

This definitely makes me think the correct approach here is simply to look at private in package.json as proposed in #63869

How would that help? That seems completely orthogonal, workspaces aren't necessarily private

[Jamesernator]

How would that help? That seems completely orthogonal, workspaces aren't necessarily private

If we're not going for the approach of this PR and allowing .ts in node_modules in all cases, then a package that uses .ts imports won't work when published anyway without a build step, that build step can always set private as needed.

Like it's important for anyone that wants to publish a module that uses .ts in Node to know that they will need to ensure the following:

• all .ts specifiers are rewritten to .js (which tsc has a flag for)
• all "exports" (and "imports") that refer to the source directory (commonly "src") are changed to refer to the build directory (commonly "dist")
  • e.g. "exports": { "./*": "./src/*" } is mapped to "exports": { "./*": "./dist/*" }
• all "exports" that have .ts extensions are remapped to .js
  • i.e. "exports": { "./*": "./core/*.ts" } is remapped to "exports": { "./*": "./core/*.js" }

Given these hazards of trying to publish a package that has .ts specifiers internally, I think it's reasonable friction to require them to remove a private: true from the package.json before publishing.

(It would also probably make sense for npm/pnpm/etc to have an option for publishing .ts directly, that performs type stripping, rewriting extensions, and fixing "exports"/"imports" based on tsconfig before publishing to the registry).