Running command that pipes to another command

[lordcodes]

Describe the problem

Currently, there is no way via the API to pipe the output of a command into another command. For example to run something similar to:

echo '{"key":"value"}' | jq '.'

Describe your solution

• Work out how to actually pipe one command into another using Process as currently it doesn't work. Likely the issue relates to the standard out and in being redirected/captured as streams.
• Commands can be a structure rather than just executed immediately as a function like now.
• A Pipeline structure can then be built from a series of Commands and then executed all together with each command piping into the next.

Checklist

• I've read the guide for contributing.
• I've checked there are no other open pull requests for the feature.
• I've checked there are no other open issues for the same feature.

Additional context

It looks like a change may be made with how the command(s) are prepared and built using ProcessBuilder so that they are executed correctly.

Workaround
Currently, you can run multiple commands by simply capturing the output of the first command into a Kotlin property and then running the second command using its value as input.

[jmfayard]

Maybe it's just me, but I prefer Kotlin typesafe syntax to Bash messy primitive obsession (everything is a string).

So I was thinking about a more functional approach where you create Commands data classes, combine them in a Pipeline data class, and only at the last step you execute it

/**
 * Pure function that represents but doesn't execute this command:
 *  $ find . -type f | grep '.kt' | xargs grep 'fun ' 2> /dev/null | wc -l
 */
fun measureKotlinFun(val output: File): Pipeline {
    val findFiles = Command("find", ".", "-type", "f")
    val onlyKotlinFiles = Command("grep", ".kt")
    val foreachSearchFun = Command("xargs", "grep", "fun ")
        .withRedirect(CommandRedirect(stderrIgnore = true))
    val countLines = Command("wc", "-l")
    val pipeline = findFiles `|` onlyKotlinFiles `|` foreachSearchFun `|` countLines
    return pipeline.withRedirect(CommandRedirect(stdout = output))
}

fun imperativeShell() {
    measureKotlinFun(File("fun.txt"))
        .executeCommand()
}

See https://github.com/jmfayard/turtle/blob/4291745b4552a65a4a7945403d7a84edf3e8c178/turtle/src/main/kotlin/com/lordcodes/turtle/Command.kt#L25-L36

What do you think?

[lordcodes]

Interesting idea. I actually investigated supporting piping in commands, but didn't get anywhere and actually realised it wouldn't be a good idea. You would be unable to use built-in commands and the result would be 'command' that is actually a chain of commands all as a String, difficult to parse and would be a bit confusing.

I really like the idea of making commands a structure that can then be executed, rather than just a function that is immediately executed. The existing commands would just create command and execute it internally. A pipeline structure can then contain a series of commands that can be executed as a chain.

The main blocker is working out how to implement the actual piping support with Process. I think it relates to checking for data being available on standard input and using that as input arguments if data is there.

[jmfayard]

The main blocker is working out how to implement the actual piping

I got it to work using https://github.com/zeroturnaround/zt-exec which seems like the right library to implement my Pipeline class

Proof of concept here: jmfayard@51aff7d?diff=split

[lordcodes]

Nice concept. We should be able to take the plumbing required to forward output of one to input of next without the dependency. Looks like a great starting point.

[jmfayard]

Yes but you would have to reinvent all the threading etc... yourself.
Personally I would start with the dependency as an implementation detail, if that works well you can always reinvent the wheel and remove it later.

[lordcodes]

Threading? I thought it was just reading from the input stream. From what I can tell in your changes it seems like command writes its output to a stream and then the next command uses this stream as its standard input. I understand your point, however, the library you are talking about is a Java library which hasn't received any updates in a long time and is also a similar library to Turtle itself. It's got quite a few dependencies itself.

[jmfayard]

Maybe I'm missing something, but to create a pipeline, don't you have to launch each program in a separate thread?
How would blocking calls work without that?

[lordcodes]

Maybe I just don't see it until I would try and work on it, however, there is no threading now and the pipeline would be running the commands sequentially. I don't see the difference to just running one command at a time in a process like they are now, but that each commands reads in from stream.

Anyway, realistically I'm not going to look into this or work on this anytime soon. If you would like this soon, then you are welcome to work on something. However, I'm really not a fan of introducing that library as a dependency, due to the fact it is a Java library with no recent changes. If threading and a lot of complexity is required to implement this, i.e. it isn't as simple as it seems to me 🤣, pipelines could be built as a separate library potentially.

[jmfayard]

Maybe I just don't see it until I would try and work on it, however, there is no threading now and the pipeline would be running the commands sequentially. I don't see the difference to just running one command at a time in a process like they are now, but that each commands reads in from stream.

There is a big difference between

yes "say yes indefinitely" | head -5

and

yes "Say yes indefinitely" > /tmp/yes.txt
head -5 < /tmp/yes.txt

The first pipeline ends in a few miliseconds and prints five times "Say yes indefinitely"

The second is an infinite loop that fills your SSD.

Here my version doesn't do an infinite loop

val inifiniteCommand = Command("yes", "say yes indefinitely")
val pipeline2 = inifiniteCommand `|` Command("head", "-5")
pipeline2.execcute()

jmfayard@51aff7d?diff=split

[jmfayard]

If the dependency is an issue, we can put the pipeline feature in a separate module. Also it was fun to discuss but I don't really need pipelines right now. Having the Command abstraction on the other hand would be really helpful. Le jeu. 11 août 2022 à 13:26, Andrew Lord ***@***.***> a écrit :

…

Maybe I just don't see it until I would try and work on it, however, there is no threading now and the pipeline would be running the commands sequentially. I don't see the difference to just running one command at a time in a process like they are now, but that each commands reads in from stream. Anyway, realistically I'm not going to look into this or work on this anytime soon. If you would like this soon, then you are welcome to work on something. However, I'm really not a fan of introducing that library as a dependency, due to the fact it is a Java library with no recent changes. If threading and a lot of complexity is required to implement this, i.e. it isn't as simple as it seems to me 🤣, pipelines could be built as a separate library potentially. — Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADQFSH4P76O2A4TYJNXMY3VYTPNVANCNFSM5Z6BSIKQ> . You are receiving this because you commented.Message ID: ***@***.***>

[lordcodes]

I agree. I have created #120 to track the Command abstraction.

[parrotcar00]

I came across this project: https://github.com/dsherret/dax which is deno/typescript library for shell command running.

I think turtle is very similar in functionality to the above.
I was looking to see how to implement the shell pipe operator in turtle but this issue tells me it's on the todo list.

The dax project above implements the pipe operator like:

const output = await $`echo foo && echo bar`
  .pipe($`grep foo`)
.text();

Maybe this can be of some help while implementing the pipe feature?