External program with pipe

Question
1

How do i perform the unix script and capture the output
ls -al | awk ‘{ if(NR!1) print $1, $9}’

In the language ruby this is peace of cake. In scala it gives me pain the head :slight_smile:

2

I would write the Bash commands to a file and then run that file as a Bash script.

In my experience, giving a command string with pipes to scala.sys.process will not work at all. Using the pipe operators provided by scala.sys.process are just linking the I/O streams inside the JVM and are not the same as the Unix pipe they are meant to allude to. In general, what scala.sys.process does is not very well defined, and same is true for the underlying Java Process class.

3

The scala.sys.process commands do not use a shell by default, so everything usually done by the shell (pipes, globbing, variable expansion) does not work directly. There are at least two ways to solve this:

Using a shell explicitly:

import scala.sys.process._
"sh -c \"ls -al | awk '{ if(NR!1) print $1, $9}'\"".lines

Add sh -c to our command line, passing your whole command as a single argument to sh to execute. Note the additional quoutes around your original command; if your command contains double quotes you’ll have to take care with escaping or use @curoli’s suggestion of writing the command to a file and call sh -c filename.sh.

Using scala.sys.process pipes:

import scala.sys.process._
("ls -al" #| "awk '{ if(NR!1) print $1, $9}'").lines

This uses the #| operator from scala.sys.process, which is semantically equivalent to the pipe on the shell, but evaluated by scala.

(PS: If you’re on Scala 2.13, you need to use lazyLines instead of lines in both variants, the latter doesn’t exist anymore)

@curoli I’m not sure I understand what you mean by “just linking the I/O streams”. A shell pipe does the same, connecting one programs standard output to the next programs standard input. This happening in the JVM should not make a difference, the only effect I can imagine is lower throughput. In my experience, the only semantic difference is that the scala operator works on windows machines without sh or bash.

4

At least in this particular case, I’d question offloading to the shell altogether.

  • It is not immediately obvious what information this code extracts.
  • It is more susceptible to typos - should this be NR!=1?
  • The awk invocation will truncate file names containing spaces.
  • Output contains . and .., which is likely not desired.
  • It likely will have to be parsed on the Scala side for further processing.

You can query the file permissions in Scala, e.g. using a library like Ammonite:

def list(path: Path): Map[Path, StatInfo] =
  (ls! path | (ch => ch -> stat(ch))).toMap

If you really need the text output as generated from awk, you can generate it from here:

def render(perms: Map[Path, StatInfo]): String =
  perms
    .map { case(f, p) => s"${if(p.isDir) "d" else "-"}${p.permissions} ${f.last}" }
    .mkString("\n")
1 Like
5

By “linking the I/O streams” I meant the pipe operators of scala.sys.process will copy the bytes over from the InputStream linked to one process into the OutputStream of another process, instead of creating a system pipe, as the Unix command would do. Whether the end result is the same is a good question. I’m not aware of a guarantee given.

6

system needed upgrade

7

What’s the output/problem?

Note, there’s also https://github.com/lihaoyi/os-lib which abstracts away much of the pain points of interacting with the OS directly and should be quite fast.

Personally, whenever I see someone use (t)csh I kind of feel sorry for them, as those shells are not really well behaved and have various oddities. I would recommend a Bourne shell instead.

8

After checking I found my java version and scala version was not updated. This was creating odd problems.
This program works fine :

#!/usr/bin/env scala
import scala.sys.process._

def func( todo:String) : Unit = {
println(todo)
val res =(Process(todo)).!!
println(res)
}

val todo="tcsh -c "ls -al | awk ‘{ if(NR!=1) print $1, $9}’ " "
func(todo)