grep No such file or directory with envoy.run
I try to build script that execute some grep search in my logs and print the results. I try to use Envoy because is more easy than subprocess but when I execute grep command it gives me back an error of no such file o directory.
The dir structure is easy:
- . # root of script
- test.py # script file
- web_logs/log/ # dir that contains log to search in
My test.py is easy:
import envoy def test(value): search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*" print(search) #check of the search string r = envoy.run(search) print(r.status_code, r.std_out, r.std_err)#check of the command response = r.std_out if __name__ == "__main__": test(2)
The output is:
grep 'cv=2' ./web_logs/log/log_* (2, '', 'grep: ./web_logs/log/log_*: No such file or directory\n')
If i run the same command:
grep 'cv=2' ./web_logs/log/log_*
I can find the occurrence of the string "cv=2" in the log files.
Where is the error?
Update after the answers The problem is in using of * that envoy cannot explode without use of glob module so I using the subprocess as it is and I try to study better the using of glob module to improve envoy.
The new code I used is:
Автор: Smilzao Источник Размещён: 08.11.2019 11:04
import subprocess def test(value): search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*" print(search) #check of the search string proc = subprocess.check_output(search, shell=True) print proc.split('\n') if __name__ == "__main__": test(2)
@baptistemm is actually right in that since you're not running bash as part of your process the globbing is not working.
However what's happening is a bit deeper.
When you run a sub process it can be done by one of several system services (system calls).
Short Answer (TLDR;)
Here's the correct way to do this:
import envoy def test(value): search = "/bin/sh -c \"grep 'cv="+str(value)+"' ./web_logs/log/log_*\"" print(search) #check of the search string r = envoy.run(search) print(r.status_code, r.std_out, r.std_err)#check of the command response = r.std_out if __name__ == "__main__": test(2)
Running the command as a shell command will take care of globbing.
Whenever a sub process is executed, it eventually gets translated into an execve system call (or equivalent).
C library there're helper functions such as
popen(3) which wrap around
execve(2) to provide easier ways of executing processes.
system launches a shell and passes its argument as is to
-c option of the shell. popen does extra magic, kinda like what envoy is doing in python.
In envoy, the argument is parsed for
def expand_args(command):) in the envoy code. and then uses the equivalent of
popen to execute the processes.
envoy is essentially what the shell does with the
| marker (splits things up across the
| and then uses popen).
What envoy is NOT doing is interpreting
* as the shell does, as in expanding it to match files using a
glob function of some sort. Bash does. Thus my answer.
A fun exercise would be for you to contribute code to envoy :-) and make it do the globbing.Автор: Ahmed Masud Размещён: 21.08.2016 05:34
Why it works in terminal but not in envoy is related to globbing (bash example).
When you run in your terminal
grep 'cv=2' ./web_logs/log/log_*
bash will parse the command line and replace the star character with every occurence of file that match. So if you have
./web_logs/log/log_foo your command will actually be
grep 'cv=2' ./web_logs/log/log_1 ./web_logs/log/log_2 ./web_logs/log/log_foo
When you execute the same thing in envoy, that'd different, it won't perform the globing of the files then it'll pass to grep a file named
./web_logs/log/log_* which doesn't exist, this is actually confirmed by the line you pasted in your question.
print r.std_err 'grep: ./web_logs/log/log_*: No such file or directory\n'
ps: there is a glob module for pythonАвтор: Baptiste Mille-Mathias Размещён: 21.08.2016 05:19