Command-line interactive programs in UNIX she...

« 上一篇: WebGUI下一篇: 这两天似乎发生了许多的事 »
Command-line interactive programs in UNIX shell-scripts
Jan @ 2005-06-22 15:11
http://www.osnews.com/story.php?news_id=10929
Like it or not, but sooner or later you realize that you‘ll have to write shell-scripts to administer UNIX. And among these scripts there certainly will be those to cooperate with interactive applications such as telnet, ftp, su, password, ssh. But it means the end of the admin‘s quiet life because while dealing with interactive programs one often come across numerous hidden traps which doesn‘t usually happen with ordinary sh-scripts. Though fortunately or may be not, but most of these problems generally turn up within first five minutes of the work under the script. The symptoms typically look like that author can‘t pass the authentication from the script. At first you feel confused because usual pipe constructions such as:
Unix scripts, Page 1/4
$ echo luser && echo TopSecret | telnet foo.bar.com
fail you and the problem which seemed so plain on the face of it grows into "mission impossible". Yet, it isn‘t all so very hopeless and quite a simple solution of the problem will come quickly enough for many of the dialogue applications include built-in mechanisms of handling scripts. See, for example, standard FreeBSD ftp-client:
$ echo ‘$FILEPUT‘ | ftp -N ftprc luser@foo.bar.com
This command will get ftp to connect with the foo.bar.com hosh with the name of luser and start FILEPUT macro described in the file ftprc. Besides the macro there must be also described the host, the login and user‘s password in this file:
$ cat ftprc
machine foo.bar.com
login luser
password TopSecret
macdef FILEPUT
binary
cd /tmp
put some_usefull_file.bin
bye
<-- Attention! There is a new-line symbol at the end of the macro.
$
If for some reason the dialogue application doesn‘t support the built-in scripts then you‘ll easily find it‘s freeware counterpart capable to act automatically. Really you are not the first to run into such a problem :-)
Another possibility to persuade an interactive program to do work by itself, without a user, is to redirect its standard input. For example, much simplified script to start Oracle database may look like this:
#!/bin/sh
su - oracle -Ó /oracle/bin/svrmgrl <
Yet we can‘t work like that with all kinds of applications. For instance, it isn‘t suitable for telnet, through the "problem of user‘s authentication" mentioned above. The difficulty to debug scripts at the moment of authentication complicates matters. So, it‘s clear that to find the way out we need some special solution. Let‘s don‘t break good traditions and google the Internet. Going through different search systems brings us some fruits in the form of indistinct mumbling about the untimely closed I/O data streams, TTYs and PTYs (pseudoterminals) and all the rest of it. But in all this incongruous abundance you‘ll certanly find the links to
expect
It‘s just what is wanted: the tool, which is traditionally used to communicate automatically with interactive programs. And as it always occurs, there is unfortunately a little fault in it: expect needs the programming language TCL to be present. Nevertheless if it doesn‘t discourage you to install and learn one more, though very powerful language, then you can stop your search, because expect and TCL with or without TK have everything and even more for you to write scripts.
If there is no expect installed in your system you should install it. On FreeBSD you‘d better do it by using port-system:
# cd /usr/ports/lang/expect
# make install clean
As a result expect and all it needs will be downloaded from the Internet and installed on your system.
Now we can work with the TCL and expect. As for the problem of intercative programs, the expect-script for a short telnet-session with host foo.bar.com (let it be SCO UnixWare-7.1.3), under login of luser with password TopSecret can look like that:
#!/usr/bin/expect
spawn telnet foo.bar.com
expect ogin {send luser\r}
expect assword {send TopSecret\r}
send "who am i\r"
send "exit\r"
expect eof
By the way, the README file of the expect says there is a libexpect library that can be used to write programs on C/C++ which allows to avoid the use of TCL itself. But I‘m afraid, this subject is beyond this article. Besides authors of expect themselves seem to prefer expect-scripts to the library.
Unix scripts, Page 2/4
However, if in spite of all the attractiveness of the foregoing method and all the arguments of the expect authors (see FAQ) you have made up your mind not to use expect, then you are either too lazy or entirely poisoned by Perl. Well, in this case your salvation lies in the installation of the corresponding Perl-module (http://sourceforge.net/projects/expectperl), which is supposed to support all the functions of the original expect, On FreeBSD you can carry it out by the installation from ports:
# cd /usr/ports/lang/p5-Expect
# make install clean
Now our example with telnet-session will be like that:
#!/usr/bin/perl
use Expect;
my $exp = Expect->spawn("telnet foo.bar.com");
$exp->expect($timeout,
[ ‘ogin: $‘ => sub {
$exp->send("luser\n");
exp_continue; }
],
[ ‘assword:$‘ => sub {
$exp->send("TopSecret\n");
exp_continue; }
],
‘-re‘, qr‘[#>:] $‘
);
$exp->send("who am i\n");
$exp->send("exit\n");
$exp->soft_close();
If I an mistaken and your attachment to Perl isn‘t very strong you can take the opportunity of using Python as a corresponding module pexpect is written for it (http://pexpect.sourceforge.net). It‘s clear that Python language should be installed on the system beforehand, otherwise the FreeBSD ports will help you again:
# cd /usr/ports/lang/python
# make install clean
And the same for the pexpect module:
# cd /usr/ports/misc/py-pexpect
# make install clean
The script of our telnet-session in Python will be like that:
#!/usr/local/bin/python
import pexpect
child = pexpect.spawn(‘telnet foo.bar.com‘);
child.expect(‘ogin: ‘);
child.sendline(‘luser‘);
child.expect(‘assword:‘);
child.sendline(‘TopSecret‘);
child.sendline(‘who am i‘);
child.sendline(‘exit‘);
child.expect(pexpect.EOF);
print child.before;
Certainly, if for some reason Python doesn‘t suit you either you can install, let us say, PHP language. Well, I think you realize that the searching of suitable solution can go on for a long time and may be only MS Visual Basic will be lacking in the list of results. So, I believe the time has already approached to put it all aside and come to
to the Point.
Well, now I‘ll tell you what really takes place when we start interactive applications from shell scripts. Though the last sentence is a bit in the stile of a conclusive speech of Hercules Poirot we are rather far from the end of narrative. In fact we are at its beginning. So let‘s forget everything suggested by expect and its worthy clones.
At first we‘ll try to make some rough model to simulate the expect-like programs. Let‘s use sh-scripts with all the standard UNIX tools trying to get our point and the target of this article. It‘s essential to note that all the experiments are valid for FreeBSD and I can‘t guarantee they‘ll give the same results on other operating systems.
To simplify out difficult task and not to fight with pipes mentioned in the first examples of the article let‘s make to FIFO-files: one for the standard input (in.fifo) and the other for the output (out.fifo) leaving alone the standard error-flow for now:
$ mkfifo in.fifo
$ mkfifo out.fifo
Then with the redirection its I/O to in.fifo and out.fifo let us execute unsafe telnet of which, I‘m afraid, you are already sick and tired because of numerous examples on expect, perl and python:
$ telnet -K localhost > out.fifo < in.fifo
A little step aside: as all the experiments are hold on the FreeBSD box, when we attempt to connect with the FreeBSD telnet server, SRA secure login mechanisms are automatically involved to secure telnet-session:
$ telnet localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is ‘^]‘.
Trying SRA secure login:
User (luser): <-- server waits for login, no new-line printed by server
Unix scripts, Page 3/4
To avoid it and return telnet its traditional behavior let‘s start telnet with the -K option:
$ telnet -K localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is ‘^]‘.
FreeBSD/i386 (unity) (ttyp1)
login: <-- server waits for login, no new-line printed by server
So let our first script (test_1.sh) consist of the following lines:
#!/bin/sh
mkfifo in.fifo
mkfifo out.fifo
telnet -K localhost > out.fifo < in.fifo
After executing the script:
$ ./test_1.sh &
we can begin our experiments. Pay attention to the & parameter which brings the script to the background. It‘s done for our possibility to continue working with the same terminal during the experiments. For example, let‘s try to read something from the out.fifo and to write something in the in.fifo:
$ cat out.fifo &
The & parameter plays the same above-mentioned role. Though, unfortunately the command cat will not show anything on the screen. Perhaps, it is caused by the blocks on reading/writing during the work with the FIFO-files: writing may be blocked till there is no one to read data from FIFO; and reading is blocked too till the other side isn‘t ready to write in it. May be, the whole process is hampered by our in.fifo, where there is nothing written. Let‘s check our guess by sending a newline symbol into the input-channel:
$ echo > in.fifo
Either our guess has been correct or the true reason lies somewhere else but the miracle has happened! The long-awaited results of the command cat out.fifo have at last appeared on the terminal:
$ Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is ‘^]‘.
FreeBSD/i386 (unity) (ttyp1)
login: login:
The only thing that is a little out of the way is the twice-repeated login:. Actually, that was the natural reaction of the telnet server on the newline symbol which was sent by echo > in.fifo command. So let‘s modify the command echo > in.fifo to avoid the additional newline character:
$ echo -n > in.fifo
May be, it will be even better to use the following combination:
$ cat > in.fifo &
In the future it will prevent the closing of the input stream, which is certainly interpreted by telnet server as the connection breakup.
Well, now we can go on with our research but first we must get rid of all the background processes, which were caused by using &. Now run fg command and then send ^C to finish the process:
$ fg
./test_1.sh
^C[2]  + Done                          cat out.fifo
The next step will be an attempt to implement the main functions of expect by filtering the output (out.fifo) to find the necessary data and sending an answer:
expect request {send answer}
At first glance something like this seems suitable:
$ cat out.fifo | grep request && echo answer
Though in our case this chain won‘t work correctly even at the grep point. The matter is the grep is designed to print strings in accordance with the pattern. So it‘s quite natural that before comparing every new line with the pattern grep always waits for the end either of line (‘\n‘) or file (‘{post.content}‘). This particular feature makes it impossible to intercept "login:" with the help of the grep because "login:" isn‘t followed by the newline (‘\n‘) character (the system still waits for user to enter his login-name on the same line). This is shown above on the example with telnet -K localhost. Thus grep will be waiting till the end of time for the end of line (EOL) and at last on telnet-server timeout it‘ll see just the end of file (EOF).
Unix scripts, Page 4/4
It‘s clear another way is needed to deal with these unfinished lines. As a possibility the dd command can be used instead of cat. In circle the dd will send data character by character to the grep. I mean the following construction, which is shown in the example of already working expect.sh script:
#!/bin/sh
while :; do
dd if=out.fifo bs=1b count=1 2>/dev/null | grep
if [ $? -eq 0 ]; then
# Match found
echo "" > in.fifo
exit 0
fi
# Match not found, let‘s play again
done
The script can be started in this way (files in.fifo and out.fifo should be already created):
$ ./expect.sh "request" "answer"
So the time has come to gather everything together in one script to make our traditional telnet-session work automatically:
#!/bin/sh
mkfifo out.fifo in.fifo
telnet -K localhost 1> out.fifo 0< in.fifo &
cat > in.fifo &
cat out.fifo > out.fifo &
pid=`jobid`
./expect.sh "ogin" "luser"
./expect.sh "word" "TopSecret"
sleep 1
echo ‘who am i > /tmp/test.txt‘ > in.fifo
sleep 1
echo "exit" > in.fifo
rm out.fifo in.fifo
kill $pid
After executing this script we may get, in the case of success, the following output:
$ ./test_2.sh
login:
Password:
Connection closed by foreign host.
And as the war trophy there will appear a file /tmp/test.txt to confirm the succeeding of our experiment:
$ cat /tmp/test.txt
luser             ttyp3    May 10 16:39 (localhost)
But if something was wrong, the command kill may be used for each process left after the failed experiment.
Unfortunately, this script is quite unstable: it very much depends on the value of the sleep parameter and the reply speed of the telnet-server. So the data don‘t always come in time to be properly filtered by dd and grep it causes script hang-ups. And you must kill these processes. It‘s also clear that such a construction doesn‘t work everywhere, for example, on Linux I didn‘t got any acceptable results. So maybe the adepts of Linux will succeed in it.
As for me
I decided to go on and try to find a tool, which can be used as an expect-substitute for the pure shell without any superstructure as TCL, Perl or Python. I realized it must be written in C and ported for as many operating systems as possible. Well, let‘s google! And after some time such a program is found. It‘s pty-4.0 written by Daniel J. Bernstein in 1992. But as far as I can see it haven‘t been developed after this release.
After a while I even succeeded to compile the source code of pty-4.0 into binary executable. And some parts of it began to run. But by that moment I have realized that it‘s much easier to write my own program than to sort out the old one.
And why not to write? So I set about studying the problem more carefully. Before long I found out that the most convenient way to communicate with interactive applications was really to imitate a terminal for them. And the place of pseudoterminals in the structure of the future program was determined clearly enough, in spite of the contradictive mumbling of specialists from Internet about expect and PTY-sessions. Next, it also because clear that it makes everything easy to start applications under the control of the PTY-sessions inside some kind of shell, for example TCL, Perl and others. Though nothing prevent us from using C and pure sh interpreter.
As a result of all this in a couple of weeks I had a working version of empty (http://www.sourceforge.net/projects/empty) which allows to start interactive programs and communicate with them using FIFO-files. For example, the FreeBSD telnet-session in the sh-script for empty will look like that:
#!/bin/sh
empty -f -i in.fifo -o out.fifo telnet -K localhost
empty -w -i out.fifo -o in.fifo -t 5 "ogin:" "luser"
empty -w -i out.fifo -o in.fifo -t 5 "assword:" "TopSecret"
empty -s -o in.fifo ‘who am i > /tmp/test.txt‘
empty -s -o in.fifo ‘exit‘
Well, it‘s much shorter then the buggy test_2.sh script and works quite stable on BSD, Linux and Solaris. Besides, it doesn‘t require TCL, Perl or Python. I admit there aren‘t so many functions in the program yet as there are in other expect-like tools, but I hope everything is still ahead.