Error: cannot open tty-output

I wrote a script whose purpose is to offer a user an option to select test or production environment when connecting to a server and then, based on the selection, script configures environment variables appropriately. To make a script a bit more user friendly I used dialog utility. All good, until user logs in and then switches to another, non-root(!), user using su command at which point the following error is reported:

cannot open tty-output

Well, a simple and quick use of strace revealed what is the problem. Namely, initial user is owner of pseudo-terminal (e.g. /dev/pts/3) and when he/she switches to another user the ownership isn't changed so opening a terminal device is unsuccessful. What caused dialog tool to try to open pseudo-terminal in the first place was --stdout switch, and if this switch isn't used then there is no error about tty-output. But then, there is another problem and that's why this option was used in the first place. Namely, I used dialog in the following way:

ANS=`dialog ... --stdout`

to catch output of the command into variable and to be able to test it within if statement. The problem was that there was no output without --stdout option, as it was going to stderr and thus invoking shell couldn't catch output and place it into variable ANS. That is, dialog tool uses stdout to draw on terminal and stderr to output user's response!

There was solution to use temporary file for that purpose, but I was reluctant to do so. The first idea was to use pipe and read statement, something like this:

dialog ... | read ANS

and in that way to avoid use of --stdout. But the problem is that pipe components (command on left and right of pipe character) are executed within subshells and thus, the result of read command isn't visible to a parent shell. In other words, this is dead end. And besides, I still have a problem of read getting the data via dialog's stdout!

In the end, I modified dialog invocation in the following way:

ANS=`dialog ...  2>&1 > /dev/tty`

What this does is the following: parent shell invokes subshell that will execute everything between backticks, but first, it redirects stdout so that it can later place output to variable ANS. Subshell then duplicates stderr so that it points to stdout (which isn't actually stdout but is a pipe to parent shell). In this way the output of the dialog command will be taken by parent bash, i.e. the one that starts subshell to execute backtick command. Then, stdout is redirected to /dev/tty that will be always possible to open/read/write and that allows dialog to control display on terminal.