LDAS logo
TclDOC logo

The genericAPI.tcl Script

Modification Date: 11/04/2009

Table of Procedures

red ball Main
red ball addNewResourcesToFile
red ball anonFtpToplevel
red ball apiDirectory
red ball bCrypt
red ball bak
red ball bakDemangle
red ball bgLoop
red ball binaryDecrypt
red ball binaryEncrypt
red ball bootLock
red ball checkLeaks
red ball checkMySetup
red ball cloneProc
red ball cmdGet
red ball cmdSets
red ball cmonResource::deleteResourcesFromFile
red ball countChannels
red ball cppBase64Init
red ball cryptFileInPlace
red ball curlGetRemoteFileSize
red ball decryptFile
red ball destructElementWrap
red ball dropStatusPage
red ball dumpData
red ball dumpFile
red ball execscp
red ball execssh
red ball expandOpts
red ball fifo
red ball fileExists
red ball fileIsBeingWritten
red ball fileType
red ball fixUrlTarget
red ball formatAndProtocol
red ball formatConversion
red ball freeMemOnBox
red ball ftpPutLocal
red ball getApiOS
red ball getCurlUrl
red ball getCurlUrlCallback
red ball getIPAddress
red ball getMddTarget
red ball getSockData (reads data from a socket, usually
red ball getUrl
red ball gifBalls
red ball gridFtpToplevel
red ball ifConfig
red ball infoVars
red ball int2roman
red ball isBinary
red ball itemCount
red ball jobDirectory
red ball lastline
red ball leakLogger
red ball leaksSummary
red ball libstdcPlusPlus
red ball macroReturnMsg
red ball managerOutputUrl
red ball mapTCLVarToCVar
red ball mapTCLVarToCVar
red ball memFlag
red ball metaOpts
red ball myIP
red ball myName
red ball newhead
red ball newtail
red ball numRange
red ball outputFormat
red ball outputUrls
red ball outputUrlsBg
red ball packageReport
red ball parseURL
red ball pingAPI
red ball pongAPI
red ball popMsg
red ball portInfo
red ball procList
red ball procServer
red ball pubDirSetup
red ball publicFile
red ball putCurlUrl
red ball putUrl
red ball randomNumber
red ball realTimeRscValues
red ball relativeDirectory
red ball revArray
red ball roVar
red ball safeRxPat
red ball saveResourceToFile
red ball setAlertDebug
red ball setAlertDebugCB
red ball setLdasSystemName
red ball setResourceLimit
red ball shellPipe
red ball sleep
red ball sourceFile
red ball sourceRsc
red ball stats
red ball touch
red ball trace
red ball traceTimeout
red ball ucase
red ball unknown
red ball unpackTarball
red ball unwrapText
red ball url2file
red ball validFilename
red ball validProc
red ball validateEtcHosts
red ball varType
red ball when
red ball wrapText

# The Laser Interferometer Gravitational Observatory Data Analysis System genericAPI.tcl script.
This module sources the following sub-modules:

  1. stack.tcl (stack and queue manipulation functions)
  2. service.tcl (service availability and addressing)
  3. log.tcl (logging functions)
  4. cmd.tcl (command formatting functions)
  5. sock.tcl (socket communication functions)
  6. key.tcl (key generation and encryption functions)
  7. ilwd.tcl (ilwd text and object manipulation functions)
  8. timers.tcl (timing routines for benchmarking, etc.)
  9. ftp.tcl (ftp functions)
  10. queue.tcl (queue management functions)
  11. gpstime.tcl (gps time conversion routine)
  12. procfs.tcl (routines for parsing /proc file systems)
  13. smtp.tcl (routines implementing SMTP)
It is anticipated, and it is the criterion for their inclusion here, that all or most of these functions will be used in all or many of the other API's comprising the LIGO Data Analysis System.

Name: Main

Description:
Define the installation directory for the LDAS modules.
This is the "main" procedure for the
genericAPI.tcl module.
The local directory structure may be defined in any of the following ways:
  1. The environment variables LDASDIR, LDASHELPDIR, LDASLOGDIR, etc. can be set to the absolute path names.
  2. The variables LDAS, LDASHELP,LDASLOG, etc. can be set in the resource file to the absolute paths.
  3. Either LDASDIR or LDAS can be set, in which case LDASHELP and LDASLOG will be inferred to be the "help" and "log" directories immediately below it.
  4. If none of the variables is set then the current working directory will be taken as the top of the LDAS tree, and the help and log directories will be assumed to exist immediately below it.
Comments:
In any case where a directory location is inferred an announcement will be made to stderr describing the default action.
set ::RCS_ID_genericAPItcl { $Id: genericAPI.tcl,v 1.671 2009/09/28 18:44:57 emaros Exp $ }
set ::RCS_ID_genericAPItcl [ string trim $::RCS_ID_genericAPItcl "\$" ]
Prints a tcl proc callstack in stderr if resource is enabled
Must reboot API to take effect
if { [ info exist ::DEBUG_PROC ] && $::DEBUG_PROC } {
	catch { info proc *proc } err
	if	{ [ lsearch $err proc ] == -1 } {
    	rename proc tcl_proc
    	set ::native_proc tcl_proc
		puts stderr "tcl command 'proc' renamed to tcl_proc"
	}
	if	{ [ lsearch [ info command tcl_proc ] tcl_proc ] != -1 } {
		tcl_proc proc { args } {
			if 	{ [ lsearch [ info command puts ] puts ] != -1 } {
				set putcmd puts
			} 	elseif { [ lsearch [ info command tcl_puts ] tcl_puts ] != -1 } {
				set putcmd tcl_puts
			}	 else {
				set putcmd ::puts
			}
	
			;## no tcl_puts
			# $putcmd "args '$args'"
			set name [ lindex $args 0 ]
			set params [ lindex $args 1 ]
			set body [ lrange $args 2 end ]
			set body [ lindex $body 0 ]
			#$putcmd stderr "no renamed puts: name=$name, params='$params', body '$body' "
	
			set text "catch { info level -1 } caller\n$putcmd \"\[ clock seconds \]: $name called by \$caller\""
			set body "$text\n$body"
	
			#$putcmd "body '$body'"
	
			eval uplevel tcl_proc [ list $name [ list $params ] [ list $body ] ]
			# $putcmd "$name proc [ info proc $name ]"
		}
	}
} else {
	set ::native_proc proc
	# addLogEntry "tcl command 'proc' is unchanged" purple
}

§   §   §
package provide generic 1.0 set genericAPI TRUE package require -exact stack 1.0 package require -exact procfs 1.0 package require -exact smtp 1.0 package require -exact service 1.0 package require -exact log 1.0 package require -exact ilwd 1.0 package require -exact cmd 1.0 package require -exact sock 1.0 package require -exact stat 1.0 ;##package require -exact gui 1.0 package require -exact key 1.0 package require -exact queue 1.0 #package require FTP 1.3 package require -exact timers 1.0 package require http 2.0 package require base64 1.0 package require gpstime 1.0 package require RawGlobus 1.0 package require RawGlobusClient 1.0
§   §   §
Create custom versions of some internal procs set ::native_trace trace if { [ lsearch [ info proc setAlert ] setAlert ] == -1 } { ##------------------------------------------------------------------- ## Use 8.3 syntax to ensure compatability ##------------------------------------------------------------------- $::native_trace variable ::TID_FINISHED rw ::setAlertDebugCB } else { ::setAlertDebugCB }
§   §   §
if { [ lsearch [ info proc *trace ] tcl_trace ] == -1 } { rename trace tcl_trace set ::native_trace tcl_trace }
§   §   §
unset ::native_trace
Name: trace

Description:
Redefine trace to allow for additional debugging
Parameters: Usage:

Comments:

proc trace { args } {
    if { [ info exists ::DEBUG_TRACE ] \
	     && $::DEBUG_TRACE == 1 \
	     && [ info exists ::operator_socket ] } {
	set msg "'[ info level -1 ]' calling Trace: trace $args"
	addLogEntry $msg purple
    }
    if { [regexp {8.3} $::tcl_version]  \
	     && ( [llength $args] > 2 ) \
	     && ![regexp {variable|vdelete|vinfo} [lindex $args 0] ] \
	     && [regexp {variable} [lindex $args 1] ] } {
	## **************************************************************
	## Provide tcl 8.4 syntax for 8.3 interpreter
	## **************************************************************
	set majorcommand [lindex $args 0]
	set tuype [lindex $args 1]
	set name [lindex $args 2]
	set ops [list]
	set minorcommand [list]
	if { [llength $args] > 2 } {
	    foreach op [lindex $args 3] {
		switch -exact $op {
		    read {
			set ops "r$ops"
		    }
		    write {
			set ops "w$ops"
		    }
		    unset {
			set ops "u$ops"
		    }
		}
	    }
	    if { [string length $ops] <= 0 } {
		set $ops [lindex $args 3]
	    }
	
	    set minorcommand [list [lindex $args 4] ]
	}
	switch -exact $majorcommand {
	    add {
		uplevel tcl_trace variable $name $ops $minorcommand
	    }
	    remove {
		uplevel tcl_trace vdelete $name $ops $minorcommand
	    }
	    info {
		uplevel tcl_trace vinfo $name
	    }
	    default {
		uplevel tcl_trace $args
	    }
	}
	
    } else {
	uplevel tcl_trace $args
    }
}

§   §   §

Name: setAlertDebug

Description:
Redefine setAlert to allow for additional debugging Do NOT call this function as setAlertDebug as it will be renamed to setAlert once the genericAPI library has been loaded.

Parameters: Usage:

Comments:

proc setAlertDebug { args } {
    uplevel ldas_setAlert $args
    if { [ info exists ::DEBUG_TRACE ] \
	     && $::DEBUG_TRACE == 1 \
	     && [ info exists ::operator_socket ] } {
	set msg "'[ info level -1 ]' setAlert $args -- [getTIDDebugInfo [ lindex $args 0] ]"
	addLogEntry $msg purple
    }
}

§   §   §

Name: setAlertDebugCB

Description:
Helper function for setting up tracing of setAlert calls
Parameters: Usage:

Comments:

proc setAlertDebugCB { args } {
    if { [ lsearch [ info proc *setAlert ] ldas_setAlert ] == -1 } {
	trace remove variable ::TID_FINISHED [list read write] ::setAlertDebugCB
	rename setAlert ldas_setAlert
	rename setAlertDebug setAlert
    }
}

§   §   §

Name: checkMySetup

Description:
Checks the LDAS installation, performing the following steps:
  1. See if the machine knows it's name. if not, check for the existence of the variable $LOCALHOST. If the machine name cannot be determined, a Tcl exception is thrown.
    No attempt is made to defeat spoofing.
  2. Check to see if the LDAS toplevel directory has been declared, and whether the subdirectory locations are set. Set the toplevel to "pwd" if it's not found, and set the required subdirs to reasonable places beneath it.
  3. Check the declarations of required resource variables, throws an exception if any required value is not found.
  4. Finally, it checks for a valid filename for the local log file, and make up a likely one based on the name of the resource file if it needs to.
Usage:
       checkMySetup
Where "API" should be declared at the top of the resource
file by a line like:

set API usr
Comments:
Should be called after sourcing the
genericAPI.tcl and the local resource file.
proc checkMySetup { } {
     if { ! [ info exists ::API ] } {
        set msg    "The required variable \"API\" has not\n"
        append msg "been set, this probably means you are\n"
        append msg "trying to source the genericAPI.tcl\n"
        append msg "without providing a dummy .rsc file.\n"
        return -code error $msg
        }
     if { ! [ info exists ::env(HOST) ] } {
        if { ! [ info exists ::LOCALHOST ] } {
           set msg    "Your machine does not know it's own name,\n"
           append msg "and you have not set the variable\n"
           append msg "\"LOCALHOST\" to your machines' name.\n"
           append msg "please set the variable \"LOCALHOST\"\n"
           append msg "in your local LDAS${::API}.rsc file."
           return -code error $msg
        }
     }
     if { ! [ info exists ::LOCALHOST ] } {
        set ::LOCALHOST $::env(HOST)
     }
     roVar LOCALHOST
     set ::MY_IP [ myIP ]
     roVar MY_IP
     set ::BAD_WORDS (rename|open|socket|proc|file|exec|cd|pwd|load|exit|source|send)
     if { $::DONT_BLOCK_BAD_WORDS } {
        set ::BAD_WORDS bad_words_are_not_blocked_so_look_out
     }
     roVar ::BAD_WORDS
     foreach dir { {} LIB HELP LOG TMP ARC MACROS } {
        set dir "LDAS$dir"
        if { ! [ info exists ::$dir ] || \
             ! [ string length [ set ::$dir ] ] } {
           switch -exact -- $dir {
              LDAS {
                   set err "The variable ::LDAS seems to be undefined.\n"
                   append err "since this variable is set using the\n"
                   append err "autoconf macro \@prefix\@, this is a\n"
                   append err "serious problem.\n\n"
                   append err "Please examine the top section of the\n"
                   append err "managerAPI and LDASgwrap executable\n"
                   append err "scripts for the place where ::LDAS is\n"
                   append err "defined!"
                   return -code error $err
                   }
           LDASLIB {
                   set ::$dir [ file join $::LDAS lib ]
                   }
          LDASHELP {
                   set ::$dir [ file join $::LDAS help ]
                   }
           LDASLOG {
                   set ::$dir [ file join $::env(RUNDIR) logs ]
                   file mkdir [ set ::$dir ]
                   file attributes [ set ::$dir ] -permissions 0755
                   gifBalls   [ set ::$dir ]
                   }
           LDASTMP {
                   set ::$dir [ file join $::env(RUNDIR) tmp ]
                   file mkdir [ set ::$dir ]
                   file attributes [ set ::$dir ] -permissions 0770
                   }
           LDASARC {
                   set ::$dir [ file join $::LDASLOG archive ]
                   file mkdir [ set ::$dir ]
                   file attributes [ set ::$dir ] -permissions 0755
                   }
        LDASMACROS {
                   set ::$dir [ file join $::LDAS share ldas macros ]
                   }
               } ;## end switch
           if { $::DEBUG } {
              puts stderr "$dir set to [ set ::$dir ]"
           }
        } ;## end of if not info exists dir
        roVar $dir
     } ;## end foreach
     pubDirSetup
     if { ! [ info exists ::REQUIRED_VARIABLES ] } {
        set msg    "No list of required variables declared!"
        append msg "please put the list \"REQUIRED_VARIABLES\"\n"
        append msg "in your local LDAS${::API}.rsc file."
        return -code error $msg
     }
     foreach var $::REQUIRED_VARIABLES {
        if { ! [ info exists ::$var ] } {
           set msg    "Variable: $var not intitialised in\n"
           append msg "your local .rsc file.\n"
           append msg "This variable must be initialised."
           return -code error $msg
        }
     }
     ;## set standard ports for all API's
     set i 0
     foreach api $::API_LIST {
        array set ::$api "host [ set ::${::LDAS_SYSTEM}($api) ]"
        foreach sock { operator emergency data } {
           array set ::$api "$sock [ expr { $::BASEPORT + [ incr i ] } ]"
        }
    }
    unset i
    if { ! [ info exists ::LOCAL_LOG_FILE ] } {
       if { [ info exists ::LOCAL_LOG ] } {
          if { ! [ regexp {LDAS[a-z]+\.log} $::LOCAL_LOG ] } {
             set msg    "Malformed log file name:\n"
             append msg "$LOCAL_LOG.\n"
             append msg "Log file name must be of the form:\n"
             append msg "\"LDASapi.log\", where api is the\n"
             append msg "name of the current api, which might\n"
             append msg "be: user, mgr, frame, etc."
             return -code error $msg
          }
          if { $::DEBUG } {
             puts stderr "Variable LOCAL_LOG_FILE not set.\n"
             puts stderr "Setting it to $::LDASLOG/LDAS${::API}.log."
          }
       } else {
          set ::LOCAL_LOG "LDAS${::API}.log"
       }
       set ::LOCAL_LOG_FILE [ file join $::LDASLOG $::LOCAL_LOG ]
    }
}

§   §   §

Name: validateEtcHosts

Description:

Parameters: Usage:

Comments:

proc validateEtcHosts { args } {
     if { [ catch {
        set data [ dumpFile /etc/hosts ]
        set data [ split $data "\n" ]
        foreach line $data {
           if { [ regexp {^\s*127.0.0.1} $line ] } {
              break
           }
        }
        if { ! [ string equal localhost $::env(HOST) ] && \
               [ lsearch $line $::env(HOST) ] != -1 } {
           set subject "$::API API: $::env(HOST) aliased to loopback!"
           set msg    "/etc/hosts file declares $::env(HOST) to "
           append msg "be an alias for the loopback address.\n"
           append msg "if all API's in this LDAS system are running\n"
           append msg "on $::env(HOST) this is not a problem, but if\n"
           append msg "any API's run on OTHER MACHINES this is likely\n"
           append msg "to cause data socket communication problems!"
           return -code error $subject
        }
     } err ] } {
        puts stderr $subject
        addLogEntry "Subject: ${subject}; Body: $msg" email
     }
}

§   §   §

Name: setLdasSystemName

Description:
Populates the value of the ::LDAS_SYSTEM resource variable.

Parameters: Usage:

Comments:

proc setLdasSystemName { } {
     if { [ catch {
        if { [ file exists /etc/ldasname ] } {
           set ::LDAS_SYSTEM [ dumpFile /etc/ldasname ]
        } else {
           set ::LDAS_SYSTEM localhost
        }
        ;## convert trailing integers to roman numerals!
        if { [ regexp {(.+)(\d+)$} $::LDAS_SYSTEM -> name int ] } {
           set ::LDAS_SYSTEM $name[ int2roman $int ]
        }
        set ::LDAS_SYSTEM [ string trim $::LDAS_SYSTEM ]
        roVar LDAS_SYSTEM
        set ::RUNCODE [ string toupper $::LDAS_SYSTEM ]
        puts stderr "::LDAS_SYSTEM set to '$::LDAS_SYSTEM'"
        ;## turn ::${::API}_API_HOST resource variables
        ;## into the system api host list.
        set hosts [ info vars ::*_API_HOST ]
        foreach host $hosts {
           regexp {::([^_]+)} $host -> api
           set host [ set $host ]
           set api [ string tolower $api ]
           set ::${::LDAS_SYSTEM}($api) $host
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: pubDirSetup

Description:
Determine public ftp and http areas, providing defaults if none are provided, as possible.

Parameters: Usage:

Comments:

Helper function for checkMySetup
proc pubDirSetup { } {
     if { ! [ info exists  ::FTPDIR ] || \
          ! [ file exists $::FTPDIR ] } {
        set ::FTPDIR  [ anonFtpToplevel ]
        ;## anon ftp may be set up badly -- it happens!
        if { [ string length $::FTPDIR ] && \
           ! [ file exists $::FTPDIR ]   && \
             [ string equal manager $::API ] } {
           file mkdir $::FTPDIR
           file attributes $::FTPDIR -permissions 0755
           puts stderr "toplevel ftp directory created: '$::FTPDIR'"
        }
        if { [ string length $::FTPDIR ] && \
           ! [ file writable $::FTPDIR ] } {
           if { [ file writable \
                [ file join $::FTPDIR pub outgoing ] ] } {
              set ::FTPDIR [ file join $::FTPDIR pub outgoing ]
           } elseif { [ file writable \
                      [ file join $::FTPDIR pub ] ] } {
              set ::FTPDIR [ file join $::FTPDIR pub ]
           } else {
              set ::FTPDIR [ list ]
           }
        }
     }
     set rel [ list ]
     set cwd $::env(RUNDIR)
     ;## no browser access to cwd!
     if { ! [ file exists index.html ] } {
        set fid [ open index.html w 0444 ]
        puts $fid <HTML>
        close $fid
     }
     regsub $::WORKING_DIRECTORY_MOUNT_POINT $cwd {} cwd
     regsub [ anonFtpToplevel ] $::FTPDIR {} rel
     if { ! [ info exists  ::FTPURL ] || \
          ! [ file exists $::FTPURL ] } {
        set ::FTPURL  ftp://$::MY_IP$rel
     }
     if { ! [ info exists  ::HTTPURL ] || \
          ! [ file exists $::HTTPURL ] } {
        set ::HTTPURL http://${::MY_IP}${cwd}/jobs
     }
     if { ! [ info exists  ::HTTPDIR ] || \
          ! [ file exists $::HTTPDIR ] } {
        set ::HTTPDIR [ file join $cwd jobs ]
     }
     if { ! [ info exists  ::PUBDIR ] || \
          ! [ file exists $::PUBDIR ] } {
        set ::PUBDIR $::HTTPDIR
     }
     set fname [ file join $::PUBDIR .htaccess ]
     if { ! [ file exists $fname ] } {
        file mkdir $::PUBDIR
        file attributes $::PUBDIR -permissions 0775
        set fid [ open $fname w 0444 ]
        puts $fid "DefaultType application/octet-stream"
        close $fid
     }
     setLdasSystemName
     set ::GRIDFTPDIR [ gridFtpToplevel ]
     if { [ string length $::GRIDFTPDIR ] } {
        set ::GRIDFTPURL gridftp:$::GRIDFTPDIR
     } else {
        set ::GRIDFTPURL [ list ]
     }
     puts stderr "::FTPDIR set to '$::FTPDIR'"
     puts stderr "::FTPURL set to '$::FTPURL'"
     puts stderr "::HTTPDIR set to '$::HTTPDIR'"
     puts stderr "::HTTPURL set to '$::HTTPURL'"
     puts stderr "::GRIDFTPURL set to '$::GRIDFTPURL'"
     puts stderr "::GRIDFTPDIR set to '$::GRIDFTPDIR'"
     puts stderr "::PUBDIR set to '$::PUBDIR'"
}

§   §   §

Name: getIPAddress

Description:
Returns the IP address of a machine, given the machine name. Will work on any UNIX system. Will do a reverse if an IP address is given!
Usage:
      set ip_address [ getIPAddress $host|$ipaddress ]
Comments:
proc getIPAddress { host } {
     set retval [ list ]
     if { [ catch {
        set sid [ socket -async $host 22 ]
        after 100
        set data [ fconfigure $sid -peername ]
        close $sid
        foreach [ list ip hostname port ] $data { break }
        if { [ string equal $host $ip ] } {
           set retval $hostname
        } else {
           set retval $ip
        }
     } err ] } {
        catch { ::close $sid }
        puts stderr "getIPAddress: failed to connect to $host port 22: '$err'"
     }
     return $retval
}

§   §   §

Name: myIP

Description:
Returns the IP address of the host machine. The version which is commented out is the canonical version, which can, under rare circumstances take VERY long to return on error.
The new version relies on the existence of sshd on the machine, which is an LDAS requirement.

Parameters: Usage:
       set ::__myip [ myIP ]
Comments:
Very slow! Should be used to create a global variable as in the usage example.
proc myIP {} {
     if { [ catch {
        set ip 127.0.0.1
        set sid [ socket -async [ info hostname ] 22 ]
        set ip  [ lindex [ fconfigure $sid -sockname ] 0 ]
        ::close $sid
     } err ] } {
        catch { ::close $sid }
        puts stderr "myIP error: '$err' on port 22 (sshd). using 127.0.0.1"
     }
     return $ip
}

§   §   §

Name: ifConfig

Description:
Returns a list of lists consisting of the interface id string, the IP address for the interface, and the aliases for the IP address from /etc/hosts.

Parameters: Usage:
  Example output:

{lo0 127.0.0.1 localhost} {ge0 10.16.0.5 gateway} {hme0 131.215.115.248 {ldas-dev ldas-dev.ligo.caltech.edu loghost}}
Comments:
proc ifConfig { args } {
     if { [ catch {
        set interfaces [ list ]
        if { [ file executable /usr/sbin/ifconfig ] } {
           catch { ::exec /usr/sbin/ifconfig -a } data
        } elseif { [ file executable /sbin/ifconfig ] } {
           catch { ::exec /sbin/ifconfig -a } data
        } else {
           return -code error "can't find 'ifconfig' executable!"
        }
        set fid [ open /etc/hosts r ]
        set hostdata [ read $fid [ file size /etc/hosts ] ]
        ::close $fid
        foreach line [ split $hostdata "\n" ] {
           array set hosts \
              [ list [ lindex $line 0 ] [ lrange $line 1 end ] ]
        }
        foreach line [ split $data "\n" ] {
           regexp {^(lo |[a-z]+\d+)} $line -> if
           set if [ string trim $if ]
           if { [ regexp {^\s+inet\s+(?:addr:)?(\S+)} $line -> ip ] } {
              if { [ info exists hosts($ip) ] } {
                 lappend interfaces [ list $if $ip $hosts($ip) ]
              } else {
                 lappend interfaces [ list $if $ip dhcp ]
              }
           }
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $interfaces
}

§   §   §

Name: validProc

Description:
Determines if a procedure is valid in the context within which it is called. Just allows a clean test for usability.

Parameters: Usage:
      if { ! [ validProc "procname" ] } { complain }
Comments:
Best made use of in the negated sense as shown above.
proc validProc { { name * } } {
     set namesp [ namespace children ]
     ;## if the argument was namespace aware
     if { [ regexp {::} $name ] } {
        set namesp {}
        }
     ;## if the arg was of the form ::foo*
     if { [ regexp {^(::[^:]+)\*} $name -> tmp ] } {
        set name ${tmp}::*
        }
     ;## examine all namespaces visible locally
     foreach ns $namesp {
        if { ! [ regexp {::$} $ns ] } {
           set ns ${ns}::
           }
        if { [ llength [ info commands $ns$name ] ] } {
           return 1
           }
        }
     ;## if the argument was a non-namespace proc
     if { [ llength [ info commands $name ] ] } {
        return 1
        }
     return 0
}

§   §   §

Name: procList

Description:
Returns a list of all the procs in the scope of the caller.
Usage:
      set ${API}procs [ procList ]
Comments:
Must be called by the interpreter whose proc list you want. Useful for knowing which interpreter to send a request to when multiple interps are available.
proc procList { { globpat * } { level 1 } } {
     return [ uplevel $level info commands $globpat ]
}

§   §   §

Name: revArray

Description:
generates reverse lookup key/value pairs for an array.
Usage:
       array set rev_array [ revArray array_name ]
Comments:
Will cause the array in the caller to be supplemented to include reverse lookup values. I could also be called so as to return the reverse values to a different array, useful when the array contents are dynamic.
proc revArray { { name "" } } {
     if { ! [ array exists $name ] } {
        return -code error "No array named \"$name\" in scope."
        }
     if { [ catch {
        array set local_array [ uplevel [ array get $name ] ]
        } err ] } {
        return -code error $err
        }
     set reverse [ list ]
     foreach {name value} [ array get local_array ] {
     lappend reverse $value $name
     }
     return $reverse
}

§   §   §

Name: dumpFile

Description:
return the contents of a file in a form suitable for "more" or "less" etc.
Usage:
      set data [ dumpFile filename ]
Comments:
This is an efficient slurper of files.
proc dumpFile { { file "" } } {
     if { ! [ string length $file ] } {
        return {}
        }
     if { ! [ file exists $file ] } {
        return {}
        }
     if { [ catch { set fid [ open $file r ] } err ] } {
        return -code error $err
        }
     set size [ file size $file ]
     if { $size == 0 } {
        set size 100000
     }
     set data [ read $fid $size ]
     catch { ::close $fid }
     set data [ string trim $data ]
     return $data
}

§   §   §

Name: publicFile

Description:
Create a new file in the ::PUBDIR of the API according to the correct naming convention for the subdirectory and filename.
Returns the correct full path to the output file, which can be immediately used by outputUrls.
The idea is, that you can pass a URI to publicFile, and then pass the return value of publicFile and the URI to outputUrls and the local file will mirror to the remote location.
This is the canonical LDAS method for creating a public space file.

Parameters: Usage:

Comments:

Fails silently on error with a log entry.
First level backups have extension .bak, second level has extension .bak2.
proc publicFile { jobid fname contents { format binary } { comp none } } {
     set seqpt {}
     if { [ catch {
        regexp {\d+} $jobid job
        set jobid $::RUNCODE$job
        ;## remove all blank spaces from dir and file names
        ;## and replace with underscores.
        regsub -all -- {\s+} $fname {_} fname
        ;## find out as much about the object to be written as
        ;## possible
        foreach [ list type contents ] [ varType $contents ] { break }
        ;## because sometimes a bug makes us try to copy
        ;## the *root* directory!!
        if { [ regexp {directory} $type ] } {
           set err "attempt to copy directory: '$contents' "
           append err "made in file context."
           return -code error $err
        }
        ;## determine the correct file extension intelligently
        set extension [ file extension $fname ]
        if { ! [ string length $extension ] } {
           if { [ regexp {unknown} $type ] } {
              set extension .txt
           } elseif { [ regexp {ilwd} $type ] } {
              set extension .ilwd
           } elseif { [ regexp {ligo_lw} $type ] } {
              set extension .xml
           }
        }
        ;## create the fully qualified path of the output file
        set rootname  [ file rootname [ file tail $fname ] ]
        set fname $rootname$extension
        set dir [ jobDirectory $jobid ]
        set fname [ file join $dir $fname ]
        ;## don't overwrite existing files
        bak $fname
        ;## do the real work of writing the file
        if { [ regexp {ilwd pointer} $type ] } {
           set seqpt {}
           ilwd::write2disk $jobid $fname $contents $format $comp
        ;## this can only happen in the frame API
        } elseif { [ regexp {framecpp pointer} $type ] } {
           set seqpt {}
           frame::writeFile $jobid $fname $contents
        } elseif { [ regexp {file} $type ] } {
           file copy -force -- $contents $fname
        } else {
           set fid [ open $fname w 0664 ]
           puts $fid $contents
           close $fid
           unset fid
        }
        if { [ file exists $fname ] } {
           file attributes $fname -permissions 0664
        }
     } err ] } {
        if { [ info exists fid ] } {
           if { [ catch {
              close $fid
           } err2 ] } {
              set err "$err: $err2"
           }
        }
        return -code error "[ myName ]:$seqpt $err"
     }
     return $fname
}

§   §   §

Name: validFilename

Description:
Detects the use of disallowed characters in a filename.
Handles cases of root filenames, filenames with an extension, and full path prepended filenames.
Usage:
      if { [ validFilename filename ] } { do }
Comments:
Unix users will find things a bit draconian.
Not exhaustive. Simply a first line of defense.
proc validFilename { { filename "" } } {
     set flag 1
     if { [ regexp {[^a-zA-Z0-9\-\_\.\~]} $filename ] } {
        set flag 0 ;## invalid filename!
        }
     set flag ;## filename is ok!
}

§   §   §

Name: pingAPI

Description:
Ping a port and see if the associated process lives.
get a little mini-report and put it in the managers file at some interval. Generate an alarm if something is awry.
Usage:
      pingAPI $api
Comments:
This works without blocking on an error!
proc pingAPI { api } {
     set busy    0
     set sid    {}
     set retval {}
     if { [ catch {
        set sid [ sock::open $api emergency ]
        fconfigure $sid -blocking off
        set pingkey [ key::time ]
        __t::start $pingkey
        puts $sid "$::MGRKEY {\${::API}::reply \$cid \[ pongAPI \]}"
        flush $sid
        set i 0;
        while { 1 } {
           incr i 20
           ;## if we get a response we're done
           if { [ regexp {\d{9,10}} [ gets $sid ] ] } {
              break
           } else {
              ;## how long have we been waiting?
              if { $i < 1000 } {
                 after 20
              } else {
                 ;## too long
                 set busy 1
                 break
              }
           }
        }
        set tmark [ ::__t::mark $pingkey ]
        ::__t::cancel $pingkey
        set tmark [ format "%.3f" $tmark ]
        set retval "mgr -> $api -> mgr in $tmark"
     } err ] } {
        set retval "$api API unreachable! $err"
     }
     if { $busy } {
        set host [ set ::${::LDAS_SYSTEM}($api) ]
        set status [ sock::diagnostic $host ]
        foreach { flag msg } $status { break }
        if { $flag } {
           set retval "$api API unreachable! $msg"
        } else {
           if { [ info exists retval ] } {
              if { [ regexp {unreachable} $retval ] } {
              } else {
                 set retval "$api emergency port is open but unresponsive"
              }
           }
        }
     }
     catch { ::close $sid }
     return $retval
}

§   §   §

Name: pongAPI

Description:
Pong back local gpsTime value Usage:

Comments:

Currently deprecated
proc pongAPI {} {
     return [ gpsTime ]
}

§   §   §

Name: popMsg

Description:
General purpose error annunciator function.
If you are running Tcl w/o Tk the message will just be puts'd, otherwise it will be in a little window located relative to the parent window.
Usage:
       popMsg msg win {delay}
Comments:
Note that to set a delay time different from the default of 2.5 seconds it is necessary to pass a "win" parameter, which may be "".
Technically this is lower priority than any of the logging functions, since no long entry is made.
proc popMsg { { msg "" } { win "" } { delay 2500 } } {
     global window; set window $win
     catch { [ winfo ] } errmsg
     if { [ string match invalid* $errmsg ] } {
        puts stderr $msg
        set msg {}
        }
     if { [ string length $msg ] } {
        destroy $win.mess
        frame   $win.mess       -class message \
                                -borderwidth 4 \
                                -relief raised
        label   $win.mess.label -foreground red \
                                -text $msg
        pack $win.mess.label -padx 12 -pady 12
        catch { ;## bury error if $win == ""
        place $win.mess -rely .1 -relx .25 -in $win
        }
        after $delay {
              destroy $window.mess
              }
        }
     return {}
}

§   §   §

Name: sourceRsc

Description:
Sources the LDASapi.rsc resource file in global context.
Usage:

Comments:

First tries to source the resource file in the current working directory. If it doesn't find it it will try to source the default resource file in $::LDAS/bin, which is probably NOT what you want!
proc sourceRsc { } {
     if { [ catch {
        set rscfile [ file join $::env(RUNDIR) LDASapi.rsc ]
        if { [ file exists $rscfile ] } {
           uplevel source $rscfile
        } else {
           uplevel source [ file join $::LDAS bin LDASapi.rsc ]
        }
        } err ] } {
        return -code error "Error sourcing LDASapi.rsc: $err"
        }
}

§   §   §

Name: sourceFile

Description:
Source a file. Limits the range of sourceable files to those found in the LDAS installation directory tree.
Usage:
      sourceFile filename subdir
Comments:
Just encapsulates a file sourcing function which has the "look and feel of the LDAS API's.
proc sourceFile { { filename "" } { subdir "" } } {
     if { ! [ string length $filename ] } {
        return -code error "sourceFile: filename not given"
        }
     if { [ string length $subdir ] } {
        set filename [ file join $subdir $filename ]
        }
     if { ! [ regexp $::LDAS $filename ] } {
        set filename [ file join $LDAS $filename ]
        }
     if { [ file exists $filename ] } {
        uplevel source $filename
        } else {
        set msg    "Tried to source:\n$filename\n"
        append msg "File not found."
        return -code error "sourceFile:\n$msg"
        }
     return {}
}

§   §   §

Name: myName

Description:
Returns the name of the proc that calls it.
Usage:
      set myname [ myName ]
Comments:
This is the canonical form for getting a procs name.
The utility of this is that a proc may be dynamically named, and the name managed much more easily this way than by trying to stack up and unstack proc names.
Note that level 0 is the context of myName, level -1 is the context of the caller. In general, you will not mess with the level. Though you could get the name of the proc that called the caller with -2, and so forth.
proc myName { { level "-1" } } {
     if { $level > 0 } {
        return -code error "myName: called with level > 0 ($level)."
     }
     if { [ catch {
        set name [ lindex [ info level $level ] 0 ]
     } err ] } {
        set name $::API
     }
     set name
}

§   §   §

Name: randomNumber

Description:
Implements a very long period uniform distribution random number generator. Returns a 31 bit random positive integer.
Usage:
      set number [ randomNumber (seed) ]
Where seed is a positive integer 0 < seed < 2147483648
and the value returned is likewise.

Comments:
Copyright 1995 by Roger E. Critchlow Jr., San Francisco, California.
All rights reserved. Fair use permitted. Caveat emptor.
The generator is one George Marsaglia, geo@stat.fsu.edu, calls the Mother of All Random Number Generators.
Modified by Philip S. Ehrens at LIGO 98.12.17
namespace eval random {
set a1 { 1941 1860 1812 1776 1492 1215 1066 12013 };
set a2 { 1111 2222 3333 4444 5555 6666 7777   827 };
set m1 { 30903 4817 23871 16840 7656 24290 24514 15657 19102 };
set m2 { 30903 4817 23871 16840 7656 24290 24514 15657 19102 };
}

§   §   §
proc random::rand16 { a m } {
    set n [ expr {
        [ lindex $m 0 ] +
        [ lindex $a 0 ] * [ lindex $m 1 ] +
        [ lindex $a 1 ] * [ lindex $m 2 ] +
        [ lindex $a 2 ] * [ lindex $m 3 ] +
        [ lindex $a 3 ] * [ lindex $m 4 ] +
        [ lindex $a 4 ] * [ lindex $m 5 ] +
        [ lindex $a 5 ] * [ lindex $m 6 ] +
        [ lindex $a 6 ] * [ lindex $m 7 ] +
        [ lindex $a 7 ] * [ lindex $m 8 ] }];
    return [ concat [ expr { $n >> 16 } ] [ expr { $n & 0xFFFF } ] [ lrange $m 1 7 ] ];
}

§   §   §
proc random::srand16 { seed } {
    set n1 [ expr { $seed & 0xFFFF } ];
    set n2 [ expr { $seed & 0x7FFFFFFF } ];
    set n2 [ expr { 30903 * $n1 + ($n2 >> 16) } ];
    set n1 [ expr { $n2 & 0xFFFF } ];
    set m  [ expr { $n1 & 0x7FFF } ];
    foreach i { 1 2 3 4 5 6 7 8 } {
      set n2 [ expr { 30903 * $n1 + ($n2 >> 16) } ];
      set n1 [ expr { $n2 & 0xFFFF } ];
      lappend m $n1;
          }
    return $m;
}

§   §   §
;## Externals
proc srandomNumber { seed } {
    set random::m1 [ random::srand16 $seed ];
    set random::m2 [ random::srand16 [ expr { 4321+$seed } ] ];
    return {};
}

§   §   §
proc randomNumber { { seed 0 } } {
    if { $seed } {
       srandomNumber [ expr { int ($seed) } ]
       }
    set random::m1 [ random::rand16 $random::a1 $random::m1 ];
    set random::m2 [ random::rand16 $random::a2 $random::m2 ];
    return [expr (( [ lindex $random::m1 1 ] << 16) + [ lindex $random::m2 1 ]) & 0xFFFFFFF ];
}

§   §   §

Name: bgLoop

Description:
Start asynchronous looping jobs. Jobs are ended by setting ::bg::jobs($name,run) to 0.
Usage:
       start: bgLoop $name $code $delay
        stop: set ::bg::jobs($name,run) 0
Comments:
Since multiple processes with the same name CAN be started, a unique name should be chosen for each process.
Process name bookkeeping must be done by the caller.
the job code MUST NOT BLOCK, and should never return explicitly.
proc bgLoop { { name NULL } { code "" } { delay 2 } } {
     if { ! [ llength [ namespace children :: bg ] ] } {
        namespace eval bg {}
        set ::bg::iterator 0
     }
     incr ::bg::iterator
     ;## register a new job if it has valid args
     if { ! [ string equal NULL $name ]      && \
            [ string length [ join $code ] ] } {
        set ::bg::jobs($name,run)   1
        set ::bg::jobs($name,code)  $code
        set ::bg::jobs($name,delay) $delay
        addLogEntry "Looping process $name started"
     }
     if { [ info exists ::bg::after ] && \
          [ lsearch [ after info ] $::bg::after ] != -1 } {
        after cancel $::bg::after
     }
     if { [ string equal NULL $name ] } {
        set dt 0
        foreach job [ array names ::bg::jobs *,run ] {
           set job [ lindex [ split $job , ] 0 ]
           if { [ string equal NULL $job ] || \
                [ string equal -1 $::bg::jobs($job,run) ] } {
              continue
           }
           ;## if the run flag == 0, unregister the job
           if { [ string equal 0 $::bg::jobs($job,run) ] } {
              foreach item [ array names ::bg::jobs $job,* ] {
                 unset ::bg::jobs($item)
              }
              addLogEntry "Looping process $job terminated"
              continue
           }
           ;## otherwise, eval!
           if { ! ($::bg::iterator % $::bg::jobs($job,delay)) } {
              set ts [ clock clicks -milliseconds ]
              if { [ catch {
                 eval $::bg::jobs($job,code)
              } err ] } {
                 set ::bg::jobs($job,run) 0
                 addLogEntry "$err ($::bg::jobs($job,code))" email
              }
              set te [ clock clicks -milliseconds ]
              set td [ expr $te - $ts ]
              set dt [ expr $dt + $td ]
              lappend data [ list $job $td ]
           }
        }
        ;## produce a timing report if required
        if { ($dt > 1000) && [ info exists ::PROFILE_BGLOOP ] && \
             [ string equal 1 $::PROFILE_BGLOOP ] } {
           addLogEntry "runtime per iteration: $dt ms ($data)" blue
        }
        set ::bg::after [ after 1000 bgLoop ]
     } else {
        ;## we are running the code block for the first time,
        ;## so we eval NOW.
        if { [ catch {
           set retval [ eval $::bg::jobs($name,code) ]
        } err ] } {
           if { [ info exists job ] } {
              set ::bg::jobs($job,run) 0
              addLogEntry "$err ($::bg::jobs($job,code))" email
           } else {
              addLogEntry "$err ('job' not defined)" email
           }
           set ::bg::after [ after 1000 bgLoop ]
           return -code error $err
        }
        set ::bg::after [ after 1000 bgLoop ]
        return $retval
     }
}

§   §   §

Name: anonFtpToplevel

Description:
Retrieve the toplevel anonymous ftp directory from the /etc/passwd file.

Parameters: Usage:

Comments:

proc anonFtpToplevel { } {
     set data [ list ]
     if { [ file readable /etc/passwd ] } {
        set fid [ open /etc/passwd r ]
        set data [ read $fid ]
        close $fid
     } else {
        return -code error "[ myName ]: /etc/passwd not readable"
     }
     foreach line [ split $data "\n" ] {
        if { [ regexp {^ftp} $line ] } {
           set tmp [ split $line ":" ]
           set dir [ lindex $tmp end-1 ]
           return $dir
        }
     }
     puts stderr "[ myName ]: no anonymous ftp on this system"
     return {}
}

§   §   §

Name: gridFtpToplevel

Description:
If a grid user exists, returns the home directory trailing slash is usually NOT attached.

Parameters: Usage:

Comments:

proc gridFtpToplevel {  } {
     set data [ list ]
     set dir  [ list ]
     if { [ file readable /etc/passwd ] } {
        set fid [ open /etc/passwd r ]
        set data [ read $fid ]
        close $fid
     } else {
        return -code error "[ myName ]: /etc/passwd not readable"
     }
     foreach line [ split $data "\n" ] {
        if { [ regexp {^grid} $line ] } {
           set tmp [ split $line ":" ]
           set dir [ lindex $tmp end-1 ]
           if { [ string length $dir ] && \
                [ info exists ::GRID_FTP_WRITABLE_SUBDIRECTORY ] } {
              set subdir $::GRID_FTP_WRITABLE_SUBDIRECTORY
              set subdir [ string trim $subdir / ]
              set dir $dir/$subdir
           }
           break
        }
     }
     if { ! [ string length $dir ] } {
        set msg "no grid ftp user found on this system.\n"
        append msg "this is ok if this system is not\n"
        append msg "supposed to be supporting grid ftp.\n"
        append msg "this error will not adversely affect\n"
        append msg "the running of LDAS unless requests\n"
        append msg "for grid ftp data transfer are made."
     } elseif { ! [ file exists $dir ] } {
        set msg "[ myName ]: grid ftp directory: '$dir' "
        append msg "not found.\nare you sure it is visible "
        append msg "from the host $::env(HOST) ?"
     } elseif { ! [ file isdirectory $dir ] } {
        set msg "[ myName ]: grid ftp directory: '$dir' "
        append msg "is not a directory."
     } elseif { ! [ file writable $dir ] } {
        set msg "[ myName ]: grid ftp directory: '$dir' "
        append msg "is not writable by user $::env(USER)"
     }
     if { [ string equal manager $::API ] && [ info exists msg ] } {
        puts stderr $msg
        set pre "$::LDAS_SYSTEM ${::API}API"
        set subject "$pre gridftp config error!"
        set msg "$pre $msg"
        addLogEntry "Subject: ${subject}; Body: $msg" email
     }
     return $dir
}

§   §   §

Name: expandOpts

Description:
Expands procedure options. If a procedure predeclares a list of arguments like this:
set opts { -user {} -host {} -command {} } then if the procedure is declared like this:
proc doThis { { args "" } } { ... } the procedure can be called like this:
doThis -u myname -h machine.domain.com -c die!
then doThis can call expandOpts, and the single letter form of the options will be expanded and used to modify the options to the procedure "in place".
Usage:
       array set opts [ expandOpts [opts] ]
     Where: opts is optional, defaulting to "opts".

Comments:
Uses an expensive but thorough "backing-up" regex pattern.
A command line *option* may NOT begin with a number.
An ambiguous option (one that matches more than one default) will throw an exception.
proc expandOpts { { opts "opts" } } {
     set options [ list ]
     set matches [ list ]
     ;## get inputs and massage them into a
     ;## well-formed list
     set inputs [ uplevel set args ]
     regsub -all -- {\s+} $inputs { } inputs
     regexp {^\{(.+)\}$}    $inputs  -> inputs
     ;## get defaults and do the same
     set defaults [ uplevel subst \$$opts ]
     regsub -all -- {\s+} $defaults { } defaults
     ;## and GO! (trimming loose spaces from values)
     foreach { opt def } $defaults {
        set opt     [ string trim $opt  ]
        set def     [ string trim $def  ]
        set matched 0
        foreach { name val } $inputs {
           set name [ string trim $name ]
           regsub -- {[-]+} $name "-" name
           set val  [ string trim $val  ]
           if { [ regexp -nocase -- ^$name $opt ] } {
              ;## next test fails if ambiguous item is last
              ;## option on command line...
              if { [ lsearch -exact $matches $name ] > -1 } {
                 return -code error "[ myName ]: ambiguous option: \"$name\""
              }
              lappend matches $name
              set options [ concat $options [ list $opt $val ] ]
              set matched 1
              break
           }
        }
        if { $matched == 0 } {
           set options    [ concat $options [ list $opt $def ] ]
        }
     }
     return $options
}

§   §   §

Name: metaOpts

Description:
Make optional arguments available to the API's, expanding them and putting them in a global array named after the unique jobid.
Note that the list of meta options is strictly limited to a very few useful arguments. The argument list is shared between the API's. User info is maintained at the manager.
After this proc sets the jobid array, the options are available to the eval'd code block in the API.
Usage:

Comments:

This will be called by the operator handler of the API.
proc metaOpts { args } {
     if { [ catch {
        set opts [ uplevel set opts ]
        array set tmp [ expandOpts ]
        set jobid $tmp(-jobid)
        #catch { unset ::$jobid }
        array set ::$jobid [ array get tmp ]
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: traceTimeout

Description:
Will put a trace on a variable. If the trigger is not set within a predetermined timeout period, the trace will be removed.

Parameters: Usage:
       traceTimeout vname doThis [ timeout ]
the argument list for command must end with "args"
to eat the arguments added by trace.

it is better for varname to be a unique name.
Comments:
"cmd" will receive three arguments from the trace, vname, {}, and the new value of $vname. These arguments will be appended to the argument list of cmd.
See the Tcl man page for "trace".
proc traceTimeout { { vname "" } { cmd "" } { timeout 10 } } {
     if { ! [ string length $cmd ] } {
        return {}
        }
     set timeout [ expr { $timeout * 1000 } ]
     if { [ catch {
     	if	{ [ regexp {8.4} $::tcl_version ] } {
        	uplevel trace add variable $vname { write } $cmd
        	uplevel after $timeout \
            	eval trace remove variable $vname [ trace info variable $vname ]
        } else {
        	uplevel trace variable $vname w $cmd
        	uplevel after $timeout trace vdelete $vname w $cmd
        }
    } err ] } {
        addLogEntry $err
        return -code error "[ myName ]:\n$err"
   	}
}

§   §   §

Name: leakLogger

Description:
Memory usage monitor which is used to log leakage of API's.

Parameters: Usage:
Called at end of job processing by each API
Comments:
Works on Solaris and Linux!
proc leakLogger { } {
     if { [ info exists ::last_leaklogger_call ] } {
        set now [ clock clicks -milliseconds ]
        if { $now - $::last_leaklogger_call < 1000 } {
           return {}
        }
     }
     set now [ gpsTime ]
     foreach [ list name pid pcpu pmem vsz rss etime ] \
        [ ps::self ] { break }
     if { ! [ info exists ::memusage_t0 ] } {
        set     ::memusage_t0 $now
        lappend ::memusage_t0 $vsz
        set init 1
        addLogEntry "inital size of $::API API: $vsz kB"
     }
     foreach { t0 x0 } $::memusage_t0 { break }
     set tn $now
     set xn $vsz
     ;## no getThreadList function in manager or cntlmon
     if { ! [ string equal manager $::API ] && \
          ! [ string equal cntlmon $::API ] } {
        eval set threads [ getThreadList ]
        set threads [ expr [ llength $threads ]/3 ]
     } else {
        set threads 0
     }
     ;## write data point to file, archive as necessary
     set megabyte [ expr { 1024 * 1024 } ]
     set fname $::LDASLOG/${::API}.mem
     set arc   $::LDASARC/memusage/${::API}.mem.$tn
     if { [ file exists $fname ] && \
          [ file size $fname   ] > $megabyte } {
        if { ! [ file exists [ file dirname $arc ] ] } {
           file mkdir [ file dirname $arc ]
        }
        file rename -force $fname $arc
     }
     set fid [ open $fname a+ ]
     if { [ info exist init ] } {
        puts $fid "$tn $xn $pcpu $threads init"
     } else {
        puts $fid "$tn $xn $pcpu $threads"
     }
     ::close $fid
     ;## nobody needs to see this, and it bloats the logs!
     if { [ info exists ::DEBUG_LEAKLOGGER ] && \
          [ string equal 1 $::DEBUG_LEAKLOGGER ] } {
        set t    [ expr { $tn - $t0 } ]
        set leak [ expr { $xn - $x0 } ]
        if { $leak && ! [ string equal manager $::API ] } {
           addLogEntry "Allocated $leak kB in $t seconds" blue
        }
     }
     set ::last_leaklogger_call [ clock clicks -milliseconds ]
     return {}
}

§   §   §

Name: memFlag

Description:
Log warning about excessive memory usage.
Returns true (1) if memory usage is excessive, false (0) otherwise.

Parameters: Usage:

Comments:

The value of ::RESTART_ON_MEMFLAG will be used as the number of seconds to wait for pending jobs on the API to complete in an attempt to avoid losing the job entirely.
proc memFlag { { max 128 } } {
     set flag 0
     set now [ clock seconds ]
     if { $::MEMFLAG_MEGS > 0 } {
        set max $::MEMFLAG_MEGS
     }
     if { [ info exists ::MEM_WARNING_FLAG_IS_SET ] } {
        set dt [ expr { $now - $::MEM_WARNING_FLAG_IS_SET } ]
        if { $dt > 3600 } {
           unset ::MEM_WARNING_FLAG_IS_SET
        }
     }
     if { [ catch {
        set size [ lindex [ sysData ] 4 ]
        if { [ string length $size ] } {
           if { $size > [ expr { $max*1024 } ] } {
              addLogEntry "memory usage by this API is: $size kB" red
              set flag 1
              if { $::RESTART_ON_MEMFLAG } {
                 set subject "memory usage at $size kB"
                 set body    "The $::API API at $::LDAS_SYSTEM is now "
                 append body "using\n$size kB of memory, which exceeds\n"
                 append body "the resource value of ::MEMFLAG_MEGS "
                 append body "($max M)\nfor this API.\n\n"
                 append body "The flag ::RESTART_ON_MEMFLAG has\n"
                 append body "been set in the LDAS${::API}.rsc file,\n"
                 append body "so this API will be restarted\n"
                 append body "automatically NOW!"
                 set msg "Subject: ${subject}; Body: $body"
                 addLogEntry $msg email
                 addLogEntry "requesting restart by manager" red
                 set sid [ sock::open manager emergency ]
                 puts $sid "$::MGRKEY NULL NULL mgr::bootstrapAPI $::API"
                 close $sid
              } elseif { ! [ info exists ::MEM_WARNING_FLAG_IS_SET ] } {
                 set ::MEM_WARNING_FLAG_IS_SET $now
                 set subject "memory usage at $size kB"
                 set body    "The $::API API at $::LDAS_SYSTEM is now "
                 append body "using\n$size kB of memory, which exceeds\n"
                 append body "the resource value of ::MEMFLAG_MEGS "
                 append body "($max M)\nfor this API.\n\n"
                 append body "The flag ::RESTART_ON_MEMFLAG has not\n"
                 append body "been set in the LDAS${::API}.rsc file,\n"
                 append body "so this API will NOT be restarted\n"
                 append body "automatically!"
                 set msg "Subject: ${subject}; Body: $body"
                 addLogEntry $msg email
              }
           }
        }
     } err ] } {
        ;## we must be pretty far gone, try and report!!
        catch { addLogEntry $err email }
        if { [ regexp {not enough memory} $err ] || \
             [ regexp {too many open files} $err ] } {
           set flag 1
           if { $::RESTART_ON_MEMFLAG } {
              if { [ catch {
                 addLogEntry "requesting restart by manager" red
                 set sid [ sock::open manager emergency ]
                 puts $sid "$::MGRKEY NULL NULL mgr::bootstrapAPI $::API"
                 close $sid
              } err ] } {
                 if { [ catch {
                    ;## try one last time to shutdown gracefully
                    ${::API}::sHuTdOwN
                 } err ] } {
                    ;## we are really hosed, so just die.
                    puts stderr "$err $::API cannot shutdown cleanly, ABORTING NOW!"
                    exit
                 }
              }
           }
        }
     }
     set flag
}

§   §   §

Name: ftpPutLocal

Description:

Parameters: Usage:

Comments:

proc ftpPutLocal { in out } {
     if { [ catch {
        if { [ catch {
           set jobid [ uplevel set jobid ]
        } err ] } {
           set jobid result
        }
        set ftpdir [ string trim $::FTPDIR / ]
        if { ! [ regexp ^/?$ftpdir/? $out ] } {
           set out [ string trim $out / ]
           set out [ file join $::FTPDIR $out ]
        }
        regsub ^$ftpdir $out /$ftpdir out
        if { [ file isdirectory $out ] } {
           set out [ file join $out $jobid.ilwd ]
        }
        set dir  [ file dirname $out ]
        set root [ file dirname $dir ]
        if { [ file exists $root ] } {
           ;## if the "root" is world readable, it is a
           ;## usable public ftp directory.
           set perms [ file attributes $root -permissions ]
           set world [ string range $perms end end ]
           if { $world } {
              if { ! [ file isdirectory $dir ] } {
                 file mkdir        $dir
                 file attributes   $dir -permissions 0775
              }
              bak $out
              file copy -force -- $in $out
           } else {
              return -code error "$root is not world readable"
           }
        } else {
           return -code error "$root does not exist"
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: parseURL

Description:
Parse URL style strings which might, for example, be used as the -returnprotocol option to a user command, and return the protocol, the target, and an optional port.

Parameters: Usage:
The URL is parsed as follows:

example 1:
http://foo.bar.edu:8080/baz/bim/buz.html protocol - http targ1 - foo.bar.edu port - 8080 targ2 - /baz/bim/buz.html example 2:
http://foo.bar.edu/baz/bim/buz.html protocol - http targ1 - foo.bar.edu/baz/bim/buz.html port and targ2 in this case are null strings.
so to recover the full URL target when a port is given:
${targ1}:$port$targ2 will always work.
Comments:
This function does NOT verify the validity of the target, the function which maps to the protocol must do that.
proc parseURL { { url "" } } {
     set protocol {}
     set targ1    {}
     set targ2    {}
     set port     {}
     set url [ string trim $url " \{\}\"\'" ]
     if { [ regexp {^gridftp(:/{0,2})?$} $url ] } {
        set url $::GRIDFTPURL
     }
     set url_rx \
       {(file|http|mailto|gridftp|ftp|port):/*([^ \/]+(:[0-9]+)?)(\/\S+)?}
     if { [ regexp $url_rx $url -> protocol targ1 -> targ2 ] } {
        set port [ lindex [ split $targ1 : ] 1 ]
        ;## for file type URL's we are trying to make both incoming
        ;## AND outgoing URL's happy... this is not trivial and
        ;## ORDER MATTERS!!
        ;## when we are dealing with output we try to agree with the
        ;## way netscape would see things.
        switch -exact -- $protocol {
           file {
                 if { ! [ file exists $targ2 ] } {
                    regexp {file:/*(\S+)} $url -> targ2
                    set original $targ2
                 }
                 if { ! [ file exists $targ2 ] } {
                    set targ2 /$targ2
                 }
                 if { ! [ file exists $targ2 ] } {
                    set targ2 $::env(RUNDIR)$targ2
                 }
                 if { ! [ file exists $targ2 ] } {
                    set targ2 $original
                 }
                 set targ1 {}
                }
           http {
                }
         mailto {
                set port {}
                }
        gridftp {
                }
            ftp {
                }
           port {
                   if { ! [ string length $targ1 ] } {
                      return -code error "parseURL: bad URL: '$url'"
                   }
                }
        default {
                return -code error "parseURL: bad URL: '$url'"
                }
        } ;## end of switch
     } else {
        return -code error "parseURL: bad URL: '$url'"
     }
     return [ list $protocol $targ1 $port $targ2 ]
}

§   §   §

Name: formatAndProtocol

Description:
Try to match -outputformat and -returnprotocol options to local file names and URL's consistent with the -returnprotocol.

Parameters: Usage:

Comments:

What level of error or warning applies when the user provides conflicting format and file extension?
This may be used to retire parseURL and url2file...
proc formatAndProtocol { jobid format protocol } {
     if { [ catch {
        set format_rx \
           {(frame|ascii|binary|ligo_?lw|ilwd)\s*(.*)}
        set \
           url_rx {(file|http|mailto|ftp|port):/*([^ \/]+(:[0-9]+)?)(\/\S+)?}
        ;## format is not required if filename has extension
        if { ! [ regexp -nocase $format_rx $format -> base extended ] } {
           set base     [ list ]
           set extended [ list ]
        }
        ;## pattern will not match a local filename
        if { ! [ regexp -nocase $url_rx $protocol -> type server -> path ] } {
           set type file
           set server [ list ]
           set path $protocol
        }
        set port [ lindex [ split $server : ] 1 ]
        if { [ regexp {\/$} $path ] } {
           set dir $path
           set file [ list ]
        } else {
           set atoms [ split $path / ]
           ;## this will be an absolute directory with a leading
           ;## and trailing "/"
           set dir [ join [ lrange $atoms 0 end-1 ] / ]/
           set file [ lindex $atoms end ]
           set ext [ file extension $file ]
           ;## user provided file extension overrides format,
           ;## but there IS the possibility of a completely
           ;## unusable file extension like '.txt'
           if { [ string length $ext ] && ! [ string equal . $ext ] } {
              set format [ list ]
           }
        }
        if { [ regexp -nocase {(http|ftp)} $type ] } {
           set type ${type}://
        }
        if { [ regexp -nocase {file} $type ] } {
           set type   [ list ]
           set server [ list ]
        }
        if { [ regexp -nocase {(mailto|port)} $type ] } {
           set type ${type}:
        }
        ;## IGWD frame format
        if { [ regexp -nocase {frame} $format ] } {
           set ext .gwf
           ;## the name should be contained in the metadata
           ;## but can be derived from the header.
           if { ! [ string length $file ] } {
              set file default_filename
           }
        }
        ;## ilwd is a lightweight pseudo-xml format
        ;## with no explicit document class descriptor
        if { [ regexp -nocase {ilwd} $format ] } {
           set ext .ilwd
           ;## default filename root: jobid-N??
           if { ! [ string length $file ] } {
              set file default_filename
           }
        }
        ;## ligo_lw is an xml format
        if { [ regexp -nocase {ligo_?lw} $format ] } {
           set ext .xml
           ;## default filename root: jobid-N??
           if { ! [ string length $file ] } {
              set file default_filename
           }
        }
        set file [ file rootname $file ]$ext
        ;## for a file type URL or local filename,
        ;## filename and URL should be identical.
        set filename $dir$file
        ;## should result in a local file path when the
        ;## 'URL' is 'file:/*' or just a local file.
        set url [ string trim $type$server$path ]
        ;## debugging helper
        if { [ info exists ::DEBUG_FORMAT_AND_PROTOCOL ] } {
           if { [ string length $::DEBUG_FORMAT_AND_PROTOCOL ] && \
           ! [ regexp -nocase {(0|false)} $::DEBUG_FORMAT_AND_PROTOCOL ] } {
              set msg    "base output format: '$base' "
              append msg "extended output format: '$extended' "
              append msg "url type: '$type' "
              append msg "remote server: '$server' "
              append msg "remote path: '$path' "
              append msg "directory component: '$dir' "
              append msg "filename component: '$file' "
              append msg "(next two should be same for a file URL: "
              append msg "calculated filename: '$filename' "
              append msg "calculated URL: '$url')"
              addLogEntry $msg purple
           }
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return [ list $filename $url ]
}

§   §   §

Name: formatConversion

Description:
Given a filename or object pointer or text object, convert it consistently with the output spec.

Parameters: Usage:

Comments:

proc formatConversion { input output } {
     if { [ catch {
        ;## 1. determine input type
        ;## 2. determine output type
        ;## 3. define conversion path
        return unimplemented
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: url2file

Description:
returns a local target name and a remotely accessible http/ftp URL parsed from the returnprotocol argument of a user command
Parameters: Usage:
url2file LDAS-DEV1 ftp://ftp.slug.org/foon/
Returns:

ftp://131.215.115.248/LDAS-DEV_0/LDAS-DEV1/foon.ilwd /ldas_outgoing/jobs/LDAS-DEV_0/LDAS-DEV1/foon.ilwd However:
url2file LDAS-DEV1 ftp://ftp.slug.org/foon/foo.ilwd Returns:
ftp://131.215.115.248/LDAS-DEV_0/LDAS-DEV1/foo.ilwd /ldas_outgoing/jobs/LDAS-DEV_0/LDAS-DEV1/foo.ilwd
Comments:
Return value "target" is the local filename.
Return value "filename" is the URL for the file.
file extension defaults to the extension on the file in the URL.
proc url2file { jobid url { ext "" } } {
     if { [ catch {
        if { [ regexp {^(http|ftp|file|mailto):} $url ] } {
           foreach { protocol t1 port t2 } [ parseURL $url ] { break }
           set target $t1$t2
        } else {
           return -code error "invalid url: $url"
        }
        ;## force filename into correct format
        if { ! [ llength $ext ] } {
           set ext [ file extension $target ]
           if { ! [ string length $ext ] } {
              set ext ilwd
           }
        }
        set ext [ string trim $ext . ]
        set ft  [ string trim [ file tail $target ] . ]
        set ft  [ file rootname $ft ]
        regsub -all -- {\s+} $ft {_} ft
        set ft $ft.$ext
        set jobdir  [ jobDirectory ]
        set reldir  [ relativeDirectory ]
        switch -exact -- $protocol {
           file {
                set filename [ file join $::HTTPDIR $reldir $ft ]
                set target [ file join $jobdir $ft ]
                }
            ftp {
                set filename [ file join $::FTPDIR $reldir $ft ]
                set target [ join "$::FTPURL $reldir $ft" / ]
                }
           http {
                set filename [ file join $::HTTPDIR $reldir $ft ]
                set target [ join "$::HTTPURL $reldir $ft" / ]
                }
         mailto {
                set filename [ file join $::HTTPDIR $reldir $ft ]
                set target [ file join $jobdir $ft ]
                }
        default {
                return -code error "invalid protocol: $protocol"
                }
            } ;## end of switch
        } err ] } {
        return -code error "[ myName ]:$err"
        }
     return [ list $target $filename ]
}

§   §   §

Name: macroReturnMsg

Description:
Formulate return message for macro code.

Parameters: Usage:

Comments:

if the return protocol is mailto, the message should consist SOLELY of a list of space or comma delimited absolute filenames, in which case the files will be attached to an e-mail.
proc macroReturnMsg { jobid protocol filenames } {
     if { [ catch {
        set files [ list ]
        set there error!
        set level 4
        set msg "Your results:\n\$files\ncan be found at:\n\$there"
        set jobdir [ jobDirectory ]
        set reldir [ relativeDirectory ]
        if { [ llength $filenames ] == 1 } {
           set filenames [ lindex $filenames 0 ]
        }
        switch -regexp -- $protocol {
           {mail} {
                   foreach file $filenames {
                      if { [ regexp {^(http|ftp)} $file ] } {
                         append files "${file}\n"
                      } else {
                         set file [ file tail $file ]
                         lappend files [ file join $jobdir $file ]
                      }
                   }
                  }
           {http} {
                   foreach file $filenames {
                      if { [ regexp {^(http|ftp)} $file ] } {
                         append files "${file}\n"
                      } else {
                         lappend files [ file tail $file ]
                      }
                   }
                   set there [ join [ list $::HTTPURL $reldir ] / ]
                   set there ${there}/
                  }
            {ftp} {
                   foreach file $filenames {
                      if { [ regexp {^(http|ftp)} $file ] } {
                         append files "${file}\n"
                      } else {
                         lappend files [ file tail $file ]
                      }
                   }
                   set there [ join [ list $::FTPURL $reldir ] / ]
                   set there ${there}/
                  }
           {file} {
                   foreach file $filenames {
                      if { [ regexp {^(http|ftp)} $file ] } {
                         append files "${file}\n"
                      } else {
                         lappend files [ file tail $file ]
                      }
                   }
                   set there $jobdir
                  }
          default {
                  set level 3
                  set msg "Unknown return protocol: '$protocol'"
                  }
        } ;## end of switch
        set msg [ list $level [ subst -nocommands $msg ] ]
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $msg
}

§   §   §

Name: countChannels

Description:
Find out which filehandles (open files, pipes and sockets)
are open, and get a list of their names.
Also returns a list of running threads and pending Tcl event loop items.

Parameters: Usage:

Comments:

stdout, stdin, and stderr will not be returned in the file list.
The list of threads is a list of alternating thread i.d.'s and the status of those threads:
RUNNING, FINISHED, or CANCELLED.
This is very fast compared to countFilehandles.
proc countChannels { } {
     set socks [ file channels *sock* ]
     set files [ file channels file* ]
     if { [ string length [ info commands getThreadList ] ] } {
        ;## we eval here because getThreadList and
        ;## getElementList return improperly listified lists.
        eval set threads [ getThreadList ]
        eval set objects [ getElementList ]
     } else {
        set threads [ list ]
        set objects [ list ]
     }
     set events [ after info ]
     return [ list $socks $files $threads $events $objects ]
}

§   §   §

Name: roVar

Description:
Make a global variable "read-only" by registering a shadow variable in the "ro" namespace.

Parameters: Usage:
       roVar ::varname
Comments:
Adds 200 microseconds overhead to variable setting.
A proc call to set a variable costs 10 us normally.
Only works on global variables!!
proc roVar { { varname "" } } {
     if { [ regexp {^$} $varname ] } { return {} }
     regsub {::} $varname "" varname
     if { ! [ llength [ info procs __roVar_x ] ] } {
        proc __roVar_x { args } {
             namespace eval ro {}
             set varname [ lindex $args 0 ]
             regsub {::} $varname {} varname
             if { [ catch { set ro::__$varname } ] } {
                set ro::__$varname [ set ::$varname ]
                }
             set ::$varname [ set ro::__$varname ]
             }
        }
     if	{ [ regexp {8.4} $::tcl_version ] } {
     	if { ! [ llength [ trace info variable ::$varname ] ] } {
        	__roVar_x $varname
        	trace add variable ::$varname { write } __roVar_x
        }
     } else {
     	if { ! [ llength [ trace vinfo ::$varname ] ] } {
        	__roVar_x $varname
        	trace variable ::$varname w __roVar_x
        }
     }
}

§   §   §

Name: unknown

Description:
Replacement for the Tcl "unknown" handler.

Parameters: Usage:
Called Internally by Tcl.

The unknown handler is (of course) only called for something unknown. So, if you want to assign to a variable which happens to be also the name of a command/proc, "unknown" would not get called:
info = Something to tell bad option "=": must be args, body, cmdcount, commands, ...
and this will not be too rare because of many common-sense command/proc names in Tcl. This is where the REBOL style ("i: 5", blank after but not before the colon) comes in handy.
info: Something to tell would only miss the unknown if you had a proc named "info:", which may not occur that often.
BTW: how's for {i: 0} {$i<$max} {i++} {...}
Comments:
Slightly modified from: Tcl "expr" change suggestion" Date: 1999/07/08 Author: Richard.Suchenwirth@kst.siemens.de
;## keep the original handler as a fallback
#if { ! [info proc _unknown] } { rename unknown _unknown }
#proc unknown { args } {
#
#     ;## allow REBOL-style assignments (foo: bar; bar: 17+4)
#     if [ regexp (.+):$ [ lindex $args 0 ] -> name ] {
#        set args [ lreplace $args 0 0 $name = ]
#        }
#
#     ;## maybe an assignment like "x = 3+4" ? (Blanks matter!)
#     if { [ lindex $args 1 ] == "=" } {
#        upvar [ lindex $args 0 ] _x
#        set rest [ lrange $args 2 end ]
#        if { [ llength [ info commands [ lindex $args 2 ] ] ] } {
#           return [ set _x [ uplevel eval $rest ] ]
#           }
#
#       ;## this should always work...
#       set _x $rest
#
#       ;## ...but maybe expr is happy
#       catch { set _x [ expr $rest ] }
#       return $_x
#
#       ;## handle things like "i++" ...
#       } elseif { [ regexp {^([^ ]+)\+\+$} $args -> vname ] } {
#       uplevel [ list incr $vname ]
#
#       ;## ... or "j--"
#       } elseif { [ regexp {^([^ ]+)--$}   $args -> vname ] } {
#       uplevel [ list incr $vname -1 ]
#
#       ;## pure expression? "(17+4)/3"
#       } elseif { [ regexp {^[]+$} $args ] } {
#       return [ expr $args ]
#
#       ;## fall back to old unknown
#       } else { eval _unknown $args }
# }

§   §   §

Name: fileExists

Description:

Parameters: Usage:

Comments:

proc fileExists { file { timeout 1000 } } {
     if { ! [ string length [ auto_execok ls ] ] } {
        return 0
     }
     if { [ catch {
        if { $timeout < 100 } { set timeout 100 }
        set a [ expr { $timeout / 100 } ]
     } err ] } {
        set err "second argument must be an integer >= 100"
        return -code error "[ myName ] $err"
     }
     set b [ expr { $a * 9  } ]
     set c [ expr { $b * 10 } ]
     set fid [ open "|ls -f $file" a+ ]
     fconfigure $fid -buffering line
     fconfigure $fid -blocking off
     foreach ms [ list $a $b $c ] {
        after $ms
        if { [ string length [ gets $fid ] ] } {
           close $fid
           return 1
        }
     }
     close $fid
     return 0
}

§   §   §

Name: numRange

Description:
Expand embedded numerical ranges, i.e. 12-24 in { 1 2 3 4 12-24 } in place to create lists of integers, i.e. { 12 13 14 15 16... } Tries to handle all sorts of badness.

Parameters: Usage:
       numRange "1 2 3 4 5-12 13..21, 23, 24"
Comments:
proc numRange { numbers } {
     set nums {}
     ;## try to handle comma delimited lists, yeccchh.
     regsub -all {\,\s*} $numbers " " numbers
     ;## and accomodate that other way of doing things.
     regsub -all {\.\.} $numbers "-" numbers
     foreach elem $numbers {
        set elem [ string trim $elem ]
        if { [ regexp {^(\d+)-(\d+)$} $elem -> begin end ] } {
           if { $end < $begin } {
              return -code error "Bad range: '$begin-$end'"
              }
           for { set i $begin } { $i <= $end } { incr i } {
               lappend nums $i
               }
           } else {
           lappend nums $elem
           }
        }
     return $nums
}

§   §   §

Name: gifBalls

Description:
Creates .gif files of red, yellow, and green balls.

Parameters: Usage:

Comments:

proc gifBalls { { where "" } } {
     set red {R0lGODlhFAAPAPMPAAAAADgFD00CEGoDGHMTJ21ZXZISK6USL7
              MoRJ1aZ9IzUvRUdNVAXscbPv9ylKqcniH5BAEAAAAALAAAAAAU
              AA8AAARwEMgpX7qPavrQ+YdBZNv0HIDCKE04kNvZMEvNeIMAU8
              mhLI7gQuEqlAwHWtCxYLgIMeRvOTwQBoHS4ICg2RCGXHZDQCIU
              aPBVMNYkBgZkSCQ2lgSDAWGfZwd2HAJ4fX6AGg9sggGLhiUPBZ
              CNJZMaEQA7}
     set orange {R0lGODlhFAAPAPUBACoRCDoYCzciG0YdDkUoHlYkEkI0L2Qp
                 FXEuGHY0HHs6JF1ANVNHQ2NcWHlkXHhwbYU5HYY6IYxFLZhI
                 LZVNM55SNqlLK6VSNrRWNIZpX5plUq5fRapgRbxkRKF1Zcdj
                 PsdlQsp1Vtt0T9p2VeN7WKeEd+uEX+WHaOuRcfmYeIiDgZOH
                 g6idmbShm62sq7WvrLuyrv+ljf+plv+vof///zoYCwAAAAAA
                 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAEAADUALAAA
                 AAAUAA8AAAaewJpwKGxpSiUWcUmEcSwYiwWiUDKHLctHRBKB
                 LpHD61qDXUCmlPrUmSQK1qXHIkrJZLFUCHxwXCUYI3YzMnp8
                 C0wvEBghKTF4KCEVCgcBVwkTHSMoKZEcEm+WTAoRFx0hqBsU
                 lAMCVxkIERMVFxQSlAUBD2QFCAkKwAkHuQIuZCsDBQfDBQMB
                 AipkQi4EAQPOAAbG0kMuDwwK0dzjS0EAOw==}
     set yellow {R0lGODlhFAAPAPMPAAAAADExBD09Kk5OAmtrDFNTKmxsWIm
                 JFLCwM5mZJaCgUsXFONPTUvb2ceXlWKennCH5BAEAAAAALA
                 AAAAAUAA8AAARqEMgpX1Lq0U3t+QdBaNz0HAmyIEk4kNyZL
                 I7DMC3xlsqBOI0gA+EylAgpoJCoK8QIPkbQxhwEjqjFbZgQ
                 WY/QFILlClw5hoEI5LUaSwO1SBc3wzaPgCC+NwfuMX5+AoA
                 lAA8GBQaFho0UEQA7}
     set green {R0lGODlhFAAPAPUAABAqCBY6CyI3GxxGDihFHiNWEjRCLz9d
                NSlkFS9xGDJ2HDp7JEdTQ1pjWGN5XHB4bTeFHTuGIUWMLUiY
                LUuVM1CeNkupK0+mMlSlOFS0NF6qRWmGX2SaUmCuRWK8RHSh
                ZWDHPmPHQm/DU3TMV3LbT3faVXvjWIOnd4LrX4jlaI7ocJHv
                cpj4d6L+f4OIgYeTg52omaK0m6utq661rLG7rqz/i7D/jrj/
                lsP/of///xAqCAAAAAAAAAAAAAAAAAAAACH5BAEAADkALAAA
                AAAUAA8AAAajwJxwKIxxTicYcUmkaSwZiwWyUDKHMQuIZCKF
                LhHE7JqjYUIoVouV8kwUBevyYyGxbjcba4QJO64SGSV3ODd7
                fQgHTDMQGSMsNjc1KiIVCwgBVwoTHiUrLCojGhJwmUwLERce
                IyMiHRSXAwJXGwkRExUYFBKXBQEPZAUJCgvFCgi+AjJkLwMF
                CMgFAwECLmRCMgQBA9M6BsvXQzIPDArW4ehLQQA7}
     set blue {R0lGODlhFAAPAPUBAAgPKgsUOhshNw4ZRh4mRRIgVhUmZBotdS
               8zQjU9XSQ3e0NGU1hZY1xieW1weB0yhSE3hi1CjC1EmDRJmitG
               qTJJpjROtDhQpT5Zx0VarERcvF9nhlJgmmVxoUJdx09r21Ztyl
               Vy2lt453eBp2iD5XGJ63iU+YGDiIOGk5mdqJuhtKurrayuta6w
               u42r/5a0/6G//////wsUOgAAAAAAAAAAAAAAAAAAAAAAAAAAAA
               AAAAAAAAAAAAAAAAAAAAAAACH5BAEAADIALAAAAAAUAA8AAAaf
               QJlwKFRxRqMUcUlsZSgWCuWhUDKHKgrmI/p4KhAD6yprXTwik5
               qkkRwK1mWH8jG9Xi4T6BJuXCMWIXYwL3p8BglMLA8WICYueCUg
               EwoGAVcHEhohJSaSGRFvl0wKEBUaIKkZlAYDAlcbBxASExcTEZ
               UFAQ5kBQcHCsEHBroCK2QoAwUGxAUDAQInZEIrBAEDzwAIx9ND
               Kw4LDNLd5EtBADs=}
     set purple {R0lGODlhFAAPAPUAACIIKiwKOC4bNTEMPTEcODgORjseRT0v
                 QkUSVk4VYlMXaVwac1M1XWIceWUke09DU2FYY3JceXVteGod
                 hWwihXEnjHstmHsyk3xfhoM1nIgrqYg2pZI0tKU+x4hSmoxG
                 pZJErp5EvJJloZt3p6ZCx6tVybhP27JdzrhV2r5Y48hf68Ro
                 5cpw6NBy79d4+YaBiI+Dk6WZqK2btK2rrbWtuN2M/92W/92h
                 /////yIIKgAAAAAAAAAAAAAAAAAAAAAAACH5BAEAADgALAAA
                 AAAUAA8AAAajQJxwKJR5RqMYcUmkfTQcjWbiUDKHMk3HlDKR
                 NBUF7YqjbUgql3oVsjQQ1qVIY3LZbLVWaUNJRK4XHCh2NzYu
                 J3wJDEw0ExwlLjV4LCUZDgkDVwsWISgtLSwnIBcLCAFXDhQb
                 ISWto5cFBFcYCxQWGRsZFQ4KphJkCAoKDsQLCQgDAjNkMAUI
                 CccIBQECL2RCMwYDBdMAB8vXQzMSDxDW4ehLQQA7}
     set mail  {R0lGODlhHwASAPMAAAAAAP7+/v7+AAB/f38Af39/AL+/v/4A
                AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAEAAAcALAAA
                AAAfABIAAARdEMhJq73gHCBC/14IjqKXSUFVkmQBFOnJrvT3
                cicQ6/o8uzDJxldb3QS5InEm5NQwFOOmp6yCmj7oRIpaWkPY
                pwXJ41K93jDamlync9q4ZaOhn+z4ux6v2fvzgCcRADs=}
     set telephone {R0lGODlhMgAkAPIEAAAAAH9/AP//AMDAwP///////wAA
                    AAAAACH5BAEAAAUALAAAAAAyACQAAAPzWLrc/jDKSau9
                    OOvNI/hg132FYJqheAFnc6aqdwpB8AnLC8QPe9Y7gQ9X
                    Qu14ixsKKBjYZgodssD62G5OQMAYiKKQPtvglAWWtLgX
                    WDzTscrpLw8wJm5nb9u9KIzRf0d3ZHomfFQif3Y0gW1o
                    hQo2I3WLkIttA45NNZEbiZWcgiZwoptHGYmUBVtPM05X
                    g2anA14moJelNaRXGnQMM6ywQKtXWqYrdb5tupvArpwY
                    qES0y8xKpZ2TLTmazNUgJJ102Y8Fzt2lxiPiyqrn6FMK
                    HwPj7rvwDPKT5+D3DvlLQNL180cn4EALMA4qXMiwocOH
                    UxIAADs=}
     if { ! [ file exists $where ] } {
        file mkdir $where
     }
     foreach color { red orange yellow green blue purple } {
        set fn [ file join $where ball_${color}.gif ]
        if { ! [ file exists $fn ] } {
           set fid [ open $fn w 0664 ]
           fconfigure $fid -translation binary
           puts -nonewline $fid [ decode64 [ set $color ] ]
           close $fid
        }
     }
     foreach glyph { mail telephone } {
        set fn [ file join $where ${glyph}.gif ]
        if { ! [ file exists $fn ] } {
           set fid [ open $fn w 0664 ]
           fconfigure $fid -translation binary
           puts -nonewline $fid [ decode64 [ set $glyph ] ]
           close $fid
        }
     }
}

§   §   §

Name: itemCount

Description:
Count number of occurrences of individual items in a list. Returns a list of items and occurrences.

Parameters: Usage:
       foreach { item ct } [ itemCount $list ] {
          ;## do something
          }
Comments:
proc itemCount { list } {
     foreach x $list {
        expr {[catch {incr ct($x)}] && [set ct($x) 1]}
        }
     return [ array get ct ]
}

§   §   §

Name: infoVars

Description:
Returns a list all of the variable names defined in the current interpreter.

Parameters: Usage:

Comments:

proc infoVars { } {
     set varlist {}
     set namespaces [ concat :: [ namespace children ] ]
     foreach namespace $namespaces {
        set varlist [ concat $varlist [ info vars ${namespace}::* ] ]
     }
     set varlist
}

§   §   §

Name: stats

Description:
Informal statistical calculations on a list of values.

Parameters: Usage:
 foreach {mean stdev precentcov} [ stats $samples ] {...}
Comments:
Returns the mean, the population standard deviation, and the coefficient of variation expressed as percent.
proc stats { samples } {
        set mean 0.0
        set S2   0.0
        set cov  0.0
        set N [ llength $samples ]
        if { $N < 3 } {
           return -code error "stats: sample size too small: ($N items)"
           }
        ;## calculate the arithmetic mean
        set mean [ expr ([ join $samples + ]) / $N. ]
        ;## calculate the standard deviation
        foreach s $samples {
           set S2  [ expr { $S2+pow(($s-$mean),2) } ]
        }
        set S2     [ expr { $S2/($N-1) } ]
        set S      [ expr { sqrt($S2) } ]
        ;## calcualte the % coefficient of variation
        if { $mean != 0 } {
           set cov [ expr { ($S/$mean)*100 } ]
        }
        ;## return the values in a formatted list
        return [ format "%.6f %.6f %3.2f" $mean $S $cov ]
}

§   §   §

Name: getCurlUrl

Description:
Exec Curl using the shellPipe wrapper to retrieve a remote file.
Will work with most valid URL types. See man curl.

Parameters: Usage:

Comments:

Note that the existence of the tarball option precludes retrieval of any other files by ftp or http!!
proc getCurlUrl { jobid url localname } {
     ;## special handling for the tarball option!
     if { [ info exists ::${jobid}::tarball ] } {
        set tarball [ set ::${jobid}::tarball ]
        if { [ string length $tarball ] && \
           ! [ string equal $url $tarball ] } {
           return [ list ]
        }
     }
     if { [ catch {
        file mkdir [ file dirname $localname ]
        set fid [ open ${localname}.lock w ]
        puts $fid [ clock seconds ]
        ::close $fid
        ;## -s : no progress meter
        ;## -N : no output buffer
        ;## -w %{size_download} : how many bytes were read?
        set fid [ open "|curl -s -N --connect-timeout 30 -m 3600 -w %{size_download} $url -o $localname" a+ ]
        fconfigure $fid -blocking off
        fileevent $fid readable [ list getCurlUrlCallback $fid $localname ]
     } err ] } {
        catch { ::close $fid }
        return -code error "[ myName ]: $err"
     }
     return $fid
}

§   §   §

Name: getCurlUrlCallback

Description:

Parameters: Usage:

Comments:

proc getCurlUrlCallback { args } {
     if { [ catch {
        set fid       [ lindex $args 0 ]
        set localname [ lindex $args 1 ]
        gets $fid retval
        ::close $fid
        if { $retval == 0 } {
           set fid [ open $localname w ]
           puts $fid "file not found!"
           ::close $fid
        }
        catch { file delete ${localname}.lock }
     } err ] } {
        catch { file delete ${localname}.lock }
        catch { ::close $fid }
        addLogEntry $err red
     }
}

§   §   §

Name: unpackTarball

Description:
Unpacks tarballs provided by users only if they are 'safe' according to LDAS's criteria, which include the requirement that there be NO absolute paths (only relative paths are permitted) and NO symlinks.

Parameters: Usage:

Comments:

This was implemented in response to PR #2705, which was opened in response to serious problems caused by a user specifying several hundred files to be uploaded via curl.
proc unpackTarball { tarball } {
     if { [ catch {
        catch { exec tar --version } report
        if { ! [ regexp {GNU tar} $report ] } {
           set msg "LDAS requires GNU tar! "
           append msg "tar program in path reports: "
           append msg $report
           return -code error $msg
        }
        set ext [ file extension $tarball ]
        switch -exact -- $ext {
           .bz2 {
                  set zip j
                }
           .gz  {
                  set zip z
                }
        default {
                  set zip [ list ]
                }
        }
        catch { set report [ exec tar t${zip}vf $tarball ] } report
        foreach line [ split $report "\n" ] {
           ;## absolute path
           if { [ regexp {^\/} [ lindex $line end ] ] } {
              set err "absolute paths are forbidden in "
              append err "user supplied tar files!! ($line)"
              return -code error $err
           }
           ;## link!!
           if { [ string equal -> [ lindex $line end-1 ] ] } {
              set err "symbolic links are forbidden in "
              append err "user supplied tar files!! ($line)"
              return -code error $err
           }
        }
        ;## drop a drone to extract the tarball in the correct place
        set fid [ open tar_wrapper w 0700 ]
        puts  $fid "cd [ file dirname $tarball ]"
        puts  $fid "tar xv${zip}f [ file tail $tarball ]"
        close $fid
        catch { set report [ exec ./tar_wrapper ] } report
        catch { file delete -force tar_wrapper }
        set msg "extracted from user tarball ${tarball}: $report"
        addLogEntry $msg blue
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $report
}

§   §   §

Name: putCurlUrl

Description:
Push a local file to a remote location.

Parameters: Usage:

Comments:

proc putCurlUrl { jobid local remote } {
     if { [ catch {
         shellPipe "|curl -s --connect-timeout 30 -m 3600 -T $local $remote"
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: getUrl

Description:
Given a URL, get the file into the job directory and return a local filename usable by any LDAS api for accessing the file.
Preserve the remote filename when possible.

Parameters: Usage:

Comments:

When the -tarball option exists for a job and is defined then all other remote files are ignored and assumed to be in the tarball. See the manager API
proc getUrl { jobid url } {
     if { [ catch {
        ;## first we create a local filename under the job
        ;## directory retaining as much of the remote URL path
        ;## as is practical (for positive identification).
        set jobdir [ jobDirectory $jobid ]
        set subdir [ file dirname $url ]
        regsub {(?:http|gridftp|ftp|file):/*} $subdir {} subdir
        set dir $jobdir/$subdir
        ;## get rid of some file globbing characters - naughty!
        ;## and construct a local filename to be used when the
        ;## URL is a valid ftp or http URL.
        regsub -all {[\~\?\+\*\\]} $dir {} dir
        set fname [ file tail $url ]
        set localfilename $dir/$fname
        bak $localfilename
        ;## file possibly local to gateway but not visible to other
        ;## LDAS machines and their API's.
        if { [ regexp -nocase {file:/*(\S+)} $url -> file ] } {
           ;## relative URL's are a no-no
           set file /$file
           if { ! [ file exists $file ] } {
              return -code error "file not found: '$file'"
           }
           ;## is someone trying to address files not in the exported
           ;## directory hierarchy?
           if { ! [ regexp ^$::env(RUNDIR) $file ] } {
              set localfilename $jobdir/[ file tail $file ]
              bak $localfilename
              file copy -force $file $jobdir
           } else {
              set localfilename $file
           }
        ;## gridftp is similar to file URL, but has specific source
        ;## directory hierarchy - and is defined to be NOT visible to
        ;## API's other than the manager.
        } elseif { [ regexp -nocase {gridftp:/*(\S+)} $url -> file ] } {
           set file /$file
           regsub ^.*$::GRIDFTPDIR $file {} file
           set file $::GRIDFTPDIR$file
           if { ! [ file exists $file ] } {
              return -code error "file not found: '$file'"
           }
           if { ! [ regexp ^$::env(RUNDIR) $::GRIDFTPDIR ] } {
              set localfilename $jobdir/[ file tail $file ]
              bak $localfilename
              file copy -force $file $jobdir
           } else {
              set localfilename $file
           }
        ;## all other cases are handled or rejected by getCurlUrl
        } else {
           getCurlUrl $jobid $url $localfilename
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $localfilename
}

§   §   §

Name: putUrl

Description:
Given a local filename and a URL, put the local file to the remote URL. Only works via ftp right now.

Parameters: Usage:

Comments:

Remote URL's which end in a directory MUST have a trailing "/". VERY IMPORTANT!!
proc putUrl { jobid filename url } {
     if { [ catch {
        ;## filename could be a calculated name
        ;## and we might reasonably be able to assume
        ;## that it is in the job directory
        set dir [ jobDirectory $jobid ]
        set tail [ file tail $filename ]
        set localfilename $dir/$tail
        if { ! [ file exists $filename ] && \
               [ file exists $localfilename ] } {
           set filename $localfilename
        }
        if { ! [ file exists $filename ] } {
           set msg "file not found: '$filename'"
           return -code error $msg
        }
        ;## handle 'file:' URL's locally
        if { [ regexp {file:/*(.+)} $url -> url ] } {
           set url /$url
           bak $url
           file copy -force -- $filename $url
        } elseif { [ regexp -nocase {gridftp:/*(\S+)} $url -> file ] } {
           set file /$file
           regsub ^.*$::GRIDFTPDIR $file {} file
           set file $::GRIDFTPDIR$file
           file mkdir [ file dirname $file ]
           bak $file
           file copy -force -- $filename $file
        } else {
           ;## remote url points to a directory,
           ;## so lift the filename from the local
           ;## object
           if { [ regexp {/$} $url ] } {
              set url $url$tail
           }
           putCurlUrl $jobid $filename $url
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return [ list $filename $url ]
}

§   §   §

Name: curlGetRemoteFileSize

Description:
Coerce a http or ftp server into telling us the remote file size in bytes.

Parameters: Usage:

Comments:

3 seconds is not an unusually long timeout value.
proc curlGetRemoteFileSize { url } {
     if { [ catch {
        set bytes [ list ]
        set data [ ::exec curl -s -S -m 15 --connect-timeout 5 -I $url ]
        if { ! [ regexp {Content-Length:\s+(\d+)} $data -> bytes ] } {
           return -code error "server did not return size for $url"
        }
     ;## in general the error condition is a timeout
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $bytes
}

§   §   §

Name: fileIsBeingWritten

Description:
Returns 1 if a file is being written, 0 if it does not exist or is not currently being written to.

Parameters: Usage:
if { [ file exists $filename ] } {
   if { ! [ fileIsBeingWritten $jobid $filename ] } {
      ;## then file is here and we can go!

} }
Comments:
No point in calling this more often than once a second.
Behaviour can be bizarre under NFS.
proc fileIsBeingWritten { filename { size "" } } {
     if { [ catch {
        ;## file not found, so it's not being written
        if { ! [ file exists $filename ] } {
           set seqpt "no file"
           set bool 0
        ;## maybe we have a size to check against?
        } elseif { [ regexp {^\d+$} $size ] } {
           set filesize [ file size $filename ]
           if { $size > $filesize } {
              set bool 1
           } else {
              set seqpt "$size <= total $filesize"
              set bool 0
           }
        ;## really dicey to go by mtime with NFS, so we
        ;## never trust files with mtimes that are not
        ;## at least two seconds old!
        } else {
           set now [ clock seconds ]
           set mtime [ file mtime $filename ]
           set delta [ expr $now - $mtime ]
           if { $delta < 2 } {
              set bool 1
           } else {
              set seqpt "delta $delta ( $now - $mtime )  not < 2"
              set bool 0
           }
        }
     } err ] } {
        return -code error "[ myName ]: '$seqpt' $err"
     }
     if { !$bool && $::DEBUG > 1 } {
        addLogEntry $seqpt purple
     }
     return $bool
}

§   §   §

Name: jobDirectory

Description:
Job directories are in subdirectories of ::PUBDIR divided up into myriads of 10,000.
If the directory hierarchy does not exist it will be created automatically and with the correct perms.
Returns the full local path to the job directory.

Parameters: Usage:

Comments:

proc jobDirectory { { jobid "" } } {
     if { [ catch {
        ;## this will throw the right error if jobid
        ;## was not defined in the caller
        if { ! [ string length $jobid ] } {
           set jobid [ uplevel set jobid ]
        }
        regexp {\d+} $jobid job
        set jobid $::RUNCODE$job
        set myriad [ expr { $job / 10000 } ]
        set myrdir ${::RUNCODE}_$myriad
        set jobdir ${::RUNCODE}$job
        set jobdir [ file join $::PUBDIR  $myrdir $jobdir ]
        set testdir [ file join $::PUBDIR $myrdir ]
        ;## create the missing subdirectories as required
        if { ! [ file exists $testdir ] } {
           file mkdir        $testdir
           file attributes   $testdir -permissions 0775
        }
        if { ! [ file exists $jobdir ] } {
           file mkdir        $jobdir
           file attributes   $jobdir -permissions 0775
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $jobdir
}

§   §   §

Name: relativeDirectory

Description:
Returns the part of the job directory relative to ::PUBDIR.
This is useful for building URL's, for example.

Parameters: Usage:

Comments:

proc relativeDirectory { { jobid "" } } {
     if { [ catch {
        ;## this will throw the right error if jobid
        ;## was not defined in the caller
        if { ! [ string length $jobid ] } {
           set jobid [ uplevel set jobid ]
        }
        regexp {\d+} $jobid job
        set jobid $::RUNCODE$job
        set jobdir [ jobDirectory ]
        set components [ file split $jobdir ]
        set reldir [ lrange $components end-1 end ]
        set reldir [ join $reldir / ]
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $reldir
}

§   §   §

Name: apiDirectory

Description:
Returns the correct working directory for an api.
Return value lacks forward slash on right end.

Parameters: Usage:

Comments:

proc apiDirectory { { api "" } } {
     if { [ catch {
        if { ! [ string length $api ] } {
           set api $::API
        }
        set apidir [ file join $::env(RUNDIR) ${api}API ]
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $apidir
}

§   §   §

Name: portInfo

Description:
Returns the process name, pid, and owner of a port service.

Parameters: Usage:
      foreach { name pid owner } [ portInfo 25 ] { break }
      (should return "sendmail" "NNN" "root")

Comments:
Uses lsof, q.v.
Adopted ::PATH_TO_LSOF per PR #2805
proc portInfo { port } {
     if { [ info exists ::PATH_TO_LSOF ] && \
          [ string length $::PATH_TO_LSOF ] } {
        set lsof $::PATH_TO_LSOF
     } elseif { [ string length [ auto_execok lsof ] ] } {
        set lsof [ auto_execok lsof ]
        set ::PATH_TO_LSOF $lsof
     } else {
        set msg    "::PATH_TO_LSOF is not defined in "
        append msg "the LDASapi.rsc file, AND the lsof "
        append msg "executable is not in my path!! Please "
        append msg "set ::PATH_TO_LSOF to be the full path "
        append msg "to the lsof executable on $::env(HOST)."
        return -code error $msg
     }
     if { ! [ file executable $lsof ] } {
        set msg    "::PATH_TO_LSOF is defined as '$lsof', "
        append msg "but this is not an executable file!"
        return -code error $msg
     }
     set name  unknown
     set pid   0
     set owner unknown
     if { [ catch {
        set info [ exec $lsof -i TCP:$port ]
     } err ] } {
        return -code error "portInfo(lsof): $err"
     }
     set info [ lindex [ split $info "\n" ] 1 ]
     foreach { name pid owner } $info { break }
     return [ list $name $pid $owner ]
}

§   §   §

Name: sleep

Description:
Does a sleep like the UNIX sleep. Evaluation is suspended at the line where sleep is called, but event loop processing continues.

Parameters: Usage:

Comments:

proc sleep { ms } {
     if { $ms < 1 } { return {} }
     set uniq [ clock clicks ]
     set ::__sleep__tmp__$uniq 0
     after $ms set ::__sleep__tmp__$uniq 1
     vwait ::__sleep__tmp__$uniq
     unset ::__sleep__tmp__$uniq
}

§   §   §

Name: cloneProc

Description:
Creates a new procedure identical in behaviour to a prototype procedure, but with a different name.

Parameters: Usage:

Comments:

proc cloneProc { proto clone } {
     set argv [ list ]
     foreach x [ info args $proto ] {
        if { [ info default $proto $x y ] } {
           lappend argv [ list $x $y ]
        } else {
           lappend argv [ list $x ]
        }
      }
      proc $clone $argv [ info body $proto ]
}

§   §   §

Name: fifo

Description:
Provide an anonymous fifo using cat.

Parameters: Usage:
trivial handler example:

proc handle { fifo } { puts [ gets $fifo ] } set fifo [ fifo handle ] puts $fifo peep!
Comments:
The input side of the fifo should never be flushed!
The fifo can be closed like an ordinary file.
proc fifo { { handler "" } } {
     if { [ catch {
     set fifo [ open |cat a+ ]
     fconfigure $fifo -blocking off
     fconfigure $fifo -buffering none
     if { [ string length $handler ] } {
        fileevent $fifo readable "$handler $fifo"
     }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     set fifo
}

§   §   §

Name: execssh

Description:
remote execution via ssh
Parameters: Usage:
a complex example:

catch { \ set pid [ execssh $metahost \ cd [ file dirname $::PUBDIR ] && \ /bin/env \ HOST=$metahost \ PATH=/ldas/bin:/ldcg/bin:$::env(PATH) \ LD_LIBRARY_PATH=/ldcg/lib:$::env(LD_LIBRARY_PATH) \ DB2INSTANCE=ldasdb \ nohup \ "$::LDAS/bin/db2utils.tcl metadata \ $::LDASLOG $::dbDelay $::MGRKEY \ >& db2utils.log &" ] \ } err
Comments:
Set ::DEBUG_EXECSSH to log all activity Unset ::DEBUG_EXECSSH to stop logging
proc execssh { host args } {
     if { [ uplevel info exists jobid ] } {
        set jobid [ uplevel set jobid ]
     }
     ;## new condition to make use of remote proc manager
     if { [ info exists ::REMOTE_PROC_SERVER_PORT ] } {
        set kill_rx {pkill\s+\-9\s+(\S+)}
        if { [ regexp -nocase -- $kill_rx $args -> api ] } {
           if { [ catch {
              procServer $host "ps::kill $api"
           } err ] } {
              addLogEntry $err red
           }
        }
     }
     set usr $::env(LOGNAME)
     set loc $::env(HOST)
     set err [ list ]
     ;## for shellPipe
     set cmd "|ssh -x -n -obatchmode=yes $host"
     regsub -all {\$} $args \\$ args
     foreach arg $args {
        set cmd "$cmd $arg"
     }
     if { ! [ regexp {\&$} $cmd ] } {
	 if { ! [ regexp {\>\&} $cmd ] } {
	     set cmd "$cmd 2>@stdout"
	 }
     } else {
	 regsub -all -- {\&$} $cmd {} cmd
	 regsub -- {-n} $cmd {-f} cmd
     }
     if { [ info exists ::DEBUG_EXECSSH ] && \
          [ string equal 1 $::DEBUG_EXECSSH ] } {
        regsub -all -- $::MGRKEY $cmd ::MGRKEY clean
        addLogEntry $clean blue
     } else {
        set ::DEBUG_EXECSSH 0
     }
     if { [ catch {
        if { ! [ string equal $host $loc ] } {
           foreach { status msg } [ sock::diagnostic $host ] { break }
           if { $status } {
              return -code error "sock::diagnostic:$msg"
           }
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     catch { shellPipe $cmd } pid
     ;## sanitize the return value so it never looks like
     ;## a broken list.
     regsub -all {\"} $pid {'} pid
     ;## get rid of some terminal type warning messages
     regsub {stdin: is not a tty} $pid {} pid
     regsub {child process exited abnormally} $pid {} pid
     regsub {mesg: cannot stat} $pid {} pid
     set pid [ string trim $pid ]
     switch -regexp -- $pid {
        {Permission denied} {
                   set err "Permission denied for ${usr}@$loc -> $host"
                            }
        {Command not found} {
                   set err $pid
                            }
 {[kK]ey [vV]erification [fF]ailed} {
                   set err $pid
                            }
 {IDENTIFICATION HAS CHANGED} {
                   set err $pid
                            }
 {standard output was redirected} {
                   set pid UNKNOWN
                            }
                    default {
                             if { [ string equal 1 $::DEBUG_EXECSSH ] } {
                                regsub -all -- $::MGRKEY $pid ::MGRKEY Pid
                                addLogEntry $Pid blue
                             }
                            }
     }
     if { [ string length $err ] } {
        regsub -all -- $::MGRKEY $err ::MGRKEY err
        return -code error "[ myName ]: $err"
     }
     return $pid
}

§   §   §

Name: procServer

Description:

Parameters: Usage:

Comments:

proc procServer { host cmd } {
     if { [ catch {
        if { [ info exists ::REMOTE_PROC_SERVER_PORT ] } {
           set port $::REMOTE_PROC_SERVER_PORT
           set sid [ socket $host $port ]
           fconfigure $sid -blocking off
           ::puts $sid $cmd
           ::close $sid
        }
     } err ] } {
        catch { ::close $sid }
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: shellPipe

Description:
Shell wrapper that redirects stderr to stdout.
Returns as soon as data are available OR after a timeout, whichever comes first.

Parameters: Usage:
 cmd must be listified
 subcommands passed to ssh MUST be enclosed in
 single quotes!!

Comments:
Can only return the FIRST error from the command
proc shellPipe { cmd { timeout 10000 } } {
    if { [ catch {
        if { ! [ regexp {^\d+$} $timeout ] } {
	    set err "second argument 'timeout' must be an integer.\n"
	    append err "did you forget to listify your command?"
	    return -code error $err
        }
        set retval [ list ]
	set mode w+
        if { [ regexp {|} $cmd ] } {
	    if { [ regexp {\>\&} $cmd ] } {
		set mode w
	    }
	    set fid [ open $cmd $mode ]
	    fconfigure $fid -blocking off
        } else {
	    set fid [ open "|/bin/sh -s 2>@stdout" $mode ]
	    fconfigure $fid -blocking off
	    fconfigure $fid -buffering line
	    puts $fid $cmd
        }
	
        switch -exact -- $mode {
            w+ {
		set i [ key::time ]
		
		set ::shellpipe_done$i 0
		
		set afterid \
		    [ after $timeout [ list set ::shellpipe_done$i 1 ] ]
		
		fileevent $fid readable [ list set ::shellpipe_done$i 1 ]
		
		if { [ set ::shellpipe_done$i ] == 0 } {
		    vwait ::shellpipe_done$i
		}
		
		fileevent $fid readable {}
		
		after cancel $afterid
		
		::unset ::shellpipe_done$i
		
		set ms 0
		while { $ms < 3000 } {
		    append retval [ read $fid ]
		    if { [ eof $fid ] } { break }
		    after 30
		    incr ms 30
		}
		
		set retval [ string trim $retval "\n" ]
	    }
	} ;# switch
        ::close $fid
    } err ] } {
        catch { after cancel $afterid }
        catch { ::unset ::shellpipe_done$i }
        catch { ::close $fid }
        return -code error "[ myName ]: $err"
    }
    return $retval
}

§   §   §

Name: execscp

Description:
Wrapper for scp
Parameters: Usage:

Comments:

proc execscp { here there } {
     if { [ uplevel info exists jobid ] } {
        set jobid [ uplevel set jobid ]
     }
     set cmd "|scp -B $here $there"
     set host [ list ]
     set loc $::env(HOST)
     set usr  [ list ]
     set err  [ list ]
     if { [ catch {
        ;## run sock::diagnostic if hostname can be parsed
        ;## from "there".
        if { [ regexp {(.+)\@(.+)\:} $there -> usr host ] } {
           if { ! [ string equal $loc $host ] } {
              foreach { status msg } [ sock::diagnostic $host ] { break }
              if { $status } {
                 return -code error "sock::diagnostic:$msg"
              }
           }
        } else {
           set usr $::env(USER)
           set host $there
        }
        catch { shellPipe $cmd } pid
        regsub -all {\"} $pid {'} pid
        regsub {stdin: is not a tty} $pid {} pid
     switch -regexp -- $pid {
        {Permission denied} {
                   set err "Permission denied for ${usr}@$loc -> $host"
                            }
        {No such file or d} {
                   set err "file not found: '$here'"
                            }
 {[kK]ey [vV]erification [fF]ailed} {
                   set err $pid
                            }
 {IDENTIFICATION HAS CHANGED} {
                   set err $pid
                            }
                    default {
                            debugPuts $pid
                            }
     }
     if { [ string length $err ] } {
        return -code error $err
     }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $pid
}

§   §   §

Name: ucase

Description:
Capitaliser for words and sentences.
Calling with -strict option forces all non-initial characters to be lower case.

Parameters: Usage:
 ucase -strict "mary Had a littLe laMB"
Mary Had A Little Lamb
ucase "mary Had a littLe laMB"
Mary Had A LittLe LaMB
Comments:
proc ucase { args } {
     set retval [ list ]
     if { [ string equal -strict [ lindex $args 0 ] ] } {
        set args [ lrange $args 1 end ]
        set args [ string tolower $args ]
     }
     if { [ llength $args ] == 1 } {
        set args [ lindex $args 0 ]
     }
     if { [ catch {
        foreach word $args {
           set first [ string range $word 0 0 ]
           set rest  [ string range $word 1 end ]
           set word [ string toupper $first ]$rest
           lappend retval $word
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $retval
}

§   §   §

Name: libstdcPlusPlus

Description:
Get the filename of the libstdc++*.so on an LDAS machine.
The manager does this so that it can start an API on a remote machine with the appropriate LD_PRELOAD option for the libstdc++.so.

Parameters: Usage:

Comments:

This is used, for example, in
manager.tcl by the mgr::bootstrapAPI function.
proc libstdcPlusPlus { api } {
     if { [ catch {
        set host [ set ::${api}(host) ]
        if { [ info exists ::STDCPLUSPLUSLIBPAT ] } {
           set lspat $::STDCPLUSPLUSLIBPAT
        } else {
           set lspat /ldcg/lib/libstdc++-3*.so*
        }
        set data [ execssh $host /bin/ls $lspat ]
        if { [ regexp -- {o match} $data ] } {
           set msg "no files matching '$lspat' "
           append msg "found on host '$host'"
           return -code error $msg
        }
        if { ! [ regexp -- {[a-z]} $data ] } {
           set msg "running /bin/ls $lspat "
           append msg "on host '$host' returned '$data'"
           return -code error $msg
        }
        foreach file $data {
           lappend tmp [ string trim $file @ ]
        }
        set data [ lsort -dictionary $tmp ]
        set lib [ lindex $data end ]
        } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $lib
}

§   §   §

Name: when

Description:
Richard Suchenwirth's "when".
Waits in the background until $cond is true, and then evaluates $body.

Parameters: Usage:
   when {file exists /home/foo/bin/killerapp} bell
Comments:
proc when { cond body } {
     if { [ eval $cond ] } {
        if { [ catch {
           eval $body
        } err ] } {
           return -code err "[ myName ]: $err"
        }
     } else {
        after 1000 [ list when $cond $body ]
     }
}

§   §   §

Name: newhead

Description:
Replace N lines at the top of a file.

Parameters: Usage:

Comments:

This can be used to, for instance, to disable automatic reloading of "old" log diles via HTTP-REFRESH by replacing the first three lines of a log file.
proc newhead { file somenewlines } {
     if { [ catch {
        set fd1  [ open $file r ]
        set fd2  [ open $file.tmp w ]
        foreach line [ split $somenewlines "\n" ] {
          gets $fd1 my_old_line
        }
        puts $fd2 $somenewlines
        fcopy $fd1 $fd2
        close $fd1
        close $fd2
        file rename -force $file.tmp $file
     } err ] } {
        return -code error "[ myName ]: $err"
     }
}

§   §   §

Name: newtail

Description:
Replace N lines at the bottom of a file.

Parameters: Usage:

Comments:

Cannot be implemented as cleanly as newhead.
NOT OPERATIONAL :TODO:
proc newtail { file somenewlines } {
     return -code error "not implemented!"
     set fd1  [ open $file r ]
     set fd2  [ open $file.tmp w ]
     foreach line [ split $somenewlines "\n" ] {
       gets $fd1 my_old_line
     }
     puts $fd2 $somenewlines
     fcopy $fd1 $fd2
     close $fd1
     close $fd2
     file rename -force $file.tmp $file
}

§   §   §

Name: bak

Description:
Create backup files as necessary to avoid overwrites.

Parameters: Usage:
before writing to a file $fname, call: bak $fname
and the file will not get overwritten.

renames like so: .bak, .ba2, .ba3, .ba4, etc.
Comments:
proc bak { fname { levels 10 } } {
     if { [ catch {
        if { [ file exists $fname ] } {
           set dir [ file dirname $fname ]
           set files [ glob -nocomplain $dir ${fname}.ba* ]
           set i $levels
           while { [ incr i -1 ] } {
              if { [ lsearch $files ${fname}.ba$i ] > -1 } {
                 file rename -force ${fname}.ba$i ${fname}.ba[ incr i ]
                 incr i -1
              }
           }
           if { [ file exists ${fname}.bak ] } {
              file rename -force ${fname}.bak ${fname}.ba2
           }
           file rename -force $fname ${fname}.bak
        }
     } err ] } {
        return -code error "bak($fname $levels): $err"
     }
}

§   §   §

Name: bakDemangle

Description:
Tries to resolve a .bak file name into the list of files that are actually in the job directory.

Parameters: Usage:

Comments:

proc bakDemangle { filename } {
     if { [ catch {
        set dir       [ file dirname $filename ]
        set filename  [ file rootname [ file tail $filename ] ]
        set filenames [ glob -nocomplain ${dir}/${filename}.* ]
        set filenames [ lsort -dictionary $filenames ]
        if { [ llength   $filenames ] == 0 } {
           set filenames $filename
        }
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $filenames
}

§   §   §

Name: realTimeRscValues

Description:
Return the current values of resouce variables .

Parameters: Usage:
call this with a list of global variables names
e.g. realTimeRscValues { ::DEBUG ::LDAS_RESOLVABLE_IP }
Comments:
handles deletion of resource vars in memory
proc realTimeRscValues { varlist } {
    set result ""
    foreach var $varlist {
        if  { [ catch {
            if  { ! [ regexp {^::} $var ] } {
                set var ::$var
            }
            ;## make array elements unique
			if	{ [ array exist $var ] } {
                append result "array set $var \[ list [ array get $var ] \]\n"
            } else {
                append result "set $var [ list [ eval set $var ] ]\n"
            }
        } err ] } {
            append result "set $var [ list ERROR: no such variable ]\n"
        }
    }
    return [ string trim $result \n ]
}

§   §   §

Name: getApiOS

Description:
Given the name of an API, returns the operating system portion of the string returned by 'uname -a'.

Parameters: Usage:

Comments:

Misconfigured ssh handled here.
proc getApiOS { api } {
     if { [ catch {
        if { ! [ info exists ::${api}(OS) ] } {
           set host [ set ::${api}(host) ]
           catch { exec uname -n } localhost
           if	{ [ string equal $host $localhost ] } {
           		catch { exec /bin/uname -a } osstuff
           } else {
           		set osstuff [ execssh $host /bin/uname -a ]
           }
           regexp {(SunOS|Linux)} $osstuff OS
           set ::${api}(OS) $OS
        } else {
           set OS [ set ::${api}(OS) ]
        }
        ;## handle misconfigure ssh
        if { ! [ info exists OS ] } {
           set msg "getApiOS: ERROR! could not run 'ssh uname -a' on"
           append msg " host: '$host': $osstuff"
           append msg " ${api}API has NOT been started!!"
           addLogEntry $msg email
           ;## pretty serious, make sure everybody knows!!
           puts stderr $msg
           set OS ssh_failed
        }
        switch -exact $OS {
             SunOS {}
             Linux {}
        ssh_failed {}
           default {
                   set msg "getApiOS: unknown operating system:"
                   append msg " $osstuff"
                   return -code error $msg
                   }
           } ;## end of switch
     } err ] } {
        return -code error "[ myName ]: $err"
     }
     return $OS
}

§   §   §

Name: dumpData

Description:
snapshot of what data structures are outstanding at the moment Usage:

Comments:

threads, objects, job array, data bucket
proc dumpData { args } {
     set jobid_rx (${::RUNCODE}\\d+)
     foreach [ list socks files threads events objects ] [ countChannels ] { break }
     set text    "[ llength $socks ] sockets: $socks\n"
     append text "[ llength $files ] files opened: $files\n"
     append text "[ expr [ llength $threads ]/3 ] threads: $threads \n"
     append text "[ llength $events ] events: $events \n"
     append text "[ llength $objects ] objects: $objects \n"
     append text "jobs:\n"
     set numJobs 0
     set numJobBuckets 0
     foreach var [ info vars ::${::RUNCODE}* ] {
        if { [ regexp $jobid_rx\$ $var -> jobid ] } {
           append text "$jobid [ array get ::${jobid} ]\n"
           incr numJobs 1
        } elseif { [ regexp ${jobid_rx}_DATABUCKET $var -> jobid ] } {
           append text "$jobid bucket has [ llength $var ] objects\n"
           incr numJobBuckets 1
        }
     }
     append text "$numJobs jobs remaining, $numJobBuckets data buckets remaining"
     if { [ array exist ::appThreads ] } {
        append text "appThreads: [ array get ::appThreads ]"
     }
     return $text
}

§   §   §

Name: destructElementWrap

Description:
wrapper for destructElement, log errors tag - for logging ignoreErr - ignore destructElement here Usage:

Comments:

useful to checking out invalid_element errors
proc destructElementWrap { elementp { tag "" } { ignoreErr 1 } } {
     set name [ lindex [ info level -1 ] 0 ]
     set client [ uplevel { namespace tail [ namespace current ] } ]
     if { [ catch {
        set id [ ilwd::getjob $elementp ]
        if { ! [ info exist id ] || ! [ string length $id ] } {
           set id none
        }
        destructElement $elementp
        if { ! $ignoreErr } {
           debugPuts "$name,$client,destroying job $id $elementp for $tag"
        }
     } err ] } {
        if { ! $ignoreErr } {
           addLogEntry "$name,$client,error destroying object $elementp, $tag: $err" red
        }
     }
}

§   §   §

Name: isBinary

Description:
Given a filename, returns 1 if the file is binary, 0 if it is a text file.
Operates by reading 1k of the file and looking for forbidden characters.

Parameters: Usage:

Comments:

requires << 1 ms to return.
proc isBinary { filename } {
     if { [ catch {
        set true  1
        set false 0
        set bin_rx {[\x00-\x08\x0b\x0e-\x1f]}
        set fid [ open $filename r ]
        fconfigure $fid -translation binary
        fconfigure $fid -buffersize 1024
        fconfigure $fid -buffering full
        set test [ read $fid 1024 ]
        ::close $fid
        if { [ regexp $bin_rx $test ] } {
           set binary $true
        } else {
           set binary $false
        }
     } err ] } {
        catch { ::close $fid }
        return -code error "[ myName ]: $err"
     }
     return $binary
}

§   §   §

Name: fileType

Description:
Fancy file type discriminator that can return types:
  1. [binary|text] ilwd [ligo frame] [link]
  2. [binary|text] xml [ligo_lw] [link]
  3. binary frame [link]
  4. binary elf executable [link]
  5. [binary|text] [*] script [link]
  6. text html [link]
  7. binary [link]
  8. text [link]
  9. binary [gif|jpeg] images [link]
  10. [binary|text] pgp message digests [link]
  11. directory [link]
  12. empty [link]
    Parameters:
    Usage:
    
    Comments:
    
    Overhead is about 2 ms. Links are detected and reported
    proc fileType { filename } {
         ;## existence test
         if { ! [ file exists $filename ] } {
            set err "file not found: '$filename'"
            return -code error "[ myName ]: $err"
         }
         ;## directory test
         if { [ file isdirectory $filename ] } {
            set type directory
            if { ! [ catch {file readlink $filename} ] } {
               lappend type link
            }
            return $type
         }
         ;## empty file test
         if { ! [ file size $filename ] } {
            set type empty
            if { ! [ catch {file readlink $filename} ] } {
               lappend type link
            }
            return $type
         }
         set bin_rx {[\x00-\x08\x0b\x0e-\x1f]}
         if { [ catch {
            set fid [ open $filename r ]
            fconfigure $fid -translation binary
            fconfigure $fid -buffersize 1024
            fconfigure $fid -buffering full
            set test [ read $fid 1024 ]
            ::close $fid
         } err ] } {
            set err $::errorInfo
            catch { ::close $fid }
            return -code error "[ myName ]: $err"
         }
         if { [ regexp $bin_rx $test ] } {
            set type binary
            set binary 1
         } else {
            set type text
            set binary 0
         }
         if { [ regexp {^\#\!\s*(\S+)} $test -> terp ] } {
            lappend type script $terp
         } elseif { $binary && [ regexp {^[\x7F]ELF} $test ] } {
            lappend type executable elf
         } elseif { $binary && [ regexp {^BZh91AY\&SY} $test ] } {
            lappend type compressed bzip
         } elseif { $binary && [ regexp {^\x1f\x8b} $test ] } {
            lappend type compressed gzip
         } elseif { $binary && [ regexp {^GIF} $test ] } {
            lappend type graphic gif
         } elseif { $binary && [ regexp {^\x89PNG} $test ] } {
            lappend type graphic png
         } elseif { $binary && [ regexp {^\xFF\xD8\xFF\xE0\x00\x10JFIF} $test ] } {
            lappend type graphic jpeg
         } elseif { $binary && [ regexp {^MM\x00\*} $test ] } {
            lappend type graphic tiff
         } elseif { $binary && [ regexp {^\%PDF\-} $test ] } {
            lappend type pdf
         } elseif { ! $binary && [ regexp -nocase {(\<html\>|\<a\s+href=)} $test ] } {
            lappend type html
            if { [ regexp -nocase {404\s+Not\s+Found} $test ] } {
               lappend type 404
            } elseif { [ regexp -nocase {Error\s+(\d{3})} $test -> errcode ] } {
               lappend type $errcode
            }
         } elseif { [ regexp {^\%\!PS\-} $test ] } {
            lappend type ps
            if { [ regexp { EPSF\-} $test ] } {
               lappend type eps
            }
         } elseif { [ regexp -nocase {\<\?xml} $test ] } {
            lappend type xml
            if { [ regexp -nocase {\<\!DOCTYPE\s+(\S+)} $test -> doctype ] } {
               lappend type [ string tolower $doctype ]
            }
         } elseif { [ regexp {BEGIN PGP MESSAGE} $test ] } {
            lappend type message pgp
         } elseif { $binary && [ regexp {^IGWD} $test ] } {
            lappend type frame
         } elseif { [ regexp -nocase {\<\?ilwd\?\>} $test ] } {
            lappend type ilwd
            if { [ regexp {LIGO::Frame} $test ] } {
               lappend type ligo frame
            }
         }
         ;## lastly, is it a link?
         if { ! [ catch {file readlink $filename} ] } {
            lappend type link
         }
         return $type
    }
    
    
    §   §   §

    Name: varType

    Description:
    Discriminate between the various things that might be passed around inside an LDAS API.
    Returns a list of two items, the second of which will be the argument as passed in, or a corrected file name if a filename was passed in some form.

    Parameters:
    Usage:
    
    Comments:
    
    The type 'file' means it is a file visible to this API.
    proc varType { item } {
         if { [ catch {
            set type unknown
            set item [ string trim $item ]
    	;## validate characters in item for filename
    	;## NOTE:
    	;##   This is necessary to prevent a memory leak found in tcl 8.4.12
    	;##   TCL appears to have problems with strings like
    	 if { ! [ string match {* *} $item ] } {
    	    ;## try to parse an ordinary filename
    	    if { ! [ file exists $item ] } {
    		if { [ file exists /$item ] } {
    		    set item /$item
    		    set type file
    		} elseif { [ file exists [ string trim $item "/" ] ] } {
    		    set item [ string trim $item "/" ]
    		    set type file
    		}
    	    } else {
    		set type file
    	    }
    	}
            ;## added 04/15/2003 - important case!!
            if { [ string equal file $type ] } {
               if { [ file isdirectory $item ] } {
                  set type directory
               }
            }
            ;## a URL, but maybe a local file specified by a URL
            if { [ regexp {(?:file|http|ftp):/*(\S+)} $item -> test ] } {
               if { [ string length [ lindex [ parseURL $item ] 3 ] ] } {
                  set type url
               }
               ;## well intentioned, but likely to cause problems
               #if { [ file exists $test ] || [ file exists /$test ] } {
               #   set type file
               #}
            }
            ;## what kind of file?
            if { [ string equal file $type ] } {
               set type [ concat [ fileType $item ] file ]
            ;## an ilwd pointer?
            } elseif { [ regexp \
               {^_[0-9a-f]+_p_Ldas(Container|ArrayBase|Element)$} $item ] } {
               set type [ list ilwd pointer ]
            ;## a framecpp pointer
            } elseif { [ regexp {^_[0-9a-f]+_p_Fr[a-zA-Z]+$} $item ] } {
               set type [ list framecpp pointer ]
            ;## maybe raw ilwd text??
            } elseif { [ regexp {<[^>]+>} $item ] } {
               set type [ list ilwd text ]
            ;## maybe a ligo lightweight pointer??
            } elseif { [ regexp {_p_LWDocument$} $item ] } {
               set type [ list ligo_lw document pointer ]
            ;## or ligo lightweight (an xml document type)
            } elseif { [ regexp -nocase {\<\!DOCTYPE LIGO_LW} $item ] } {
               set type [ list ligo_lw text ]
            }
            ;## try dereferencing in uplevel context
            ;## and recurse once.
            if { [ string equal unknown $type ] } {
               set caller [ uplevel myName ]
               if { ! [ string equal varType $caller ] } {
                  if { [ llength $item ] == 1 && \
                       [ uplevel info exists $item ] } {
                     set item [ uplevel set $item ]
                     set type [ varType $item ]
                  }
               }
            }
         } err ] } {
            return -code error "[ myName ]: $err (item: '$item')"
         }
         return [ list $type $item ]
    }
    
    
    §   §   §

    Name: packageReport

    Description:
    Writes a file named $::API.packages into the working directory of each API at startup.
    This file is linked from the api status page.

    Parameters:
    Usage:
    
    Comments:
    
    proc packageReport { args } {
         if { [ catch {
            set dom $::LDAS_SYSTEM
            set head "<html>
                      <head>
                      <!-- report of packages and versions -->
                      <title>LDAS $::API API Package Info at $dom</title>
                      </head>
                      <BODY BGCOLOR='#DDDDDD' TEXT='#000000'>
                      <h2>LDAS $::API API Package Info at $dom</h2>
                      <p>
                      <ul>"
            set foot "</ul></body></html>"
            regsub -all {[ ]+} $head { } head
            set body [ list ]
            set pkgs [ lsort -dictionary [ package names ] ]
            foreach pkg $pkgs {
               if { [ catch {
                  set version [ package present $pkg ]
               } err ] } {
                  continue
               }
               set version "<font color=red>$version</font>"
               if { [ string equal Tcl $pkg ] } {
                  set version $::tcl_patchLevel
                  set version "<font color=red>$version</font>"
               }
               append body "<li><b>$pkg</b>  - $version\n"
            }
            if { [ info exists ::PATH_TO_LSOF ] && \
               [ string length $::PATH_TO_LSOF ] } {
               set lsof $::PATH_TO_LSOF
            } elseif { [ string length [ auto_execok lsof ] ] } {
               set lsof [ auto_execok lsof ]
               set ::PATH_TO_LSOF $lsof
            } else {
               set lsof lsof
            }
            ;## added lsof report to packages page 08/12/03
            catch { exec $lsof -b -p [ pid ] } loaded
            if { [ string length $loaded ] } {
               append body "<p>\n<pre>\n$loaded\n</pre>\n<p>\n"
            }
            ;## write in the current directory
            set filename ${::API}.packages
            set fid [ open $filename w ]
            puts $fid $head
            puts $fid $body
            puts $fid $foot
            ::close $fid
            file attributes $filename -permissions 0644
         } err ] } {
            catch { ::close $fid }
            return -code error "[ myName ]: $err"
         }
    }
    
    
    §   §   §

    Name: safeRxPat

    Description:
    Make a string safe for use as a regexp pattern by escaping dangerous chars.

    Parameters:
    Usage:
    
    Comments:
    
    proc safeRxPat { string } {
         if { [ catch {
            set pat_rx {[\(\)\{\}\[\]\.\+\-\*\?]}
            regsub -all $pat_rx $string {\\&} string
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $string
    }
    
    
    §   §   §

    Name: outputUrlsBg

    Description:
    Will wait until a file is finished writing before connecting to the manager and telling it to push the file along to a URL.

    Parameters:
    Usage:
    
    Comments:
    
    done in the background after if file is not immedidately done
    proc outputUrlsBg { var jobid urls { seconds 0 } } {
         if { [ catch {
            set repeat 0
            set bad_rx \[^\\\s\\\/\\\'\\\"\\\}\\\{\\\)\\\(\\\]\\\[\\\>\\\<\]+
            set sane_rx (file|ftp|http):\\\/+$bad_rx\(\\\/$bad_rx\)+
            set fifteen_minutes [ expr { 15 * 60 } ]
            foreach [ list file url ] $urls {
               set rc1 [ file exists $file ]
               set rc2 [ fileIsBeingWritten $file ]
               if { ! $rc1 || $rc2 } {
                  incr seconds
                  if { $seconds > $fifteen_minutes } {
                     set msg "output file: '$file' still not available, file exist ? $rc1, fileIsBeingWritten ? $rc2"
                     append msg " after 15 minutes! giving up!"
                     error $msg
                  }
                  set repeat 1
               }
               if   { ! $repeat } {
                    if  { [ regexp $sane_rx $url ] } {
                        managerOutputUrl $jobid [ list $file $url ]
                    } else {
                        debugPuts "invalid URL: '$url'"
                    }
                }
            }
         } err ] } {
            set $var "Error: $err"
            return -code error [ set ::$var ]
         }
         if { $repeat } {
            after 1000 [ list outputUrlsBg $var $jobid $urls $seconds ]
         } else {
            set $var OK
         }
    }
    
    
    §   §   §

    Name: outputUrls

    Description:
    Will wait until a file is finished writing before connecting to the manager and telling it to push the file along to a URL.

    Parameters:
    Usage:
    
    Comments:
    
    waits for result so failure is captured back to user esp. if outputUrls is called again in after
    proc outputUrls { jobid urls { seconds 0 } } {
         if { [ catch {
            set result ""
            set unique_key [ key::time ]
            outputUrlsBg ::$unique_key $jobid $urls $seconds
            vwait ::$unique_key
            set result [ set ::$unique_key ]
            unset ::$unique_key
            if  { [ regexp {^Error:.+}  $result ] } {
                error $err
            }
         } err ] } {
            # addLogEntry "[ myName ]: $err" email
            return -code error $err
            catch { unset ::${unique_key} }
         }
         return $result
    }
    
    
    §   §   §

    Name: managerOutputUrl

    Description:
    Generic routine for connecting to manager API and passing URL output information.
    urls should be a flat list of pairs of local filenames and URL's to push them to.

    Parameters:
    Usage:
    
    Comments:
    
    proc managerOutputUrl { jobid urls } {
         if { [ catch {
            set cmd    "$::MGRKEY NULL NULL "
            append cmd "mgr::delocaliseOutputURLs $jobid \{$urls\}"
            set sid [ sock::open manager emergency ]
            puts $sid $cmd
            ::close $sid
         } err ] } {
            catch { ::close $sid }
            return -code error "[ myName ]: $err"
         }
    }
    
    
    §   §   §

    Name: fixUrlTarget

    Description:
    wraps parseURL to append 2nd target name to 1st if any is returned.
    returns protocol,target and port Usage:
    
    Comments:
    
    call this in place of parseURL
    proc fixUrlTarget { url } {
        if { [ catch {
           foreach [ list protocol targ1 port targ2 ] [ parseURL $url ] {
              break
           }
           if { [ string length $targ2 ] } {
              set target $targ1/$targ2
              regsub -all {/+} $target {/} target
           } else {
              set target $targ1
           }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return [ list $protocol $target $port ]
    }
    
    
    §   §   §

    Name: getMddTarget

    Description:
    determine mdd target from -mddapi or -multiDimDataTarget Usage:
    
    Comments:
    
    returns null if data
    proc getMddTarget { jobid } {
         set mddtarget [ list ]
         if { [ info exist ::${jobid}(-multidimdatatarget) ] && \
            [ string length [ set ::${jobid}(-multidimdatatarget) ] ] } {
            set mddtarget [ set ::${jobid}(-multidimdatatarget) ]
         } elseif { [ info exist ::${jobid}(-mddapi) ] && \
            [ string length [ set ::${jobid}(-mddapi) ] ] } {
            set mddtarget [ set ::${jobid}(-mddapi) ]
         }
         return $mddtarget
    }
    
    
    §   §   §

    Name: dropStatusPage

    Description:
    Write file used by manager API to generate status page.

    Parameters:
    Usage:
    
    Comments:
    
    Could not use dumpFile on /proc/pid because it's size is 0 until you read it!!
    proc dropStatusPage { } {
         if { [ catch {
            set props [ sysData $::API ]
            set chans [ countChannels ]
            set filename [ file join $::LDASLOG ${::API}.status ]
            if { [ file exists $filename ] } {
               set fid [ open $filename r ]
               set olddata [ read $fid [ file size $filename ] ]
               ::close $fid
               if { [ catch {
                  set olddata [ lindex $olddata 0 ]
                  if { [ string length $olddata ] && \
                     ! [ string length $props ] } {
                     set props $olddata
                  }
               } err ] } {
                  set err "ignoring corrupt status file '$filename': '$err'"
                  addLogEntry $err red
               }
            }
            set fid [ open ${filename}.tmp w 0600 ]
            puts $fid [ list $props $chans [ array get ::$::API ] ]
            ::close $fid
            file rename -force ${filename}.tmp $filename
         } err ] } {
            catch { ::close $fid }
            addLogEntry $err red
         }
    }
    
    
    §   §   §

    Name: cmdSets

    Description:
    supports execution of a set of cmds and return one result
    Parameters:
    Usage:
    examples of cmds:
    
    set cmd "puts \$cid \[ cmdSet \{ \{ mgr::updateUserInfo patty junk \{$email\} $flag \} \ \{ mgr::updateUserInfo juno xxxyy \{$email\} $flag \} \ \{ mgr::updateUserInfo jaco xxx \{$email\} $flag \} \ \} \]"
    Comments:
    useful for emergency data request
    proc cmdSet { cmdlist } {
         set result ""
         foreach cmd $cmdlist {
            if { [ catch {
                append result "[ ::eval $cmd ]; "
            } err ] } {
               append result "$cmd : $err\n"
            }
         }
    	return $result
    }
    
    
    §   §   §

    Name: cmdGet

    Description:
    supports execution of a set of cmds and return all results from all commands
    Parameters:
    Usage:
       examples of cmds:
    
    cmd={ puts $cid [ cmdGet [ list { [ lindex [ set \ ::QUEUE(LDAS-DEV11104869,cmd) ] 0 ]} \ { [ lindex [ set ::QUEUE(LDAS-DEV11104883,cmd) \ ] 0 ]} ] ] }
    Comments:
    useful for emergency data request
    proc cmdGet { cmdlist } {
         set result [ list ]
         foreach cmd $cmdlist {
            if { [ catch {
               append result "[ eval $cmd ]\n"
            } err ] } {
               append result "$cmd : $err\n"
            }
         }
    return $result
    }
    
    
    §   §   §

    Name: cryptFileInPlace

    Description:

    Parameters:
    Usage:
    
    Comments:
    
    proc cryptFileInPlace { filename salt } {
         if { [ catch {
            set data [ dumpFile $filename  ]
            set data [ encrypt $data $salt ]
            set fid  [ open $filename w    ]
            puts $fid $data
            ::close $fid
            file attributes $filename -permissions 0600
         } err ] } {
            catch { ::close $fid }
            return -code error "[ myName ]: $err"
         }
    }
    
    
    §   §   §

    Name: decryptFile

    Description:

    Parameters:
    Usage:
    
    Comments:
    
    proc decryptFile { filename salt } {
         if { [ catch {
            set data [ dumpFile $filename  ]
            set data [ decrypt $data $salt ]
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $data
    }
    
    
    §   §   §

    Name: binaryEncrypt

    Description:
    simple encryption Usage:
    
    Comments:
    
    news.comp.lang.tcl
    proc binaryEncrypt {passphrase cleartext} {
        set r {}
        binary scan $passphrase c* l
        binary scan $cleartext c* d
        set pmax [llength $l]
        set cn 0
        foreach c $d {
            set cp [lindex $l $cn]
            append r [format %c [expr {($c & 0xff) + ($cp & 0xff)}]]
            incr cn
            if {$cn >= $pmax} { set cn 0 }
        }
        return $r
    }
    
    
    §   §   §

    Name: binaryDecrypt

    Description:
    simple decryption Usage:
    
    Comments:
    
    from news.comp.lang.tcl, hence the opaque variable names.
    proc binaryDecrypt { passphrase str } {
         if { [ catch {
            set r {}
            binary scan $passphrase c* l
            binary scan $str c* d
            set pmax [ llength $l ]
            set cn 0
            foreach c $d {
               set cp [ lindex $l $cn ]
               append r [ format %c [ expr {($c & 0xff) - ($cp & 0xff)} ] ]
               incr cn
               if { $cn >= $pmax } { set cn 0 }
            }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $r
    }
    
    
    §   §   §

    Name: bCrypt

    Description:
    Simple binary encrypt/decrypt
    Parameters:
    Usage:
    
    Comments:
    
    proc bCrypt { passphrase data { action decrypt } } {
         if { [ catch {
            if { [ string equal encrypt $action ] } {
               set op +
            } else {
               set op -
            }
            set result {}
            set cn     0
            binary scan $passphrase c* l
            binary scan $data       c* d
            set pmax [ llength $l ]
            foreach c $d {
               set cp   [ lindex $l $cn ]
               append result \
                  [ format %c [ expr { ($c & 0xff) $op ($cp & 0xff) } ] ]
               if { [ incr cn ] >= $pmax } {
                  set cn 0
               }
            }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $result
    }
    
    
    §   §   §

    Name: bootLock

    Description:

    Parameters:
    Usage:
    
    Comments:
    
    proc bootLock { state args } {
         if { [ catch {
            set sid [ sock::open manager emergency ]
            set pause "$::MGRKEY NULL NULL mgr::bootstrapLock   $::API"
            set run   "$::MGRKEY NULL NULL mgr::bootstrapUnlock $::API"
            if { [ string equal -nocase on $state ] } {
               puts $sid $pause
            } else {
               puts $sid $run
            }
            ::close $sid
         } err ] } {
            catch { ::close $sid }
            set subject "$::LDAS_SYSTEM $::API API error!!"
            set msg "[ myName ]: error contacting manager API: '$err'"
            addLogEntry  "Subject: ${subject}; Body: $msg" email
         }
    }
    
    
    §   §   §

    Name: int2roman

    Description:
    Convert natural numbers to roman numerals.

    Parameters:
    Usage:
    
    Comments:
    
    Code by Richard Suchenwirth from http://mini.net/tcl/1749
    proc int2roman { i } {
         if { [ catch {
            if { $i < 1 } {
               set err "only natural numbers can be represented "
               append err "as Roman Numerals. Not a natural "
               append err "number: '$i'"
               return -code error $err
            }
            set result [ list ]
            set keys [ list 1000 M 900 CM 500 D 400 CD 100 C \
                     90 XC 50 L 40 XL 10 X 9 IX 5 V 4 IV 1 I ]
            foreach [ list value roman ] $keys {
               while { $i >= $value } {
                  append result $roman
                  incr i -$value
              }
           }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $result
    }
    
    
    §   §   §

    Name: outputFormat

    Description:

    Parameters:
    Usage:
    
    Comments:
    
    proc outputFormat { format } {
         if { [ catch {
            switch -regexp -- $format {
               {[Ff][Rr][Aa][Mm][Ee]} {
                  set type frame
                  if { [ regexp -nocase {ilwd} $format ] } {
                     set type [ list frame ilwd ]
                     if { [ regexp -nocase {binary|ascii} $format type2 ] } {
                        lappend type $type2
                     } else {
                        set type [ list frame ilwd ascii ]
                     }
                  } elseif { [ regexp -nocase {ligo_?lw} $format ] } {
                     if { [ regexp -nocase {base64} $format type ] } {
                        set type [ list frame LIGO_LW base64 ]
                     } else {
                        set type [ list frame LIGO_LW ]
                     }
                  }
                  set format $type
                 }
               {[Ii][Ll][Ww][Dd]} {
                  if { [ regexp -nocase {binary|ascii} $format type ] } {
                     set format [ list ilwd $type ]
                  }
                 }
               {[Ll][Ii][Gg][Oo]_?[Ll][Ww]} {
                  if { [ regexp -nocase {base64} $format ] } {
                     set format [ list LIGO_LW base64 ]
                  }
                 }
               {[Aa][Ss][Cc][Ii][Ii]} {
                    set format [ list ilwd ascii ]
                 }
               {[Bb][Ii][Nn][Aa][Rr][Yy]} {
                    set format [ list ilwd binary ]
                 }
               {[Bb][Aa][Ss][Ee]64} {
                    set format [ list LIGO_LW base64 ]
                 }
               default {
                        set err "unrecognised output format: '$format'"
                        return -code error $err
                       }
            }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $format
    }
    
    
    §   §   §

    Name: cppBase64Init

    Description:
    Initialise base64 character per line limit for XML base 64 output format.

    Parameters:
    Usage:
    
    Comments:
    
    This should be called immediately after loading the libgeneral.so shared library.
    proc cppBase64Init { args } {
         ;## default is specifically for compatibility with Daniel
         ;## Sigg's DTT XML parsing library.
         if { ! [ info exists ::BASE_64_CHARACTER_PER_LINE_LIMIT ] } {
            set ::BASE_64_CHARACTER_PER_LINE_LIMIT 64
         }
         set limit $::BASE_64_CHARACTER_PER_LINE_LIMIT
         if { [ catch {
            if { [ expr { $limit %  4 } ] } {
               set err "::BASE_64_CHARACTER_PER_LINE_LIMIT bad value: "
               append err "'$limit' must be integral multiple of 4."
               return -code error $err
            } else {
               setBase64CharacterPerLineLimit $limit
            }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
    }
    
    
    §   §   §

    Name: checkLeaks

    Description:
    check for var leaks in namespaces and swig name leaks error codes
    Parameters:
    Usage:
    
    Comments:
    
    proc checkLeaks { args } {
         set out "--- channels:\n[ dumpData ]\n\n--- globals vars: [ info globals * ]\n"
         append out "\nChecking namespace leaks:\n"
         foreach name [ namespace children ] {
            append out "--- $name namespace:\n"
            set vars [ namespace eval $name info vars ::${name}::* ]
            foreach var $vars {
                append out "$var,value "
                if { [ namespace eval $name array exist $var ] } {
                   append out "[ namespace eval $name array get $var ]\n"
                } else {
                   append out "[ namespace eval $name set $var ]\n"
                }
            }
        }
        set cmds [ info commands ]
        append out "\nChecking swig name leaks in commands:\n"
        set cmdLeaks 0
        foreach cmd [ info commands *_p_* ] {
           if { [ regexp {_[a-f\d]+_p_} $cmd ] } {
              incr cmdLeaks 1
              append out "$cmd\n"
           }
        }
        append out "$cmdLeaks command leaks"
        set fd [ open leaks.log w ]
        puts $fd $out
        close $fd
    }
    
    
    §   §   §

    Name: freeMemOnBox

    Description:

    Parameters:
    Usage:
    
    Comments:
    
    proc freeMemOnBox { args } {
         if { [ catch {
            set OS [ getApiOS $::API ]
            switch -exact $OS {
                 SunOS {
                        set freemem [ lindex [ exec vmstat 1 2 ] end-17 ]
                       }
                 Linux {
                        set freemem [ dumpFile /proc/meminfo ]
                        regexp {MemFree\D+(\d+)} $freemem -> memfree
                        regexp {Cached\D+(\d+)}  $freemem -> cachemem
                        set freemem [ expr { $memfree + $cachemem } ]
                       }
               default {
                        return -code error "Unknown OS!"
                       }
            }
         } err ] } {
            return -code error "[ myName ]: $err"
         }
         return $freemem
    }
    
    
    §   §   §

    Name: wrapText

    Description:
    wrap long text with continuation chars and new lines and remove leading and trailing blanks.

    Parameters:
    Usage:
    
    Comments:
    
    proc wrapTextX { text } {
        ;## delistify
        if  { [ string match "\{*" $text ] } {
            while 1 {
                set text [ string trim [ lindex $text 0 ] ]
                if  { [ regexp {^\{} $text ] } {
                    set value [ lindex $text 0 ]
                } else {
                    break
                }
            }
        }
        if  { [ string length $text ] < 30 } {
            return $text
        } else {
            set newtext ""
            set newlen 0
            foreach word $text {
                incr newlen [ string length "$word " ]
                if  { $newlen > 30 } {
                    append newtext " \\\n$word "
                    set newlen 0
                } else {
                    append newtext " $word"
                }
            }
            set newtext [ string trim $newtext ]
            set newtext [ string trim $newtext \n ]
            return $newtext
        }
    }
    
    
    §   §   §
    proc wrapText { text } {
    	set columnLen 70
    	if  { [ string length $text ] <= $columnLen } {
            return $text
        } else {
            set newtext ""
            set newlen 0
            ;## dont use foreach - will delistify
            set strlen [ string length $text ]
            set newtext ""
           	for { set i 0 } { $i < $strlen } { incr i $columnLen} {
            	set index 0
            	set limit [ expr $i + $columnLen ]
                if	{ $limit > $strlen } {
                	set limit end
                    append newtext "[ string range $text $i $limit ]"
                } else {
                	set segment [ string range $text $i $limit ]
                	set index [ string last " " $segment ]
                	if	{ $index > -1 } {
                    	set fromend [ expr $limit - ( $index + $i ) ]
                        incr limit -$fromend
                        set index -$fromend
                        append newtext "[ string range $text $i $limit ]\\\n"
                	} else {
                        set index 0
                        append newtext "[ string range $text $i $limit ]"
                    }
               	}
                incr i $index
                incr i
            }
            set newtext [ string trim $newtext \\\n ]
    		set unwraptext [ unwrapText $newtext ]
            set origlen [ string length $text ]
            set newlen [ string length $unwraptext]
            if	{ $origlen != $newlen } {
            	addLogEntry "error: original text len $origlen, unwraptext text length $newlen\n'$text'\n'$unwraptext'\n'$newtext'" orange
            }
            return $newtext
        }
    }
    
    
    §   §   §

    Name: unwrapText

    Description:
    unwrap long text by removing newlines
    Parameters:
    Usage:
    
    Comments:
    
    proc unwrapText { text } {
        regsub -all {\\\n} $text "" text
        return [ string trim $text ]
    }
    
    
    §   §   §

    Name: saveResourceToFile

    Description:
    update resource file with user modifications
    Parameters:
    • client
    Usage:
    Comment:
    
    resource file can be LDASapi.rsc or the API's resource file
    proc saveResourceToFile { rscapi fname patterns } {
        if  { [ catch {
            if  { [ llength $patterns ] } {
    			set localname [ file tail $fname ]
    			if	{ [ string match LDASapi.rsc $localname ] } {
    				set localrsc [ file join $::TOPDIR $localname ]
    			} else {
    				set localrsc [ file join $::TOPDIR ${rscapi}API $localname ]
    			}
    			if	{ [ regexp "$::LDAS/(lib|bin)" $fname ] } {
    				file copy $fname $localrsc
    			}
    		    set fd [ open $localrsc r ]
    		    set data [ read -nonewline $fd ]
    		    set data [ split $data \n ]
    		    close $fd
    		    set i 0
                set rsctext ""
                set longlines [ list ]
                set origlines [ list ]
                ;## ensure uniqueness of each pattern
    			
    		    foreach line $data {
                    ;## if line has continue, get all of it first
                    if	{ [ regexp {\\[\t\s\n]*$} $line ] } {
    			        lappend longlines $line
                        continue
    	            } elseif { [ llength $longlines ] } {
                        lappend longlines $line
                        regsub -all {\\} [ join $longlines ] {} line
                        set origlines $longlines
                        set longlines [ list ]
                    }
                    set idx 0
    				set found 0
    			    foreach pat $patterns {
                        ;## make sure array vars are detected
            			regsub -all {\(} $pat "\\(" pat1
            			regsub -all {\)} $pat1 "\\)" pat1
    				    if	{ [ regexp "^${pat1}(\\s|\\t)" $line ] } {
    						if	{ [ regexp {^array set (\S+)} $line -> arrname ] } {
    							set value [ array get $arrname ]
                                set value [ wrapText $value ]
    							set line "$pat \[ list $value \]"
    						} else {
    							set value [ eval $pat ]
                                set len [ llength $value ]
    							if	{ [ llength $value ] > 1 } {
                                    set value [ wrapText $value ]
    					    		set line "$pat \[ list $value \]"
    							} elseif { $len == 1 } {
    								set line "$pat $value"
    							} else {
                                    set line "$pat {}"
                                }
    						}
                            set found 1
                            break
    				    }
                        incr idx 1
    			    }
                    if	{ $found } {
    					set patterns [ lreplace $patterns $idx $idx ]
                        set origlines [ list ]
    				} elseif  { [ llength $origlines ] } {
                        set line [ join $origlines \n ]
                        set origlines [ list ]
                    }
                    append rsctext $line\n	
    		    }
                bak $localrsc $::RESOURCE_BAK_LEVEL
    		    set fd [ open $localrsc w ]
    			set rsctext [ string trim $rsctext \n ]
    		    puts $fd $rsctext
    		    close $fd
            }
        } err ] } {
            catch { close $fd }
            return -code error $err
        }
        return $localrsc
    }
    
    
    §   §   §

    Name: addNewResourcesToFile

    Description:
    add a new resource to API resource file resource file
    Parameters:
    Usage:
    
    Comments:
    
    proc addResourcesToFile { fname descs cmds } {
        set localrsc ""
        if  { [ catch {
            if  { [ llength $cmds ] } {
                set localname [ file tail $fname ]
    	        if	{ [ string match LDASapi.rsc $localname ] } {
    			    set localrsc [ file join $::TOPDIR $localname ]
    			} else {
    			    set localrsc [ file join $::TOPDIR ${::API}API $localname ]
                }
    			if	{ [ regexp "$::LDAS/(lib|bin)" $fname ] } {
    				file copy $fname $localrsc
    			}
                set fd [ open $localrsc r ]
                set rsctext [ read $fd ]
                close $fd
                set index 0
    			foreach desc $descs {
                    regexp {(set \S+|array set \S+)(.+)} [ lindex $cmds $index ] -> pattern value
                    set len  [ llength $value ]
                    set value [ wrapText [ string trim $value ] ]
    	            if	{ [ llength $value ] > 1 } {
                        set value [ wrapText [ string trim $value ] ]
    					set cmd "$pattern \[ list $value \]"
    				} elseif { $len == 1 } {
    					set cmd "$pattern $value"
    				} else {
                        set cmd "$pattern {}"
                    }
                    append rsctext "\n;## desc=$desc\n$cmd\n"
                    incr index 1
                }
                bak $localrsc $::RESOURCE_BAK_LEVEL
    		    set fd [ open $localrsc w ]
                set rsctext [ string trim $rsctext \n ]
    		    puts $fd $rsctext
    		    close $fd
            }
    	} err ] } {
            addLogEntry $err red
            return -code error $err
        }
        return $localrsc
    }
    
    
    §   §   §

    Name: cmonResource::deleteResourcesFromFile

    Description:
    delete resources from API resouce file
    Parameters:
    • parent widget
    • page name
    Usage:
    
    Comments:
    
    proc deleteResourcesFromFile { fname descs patterns } {
        if  { [ catch {
            set index 0
            if  { [ llength $patterns ] } {
                set localname [ file tail $fname ]
    	        if	{ [ string match LDASapi.rsc $localname ] } {
    			    set localrsc [ file join $::TOPDIR $localname ]
    			} else {
    			    set localrsc [ file join $::TOPDIR ${::API}API $localname ]
                }
    			if	{ [ regexp "$::LDAS/(lib|bin)" $fname ] } {
    				file copy $fname $localrsc
    			}
     		    set fd [ open $localrsc r ]
    		    set data [ read -nonewline $fd ]
    		    set data [ split $data \n ]
    		    close $fd
    		    set i 0
    		    set rsctext ""
                set longlines [ list ]
                set origlines [ list ]
                ;## ensure uniqueness of each pattern
    		    foreach line $data {
                    ;## if line has continue, get all of it first
                    if	{ [ regexp {\\[\t\s\n]*$} $line ] } {
    			        lappend longlines $line
                        continue
    	            } elseif { [ llength $longlines ] } {
                        lappend longlines $line
                        regsub -all {\\} [ join $longlines ] {} line
                        set origlines $longlines
                        set longlines [ list ]
                    }
                    ;## remove the desc line also
                    if  { [ regexp {desc=(.+)$} $line -> rscdesc ] } {
                        set idx 0
                        set found 0
                        foreach desc $descs {
    				        if	{ [ string match $desc $rscdesc ] } {
                                set found 1
                                break
                            }
                            incr idx 1
                        }
                        if  { !$found } {
                            append rsctext $line\n
                        } else {
                            set descs [ lreplace $descs $idx $idx ]
                        }
                        continue
    			    }
                    set idx 0
                    set found 0
    			    foreach pat $patterns {
                        ;## make sure array vars are detected
            			regsub -all {\(} $pat "\\(" pat1
            			regsub -all {\)} $pat1 "\\)" pat1
    				    if	{ [ regexp "${pat1}(\\s|\\t)" $line ] } {
                            set found 1
                            break
                        }
                        incr idx 1
    			    }
                    if  { $found } {
                        set patterns [ lreplace $patterns $idx $idx ]
                        set origlines [ list ]
                    } else {
                        if  { [ llength $origlines ] } {
                            set line [ join $origlines \n ]
                            set origlines [ list ]
                        }
                        append rsctext $line\n
                    }
                }
                bak $localrsc $::RESOURCE_BAK_LEVEL
                set fd [ open $localrsc w ]
                puts $fd [ string trim $rsctext \n ]
                close $fd
            }
    	} err ] } {
            addLogEntry $err red
            return -code error $err
        }
        return $localrsc
    }
    
    
    §   §   §

    Name: setResourceLimit

    Description:
    delete resources from API resouce file
    Parameters:
    • parent widget
    • page name
    Usage:
    
    Comments:
    
    proc setResourceLimit {} {
    	if	{ [ catch {
    		set text ""
    		set color purple
    		if	{ [ array exist ::RESOURCE_LIMIT ] } {
    			foreach { name value } [ array get ::RESOURCE_LIMIT ] {
    				if	{ ! [ string equal default $value ] } {
    					if	{ [ catch {
    						setRLimit $name $value
    					} err ] } {
    						addLogEntry "setRLimit on $name to $value failed: $err" orange
                            set color orange
    					}
    				}
    				if	{ [ catch {
    					set curvalue [ getRLimit $name ]
    					append text "$name=$curvalue; "
    				} err ] } {
    					append text "getRLimit on $name error:'$err'; "
    					set color orange
    				}
    			}
    			addLogEntry [ string trim $text "; "] $color
    		}
    	} err ] } {
    		addLogEntry $err orange
    	}		
    }
    
    
    §   §   §

    Name: touch

    Description:
    reserve a file similar to touch shell command but does not need to do exec
    Parameters:
    • file name
    Usage:
    
    Comments:
    
    proc touch { filename } {
    	if	{ [ catch {
        	set fd [ open $filename w ]
            close $fd
        } err ] } {
        	return -code error "failed to touch $filename: $err"
        }
    }
    
    
    §   §   §

    Name: lastline

    Description:
    Replacement for 'exec tail -1' to avoid a fork.

    Parameters:
    Usage:
    
    Comments:
    
    Obviously not robust, but fast and efficient.
    proc lastline { file } {
         set lastline [ list ]
         if { [ file exists $file ] } {
            set size [ file size $file ]
            set fid  [ open $file r ]
            if { $size < 100000 } {
               set data [ read $fid $size ]
            } else {
               seek $fid -100000 end
               set data [ read $fid 100000 ]
            }
            close $fid
            set lastline [ lindex [ split $data "\n" ] end-1 ]
         }
         return $lastline
    }
    
    
    §   §   §

    Name: leaksSummary

    Description:
    list counts of threads,jobs,objects and data buckets Usage:
    
    Comments:
    
    threads, objects, job array, data bucket
    proc leaksSummary { args } {
         set jobid_rx (${::RUNCODE}\\d+)
         set numJobs 0
         set numJobBuckets 0
         foreach var [ info vars ::${::RUNCODE}* ] {
            if { [ regexp $jobid_rx\$ $var -> jobid ] } {
               incr numJobs 1
            } elseif { [ regexp ${jobid_rx}_DATABUCKET $var -> jobid ] } {
               incr numJobBuckets 1
            }
         }
         foreach [ list socks files threads events objects ] [ countChannels ] { break }
         set text 	"threads=[ expr [ llength $threads ]/3 ] "
         append text "jobs=$numJobs "
         append text "objects=[ llength $objects ] "
         append text "data buckets=$numJobBuckets "
         append text "sockets=[ llength $socks ] "
         append text "events=[ llength $events ] "
        	
         return $text
    }
    
    
    §   §   §

    Name: mapTCLVarToCVar

    Description:
    Usage:
    
    Comments:
    
    proc updateCVarWithTCLVar { varname vardefault resetfunc } {
        if { ! [ info exists $varname ] } {
    	;## Reestablish the variable
    	set $varname $vardefault
        }
        if { [ string length [ trace info variable $varname ] ] == 0 } {
    	;## re-register the trace
            if { [ regexp {8.4} $::tcl_version ] } {
                trace add variable $varname { write } $resetfunc
                trace add variable $varname { unset } $resetfunc
            } else {
                trace variable $varname wu $resetfunc
            }
    	$resetfunc
        }
    }
    
    
    §   §   §

    Name: mapTCLVarToCVar

    Description:
    Usage:
    
    Comments:
    
    proc logVariableChange { varname previous } {
        addLogEntry "Changed $varname from $previous to [ set $varname ]" purple
    }
    
    
    §   §   §
    namespace eval GenericAPI::${::API}::local { ## ******************************************************** ## ## Name: resetDebugDeadLockDetectorLevel ## ## Description: ## ## Usage: ## ## ## Comments: proc resetDebugDeadLockDetectorLevel { args } { set previous [setDebugDeadLockDetectorLevel \ $::DEBUG_DEADLOCK_DETECTOR_LEVEL] logVariableChange ::DEBUG_DEADLOCK_DETECTOR_LEVEL $previous } ## ******************************************************** ## ## Name: updateRWLockInterval ## ## Description: ## ## Usage: ## ## ## Comments: proc updateDebugDeadLockDetectorLevel { args } { updateCVarWithTCLVar \ ::DEBUG_DEADLOCK_DETECTOR_LEVEL \ 0 \ GenericAPI::${::API}::local::resetDebugDeadLockDetectorLevel } }
    §   §   §

    Name: getSockData (reads data from a socket, usually
    emergency when a cmd has been issued to that socket
    Description:
    Usage:
    
    Comments:
    
    proc getSockData { sid var } {
    	if	{ [ catch {
    	    set seqpt "cmd::result $sid"
    	    set $var [ cmd::result $sid ]
        } err ] } {
            set $var $err
    	}
        catch { close $sid }
    }
    
    
    §   §   §
    getEmergData Description connects to an LDAS API emergency and issues a cmd for API Parameters host - ldas host port - ldas port number Usage set target [ getEmergData ldas-dev.ligo.caltech.edu 10002 ] Comments:
    uniqid does not have :: in front get data from an API's emergency socket msg format e.g. set msg "\{puts \$cid \[ validService cntlmon operator \]\}" same code in controlmon client works more reliably than msg2mgr
    proc getEmergData { api msg { wait 1 } } {
    	
        if	{ ! [ regexp $api $::API_LIST ] } {
        	return -code error "$api is not on LDAS API list ( $::API_LIST )"
        }
        set uniqid [ key::time ]
        set ::$uniqid {}
        if 	{ [ catch {
            set sid [ sock::open $api emergency ]
        } err ] } {
            catch { close $sid }
            addLogEntry "connect error $api: $err" red
            return -code error "connect error $api: $err"
        }
        if  { [ catch {
            fconfigure $sid -buffering line
            ;## do puts in background so timeout can be set up
            if  { [ string match manager $api ] } {
                set cmd "$::MGRKEY NULL NULL eval $msg"
            } else {
                set cmd "$::MGRKEY $msg"
            }
            puts $sid $cmd
            fileevent $sid readable \
                [ list getSockData $sid ::$uniqid ]
            if  { $wait } {
                vwait ::$uniqid
            }
        } err ] } {
            catch { close $sid }
            return -code error "connect error $api: $err"
        }
        if  { ! $wait } {
            return
        }
        set retval [ set ::$uniqid ]
    	unset ::$uniqid
    	
    	if	{ [ regexp {emergency.+error} $retval ] } {
    		return -code error $retval
    	}
    	set retval [ split $retval \n ]
    	;## remove the emergency reply line
    	debugPuts $retval
    	set result [ list ]
    	foreach line $retval {
    		if	{ [ regexp {emergency.+executed} $line ] } {
                break
            }
            lappend result $line
    	}
    	set result [ join $result \n ]
        return $result
    }
    
    
    §   §   §
    execOverload Description overloads the exec function to track caller of execs Parameters Usage Comments:
    do this only when logging is established
    proc execOverload {} {
        if  { ! [ string length [ info command tcl_exec ] ] } {
            rename ::exec tcl_exec
            addLogEntry "overloaded exec" purple
            proc exec { args } {
                if  { [ info exist ::LOG_EXEC ] && $::LOG_EXEC } {
            	    addLogEntry "exec [ join $args ]" purple
                }
    			set rc [ catch { eval tcl_exec $args } err ]
         		if 	{ ! $rc } {
            	   	return $err
         		} else {
            		return -code error $err
         		}
            }
        }
    }
    
    
    §   §   §

    up arrow Back to Top up arrow