newspaint

Documenting Problems That Were Difficult To Find The Answer To

Category Archives: Programming

Serialising Arrays and Hashes in PowerShell 2.0

So you want to Invoke-Command a scriptblock on another Windows computer but are struggling to communicate results back to the caller because of a lack of serialisation routines in PowerShell 2.0? Yes, PowerShell 3.0 did introduce the ConvertFrom-Json and ConvertTo-Json cmdlets. But if you’re stuck on PowerShell 2.0 then you need another way to send hashes and lists.

Non-Recursive

Why not convert your hash-and-array data structure into a string – one that can be parsed by Invoke-Expression? This is a function that will do exactly that – and it is non-recursive for reasons that will be explained further down (and there’s a simpler recursive function provided later, too):

Function Serialise-Object {
    Param( $Root )

    Function AddAfter-ListNode {
        Param( $LinkedList, $AfterNode, $NewNode )
        if ( $AfterNode -eq $null ) {
            $LinkedList.AddLast( $NewNode )
        } else {
            $LinkedList.AddAfter( $AfterNode, $NewNode )
        }
    }

    Function Escape-SingleQuoted {
        Param( $Source )
        $Source -replace "'", "''"
    }

    # create lists
    $TodoStack = New-Object "System.Collections.Generic.Stack[Object]"
    $StringsList = New-Object "System.Collections.Generic.LinkedList[String]"

    # set up first element
    $TodoStack.Push( @( $Root, $StringsList.Last ) )

    while ( $true ) {
        try {
            $NextTodo = $TodoStack.Pop()
        } catch {
            break
        }

        ( $Item, $Node ) = @( $NextTodo )
        if ( $Item -eq $null ) {
            $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" "`$null"
            AddAfter-ListNode $StringsList $Node $NewStringNode
        } elseif ( $Item.getType().FullName -eq "System.Collections.Hashtable" ) {
            $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" "@{"
            AddAfter-ListNode $StringsList $Node $NewStringNode
            $LastStringNode = $NewStringNode

            $First = $true
            $Item.Keys |ForEach-Object {
                $keyname = ""
                if ( $First ) {
                    $First = $false
                } else {
                    $keyname += ";"
                }
                $keyname += $( "'" + (Escape-SingleQuoted $_) + "'=" )

                $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" $keyname
                AddAfter-ListNode $StringsList $LastStringNode $NewStringNode
                $LastStringNode = $NewStringNode

                $TodoStack.Push( @( $Item[$_], $LastStringNode ) )
            }

            $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" "}"
            AddAfter-ListNode $StringsList $LastStringNode $NewStringNode
        } elseif ( $Item.getType().FullName -eq "System.Object[]" ) {
            $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" "@("
            AddAfter-ListNode $StringsList $Node $NewStringNode
            $LastStringNode = $NewStringNode

            $First = $true
            $Item |ForEach-Object {
                if ( $First ) {
                    $First = $false
                } else {
                    $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" ","
                    AddAfter-ListNode $StringsList $LastStringNode $NewStringNode
                    $LastStringNode = $NewStringNode
                }

                $TodoStack.Push( @( $_, $LastStringNode ) )
            }

            $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" ")"
            AddAfter-ListNode $StringsList $LastStringNode $NewStringNode
        } else {
            if ( $Item.GetType().FullName -eq "System.String" ) {
                $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" $( "'" + (Escape-SingleQuoted $Item) + "'" )
                AddAfter-ListNode $StringsList $Node $NewStringNode
            } else {
                $NewStringNode = New-Object "System.Collections.Generic.LinkedListNode[String]" $( "[" + $Item.GetType().FullName + "]'" + (Escape-SingleQuoted $Item.ToString()) + "'" )
                AddAfter-ListNode $StringsList $Node $NewStringNode
            }
        }
    }

    @(
        $StringsList.GetEnumerator() |ForEach-Object { $_ }
    ) -join ""
}

Some examples of output:

> Serialise-Object $null
$null

> Serialise-Object 14.25
[System.Double]'14.25'

> Serialise-Object "Four o'clock"
'Four o''clock'

> Serialise-Object @( "First", "Second", @( "Inner1", "Inner2" ) )
@('First','Second',@('Inner1','Inner2'))

> Serialise-Object @{ "ArrayA" = @( 1, 2.5 ); "ArrayB" = @( 'e', 'f', 'g' ) }
@{'ArrayA'=@([System.Int32]'1',[System.Double]'2.5');'ArrayB'=@('e','f','g')}

> Serialise-Object @{ "OuterH" = @{ "InnerH" = @{ "key1" = [long]0xff } } }
@{'OuterH'=@{'InnerH'=@{'key1'=[System.Int64]'255'}}}

The resulting string can be fed directly into Invoke-Expression and the result is going to be very similar if not identical to the serialised object.

So, how does it work? It iterates over the object it is given. If it is a simple scalar type ($null, string, or non-list/hash) then it is converted to a string, prepended with its type if not a string, and output. If the object is an array or hash then each element or element-pair is iterated through and that object is recursively processed.

This function would have looked a lot simpler as a recursive function. So why was it implemented using lists and stacks instead of recursion? Because I wanted to send this function as a string through an Invoke-Command cmdlet and have it rebuilt as a scriptblock on the remote side; but one problem – how does one call an anonymous scriptblock recursively? Perhaps there’s a way but I don’t know how.

For example:

$remote_scriptblock = {
    Param( [String]$FnSerialiseStr )

    $FnSerialise = [scriptblock]::create( $FnSerialiseStr )

    $Start = Get-Date
    Start-Sleep -Milliseconds 1500

    & $FnSerialise @{ "time"=((Get-Date) - $Start).TotalSeconds }
}

Invoke-Command $remote_scriptblock -ArgumentList @(${function:Serialise-Object})

This outputs:

@{'time'=[System.Double]'1.5'}

Pretty neat, huh? You can send this function to the other side and run it!

Recursive

Ah.. but what if you want to send several named functions to the other side?

$remote_scriptblock = {
    Param( [String]$PreBlockStr )

    $PreBlock = [scriptblock]::create( $PreBlockStr )
    & $PreBlock

    $Start = Get-Date
    Start-Sleep -Milliseconds 2500

    Serialise-Object @{ "time"=((Get-Date) - $Start).TotalSeconds }
}
Invoke-Command $remote_scriptblock -ArgumentList @("Function Serialise-Object { ${function:Serialise-Object} }")

This outputs:

@{'time'=[System.Double]'2.5'}

Well that solves the problem of a recursive function trying to call itself.

Let’s rewrite the serialisation function as the simpler recursive form:

Function Serialise-Object {
    Param( $Root )

    Function Escape-SingleQuoted {
        Param( $Source )
        $Source -replace "'", "''"
    }

    if ( $Root -eq $null ) {
        "`$null"
    } elseif ( $Root.getType().FullName -eq "System.Collections.Hashtable" ) {
        $out = "@{"

        $First = $true
        $Root.Keys |ForEach-Object {
            if ( $First ) {
                $First = $false
            } else {
                $out += ";"
            }

            $out += $( "'" + (Escape-SingleQuoted $_) + "'=" )
            $out += Serialise-Object $Root[$_]
        }
        $out + "}"
    } elseif ( $Root.getType().FullName -eq "System.Object[]" ) {
        $out = "@("

        $First = $true
        $Root |ForEach-Object {
            if ( $First ) {
                $First = $false
            } else {
                $out += ","
            }

            $out += Serialise-Object $_
        }
        $out + ")"
    } else {
        if ( $Root.GetType().FullName -eq "System.String" ) {
            $( "'" + (Escape-SingleQuoted $Root) + "'" )
        } else {
            $( "[" + $Root.GetType().FullName + "]'" + (Escape-SingleQuoted $Root.ToString()) + "'" )
        }
    }
}

With this simpler code we can use it remotely as follows:

$remote_scriptblock = {
    Param( [String]$PreBlockStr )

    $PreBlock = [scriptblock]::create( $PreBlockStr )
    & $PreBlock

    $Start = Get-Date
    Start-Sleep -Milliseconds 3500

    Serialise-Object @{ "time"=((Get-Date) - $Start).TotalSeconds }
}
Invoke-Command $remote_scriptblock -ArgumentList @("Function Serialise-Object { ${function:Serialise-Object} }")

This outputs:

@{'time'=[System.Double]'3.5'}

… using simpler code.

Allowing Powershell Scripts to Run on Windows

So you want to run a PowerShell script on your Windows system but get the following message:

File myscript.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please
see "get-help about_signing" for more details.
At line:1 char:23
+ .\myscript.ps1 <<<<
    + CategoryInfo          : NotSpecified: (:) [], PSSecurityException
    + FullyQualifiedErrorId : RuntimeException

A fix is to change the execution policy. To see the current state of execution policies on your system:

PS C:\> Get-ExecutionPolicy -List |Format-Table -AutoSize

        Scope ExecutionPolicy
        ----- ---------------
MachinePolicy       Undefined
   UserPolicy       Undefined
      Process       Undefined
  CurrentUser       Undefined
 LocalMachine       Undefined

If you do not have administrator rights on your system the easiest way to allow scripts to be executed is to allow the current user to run scripts.

PS C:\> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
you to the security risks described in the about_Execution_Policies help topic. Do you want to change the execution
policy?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y

Note that “RemoteSigned” means that all scripts and configuration files downloaded from the Internet must be signed by a trusted publisher, but locally authored scripts can be run without a signature.

Using Perl to Make Signed Requests to Public Transport Victoria Timetable API

So you want to create an application to access the Public Transport Victoria (PTV) Timetable API.

You’ve followed the instructions and e-mailed a request for a developer ID and an access key; and you’ve received both in an e-mail that contained text similar to the following:

Thank you for your interest in the PTV Timetable API.

Your email address has now been registered and your user Id and API key are
below.

User Id: 2912345

API Key: 4cc12345-ff11-2222-a00a-dd1297cd04aa

Now you want to create a signed request using Perl to access the API.

The following function will take a URL and return a signed URL that can be used to access that URL:

#!/usr/bin/perl -w

use Digest::HMAC;
use Digest::SHA;

use strict;

my $devid = "2912345";
my $apikey = "4cc12345-ff11-2222-a00a-dd1297cd04aa";

sub sign_url {
  my ( $url ) = @_;

  # add ?devid=$devid or &devid=$devid to URL
  $url .= ( index($url, "?") < $[ ) ? "?" : "&";
  $url .= "devid=$devid";

  # strip out base URL from part used in signing
  my $request = $url;
  $request =~ s{^(([a-z]+)://+)?[^/]+}{};

  # calculate signature using API key and URL without base
  #   e.g. sign over a string like "/v3/routes?devid=2912345"
  my $signature = Digest::HMAC::hmac_hex(
      $request,
      $apikey,
      \&Digest::SHA::sha1
  );

  $url .= "&signature=" . $signature;
  return $url;
}

print sign_url( "http://timetableapi.ptv.vic.gov.au/v3/routes" );

This should output the following:

$ perl -w signtest.pl
http://timetableapi.ptv.vic.gov.au/v3/routes?devid=2912345&signature=b7ee928f05499a0016746daef5013dba35224d8e

The example is using an invalid signature and devid, so this example URL will not actually return a page; you will have to provide the access key and devid you received in your e-mail.

Note: it is absolutely essential that you do not change the capitalisation of the access key in this script. While it looks like a hex string it is actually treated as case-sensitive text.

Rust/Cargo Outputting Dollar Angle Bracket Symbols

So you’re compiling source using Cargo/Rust and you see output similar to the following:

error[E0046]$<2>: not all trait items implemented, missing: `decode`, `encode`$<2>
  $<2>--> $<2>src/main.rs:7:1$<2>
   $<2>|$<2>
7$<2>  $<2>| $<2>impl tokio_core::io::Codec for LineCodec {$<2>
   $<2>| $<2>^$<2> missing `decode`, `encode` in implementation$<2>

error$: aborting due to previous error$

This is full of dollar, angle bracket, number, angle bracket characters.

The fix is to change your terminal, e.g.:

TERM=xterm

On doing this I had the output in full colour without the mysterious symbols.

Implementing Custom Sort For BTree In Rust

So you want to use a std::collections::btree_map::BTreeMap but you want your keys sorted in a different way to the default for the type.

In this example I show how you can provide a BTreeMap that has String keys but sorted in reverse order. But, by altering the cmp() function for the std::cmp::Ord trait for a custom type, you can make the sort in any order you desire.

It is a bit of a pain, however. As this post points out you have to define partial_cmp() and partial_eq() functions from other traits as well, as well as adding the Eq trait to your struct.

// http://rustbyexample.com/custom_types/structs.html
struct RString {
    s: String,
}

use std::cmp::Ord;
use std::cmp::Ordering;

impl PartialEq for RString {
    fn eq(&self, other:&Self) -> bool {
        if self.s == other.s {
            true
        } else {
            false
        }
    }
}

// this does not actually have any methods, it's just a flag on the type
impl Eq for RString { }

// make partial_cmp() just return result from cmp()
impl PartialOrd for RString {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        let me: &String = &(self.s);
        let them: &String = &(other.s);

        Some( me.cmp( them ) )
    }
}

impl Ord for RString {
    fn cmp(&self, other:&Self) -> Ordering {
        let me: &String = &(self.s);
        let them: &String = &(other.s);

        if me > them {
            Ordering::Less
        } else if me < them {
            Ordering::Greater
        } else {
            Ordering::Equal
        }
    }
}

use std::env;
use std::collections::btree_map::BTreeMap;
fn main() {
    // collect environment variable keys into a vector we can sort
    let mut sortedmap : BTreeMap<Box<RString>,Box<String>> = BTreeMap::new();

    for (key, value) in env::vars() {
        sortedmap.insert(
            Box::<RString>::new( RString { s: key } ),
            Box::<String>::new( value )
        );
    }

    for (key, value) in sortedmap {
        println!( "{} => {}", (*key).s, *value );
    }
}

This could also be implemented using a tuple containing a single element:

// http://rustbyexample.com/custom_types/structs.html
struct RString(String);

use std::cmp::Ord;
use std::cmp::Ordering;

impl PartialEq for RString {
    fn eq(&self, other:&Self) -> bool {
        if self.0 == other.0 {
            true
        } else {
            false
        }
    }
}

// this does not actually have any methods, it's just a flag on the type
impl Eq for RString { }

// make partial_cmp() just return result from cmp()
impl PartialOrd for RString {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        let me: &String = &(self.0);
        let them: &String = &(other.0);

        Some( me.cmp( them ) )
    }
}

impl Ord for RString {
    fn cmp(&self, other:&Self) -> Ordering {
        let me: &String = &(self.0);
        let them: &String = &(other.0);

        if me > them {
            Ordering::Less
        } else if me < them {
            Ordering::Greater
        } else {
            Ordering::Equal
        }
    }
}

use std::env;
use std::collections::btree_map::BTreeMap;
fn main() {
    // collect environment variable keys into a vector we can sort
    let mut sortedmap : BTreeMap<Box<RString>,Box<String>> = BTreeMap::new();

    for (key, value) in env::vars() {
        sortedmap.insert(
            Box::<RString>::new( RString(key) ),
            Box::<String>::new( value )
        );
    }

    for (key, value) in sortedmap {
        println!( "{} => {}", (*key).0, *value );
    }
}

Note that a tuple’s single element can be accessed through the .0 field.

Creating a Basic CGI Executable Using Rust

I created a new cargo project called “cgitest”:

~$ cargo new cgitest --bin
~$ cd cgitest
~/cgitest$

Then I added the following Rust code into src/main.rs:

// Use the following as a guide:
//   http://httpd.apache.org/docs/current/howto/cgi.html

use std::io::Write;
use std::env;
use std::collections::btree_map::BTreeMap;

fn write_stderr( msg : String ) {
    let mut stderr = std::io::stderr();
    write!(&mut stderr, "{}", msg).unwrap();
}

fn write_stderr_s( msg : &str ) {
    write_stderr( msg.to_string() );
}

fn write_stdout( msg : String ) {
    let mut stdout = std::io::stdout();
    write!(&mut stdout, "{}", msg).unwrap();
}

fn write_stdout_s( msg : &str ) {
    write_stdout( msg.to_string() );
}

fn html_escape( msg : String ) -> String {
    let mut copy : String = String::with_capacity( msg.len() );

    for thechar in msg.chars() {
        if thechar == '&' {
            copy.push_str( "&amp;" );
        } else if thechar == '<' {
            copy.push_str( "&lt;" );
        } else if thechar == '>' {
            copy.push_str( "&gt;" );
        } else if thechar == '\"' {
            copy.push_str( "&quot;" );
        } else {
            copy.push( thechar );
        }
    }

    return copy;
}

fn main() {
    write_stdout_s( "Content-type: text/html\n" );
    write_stdout_s( "\n" );
    write_stdout_s( "<html>\n" );
    write_stdout_s( "  <head>\n" );
    write_stdout_s( "    <title>Rust CGI Test</title>\n" );
    write_stdout_s( "    <style type=\"text/css\">\n" );
    write_stdout_s( "      td { border:1px solid black; }\n" );
    write_stdout_s( "      td { font-family:monospace; }\n" );
    write_stdout_s( "      table { border-collapse:collapse; }\n" );
    write_stdout_s( "    </style>\n" );
    write_stdout_s( "  </head>\n" );
    write_stdout_s( "  <body>\n" );
    write_stdout_s( "    <h1>Environment</h1>\n" );
    write_stdout_s( "    <table>\n" );
    write_stdout_s( "      <tr><th>Key</th><th>Value</th></tr>\n" );

    // copy environment into a BTreeMap which is sorted
    let mut sortedmap : BTreeMap<String,String> = BTreeMap::new();
    for (key, value) in env::vars() {
        sortedmap.insert( key, value );
    }

    // output environment into HTML table
    for (key, value) in sortedmap {
        write_stdout(
            format!(
                "      <tr><td>{}</td><td>{}</td></tr>\n",
                html_escape( key ),
                html_escape( value )
            )
        );
    }
    write_stdout_s( "    </table>\n" );
    write_stdout_s( "  </body>\n" );
    write_stdout_s( "</html>\n" );
}

Compiling it using the cargo build --release command produced a binary in target/release/cgitest which can be launched from Apache as a CGI script.

Where Is The Std Crate Located In A Rust Installation

I downloaded Rust (the programming language) from the Rust downloads page. Specifically I downloaded the 64-bit Linux .tar.gz file and then extracted it to /opt/ on my filesystem and symlinked /opt/rust to the untarred directory.

However when I went to compile a test Rust program I got the following error:

$ /opt/rust/rustc/bin/rustc test.rs
error: can't find crate for `std` [E0463]
error: aborting due to previous error

I searched for the std crate by running:

$ find /opt/rust/ -name 'libstd*'
/opt/rust/rustc/lib/libstd-d16b8f0e.so
/opt/rust/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d16b8f0e.so
/opt/rust/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d16b8f0e.rlib
/opt/rust/rust-docs/share/doc/rust/html/src/std/up/src/libstd

I tried then compiling my Rust program using the first path, “/opt/rust/rustc/lib/“:

$ /opt/rust/rustc/bin/rustc -L /opt/rust/rustc/lib test.rs
error:error: can't find crate for `core` which `std` depends on [E0463]
error:error: aborting due to previous error

So, where is libcore?

$ find /opt/rust/ -name 'libcore*'
/opt/rust/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-d16b8f0e.rlib
/opt/rust/rust-docs/share/doc/rust/html/src/core/up/src/libcore

I then tried compiling using the path to libcore.

$ /opt/rust/rustc/bin/rustc -L /opt/rust/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib test.rs
$

Success!

If you want to pass command line options for rustc to cargo you can do it as follows:

$ RUSTFLAGS="-L /opt/rust/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" cargo
$

Getting Statistics from HG612 Revision 3B VDSL Modem Using Node

The HG612 Revision 3B VDSL modem commonly provided by BT OpenReach in the United Kingdom for fibre-to-the-cabinet (FTTC) connected households can be unlocked with alternative firmware.

For those modems that are unlocked they can be connected to on the modem’s default IP address of 192.168.1.1 with a username of admin, password of admin, using web (if enabled), telnet, and ssh.

Such a session entails getting into the shell, then running a Linux command to show the DSL statistics, like the following:

me@home:~$ ssh admin@192.168.1.1
admin@192.168.1.1's password: admin
PTY allocation request failed on channel 0

ATP>sh
sh


BusyBox v1.9.1 (2010-10-15 17:59:06 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# xdslcmd info --stats
xdslcmd: ADSL driver and PHY status
Status: Showtime
Retrain Reason: 0
Max:    Upstream rate = 30992 Kbps, Downstream rate = 90516 Kbps
Path:   0, Upstream rate = 20000 Kbps, Downstream rate = 79999 Kbps
...
# exit
exit

ATP>exit
exit
exit from configuration console.
[6408] Jan 03 23:45:49 exit after auth (admin): child failed
Connection to 192.168.1.1 closed.

I wanted a way of dumping this information on a regular basis automatically but it wasn’t as simple as merely executing a command from the Linux command line, e.g. ssh admin@192.168.1.1 “sh\nxdslcmd info –stats\nexit\nexit”.

So I wrote the following Node.JS script to open a ssh connection to the modem, and execute the appropriate instructions, and dump the output from the xdslcmd command.

'use strict';

/*

  You will need to run the following in the directory you place this script:
    npm install ssh2
    npm install bluebird

*/

var Promise = require("bluebird");
var ssh2 = require("ssh2");

var modem_ip = "192.168.1.1";
var modem_user = "admin";
var modem_pass = "admin";

var debugging = false;

function debug( msg ) {
  if ( debugging ) {
    console.log( msg );
  }
}

function connect() {
  return new Promise( function( resolve, reject ) {
    var conn = new ssh2();

    conn.setMaxListeners(0);

    conn.on(
      'connect',
      function () {
        debug( "ssh connect" );
      }
    );

    conn.on(
      'ready',
      function () {
        debug( "ssh ready" );

        resolve( conn );
        return;
      }
    );

    conn.on(
      'error',
      function (err) {
        debug( "error: " + err );

        reject();
        return;
      }
    );

    conn.on(
      'end',
      function () {
        debug( "end" );

        //reject();
      }
    );

    conn.on(
      'close',
      function (had_error) {
        debug( "close: " + had_error );

        reject();
        return;
      }
    );

    conn.connect(
      {
        "host": modem_ip,
        "port": 22,
        "username": modem_user,
        "password": modem_pass,
        "debug": function ( msg ) { debug( msg ); },
        "algorithms": {
          "kex": [
            'ecdh-sha2-nistp256',
            'ecdh-sha2-nistp384',
            'ecdh-sha2-nistp521',
            'diffie-hellman-group-exchange-sha256',
            'diffie-hellman-group14-sha1',
            'diffie-hellman-group1-sha1', /* required for modem */
          ],
          "cipher": [
            'aes128-ctr',
            'aes192-ctr',
            'aes256-ctr',
            'aes128-gcm',
            'aes128-gcm@openssh.com',
            'aes256-gcm',
            'aes256-gcm@openssh.com',
            '3des-cbc', /* required for modem */
          ],
        },
      }
    );

    debug( "Starting..." );
  } );
}

function interactive( conn ) {
  return new Promise( function( resolve, reject ) {
    debug( "Starting shell..." );
    conn.exec( '',
      function( err, stream ) {
        if ( err ) {
          reject( err );
          return;
        }

        var data = "";

        stream.on(
          'close',
          function( code, signal ) {
            debug( "close stream " + code + " " + signal );
            conn.end();
            resolve( data );
            return;
          }
        );

        /* use a state machine */
        var stage = "ATPprompt";

        stream.on(
          'data',
          function ( chunk ) {
            debug( "Chunk: " + chunk );
            data += chunk;

            /* determine state we are in and if expected prompt is seen */
            if ( ( stage == "ATPprompt" ) && /^ATP>/m.test( data ) ) {
              debug( "writing sh" );
              data = ""; /* clear buffer */
              stream.write( "sh\n" );
              stage = "shprompt";
              return;
            }

            if ( ( stage == "shprompt" ) && /^#/m.test( data ) ) {
              debug( "writing xdslcmd info --stats" );
              data = ""; /* clear buffer */
              stream.write( "xdslcmd info --stats\n" );
              stage = "shprompt2";
              return;
            }

            if ( ( stage == "shprompt2" ) && /^#/m.test( data ) ) {
              debug( "writing exit" );
              console.log( data );
              data = ""; /* clear buffer */
              stream.write( "exit\n" );
              stage = "ATPprompt2";
              return;
            }

            if ( ( stage == "ATPprompt2" ) && /^ATP>/m.test( data ) ) {
              debug( "writing exit" );
              data = ""; /* clear buffer */
              stream.write( "exit\n" );
              stage = "none";

              /* we now expect connection to close, so resolve when we do */
              conn.on(
                'close',
                function (had_error) {
                  resolve();
                  return;
                }
              );

              return;
            }
          }
        );
      }
    );
  } );
}

/* set out abort timer in case something goes wrong */
setTimeout(
  function () {
    console.error( "ERROR: Timeout" );
    process.exit( 1 );
  },
  5000 /* 5.000 seconds */
);

/* main routine as a chain of promises */
connect().then(
  function ( conn ) {
    return interactive( conn );
  }
).then(
  function () {
    debug( "Done." );
    process.exit();
  }
).catch(
  function ( err ) {
    debug( "Failed to connect." );
    process.exit();
  }
);

Which can be run simply as node xdslcmd.js.

JMeter – User Parameters or User Defined Variables?

You almost always want User Parameters because these are recomputed for every thread and, optionally, every iteration. Whereas User Defined Variables are processed at the start and copied to all threads.

Thread Pool Pattern for Node.JS using Q.js Promises

I wanted to emulate the Thread Pool Pattern in Node.JS using Q.js promises.

Of course Node.JS does not have multiple threads for the user to use. Which is why Q.js is used in the first place – to make concurrency programming easier using the event/callback model that Node.JS implements.

Given an array of functions which return promises I wanted a pattern function that provided a customisable n number of workers. These workers would run asynchronously processing one function from the list at a time, and when each returned promise resolved it would then fetch the next function from the list – until no functions were left to process. When the last worker finished processing it would then resolve the promise returned by the pattern function to the caller. Usually there would be many tasks and limited number of workers.

Why would I want to do this? Let’s say I have a hundred or thousand URLs I want to fetch. If I put them all into a list and called Q.allSettled then Node.JS would attempt to fetch them all at the same time – this could result in excessive delays, a peak in network traffic, or might be plain impolite on the target webserver. Instead it may be better to download a maximum of 4 web pages at any one time.

For simplicity’s sake here is an example. Let’s create a function that returns a promise:

// delayms() - returns a promise that resolves approximately n milliseconds into the future
//   - ms - milliseconds to wait before resolving promise
function delayms( ms ) {
  var deferred = Q.defer();

  setTimeout( function () { deferred.resolve( "Finished after " + ms + "ms" ); }, ms);
  return deferred.promise;
}

Now let’s create a list of functions that return a promise (we could call these functions “tasks”). It’s important that the array contains functions, not the result of a function that returns a promise (i.e. a promise).

For example:

// create array of functions to process
var tasklist = [ ];
for ( var item = 0; item < 12; item++ ) {
  tasklist.push(
    function () {
      console.log( "Going to wait 1 second..." );
      return delayms( 1000 ); // return promise as last statement
    }
  );
}

// let's throw in an error, too
tasklist.push(
  function () {
    throw new Error( "Deliberate error!" );
  }
);

Now we have an array of tasks to process. How about we process them all – but a maximum of 3 running at any one time? We would want to do something like:

var Q = require('./q.js'); // load Q.js library

Q.fcall(
  // call a function, this function resolves immediately
  function () {
    console.log( "Starting" );
  }
).then(
  // do the list of tasks, max concurrency of 3
  function () {
    // return the promise
    return workermodel( 3, tasks );
  }
).then(
  // output the list of results
  function ( results ) {
    console.log( "End: " + JSON.stringify( results, null, 2 ) );
  }
).fail(
  function ( reason ) {
    console.error( "Error: " + reason );
  }
).done();

Well all we need now is the workermodel() function to run through our list of tasks:

// workermodel() - returns a promise that resolves after task list processes with max concurrency
//   - workers - maximum concurrency or number of workers
//   - tasklist - array of functions that return a promise
var debug = true;
function workermodel( workers, tasklist ) {
  var taskidx = 0;
  var workersfree = 0;

  // if no tasks to perform return a completed promise with empty array
  if ( tasklist.length == 0 ) {
    return Q.fcall( function () { return []; } );
  }

  var deferred = Q.defer();

  // if less tasks than workers, limit workers to task size
  if ( tasklist.length < workers ) {
    workers = tasklist.length;
  }

  // results will go into this array
  var resultsarray = new Array( workers );

  var getNextTaskGenerator = function () { }; // place holder

  var startNextTask = function ( innerworkeridx, innertaskidx ) {
    if ( debug )
      console.error( new Date() + " task[" + innertaskidx + "/" + tasklist.length + "] assigned to worker[" + innerworkeridx + "]" );
    var nextTask = Q.fcall(
      function () {
        return tasklist[innertaskidx]();
      }
    );
    nextTask.then(
      function ( value ) {
        if ( debug )
          console.error( new Date() + " task[" + innertaskidx + "] resolved on worker[" + innerworkeridx + "]" );
        resultsarray[innertaskidx] = { state: "fulfilled", value: value };
        return getNextTaskGenerator( innerworkeridx )();
      },
      function ( reason ) {
        if ( debug )
          console.log( new Date() + " task[" + innertaskidx + "] rejected on worker[" + innerworkeridx + "]" );
        resultsarray[innertaskidx] = { state: "rejected", reason: reason };
        return getNextTaskGenerator( innerworkeridx )();
      }
    );

    return nextTask;
  };

  getNextTaskGenerator = function ( workeridx ) {
    return function () {
      if ( debug )
        console.error( new Date() + " getnext task[" + taskidx + "] for worker[" + workeridx + "]" );
      if ( taskidx < tasklist.length ) {
        var nextTask = startNextTask( workeridx, taskidx );
        taskidx++;
        return nextTask;
      } else {
        workersfree++;
        if ( workersfree == workers ) {
          if ( debug )
            console.error( new Date() + " workermodel RESOLVE" );
          deferred.resolve( resultsarray );
        } else {
          if ( debug )
            console.error( new Date() + " no more work but " + ( workers - workersfree ) + " workers busy" );
        }
      }
    };
  };

  // start workers
  for ( var workeridx = 0; workeridx < workers; workeridx++ ) {
    startNextTask( workeridx, taskidx );
    taskidx++;
  }

  if ( debug )
    console.error( new Date() + " RETURNING PROMISE" );
  return deferred.promise;
}

Like Q.allSettled() this function always resolves and when it does the value is a list of associative arrays, each row (related to the function in the tasklist provided) contains a state field of “fulfilled” or “rejected” and if “fulfilled” then the “value” field will contain the result. Otherwise the “reason” field will contain the error.

What is the output of this script (with debugging turned on)?

Starting
Sat Sep 12 2015 19:48:29 GMT+0100 (BST) task[0/13] assigned to worker[0]
Sat Sep 12 2015 19:48:29 GMT+0100 (BST) task[1/13] assigned to worker[1]
Sat Sep 12 2015 19:48:29 GMT+0100 (BST) task[2/13] assigned to worker[2]
Sat Sep 12 2015 19:48:29 GMT+0100 (BST) RETURNING PROMISE
Going to wait 1 second...
Going to wait 1 second...
Going to wait 1 second...
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[0] resolved on worker[0]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) getnext task[3] for worker[0]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[3/13] assigned to worker[0]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[1] resolved on worker[1]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) getnext task[4] for worker[1]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[4/13] assigned to worker[1]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[2] resolved on worker[2]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) getnext task[5] for worker[2]
Sat Sep 12 2015 19:48:30 GMT+0100 (BST) task[5/13] assigned to worker[2]
Going to wait 1 second...
Going to wait 1 second...
Going to wait 1 second...
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[3] resolved on worker[0]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) getnext task[6] for worker[0]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[6/13] assigned to worker[0]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[4] resolved on worker[1]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) getnext task[7] for worker[1]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[7/13] assigned to worker[1]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[5] resolved on worker[2]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) getnext task[8] for worker[2]
Sat Sep 12 2015 19:48:31 GMT+0100 (BST) task[8/13] assigned to worker[2]
Going to wait 1 second...
Going to wait 1 second...
Going to wait 1 second...
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[6] resolved on worker[0]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) getnext task[9] for worker[0]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[9/13] assigned to worker[0]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[7] resolved on worker[1]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) getnext task[10] for worker[1]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[10/13] assigned to worker[1]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[8] resolved on worker[2]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) getnext task[11] for worker[2]
Sat Sep 12 2015 19:48:32 GMT+0100 (BST) task[11/13] assigned to worker[2]
Going to wait 1 second...
Going to wait 1 second...
Going to wait 1 second...
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) task[9] resolved on worker[0]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) getnext task[12] for worker[0]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) task[12/13] assigned to worker[0]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) task[10] resolved on worker[1]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) getnext task[13] for worker[1]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) no more work but 2 workers busy
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) task[11] resolved on worker[2]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) getnext task[13] for worker[2]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) no more work but 1 workers busy
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) task[12] rejected on worker[0]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) getnext task[13] for worker[0]
Sat Sep 12 2015 19:48:33 GMT+0100 (BST) workermodel RESOLVE
End: [
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "fulfilled",
    "value": "Finished after 1000ms"
  },
  {
    "state": "rejected",
    "reason": {}
  }
]

All of the above is my own work. It is free for others to use.

See Also

  • bluebird Javascript promises library has a concurrency option which would appear to be similar in functionality