newspaint

Documenting Problems That Were Difficult To Find The Answer To

GenieACS – accessing DeviceID and Tags from NBI API

GenieACS offers a Northbound Interface (NBI) which provides a REST API.

The GET /devices/?query=<query>&projection=<list> URL permits select parameters to be returned (where <list> is a comma-separated list of parameters).

This works well enough for parameters like InternetGatewayDevice.DeviceInfo.Description. However some parameters don’t seem to be accessible like DeviceID.* and Tags.*.

They can be accessed as follows:

Parameter NBI Reference Note
DeviceID.Manufacturer _deviceId._Manufacturer all the _deviceId values can be returned with just a projection of _deviceId which returns an object of key/value pairs
DeviceID.OUI _deviceId._OUI
DeviceID.ProductClass _deviceId._ProductClass
DeviceID.SerialNumber _deviceId._SerialNumber
Tags.* _tags is an array from the NBI
_lastInform

Values like InternetGatewayDevice.DeviceInfo.Description are returned as an object with the following keys:

Key Purpose
_value the actual parameter value (if not an object)
_type parameter value type, e.g. xsd:string, xsd:boolean (if not an object)
_writable whether parameter value can be set through NBI
_timestamp last time GenieACS was updated with the value
_object true or false, whether this is an object (with parameters branched from this one)

Growatt Server Time Change for Daylight Savings

Incredibly, the Growatt server does not keep track of timezone changes and mis-attributes energy data when daylight savings begins or ends.

To that end one must:

  • login to server.growatt.com
  • choose the Energy button (there are 4 buttons: Dashboard, Energy, Log, Setting)
  • choose the Plant Management tab
  • in the far-right cell of the table for the configured plant (usually “PlantA”) choose the Edit button (with the pencil)
  • select the timezone appropriate for whether daylight savings is in effect or not

Compilers Treating Unused Variables as Errors

It seems to be a recent phenomenon where some new programming languages have decided to treat the presence of an unused variable in code as an error that causes compilation to fail.

This never used to be an issue. The presence of a declared variable that had never been read/used could produce a warning but it certainly wasn’t a compiler error.

Arguments in favour of this behaviour are along the lines of “it helped me identify bugs in my code”. Such arguments ignore that exactly the same could be achieved by paying attention to warnings and changing the code until those warnings were no longer produced.

Arguments against this behaviour, however, hold much greater weight. It is a common debugging pattern to comment out a block or line of code to observe what difference this makes at runtime. Having to identify a variable declaration elsewhere, however, in order to comment it out or to change it to a placeholder requires significantly more effort and also necessitates remembering those changes which will need reversing when the code block is uncommented/restored.

Ultimately what is the goal of turning all warnings into errors? The sole reason must be to avoid broken code being published in that language. It is reputational damage control.

These language designers are fearful of the reputation of languages like Perl or C – which allow the developer the freedom to write as clean or messy code as they wish. If, however, they could make the compiler so strict that no broken code could ever be published then the language’s reputation may remain strong.

It is a propaganda tool. Nothing more.

And like any propaganda tool it is half truths mixed with half lies. It does not result in “better” code. Only code written by developers who do not have a thorough understanding of the difference between development and production. In other words mediocre programmers. Experienced developers will abandon nannying languages and use those that offer flexibility and pragmatism.

GenieACS Provision Script Helper Functions

The following are some helper functions that can be added to a GenieACS provision script to make things easier.

// getFirstValue() - takes iterator and returns the first value found
//
// Usage (to create new branch and return the path to it):
//   var softwareVersion = getFirstPath(
//     declare( "InternetGatewayDevice.DeviceInfo.SoftwareVersion", {value:0} )
//   );
//
//   if ( ! softwareVersion ) {
//     softwareVersion = getFirstPath(
//       declare( "Device.DeviceInfo.SoftwareVersion", {value:0} )
//     );
//   }

function getFirstValue( iterator ) {
  for (let element of iterator) {
    if ( element.value && ( element.value[0] != undefined ) ) {
      return element.value[0];
    }
  }

  return undefined;
}
// getFirstPath() - takes iterator and returns the first path found
//
// Usage (to create new branch and return the path to it):
//   const newPathWANPPPConnection = getFirstPath(
//     declare(
//       `${pathWANConnectionDevice}.WANPPPConnection.[Name:MyNewPPP]`,
//       {path: now},
//       {path: 1}
//     )
//   );

function getFirstPath( iterator ) {
  for (let element of iterator) {
    if ( element.path ) {
      return element.path;
    }
  }

  return undefined;
}

Keyboard Ignored After Waking From Suspend In Xubuntu/XFCE

Every time after waking from suspend on my Lenovo ThinkPad laptop I would enter my password to unlock the session – but then the keyboard would not work (no input would go into the active window and even session shortcuts like alt-arrow would not work)… until I used the mouse to click on another window for it to gain focus. After that the keyboard would work fine.

I found the solution on this thread. It was as simple as running:

sudo apt-get remove light-locker

This was on Ubuntu 20.04 running XFCE (Xubuntu).

BASH Script to Back Up GenieACS

The following script will back up the following collections from the north bound interface (REST API):

  • tasks
  • devices
  • presets
  • objects
  • provisions
  • virtualParameters
  • faults

And will back up the following collections from the web user interface after logging in as admin:

  • operations
  • permissions
  • users
  • config
  • cache
#!/bin/bash

###############################################################################
# IMPORTANT - put your web UI admin username into environment variable
#   GENIEACS_PASSWORD_ADMIN before calling this script!
###############################################################################

NBI_COLLECTIONS="\
  tasks devices presets objects provisions virtualParameters faults"

UI_COLLECTIONS="\
  operations permissions users config cache"

BASE_URL_NBI="http://localhost:7557"
BASE_URL_UI="http://localhost:3000"
DATESTR=$(date +%Y-%m-%d-%H%M%S)

###############################################################################
# fetch from UI (web interface)
###############################################################################
# login to the UI and get token
TOKEN=$(curl -f -X POST -H "Content-Type: application/json" -d "{\"username\": \"admin\", \"password\": \"${GENIEACS_PASSWORD_ADMIN}\"}" ${BASE_URL_UI}/login)
if [ $? -ne 0 ]; then
    echo "Failed to login"
    exit
fi

# remove leading and trailing double-quotes from the token
TOKEN="${TOKEN#\"}"
TOKEN="${TOKEN%\"}"

echo "Got token ${TOKEN}"

# fetch from UI
for i in ${UI_COLLECTIONS}; do \
    DEST="${DATESTR}-backup-genieacs-${i}.json.gz"
    echo "==${i}== => ${DEST}"
    /usr/bin/curl -b "genieacs-ui-jwt=${TOKEN}" "${BASE_URL_UI}/api/${i}/" |/usr/bin/nice /usr/bin/gzip -9 -c >"${DEST}"
done

###############################################################################
# fetch from NBI (North Bound Interface/API)
###############################################################################
for i in ${NBI_COLLECTIONS}; do \
    DEST="${DATESTR}-backup-genieacs-${i}.json.gz"
    echo "==${i}== => ${DEST}"
    /usr/bin/curl "${BASE_URL_NBI}/${i}/" |/usr/bin/nice /usr/bin/gzip -9 -c >"${DEST}"
done

LUA Socket.connect Function Never Returns in HAProxy

I wanted a front-end to pass a copy of an HTTP request to a custom webservice that would inspect the request and decide whether to authorise the request to be passed to a backend.

So I started out with a configuration for front-end and back-end:

global
    lua-load /etc/haproxy/lua/mycode.lua

    -- need bigger buffer to fit entire request in txn.req:dup()
    tune.bufsize 131072

frontend https_frontend
    # external interface for incoming requests (offloading SSL)
    bind 1.2.3.4:7547 ssl crt /etc/haproxy/ssl/haproxy.pem
    
    mode http

    # buffer the request before passing onto backend
    option http-buffer-request

    # give time for request headers and body to arrive
    timeout http-request 30s

    # call LUA script to forward request copy to webservice
    http-request lua.action_auth_check

    # if the LUA script did not set var to 200 then deny
    http-request deny unless { var(txn.auth_check_status) -m str "200" }

    default_backend backend_http

backend backend_http
    mode http
    option forwardfor # set header saying who the request was from
    http-reuse never
    server backend_server 127.0.0.1:8888 maxconn 32

Then I created my code:

local function action_auth_check(txn)
   local prefix = "action_auth_check: "

   local socket = core.tcp()
   socket:settimeout(3)

   txn:log(core.debug, prefix .. "connecting")

   local result, errorMessage = socket:connect("127.0.0.1", 9999)
   if not result then
      txn:log(core.debug, prefix .. "failed to connect: " .. errorMessage);
      return
   end

   -- Send the entire request to the TCP socket
   txn:log(core.debug, prefix .. "sending request")
   socket:send(txn.req:dup())

   -- Wait for the response from the TCP socket
   txn:log(core.debug, "waiting for response")
   local response, errorMessage2 = socket:receive("*a")

   -- Close the TCP connection
   txn:log(core.debug, "closing connection")
   socket:close()

   if not response then
      txn:log(core.debug, prefix .. "failed to receive: " .. errorMessage2)
      return
   end

   txn:log(core.debug, prefix .. "received: " .. response);

   -- Check the response status
   local status = string.match(response, "HTTP/%d%.%d (%d+)")
   if status and string.len(status) == 3 then
      txn:log(core.debug, prefix .. "setting auth_check_status = " .. status);
      txn:set_var("txn.auth_check_status", status)
   end

   txn:log(core.debug, prefix .. "finished");
end

Well, the above all works. So what went wrong?

Initially I started out using txn.req:get() to get the contents of the buffer when I should have been using txn.req:dup(). Looking at some documentation it is apparent that the get() attempts to read and remove data from the buffer.

I had a log message that called txn.req:get() and so subsequent calls to socket:connect() failed to ever return, even though I could see (with tcpdump) a valid 3-way handshake, followed by timeout seconds, and haproxy sending a FIN.

For some reason calling txn.req:get() caused other functions to block. Being a channel there might be a correct way of emptying or indicating that reading from the channel has finished. But the easier way is just to use txn.req:dup() which doesn’t appear to block subsequent calls.

GenieACS Provision Scripts and Making Sense of Declare

GenieACS provisioning scripts have a function named declare() but what that actually returns is extremely poorly documented.

This is my attempt to understand what this actually returns.

Fetching Values

Fetching values is important because it allows a script to determine what to do based on how a device is currently configured.

There are two scenarios. Fetching a non-wildcard path, and fetching a path containing a wildcard.

Non-Wildcard (or Scalar)

let obj = declare("DeviceID.Manufacturer", {value:now});

This returns a variable of type “object”. This consists of the keys:

{
  value: objValue,               // object (containing the value and value type)
  path: "DeviceID.Manufacturer", // path (string)
  size: 1
}

Where objValue is:

{
  0: "TP-Link",    // string (the value)
  1: "xsd:string", // string (the type)
  length: 2        //number
}

So this is pretty straightforward to fetch the value:

log( obj.value[0] );

Wildcard

How can I list all the leaf paths from a branch? Treat the result as an “iterator“.

let obj = declare("InternetGatewayDevice.WANDevice.*", {value:now});

for (let iter of obj) {
  log( iter.path );
}

// OUTPUTS:
//   InternetGatewayDevice.WANDevice.1
//   InternetGatewayDevice.WANDevice.2
//   InternetGatewayDevice.WANDevice.3
//   InternetGatewayDevice.WANDevice.X_TP_DSLNumberOfEntries

We can combine this with a regular expression to only display the enumerated leaves:

let obj = declare("InternetGatewayDevice.WANDevice.*", {value:now});

for (let iter of obj) {
  if ( ! /[.][0-9]+$/.test( iter.path ) )
    continue;

  log( iter.path );
}

// OUTPUTS:
//   InternetGatewayDevice.WANDevice.1
//   InternetGatewayDevice.WANDevice.2
//   InternetGatewayDevice.WANDevice.3

Deleting All Records

All the records under a branch can be deleted:

// path: 0 means zero instances under this branch
declare(`${wantedWANConnectionDevice}.WANPPPConnection.[]`, {path: now}, {path: 0});

Note that if the value of path is changed to a positive non-zero number then that specifies how many instances of that branch should exist/remain after that command (e.g. if path is 1 and there are no instances then create a new instance).

Creating New Record

(this section is untested)

A new record can be created using filter syntax which will cause the filtered parameters to be created as part of the record.

e.g.

declare(`${wantedWANConnectionDevice}.WANPPPConnection.[Username:${username},Password:${password},Enable:true]`, {path: now}, {path: 1});

Cannot See DHCP Requests in accel-ppp Logs

I was trying to get ipoe working with accel-ppp. My interfaces were being brought up – but no DHCP packets were being logged even though debugging level was set to the maximum (5).

The solution? The firewall, iptables, did not let incoming DHCP packets through. So even though I could see the DHCP requests using tcpdump they were not getting through the firewall to the accel-ppp daemon.

Add something like the following to your /etc/iptables/rules.v4 file:

-A INPUT -d 255.255.255.255/32 -p udp --dport 67 -m comment --comment "DHCP requests" -j ACCEPT

You should then see in your accel-ppp logfile something like:

e0.1234.5678: : recv [DHCPv4 Discover xid=ea123456 chaddr=00:11:22:33:44:55 <Message-Type Discover>  <Request-List Subnet,Router,DNS,Host-Name,Domain-Name,Broadcast,NTP,Classless-Route> <Host-Name myrouter>  <Relay-Agent {Agent-Circuit-ID AVC123456789012}>]

PJSIP and Layers of NAT with Asterisk v16.28.0

For various reasons my Asterisk server is behind two sets of NAT, the first being a router connected to the public Internet, and the other being a container on a server on my local network.

This results in Asterisk struggling with determining where a phone call is coming from – and thus what context to use in the dialplan.

I was getting an error message like:

NOTICE[123456]: res_pjsip/pjsip_distributor.c:676 log_failed_request: Request 'INVITE' from '"0123456789" ' failed for '192.168.9.1:12345' (callid: 00325ab3-222f-322e1522-333bf1114e09) - No matching endpoint found

The short story is that PJSIP could not identify the endpoint this incoming call was destined for.

The problem was that the incoming INVITE SIP message only had the IP address of the originating server in the Contact header – which Asterisk seems to ignore. All the other IP addresses (the source, the From header, etc) had various VPN (local) IP addresses.

I experimented with the identify_by option in the endpoint section of pjsip.conf and was not getting anywhere until I came across this forum post.

The solution for me was to add authentication in both directions of my Asterisk servers. And not just that but for the username of that authentication to be the name of the endpoint on the other end.

Importantly I had to specify auth_username in the identify_by option:

identify_by=auth_username,username,ip

Let’s say I had the originating server configured as:

[homeuser]
type=endpoint
transport=tlstransport ; use TLS transport
context=from_homeuser ; context for dialplan
disallow=all
allow=alaw,ulaw,g722 ; helps to be specific
auth=homeuser ; incoming auth required
aors=homeuser
media_encryption=sdes
direct_media=no
outbound_auth=orig_server ; outgoing call auth

[homeuser]
type=auth
auth_type=userpass
username=homeuser ; required when registering with us
password=abcd1234 ; required when registering with us

[orig_server]
type=auth
auth_type=userpass
username=orig_server ; name of endpoint on other server
password=abcd1234

Then on my home server behind several layers of NAT I would have:

[orig_server]
type=registration
outbound_auth=orig_server
server_uri=sip:mysip.origserver.com:5061
client_uri=sip:homeuser@mysip.origserver.com:5061
transport=tlstransport
auth_rejection_permanent=no
max_retries=100

[orig_server]
type=endpoint
transport=tlstransport ; use TLS transport
context=from_orig_server ; context for dialplan
disallow=all
allow=alaw,ulaw,g722 ; helps to be specific
outbound_auth=orig_server ; outgoing call auth
aors=orig_server
media_encryption=sdes
direct_media=no
auth=homeuser ; incoming auth required
identify_by=auth_username,username,ip ; allow identification by auth username

[orig_server]
type=auth
auth_type=userpass
username=homeuser ; required when registering with us
password=abcd1234 ; required when registering with us

[homeuser]
type=auth
auth_type=userpass
username=orig_server ; name of endpoint on other server
password=abcd1234

So now when there was an incoming call my home server would initially reply with unauthorised, and the originating server would then authenticate, and the username used in that authentication would be used as the endpoint for that call.