Browsed by
Category: Recent Projects

Historical ONT Light Levels

Historical ONT Light Levels

This feature has been requested for years by our team and I have never had the time to really dive in and add the functionality. Since the FMS has been transitioning to a microservice backend, it was further put off and continued to be put off as the FMS core functionality was migrated. Upon discussion in a Calix Community thread, it brought the idea back to the front burner.

In addition, we’ve had a higher number of devices reporting low light levels and they seems to be more related to drops in terminals or at the CSP, rather than the ONT splice point. Cleaning the fiber fixes the issue, but having a way to monitor or pull the levels daily for analysis is becoming increasingly important in our office.

So, as I set to figure this out, I knew I already had a pretty extensive system built that would be very simple to implement. Already having Customer, Inventory and Calix API services tied together, it would just be a simple UI edit for a graph, an API to pull information based on date, and then running a scheduled daily task to pull all of the ONTs.

There were a few things to consider. The FMS is built in a relational database, meaning that it’s a bit slower and archiving information is not always the most performance or resource optimized. For the sake of timeline, company size and resource demands, I made the choice that switching to a different DB or key/value store would be a future project.

DB Model

Our existing database already had inventory records and customer records that were joined as a 1:Customer to n:Device relationship. Introducing a new table to hold light level information would have to reference the inventory records. Further, we are only pulling devices that are assigned to active customers, and provisioned via CMS or SMx. Without making the join too busy for this post, I have condensed it to Customer joined to Inventory ONTs where the customer is active and the ONT is provisioned via CMS.

Now that we can store and query the data, we have to write the services to connect to the device. For background, the UI makes http requests to the API for information. The API Gateway will then direct the request to the microservice. Depending on the request, it may make additional http calls internally to process the request. For example, to provision an ONT on a customer profile: the UI would send an http request from the client to the API Gateway, the gateway will direct it to the customer module, where it will begin processing the request. It will make another call to the inventory module to look up the inventory record and make sure the FSAN exists in inventory, if it does, the inventory record is updated to assign the unit to the customer record and the inventory record is sent back to the customer module, and joined to the provisioning parameters. Then the customer module will make a call to the CMS/SMX/Cloud API and process a new ONT. The Calix API service will query for existing ONTs and resolve service tags, ONT models and bandwidth profiles, then process a new ONT for the appropriate host. If it is an RG service and not a data service, a Calix Cloud profile is created as well and then a response is sent back to the customer service, which updates the customer record and returns the result.

Now, this is a pretty complicated example, but it is suppose to show the idea of workflow that involves inter-service dependencies as well as public/secure API requests as well. In our example, there are a couple API calls made. One is requesting historical information, the other is actually getting the data from the ONT and storing it in the DB.

Lets start with ONT querying. We have a task scheduler that will run at 5am every morning, to query all ONTs that are assigned to an active customer and provisioning using CMS or SMX. It will be housed in the inventory service.

Backend Services

1. Task Scheduler runs at 5am, every morning. It will send an internal http request to internal/telecom/inventory/v1/equipment/pull-ont-light-levels. The image below shows the task row, and the cron pattern that is different considering it is in our development environment.

2. The inventory service receives the request and generates a process job, which it returns with an OK, job received. The job starts to spin up workers and processes all inventory records that are assigned to an active customer.

3. Inventory Service does its best to perform nodejs ‘multithreading’, and run multiple requests at once. It will query the CMS API for each device to find an ONT and pull the light levels, respond back and write the results to the DB. Finally, the job will update the task scheduling service every 30 seconds with updates.

4. The CMS API requires the serno, and must perform a lookup to get the ONT id, on a system. Once an ONT ID and hostname have been identified, we can query the ONT for data.

5. Finally, data is returned to the task worker and stored into the database!

The UI

Now, pulling the light level data, requires us to just make a db call that will return a list of information. The UI can graph the information, or you could use it to compute averages and generate alerts or reports of people will low light levels. CMS will give you live alerts, but being able to compute change over time has the ability to provide more meaningful stats.

My service hasn’t been running for 30 days, so I generated some fake data to show a potential trend.

I figured that I’ll make a post later on to help in determining reports and thresholds later.

Alpha 221c LED Sign

Alpha 221c LED Sign

Covid-19 is here, and I’ve been keeping to myself a lot more after work. For reference, our office is currently an old school that we have been squatting in for a couple years, and one of the old buildings had an LED sign that dated back to the late 80’s. This thing came with a power supply and nothing else. No documents and no keyboard to program it, nada. Also, no documentation was found for this specific model online.

After digging through any and all archives, taking whatever model numbers or post information I could obtain from the screen, I was able to locate a couple of things. These signs have the ability to be networked together, using almost any medium and also a lot of others with similar home projects. I found a couple of documents that were essential to figuring out how this dumb thing operated.

Turns out the networking document from Adaptive Displays gives you enough info to get started. All examples and documentation I could find point me to using a serial connection or RJ11 cable to interface with. My stupid sign had a DIN connector. So, I popped it off. I figured if it was a serial connection, it need a +/- and a ground, right? I guess so. The sign was fed from an AC power supply, and offered a 13 volt, AC connection to the sign itself. The DIN connector had three wires that were not AC power related, which I assumed to be the needed for a serial connection. I knew from previous documents that networking was possible, so I had a hunch that the DIN would have to be the connector. It was easy to identify the ground, it was the smallest gauge wire and was soldered to a separate location on the board. The networking document stated that the serial connection would require a RS-485 adapter, so I went digging for a USB to serial adapter. I actually had an RS-485 serial to USB adapter available, and got to work. Pin 5 was ground and Pin 1 and 2 were +/- 5v. I soldered a cat5e cable into the back of the DIN port and wired it to the USB adapter.


Using a document from MIT, I was able to find the serial information for the connection. My sign used the EZKEY II or EZKEY protocol, and required a 1200, 2400 or 4800 baud rate. I used the AlphaNET software trial to see if I could get the sign to interface with the serial connection. It could.

I had found the appropriate connection settings. 2400 Baud, 7 data bits, two stop bits and even parity. I could send text to the sign.

Now it was time to use the dumb thing. I found out through a lot of trial and error that my sign is too old to use strings, and must only write and read TEXT files. I began working on my own interface to handle sign text in nodejs. Once I was able to write string buffers of hex ASCII codes, I could use strings and convert them into packets. But whats the point of that? Why not make an express HTTP server to handle requests and use parameters to create messages that do things?


Great! Now we just need a front end to interface with the user and hang it in our office


Calix AXOS Console Cable

Calix AXOS Console Cable

Recently we ran into an issue where we were required to get into a E3-16F with a console cable. This was due to a downgrade from R3.1.4 to R2.3.3.2. Such a downgrade causes the updated configuration file to not be read correctly and throws a running suspect alarm in the unit. The unit boots with no problem, but it was totally inaccessible from any interface. The solution was to use a console cable. Calix does not provide console cables, nor do they provide a pin out of the correct one to use.

The requirements they state are:

  • 115200 Baud
  • 8 Data bits
  • 1 Stop bit
  • No flow control

Further an RS-232 DB9(Female) to RJ11m or RJ12 connector is possible, but no pin out diagram exists. Luckily the DB9 only needs three wires. TX/RX and a ground. And a RJ11m only has four usable pins, so it just becomes a process of elimination. I have attached a diagram that shows how to wire it up, so if you need to purchase one or make one yourself, you can know how or what to get. Since there is no standard for this kind of connector set up, it seems crazy for Calix to not put the pin-out on their site.


Calix Success Story

Calix Success Story

On the 8th, Calix flew out three people to Sandy to interview employees and customers about our fiber network. I finished my organizational behavior presentation by 10am and arrived in Sandy at 12. I was interviewed for 30 minutes about how I have utilized the northbound interfaces on different Calix software’s. No more than two weeks later, Calix released the video online, so I have embedded it here. Granted, I only appear for 30 seconds and ramble on about our Fiber Management System, but I am hoping that with other recent events I will be able to move forward a bit more, link my name OSS fiber management solutions.

Calix 844-G Gigacenter CLI Command List

Calix 844-G Gigacenter CLI Command List

I have continued to update this page every time find more information regarding a command. A lot of these commands are explained here.

? – Displays help list
help – Displays help list
logout – closes connection
exit – closes connection
quit – closes connection
reboot – soft reboot on ONT
brctl – bridge related commands
cat – cat file
df – prints filesystem usage statistics
loglevel – set or view current logging level for appnames
dumpcfg – dumps the contents of the config flash to screen
dumpmdm – dumps entire contents of the MDM
dumpeid – request smd to dump its Entity Info database
meminfo – dumps memory stats
kill – kills a process
dumpsysinfo – dumps all system information to screen
exitOnIdle – allows user to terminate the session after being idle for n seconds
syslog – syslog related commands
echo – seriously, you should know this
ifconfig – shows the interface configurations
ping – unleashes little robots that contact other things
ps – lists currently running processes
pwd – prints the working directory
sntp – secure network time protocol
sysinfo – provides brief ONT system information
tftp – trivial file transfer protocol commands
voice – used for voice related service
arp – address resolution protocol commands
defaultgateway – default gateway related commands
dhcpserver – dynamic host configuration protocol related commands
dns – domain name resolution related commands
lan – commands relating to the LAN interface(s)
lanhosts – lists current LAN devices
passwd – changes the current password
ppp – access to point to point protocol commands
restoredefault – performs factory reset on ONT
route – commands related to L3 routing
save – save the configuration
swversion – prints the current software version
uptime – displays the ONT uptime
wan – commands related to the WAN interface
calixdebug – dumps calix related information
calixfile – commands related to the calix files


Commands Expanded


addbr <bridge> add bridge
delbr <bridge> delete bridge
addif <bridge> <device> add interface to bridge
delif <bridge> <device> delete interface from bridge
setageing <bridge> <time> set ageing time
setbridgeprio <bridge> <prio> set bridge priority
setfd <bridge> <time> set bridge forward delay
sethello <bridge> <time> set hello time
setmaxage <bridge> <time> set max message age
setpathcost <bridge> <port> <cost> set path cost
setportprio <bridge> <port> <prio> set port priority
enableportsnooping <bridge> <value> 0-disable 1-standard 2-blocking
enableproxymode <bridge> <value> To enable 1 or disable 0
show show a list of bridges
showmacs <bridge> show a list of mac addrs
addmacs <bridge> <ifname> <mac> add mac addresses to the bridge table
delmacs <bridge> <ifname> <mac> remove mac addresses from the bridge table
deldynmacs <bridge> <ifname> remove all dynamic mac address from the bridge table
showstp <bridge> show bridge stp info
stp <bridge> {on|off} turn stp on/off
mldenableportsnooping <bridge> <value> 0-disable 1-standard 2-blocking
mldenableproxymode <bridge> <value> To enable 1 or disable 0
enableigmplan2lan <bridge> <value> 0-disable 1-enable
enablemldlan2lan <bridge> <value> 0-disable 1-enable
flows <bridge> <rxif> <txif> To setup layer 2 flows to the path (rxif->txif)
enableigmpratelimit <bridge> <value> 0-disable, 1..500-packet rate
eponuniunictrl <bridge> <value> 0-disable 1-enable
showmaclmt <bridge> show a mac limit of bridge


 -P POSIX output format
 -k 1024-byte blocks (default)
 -a Show all filesystems
 -i Inodes
 -B SIZE Blocksize


Usage: loglevel get appname
       loglevel set appname loglevel
where appname is one of: httpd, tr69c, smd, ssk, telnetd, sshd, consoled, upnp, dnsproxy, , mcpd, vodsl, dectd, wlmngr, linmosd
loglevel is one of "Error", "Notice", or "Debug" (use these exact strings).


Usage: logdest get appname
       logdest set appname logdest
where appname is one of: httpd, tr69c, smd, ssk, telnetd, sshd, consoled, upnp, dnsproxy, linmosd
loglevel is "Standard Error", "Syslog" or "Telnet".


Usage: mdm setpv <full path to parameter value> <param value>
       mdm getpv <full path to parameter value>
       mdm addobj <full path to object>
       mdm delobj <full path to object instance>
       mdm setnonpersistent <full path to object instance>
more subcommands will be added later.


Usage: meminfo [app name] [operation]
 App name can be httpd, tr69c, or ssk. If app name is omitted, then the operation is done for CLI app.
 operation is one of stats, traceAll, trace50, traceClones. If operation is omitted, then stats.
 meminfo : dumps the memory stats as seen by the CLI app. Same as meminfo self stats.
 meminfo ssk : send a message to ssk to tell it to dump its memory stats. Same as meminfo ssk stats.
 meminfo httpd trace50 : send a message to httpd to tell it to dump last 50 leak tracing records
 meminfo traceClones : tell this CLI app to dump leak trace records with 5 or more clones.

psp – I don’t know what this command does

Usage: psp list
       psp dump xxx
       psp delete xxx
       psp clearall
       psp help


Usage: tftp [OPTIONS] HOST [PORT]
Transfer a file from/to tftp server
 -l FILE Local FILE
 -r FILE Remote FILE
 -g Get file
 -p Put file
 -g -t i -f filename server_ip Get (flash) broadcom or whole image to modem
 -g -t c -f filename server_ip Get (flash) config file to modem
 -p -t f -f filename server_ip Put (backup) config file to tftpd server


voice --help - show the voice command syntax
voice show - show the voice parameters
voice show stats - show call statistics
voice show cctkmemstats - shows memory allocation statistics
voice show cctkcmstats - shows Call Manager & CCTK statistics
voice start - start the voice application
voice sendUpldComplete - send the upload complete message to ssk
voice stop - stop the voice application
voice save - store voice params to flash
voice reboot - restart the voice application
voice set <param> <arg1> <arg2>.. - set a provisionable parameter
List of voice set params and args:
defaults <None> - Default VoIP setup
boundIfname <LAN|Any_WAN|(WAN IfName, e.g. nas_0_0_35)> - vodsl network interface
ipAddrFamily <IPv4|IPv6> - IP address family
pstnDialPlan <pstn line#> <dialPlan> - PSTN dial plan
pstnRouteRule <pstn line#> <Auto|Voip|Line> - PSTN Route rule
pstnRouteData <pstn line#> <line #|URL for VOIP> - PSTN Route data
locale <srvPrv#> <region> - 2 or 3 character code
DTMFMethod <srvPrv#> <InBand|RFC2833|SIPInfo> - DTMF digit passing method
hookFlashMethod <srvPrv#> <SIPInfo|None> - Hook flash method
transport <srvPrv#> <UDP|TCP|TLS> - transport protocol
srtpOption <srvPrv#> <Mandatory|Optional|Disabled> - SRTP usage option
regRetryInt <srvPrv#> <seconds> - SIP register retry interval
regExpires <srvPrv#> <seconds> - Register expires hdr val
rtpDSCPMark <srvPrv#> <mark> - RTP outgoing DSCP mark
logServer <srvPrv#> <hostName|IP> - Log server
logPort <srvPrv#> <port> - Log server port
digitMap <srvPrv#> <digitmap> - dial digit map
T38 <srvPrv#> on|off - enable/disable T38
V18 <srvPrv#> on|off - enable/disable V.18 detection
reg <srvPrv#> <hostName|IP> - SIP registrar server
regPort <srvPrv#> <port> - SIP registrar server port
proxy <srvPrv#> <hostName|IP> - SIP proxy server
proxyPort <srvPrv#> <port> - SIP proxy server port
obProx <srvPrv#> <hostName|IP> - SIP outbound proxy
obProxPort <srvPrv#> <port> - SIP outbound proxy port
sipDomain <srvPrv#> <CPE_domainName> - SIP user agent domain
sipPort <srvPrv#> <port> - SIP user agent port
sipDSCPMark <srvPrv#> <mark> - SIP outgoing DSCP mark
musicServer <srvPrv#> <hostName|IP> - SIP music server
musicSrvPort <srvPrv#> <port> - SIP music server port
confURI <srvPrv#> <hostName> - SIP conferencing URI
confOption <srvPrv#> <Local|Refer participants|Refer server> - SIP conferencing option
tagMatching <srvPrv#> <on|off> - SIP to tag matching
timerB <srvPrv#> <time in ms> - SIP protocol B timer
timerF <srvPrv#> <time in ms> - SIP protocol F timer
lineStatus <srvPrv#> <accnt#> <on|off> - Activate line
physEndpt <srvPrv#> <accnt#> <id> - Phys Endpt
extension <srvPrv#> <accnt#> <URI> - SIP extension
dispName <srvPrv#> <accnt#> <Name> - SIP Display Name
authName <srvPrv#> <accnt#> <name> - SIP auth name
authPwd <srvPrv#> <accnt#> <pwd> - SIP auth password
MWIEnable <srvPrv#> <accnt#> <on|off> - Msg Waiting Indication
cfwdNum <srvPrv#> <accnt#> <number> - call forward number
cfwdAll <srvPrv#> <accnt#> <on|off> - call forward all
cfwdNoAns <srvPrv#> <accnt#> <on|off> - call forward no answer
cfwdBusy <srvPrv#> <accnt#> <on|off> - call forward busy
callWait <srvPrv#> <accnt#> <on|off> - call waiting
anonBlck <srvPrv#> <accnt#> <on|off> - Anonymous call rcv blcking
anonCall <srvPrv#> <accnt#> <on|off> - Anonymous outgng calls
DND <srvPrv#> <accnt#> <on|off> - do not disturb
CCBS <srvPrv#> <accnt#> <on|off> - Call completion on busy
speedDial <srvPrv#> <accnt#> <on|off> - Speed dial
warmLine <srvPrv#> <accnt#> <on|off> - Warm line
warmLineNum <srvPrv#> <accnt#> <number> - Warm line number
callBarring <srvPrv#> <accnt#> <on|off> - Call barring
callBarrPin <srvPrv#> <accnt#> <number> - Call barring pin
callBarrDigMap <srvPrv#> <accnt#> <digitmap> - Call barring digit map
netPrivacy <srvPrv#> <accnt#> <on|off> - Network privacy
vmwi <srvPrv#> <accnt#> <on|off> - Visual message waiting indication
vad <srvPrv#> <accnt#> <on|off> - enable vad
pTime <srvPrv#> <accnt#> <pTime> - packetization period
codecList <srvPrv#> <accnt#> <codec(1)[,codec(2)]> - codec priority list
rxGain <srvPrv#> <accnt#> <rxGain> - rxGain (dB)
txGain <srvPrv#> <accnt#> <txGain> - txGain (dB)
cctktracelvl <Info|Warn|Debug|Off> - CCTK tracelevel ( stop/start reqd)
cctktracegrp <CCTK|SCE|Trans|SDP|SIP|Misc|All|None> - CCTK concat tracegroups( stop/start reqd)
mgtProt <TR69|OMCI> - Protocol used to manage Voice
loglevel <general|cmgr|dispatch|sipcctk> <Error|Notice|Debug> - Vodsl module-specific log level


dect get <ac|mode|prom|linesetting|systemsetting|contactlist> [value] ...
dect set <ac|mode|prom|default> [value] ...
dect add <contact|call|handset> [value] ...
dect del <contact|call> [value] ...
dect hs <reg|ping|del|info> [value] ...
dect test <synctime|vmwi> [value] ...
dect start
dect stop
dect save
dect help [cmd]
dect info


There is a lot of commands for wlctl. Use this for reference.


laser param --load [<filename>]
laser param --dump default | current
laser power
laser power --rxinit <initial rx reading> <rx offset>
laser power --txinit
laser power --rxread
laser power --txread
laser txbias --read
laser temperature --read
laser voltage --read


wan add interface <atm|ptm|eth>
wan add service <interfacename> --protocol <bridge|ipoe|pppoe|ipoa|pppoa>
wan delete interface atm <port.vpi.vci>
wan delete interface ptm <port> --priority <normal|high|both>
wan delete interface eth <ethx>
wan delete service L3IfName
wan show interface
wan show [<port.vpi.vci>]
wan --help <bridge|pppoe|pppoa|ipoe|ipoa>


objinfo -Display mcpd object tree information.
meminfo -Display mcpd object memory usage information.
mcgrpmode -Set Multicast Group rules for categorizing ASM vs SSM. (iana|firstin)
configinfo -Display mcpd configuration.
reload -Reload mcpd configuration.
allinfo -Display all information


Usage: sys <gphytest|usbtest|usbchk|btt|ledctl|atsh|atwz|atqz|atsn|atgs|atgp|atmg|atsw|gpio|atbd|atrg|athp|atmi|atei|atri|ups> [sys command option]
            sys show
            sys help
  •  gphytest – More info here
    • tests LAN status I guess?
  • usbtest
    • tests the USB port status
  • usbchk
    • Shows the status of the USB port
  • btt
    • Controls a button status, but I am not sure if it is reset or WPS
  • ledctl – This is a fun one
    • AG – Turns on all green LED’s
    • AF – Turns on all red LED’s
    • B1 – Blinks all green LED’s at 1Hz
    • B2 – Blinks all green LED’s at 2Hz
    • B4 – Blinks all green LED’s at 4Hz
    • B8 – Blinks all green LED’s at 8Hz
    • BF – Blinks all green LED’s at 16Hz
    • ALL – Turns on all controlled LED’s
    • AO – Turns off all controlled LED’s
    • AA – Turns on all amber LED’s
    • BR1- Blinks all red LED’s at 1Hz
    • BR2 Blinks all red LED’s at 2Hz
    • BR4 – Blinks all red LED’s at 4Hz
    • BR8 – Blinks all red LED’s at 8Hz
    • BR16 – Blinks all red LED’s at 16Hz
    • BA1- Blinks all amber LED’s at 1Hz
    • BA2 Blinks all amber LED’s at 2Hz
    • BA4 – Blinks all amber LED’s at 4Hz
    • BA8 – Blinks all amber LED’s at 8Hz
    • BA16 – Blinks all amber LED’s at 16Hz
    • WpsInpro – Blinks the WPS LED
    • WpsOverLap – Pulsates the WPS LED
    • WpsError – Blinks the WPS LED red
    • WpsInPro_IPTV Blinkes the WPS LED amber
  • atsh
    • Displays information about the ONT unit
  • atwz
    • Shows the ONT MAC
  • atqz
    • Shows all the ONT MAC addresses
  • atsn
    • Shows the ONT serial number
  • atgs
    • Show the ONT GPON serial number
  • atgp
    • Shows the ONT GPON password
  • atmg
    • Show the ONT manufacturer information
  • atsw
    • Dunno, it breaks the session
  • gpio
    • Dunno, not a very good help prompt
  • atbd
    • Turns on or off the battery signal detection
  • atrg
    • Show the ONT region code
  • athp
    • Show the ONT part nubmer
  • atmi
    • Show the ONT module number
  • atei
    • Show the ONT CLEI code
  • atri
    • Show the ONT remote inventory version
  • ups
    • Show the ONT battery status


This command allows the user to perform basic file maintenance operations on the Calix generated persistent debug files stored on the 800SG series ONTs
Usage: calixfile <target>:[action]
       target [ help | all | port | global | panic | diag |logs ]
       action [ trim | delete | ls | dump ]
       example: calixfile port:ls logs:ls
        help or ? outputs this help screen
        all, port, global, panic, diag, logs are types of persistent data stored in the FLASH memory on the unit. At least one of these targets is required. Entering the command: calixfile without a target outputs this help screen
               trim - clears the current contents of the file
               delete - removes the file from the FLASH, some files are not removable and will be resized to 0 bytes instead.
               ls - performs the linux ls -l command for a given type of file
               dump - outputs the contents of the specified target type files

Commands are executed from right to left on the command line



Hoodland Library Transition

Hoodland Library Transition

Last month, SandyNet had to get Internet connectivity from the old Hoodland library, across the street, to the new one. Part of the switch required a new line to be drug from our splice case on Welches road to the new library. Northsky got an OFS 24 pair (I think?) to the site, but left it coiled up outside. So, the team and I performed the first ever SandyNet aerial splice.

In addition, Chris and I did a tower climb a week or so later and drug 16 Ethernet cables up our Waybill tower. From there, we performed a few rescue training exercises, and called it a day.


It has been a while since my last post, and a lot has happened since then. I have spent a lot of time writing a piece of software and customer, record, helpdesk management system.

Introducing Speedtest Stats!

Introducing Speedtest Stats!

Within the past couple of months, we set up a speedtest server, that our customers, and everyone else can test to, to see how fast their internet is to Sandy, OR. All tests and accessible information is stored on OOKLA’s reporting site, and can be downloaded in a CSV format. What we wanted to do is graph this data. Instead of using an MS product like Excel to graph data, and updating records and adding new tests to an existing table, I thought I would make a PHP/MYSQL driven site to graph statistics. The project was fun, and when I was done, I waned it to be publicly accessible and usable. For that reason, I got approval from my boss to place the project on GitHub.

Version 0.1.1 is out, and it will probably stay that way. I figured, maybe someone else out there wants to be able to graph their speedtest data. Maybe. We have over 15,000 tests in the past couple of months, making a sql driven approach mighty attractive. Take a look at the screenshots below to see what it all does!

speedtest-frontpage speedtest-cityofsandy


Now, I have never really polished a project, and this is the closest I have ever been to one. It gets the job done, and take a little bit of configuration, but it does work, and people can download it and modify it to their liking.

The Xen Struggle is Real

The Xen Struggle is Real

I woke up at 6am this morning to a phone call from my boss. Barely awake, I answered in a very tired voice, “Hello.” His response was “The Xen environment is down.” I mean, it is read only Friday, this isn’t suppose to happen. Now, our environment is pretty small, but it runs all of our critical services, DNS, DHCP, AD, Monitoring, File storage. I had my boss ssh into the box and see if there were any zombie processes. Sure enough, there were. Now, before, we had processes become zombies when our log files filled up the  log partition and made everything choke. Now this was on XenServer 5.6 and things are different now. I was able to assign myself a static IP from my room and get into XenCenter, to find all of our hosts in maintenance mode. This would explain the zombie processes and the reason no VM’s were on the hosts. I attempted to bring each one out of maintenance mode, but received an error (see below).


I got into console of the master, and ran xe pool-ha-disable and boom, I was able to spin up all of our VM’s. Once we restored service to the city, I attempted to think about what caused this issue. Obviously it was related to HA, but why would that cause all of our VM’s to stop running. Part of the solution was found in the alerts section of XenCenter.



After cycling through each host, HA ran out of working hosts to break, so it just killed all of our VM’s and placed all of the servers into maintenance mode. Since I as still really tired, and wanted to get some sleep before my classes, I told my boss to open a case with Citrix, and have them dig through the logs. I went back to bed.

Turns out our NIC drivers were out of date, and it caused instability within our hosts. The resolution was to install the some updated drivers from XenServer 6.2. It would seem that the upgrade to 6.5 wiped the already updated drivers, and they needed to be re-installed. Woot! Same, day I made my drive back up to Sandy, and did a late night BIOS upgrades of our IBM and Dell hosts, and installed updated Broadcom and Intel NIC drivers. I followed the guide from a Citrix support page to upgrade them. The upgrade took no time at all, but the migration of VM’s over a 1Gbps connection was more than slow. After rebooting each host, the applied drivers should resolve our issue. This upgrade was performed on 4/3/15 and we have not had any reported issues yet.

We still have a case open with Citrix though, and we have not re-enabled HA just yet. I am waiting to find time to call and chat with them. According my boss, if call back in, they can assist and help get HA configured, tested and stabilized. I’ll update this post when that is completed, and show the results and process.

In addition, I made a post over at /r/citrix regarding my frustrations. The responses didn’t quite yield the response I was looking for, but were none the less interesting.


Migrating from IE9 to IE11

Migrating from IE9 to IE11

Now, I have put off this task for a long time. IE9 is old and crappy. I remember when it came out in 2011, and promised speeds as fast as Firefox 3.6… Well it turns out, every federal or state law enforcement site requires its users to use IE8. Custom active x applets and other junk that they use to ‘secure’ data, is so custom that it is only compatible with specific version of IE, which is not secure. The fact that I have to use an old, non supported version of IE to ‘securely’ which is incredibly stupid. Diverting from Law Enforcement, other sites that are not standardized require the use of IE9 in compatibility mode just to display data correctly (I am looking at you Granicus).


Now, the form of this post is step by step instructions of what I have done. As you read through it, I may make a change, and then end up reverting it, because it sucked. Do not follow this a guide, but rather use it as documentation. I will most likely provide some form of process once I finish the migration.

3/11/15 – I decided to actually start preparing for the migration. For the longest time our WSUS server has denied the deployment of IE10 and IE11.

3/12/15 – Microsoft announced that an update, KB3033929 causes boot loops in Windows 7. This scares me, since I thought the installation of IE11 caused issues.

3/17/15 – It seems that Microsoft has been releasing a bunch of updates lately, and there is a huge bundle of them that need to be installed. IE11 has currently caused no problems, and is slowly being installed on other computers. Let’s continue to hope, as things move along smoothly. We have only had a few cases of our servers logging in with this error. It seems to have been resolved by updating the remaining updates for IE11 after the initial install.

3/22/15 – According to Lansweeper, most of our computers have been upgraded to IE11, and we haven’t had any issues so far. Fingers crossed that it won’t cause any more issues.


As of 3/22/15 it looks like there are no current issues. Either law enforcement and states sites are being updated, or IE is working better or compatibility modes is working well enough. It was decided that if a site has issues with IE11, we will set users up with a XenApp subscription where they can use a remote app for IE9. With this setup, we hope we can keep IE up to date and secure, while still allow access to any site.

XenServer upgrade from 6.2 to 6.5

XenServer upgrade from 6.2 to 6.5

XenServer 6.5 debuted in January of 2015, and naturally I wanted to a little before upgrading to a new major release in our production environment. That time came on March 21st, 2015. I scheduled to perform a pool upgrade to 6.5 after hours. This post will consist of the upgrade process, any issues that arose and the result and thoughts about the new version of XenServer. Release notes for XenServer can be found here. 

Prepping for the upgrade – I learned the hard way when doing  a rolling pool upgrade from 6.02 to 6.1, Always read the documentation before upgrading. I logged into Citrix and overlooked the upgrade process for XenServer 6.5 under the XenServer Installation Guide. I performed the database backup and offloaded it through SFTP. Then I downloaded XenServer 6.5 and uploaded the extracted files to an FTP site. After doing a few small housekeeping tasks (shutting down non critical VM’s), I was ready to begin the upgrade

Attempt 1 – I first tried to do an automatic rolling pool upgrade via FTP. After applying one last hotfix, I began to start the pool upgrade. The master rebooted, and said it was installing. After about 20 minutes an error came up that it had failed. Instead of trying to troubleshoot it, I burned XenServer to a CD and began the upgrade again only this time, in manual mode.

Attempt 2 – This process is a little longer, but I feel more confident upgrading via CD rather than an FTP site, which after beginning the upgrade process, realized that my FTP site was on a virtual machine hosted by the Xen hosts. Shouldn’t have really mattered, but still not super settling. The master took the upgrade no problem, and I then began on the other two hosts. As I waited for machines to migrate, I spent a large amount of the time listening to music and surfing reddit. Each Xen hosts upgraded, and there were no other problems.

My thoughts on why the pool failed to update automatically, was that I didn’t point the Xen Installer to the proper FTP directory. Either way, the manual upgrade worked flawlessly, it just took a bit more time and I expected.


The only thing that annoyed me was the amount of alerts and notifications that showed up.

I started the upgrade at 11pm, and finished up around 2:30am. It was by far that easiest XenServer upgrade I ever performed, and I am hoping that we will see noticeable performance increase in performance. Having a x64 Dom should be nice, and updated templates will allow us to begin upgrading our Ubuntu servers to 14.04 LTS.

Although, I don’t know if anything is really broken. Our monitoring systems are having no issues, but we’ll see on Monday if users have any issues.