Note: This post was was written at 2am, and probably contains many spelling and grammar errors.
I have been with the SandyNet team for over five years now, and even when I went off to college, I was asked to work remotely and take calls after hours. It has been an awesome experience, and I am very lucky to still continue to learn and expand my knowledge by working with this company.
SandyNet has changed a lot over the past five years that I have worked there. What was originally started by contractors, was taken over by Scott Brown whom left and was replaced by Joe Knapp. After Joe has been in place for a few years, I was introduced as an Intern. What started out as organizing parts, prepping Ubiquiti legacy nanostations and best of all, making 50’ Ethernet cables, turned into a full blown job, where I had responsibility and I could rise as high as I wanted. Going back, I was absorbed by the city after spending a year volunteering as the IT guy for the Sandy Police Department. I knew basics about networks, and mainly focused on physical computers. Learning how the CPU works, and how memory is allocated. It was 2009 and I was finishing my freshman year of high school and began working every day after school. Anyway, I began to learn the basics of wireless and got a better understanding of routing, switching and TCP/IP. After a month or two, a new employee was brought on, Carlos Manzano. I grew interested in datacenters, servers and Microsoft Windows. I dinked around with servers and tore up my house dragging Ethernet. About a year later, I obtained my license, and was able to start repairing and setting up customer equipment. Soon enough, a Cisco IP phone showed up on my desk. I remember excitingly telling my parents that I got a phone and was earning more trust in the company. This opportunity came at a price. Helpdesk. The worst job at an ISP. What this boils down to is, you only get a call when something isn’t working. This ensures that the customer is usually in a bad mood, which means that is my fault that they cannot play Call of Duty. Remember, we are a wireless ISP, with three people. We have never had the man power to install every customer with a wireless radio. We usually had the customer pick up the equipment at city hall and install it themselves. Because the customer does not know how a point to multipoint wireless network works, it usually means the equipment was installed incorrectly, resulting in poor performance.
Come late 2010, I started to attend the Center for Advanced Learning. A charter school in Gresham that allowed me to focus on computer related learning. I spent half the day at CAL and the other half at Sandy High School. It was at CAL where I began studying and learning the Cisco CCNA course. In addition to networking, I was introduced to programming in PHP and Java. I also learned how to install, operate, distribute and manage Linux operating systems. Later we moved to the Microsoft learning academy, where I was properly learning how to operate Active Directory environments, DNS, DHCP and other services that Windows server offered. I used this knowledge to set up my own AD environment at home, which later helped me get into managing our internal network at the City of Sandy. By now, Scott Brown has made his way back to the City and we expanded to four employees. Also, at this time, the customer count was also rapidly increasing towards 1000 customers. Now, Sandy has a population of around 11,000, with about 3600 houses. Four people had taken almost 1/3 of the internet customer base in the whole city. I continued to rise slowly, taking on more responsibility. I began to host servers out of the City, which I used to learn more Linux and how to host websites. How DNS really worked and got to deal with security and learning how to properly secure data and sites. I started a little business, where I could host websites and manage them. I mean, we had the bandwidth and power in our datacenter, so why not?
After graduating high school in 2012, I moved onto my first year of College at Mount Hood Community College. I believe CAL helped prepare me for college and made the transition fairly smooth. I also noticed I was changing as well. I felt I needed to be more professional. I had spent two years learning how to talk with people on the phone, and deal with them in person, but I lacked the professionalism. Well, I felt I wanted more of a challenge, and began to ask to start managing servers and computers internally. I started with the deployment of Sophos antivirus to about 80 machines city wide. After that came active directory and managing our Citrix environment. XenApp, XenDesktop, XenServer and Netscaler. Learning to manage and run these systems made the main internal guy. Any problem with a server, or a client device was now my problem. I moved less and less away from phones, and focused more on data, servers and system administration. Currently, it is what I am still doing. After spending a year at OSU(Second year at college) working over a VPN and phone, my main focus was no longer going out and fixing customers problems, but rather me fixing logical, or system problems. This has lead me to spend less time on the SandyNet side, and more on the City of Sandy side. Both sides have benefits and detriments. My most recent task was deploying a new tablet solution to replace our aging CF-29 toughbooks, along with migrating storage data to another server. Fun…
Beginning to get burnt out on system administration, I wanted to get all the piss and vinegar out of my veins before was confined to a desk and phone and left to weep in a corner as stress and boredom got the best of me. I wanted to still have some ties to SandyNet, because it was where we were all trying to go. It was department that was changing all the time. New problems, new solutions. I decided to blend the two. I asked to be put through tower training class, and bucket truck training. This would enable me to rise from being a paid intern, and become an official City of Sandy employee. It took four years, but my responsibility rose from making cables, to managing all internal servers, switches, systems and users. To climbing towers, and replacing broken access points at night. To deploying and ensuring police officers have a system that they can use without worrying about possible problems.
With a department of four people, we have no predefined roles. I am not limited to helpdesk. I am not limited to internal city administration. I am free to move around and show interest in other sections. Because of where I work, there is no written way to do your job. There is no written solution. When a problem shows up, it is my, and everyone else’s job to come up a solution that best suites our needs. The factors of cost, effort, urgency, efficiency and management all play a role in problem solving at SandyNet. We are constantly changing, and we are all moving around from task to task. A project I started may be handed off, or later fixed by my coworker, but we are all in the same boat. We all know the same basic stuff, and all employees can almost pick up any project and work with it. I believe this is one of the main reasons SandyNet has been as successful as it has.
Currently SandyNet is still mainly wireless. Our traffic continues to grow higher and higher, and our equipment continues to fail and buckle under heavy loads. But we have a solution in place, and it is going to solve all of our problems. Fiber.
The whole SandyNet team has been crying for fiber for the past three years. We saw that internet was quickly becoming a necessity and people needed more of it. We knew wireless had limitations, and we had reached our carrying capacity. If SandyNet wanted to have any future at all, it needed a new medium on which it can deliver service. We all decided that fiber was needed.
Well the first two attempts were complete flops. We tried to do it ourselves. We tried to get it in order, and we tried find money to support the project. It was a huge gamble, and nobody would open up. We quickly realized that we needed the help of another company. So we sought out i3 America. A UK based company which held a patent on a fiber technology that was capable of being placed in sewer or water lines. As a local government agency, we were looking for the least amount of production cost, and we saw this as a way to decrease the cost to implement fiber. Things stated moving along. We got in touch, and plans were being drawn out. A few problems did occur. Based on our numbers, we would need 50% take rate from the city for the project to be successful. This was quite a high number. If we could not meet that take rate, we would flop, and SandyNet would most likely go under. I mean, with a project that costs over 8 million dollars, you can’t just hope for the best. So we started advertising. Presenting to people that we have a solution for the stupid slow internet in Sandy, and we need your help to get it going. Orders flowed in. We were getting a lot of people, but it still was not enough. The gamble was that we were going to go for it. So plans for constructions were being drawn out by engineers. Then one day, all communications stopped. They stopped responding to emails and phone calls. No one could get ahold of them. We knew something was wrong. We began to look for a way out, a loophole in the contract. We had a deadline, which was rapidly approaching, and they were not going to meet it unless they got started soon. The deadline was reached, and we still did not have any communication.
Communication was finally re-established. The company has been bought out and was under reconstruction. The UK company went from i3 to SCIFI networks, or something along those lines. We didn’t really care, we just wanted fiber. We began from scratch again. Starting with contracts, the long process of coming to an agreement was difficult. That is about as far as we made it with that company. They could not provide a good enough agreement for us to sign off on. So we dropped them, and went with another company, OFS, which at this time was able to do a traditional fiber build for an even lower price. No mucking around with sewers now, it can be straight in the ground! We quickly jumped on board, and things started rolling. Now, I was in Corvallis when everything started getting back on track. Contracts were signed, plans were done, and equipment was arriving. I came back to have my cubical trashed and filled with modems and fiber from OFS.
Construction had official started a week before I made it back home. It was extremely exciting to see, as fiber was actually going in the ground. It was no longer just talk, or this or that, I could go out and see it with my eyes. One major project upon my return was getting a new datacenter built and prepped for the fiber deployment. Construction on our old office at City Hall was underway while I was in Corvallis, and electrical equipment was installed, and Chatsworth products arrived three weeks into my return for the summer. I then spend another three weeks assembling racks, hanging ladder systems, dragging cable, fiber and mounting racks in the ground. PDU’s were in every rack, and two Star bus systems were in place to distribute 90% of the power we used in the room. New Cummins generator, and fancy Eaton battery backup system. It was extremely awesome to actually build a datacenter. While I did not do electrical or construction the room itself. I did assemble and mount all equipment inside. Once it was all done, we performed our generator and battery tests and spent two nights, a week apart, migrating all equipment from one room to the other. With my two hands, I had actually mounted $250,000 worth of Calix equipment in a matter of hours. I set up, blew up and then helped repair a DC plant, and wired all Calix gear to it. It was an awesome experience.
Now, I have had to leave out so many details about this project, and my history with SandyNet. There is enough experience and memories within these five years to write a book. I am very glad to have this job, and to continue working and rising up in the company. It is ever changing, and by staying here, I am getting awesome experience in one job, that five normal jobs could not even begin to scratch the surface of.
If you speak to our customers, you will get both good and bad information. Yes, wireless sucks. For all that is good and holy, stop reminding me. I work with it every day, I know your pains. But all is not lost! Fiber is around the corner. Hopefully by the end of 2014, fiber will in the ground and the city will be deployed. The process of hooking people up then finding out what to do with all the money that will be pouring in.
I fear that after the fiber project, things will get boring. I hope this is not the case, but having spent so much time with the city, it is hard to say what exactly will happen. Only time will tell. And if I do not like where it is going, I can always quit and go somewhere else. Either way, the experience alone is a story I can tell for the rest of my life. Not a whole lot of people get to experience what I get to. I am grateful to have this opportunity, and to be a part of this small time ISP. As of right now, 8/16/14, there are not a lot of municipal ISP’s that have fiber. I am excited to see how this goes, and how I can keep growing and learning new things. I have been gifted with the want to continue learning. To want to learn more, and to keep moving and not get bored. I hope to be able to brag later in my life, that four people, ran an entire city’s internet.
Next week, we are hoping to get our Calix gear provisioned and ready to roll out. If we are lucky, we will be turning on our first customers in the Snowberry subdivision. I hope to be the guy that turns on the first fiber customer in the small city that is Sandy. 😀
I got a few photos of our data center build. We had contractors come in and smash a couple of walls, then build a few walls, put in AC, and wire up an automatic transfer switch and install a star bus power line. My job was to unbox, assemble, and mount all the Chats Worth hardware that arrived. Which there was quite a bit of.
Once everything was unpacked, we began install the ladders and racks. We spent a lot of money to get some good cable management so that everything would look nice. We barely used any Velcro or zip ties to secure wires which is awesome.
The next task was to get Ethernet to each rack. Each of our three 4-post racks had 48 strands of Cat 6 connecting back to a two post rack for distribution.
Then the electrical was finished up, and we got all of the racks mounted with PDU’s and connected to the Star Line power bus.
The Calix gear was mounted and ready to be turned on. At this point we had to wait a week or so until the DC plant arrived.
After blowing up three rectifiers, we finally got all of our DC equipment setup and things were running. We then migrated all of our existing hardware from our old server room to our new one.
On 8/19/14 we got our first ONT setup in my truck, and tested the Calix gear to make sure it works. Soon enough we will be able to start hooking up customers, and making people happy!
So we went from this
and now the old server room looks like this
We’ll clean it up someday. I promise.