Nearly 2 years ago I wrote Starting an Internet Service Provider. When I sat down to write that post I initially intended it to be a weekly or monthly log of events to look back on, but it turned into one long blog post about the struggles of starting an ISP. That post ended up receiving over 20,000 visits in one day while at the top of Hacker News and roughly 40,000 total. I obviously haven't written weekly or monthly since then; mainly due to lack of time.

NEPA Fiber has come a long way since I first wrote about it 2 years ago. Perhaps the biggest change is our name. I decided to rebrand from NEPA Fiber to Loop Internet. Most out-of-towners didn't know what NEPA was (an acronym for Northeastern Pennsylvania) and I also wanted to eventually expand outside of NEPA. I don't remember how the name Loop came to be, but I do remember bouncing the name off of a girl I was seeing at the time. She liked it, I liked it, and that was good enough for me. Loop Internet was born (or reborn?).

I've had some second thoughts about the name since then. At the time, deploying physical fiber didn't seem like it was going to happen anytime soon and having the word "fiber" in our name just seemed wrong. On the other hand, you'd be surprised at how many people don't even know what fiber is. Regardless, I committed to Loop, so Loop it is.

What's changed?

• Our MRR (Monthly Recurring Revenue) has quadrupled.
• We've grown to nearly 200 subscribers (mainly businesses)
• We just launched in our 2nd city; Scranton, PA.
• Our network coverage area has grown drastically. We went from being able to service maybe 10-20% of our first city, Wilkes-Barre, PA, to now covering nearly 80-90%.
• Our wireless reliability doesn't suck anymore.
• We now offer Gigabit service in several properties.
• We now offer fiber service in several dozen properties.

What hasn't changed?

• Wireless spectrum is still a huge issue. This is one of the main reasons we're focusing on fiber now.
• Wireless reliability still kind of sucks (I'm aware I contradicted myself from above. Read on to "the wireless struggle" in the next section for an explanation under the pro's/con's list. Tl;dr; It's a spectrum (or lack-thereof) issue.)
• This shit's expensive.

The Wireless Struggle

I've learned A LOT about wireless since my last post. A lot about interference, shielding, spectrum, and wireless network design.

I try not to hate on wireless too much because it got us to where we are, but it's a pain in the ass to maintain a reliable network.

Pros:

• Wireless is affordable.
• Speeds are competitive with cable (for now).

Cons:

• Spectrum is and probably always will be a major barrier for WISP's. (See more on my wireless spectrum rant below).
• Wireless is completely dependent on the terrain. Wireless works okay in some areas and is totally unfeasible in most other areas (trees = bad).
• Wireless radios are susceptible to various environmental factors. We've had squirrels chew through wires, tornado gusts and nearby lightning strikes take out antennas.
• All PoP's need battery backup units that can run for extended periods of time (>24 hrs). The costs of deploying and maintaining extended runtime battery backup units really add up when you have over a dozen micro-PoP's.
• WISP equipment is cheaply made in order to make it affordable; failure rates are fairly high over time.
• Everyone wants a piece of the cake. Property owners with buildings ideal for wireless equipment want hundreds or thousands of dollars a month to place our equipment on their roofs. It takes 10 subscribers at $50/m just to break even on a $500/m roof lease (not including the equipment, labor, materials, etc). Most of our antennas have less than 10 subscribers on them. The math obviously doesn't work out.

I'm clearly a little bias against wireless, but I feel it's for good reason. When you're operating in an urban environment, unlicensed equipment makes life hell. We frequently have to move wireless channels on antennas (every few days in some cases) in order to resolve speed/service issues. Some channels will be wide open for a week and then out of no where one end of the link will pick up interference at -60dBm. We then have to move the equipment to another channel and repeat every few days. We've been able to react quickly to these sorts of issues in order to continue growing, but it's not ideal.

It's also fairly difficult to find and train techs who are willing and able to install equipment in some fairly sketchy locations. I've literally jumped from the top of a 32ft ladder onto a hook-ladder in order to install a receiver on the peak of a roof before. I would obviously never ask, nor allow, a tech to do that on their own; but when you're growing and you have everything on the line, sometimes the juice is worth the squeeze (I'd never do that again, btw. I'm lucky I didn't die.)

In general, wireless creates a whole slew of challenges. We've had to mount receivers on the tops of 30ft monopoles. Buying a bucket truck doesn't really make sense unless we were to use it daily. Renting one makes these types of one-off installations fairly expensive and kill our ROI.

Wireless Spectrum Rant

There are two "genre's" of wireless for point-to-multipoint applications. Licensed and Unlicensed. If you're operating in a rural area, you may be looking at obtaining a 3.5Ghz or 3.65Ghz license. You'll have a leg up in interference, reliability, and signal penetration (trees), but you'll likely be far behind the competition in terms of speed. From my real-world testing, you shouldn't really expect to get more than 30-40 Mbps down and 5 Mbps up in a typical scenario. If you're competing with DSL, that's perfectly acceptable and you can win over most subscribers. If you're competing with cable services you probably won't be able to make much of a dent; especially with DOCSIS 3.1 being deployed now.

If you're like us and are deployed in a more urban environment, the only spectrum that can truly compete with cable and fiber is unlicensed spectrum (unless you have millions of dollars and can win some in an FCC auction). Our network runs on a combination of 5 Ghz, 11 Ghz, 24 Ghz, 60 Ghz, and 80 Ghz equipment. 11 Ghz is licensed and 80 Ghz is lightly licensed. You can only use that spectrum in Point to Point links, so we use that primarily for our back-haul to connect our PoP's together.

As wireless and 5G become more prominent and as more services move to mobile, I believe WISP's will be left in the dark as large mobile carriers use their influence and money to horde high-bandwidth "5G" spectrum.

Transitioning to Fiber

My dream has always been to deploy fiber. It has taken nearly 3 years of persistent nagging, emails, and meetings to finally get the chance to do so. I can't thank Wilkes-barre City Councilman, Tony Brooks and Wilkes-Barre's Director of Operations, Butch Frati enough for their help in making this dream come true.

When we first began our transition from a pure wireless provider to a fiber provider towards the end of 2017 and into the Spring of 2018, we had run some fiber from a building already serviced by a 80Ghz 5 Gbps point-to-point link, down the pipe-chase of the building, and then out the back of the building to the neighboring buildings which are connected. That's one of several short-cuts we've taken in order to provide <$1,000 1 Gbps circuits. The service is just as reliable as it's pure fiber counterpart, but at a substantial costs savings to the end-user and ourselves.

About 3-4 months ago we began deploying fiber conduit in the public right of way in Wilkes-Barre, PA. The orange line in the below map shows our current fiber conduit network from the project. The original project spanned 3 blocks and we are now beginning to build off of that network. The purple dots are some of the multi-tenant buildings in our network, most of which are serviced via wireless.

I always knew running fiber was expensive. As an entrepreneur your job is to figure out how to do something better, faster, and for less. I thought we'd be able to do that. I was wrong. Not wrong in the sense that we can't do it, but wrong that we could do it on our first try. We made a number of mistakes. The good news is we learned from them and I'm confident we can avoid them as we move forward. The bad news is, those mistakes cost money; a lot of it.

When we started this project I budgeted around $30,000. I've honestly lost track of the costs because we've been working to complete it under whatever means necessary due to our time constraints. I think when it's all said and done that it'll end up costing around twice that.

My biggest mistake was starting the project without ample funding. I tried to fund the project primarily through our existing subscription revenue and through a large cabling contract I had received months earlier. Realistically, we required around $100k to buy the equipment we needed. Had we had adequate funding I would have bought the equipment we rented: a skid steer breaker attachment, a concrete saw, and also a few other tools that we didn't rent but plan to buy in the future to make the process faster and more efficient; such as a vacuum excavator and boring machine. The project would have cost around the same, but we'd own several assets that we'd be able to use again in the future. Instead, we have nothing to show for the nearly $10,000 in rental fees we were charged over the period of 3-4 months.

The project began as a 3-block run, approximately 3,200 ft in all. The local gas company was replacing a leaky gas line that runs down the street. Per an ordinance put into place by City Council, once a street is repaved no one can work in it again for ~5 years without repaving it curb to curb. For us, this was a unique opportunity to get some conduit in the ground without having to pay to restore the street. However, the street was being restored immediately after the gas company's project was completed, so we had a limited window to get in and out.

The project began as a 3-block, 3,200ft, fiber run. By the time we're done we will have dug up about a mile of road and deployed approximately 8,000ft of conduit. As you can tell by that math, we decided early on that if we were going to dig up the road we might as well put ample conduit in the ground for future expansion purposes. The last thing we want to do is run out of room in the conduit and have to dig up the same road again in the future. We put two 2" pieces of conduit along the the majority of the route; while most end-user connections have a single piece of 2" ran to them.

Rough Beginnings

I ran into one HUGE issue that delayed us for over a month, as well as some other minor inconveniences.

First, the major issue. I'm a firm believer that buying equipment you plan to use regularly, or for a long duration, is almost always better than leasing. You own the physical equipment, you can use it whenever you want, and it pays for itself overtime. You have a physical asset in which you can chose to either hold onto or sell, usually at a net profit compared to renting.

I don't have the best credit because I hold a lot of the business' debt under my own name. Let's just say that I have every ounce of skin in the game. Therefore, getting financing to buy the equipment we need generally isn't an option.

This entire project was put together at the last minute so I contacted our local Cat dealer about 3 weeks before our planned start date to reserve a wheel saw for our skid steer. A wheel saw is basically a giant saw blade with teeth that cuts through the ground. You may be familiar with the term "micro-trenching". He said they had one and to come pick it up in 2-3 weeks when we're ready. So 3 weeks later I call back to pick it up and they don't have it; it's out on rent. I ask if they can find me another one. He tells me that he can't search other dealers inventory. I called another dealer, they don't have one either, but he tells me he sees that another dealer in Pittsburgh has one (not sure why the hell the first guy told me he can't search other dealer inventory). I then fill out some financial paperwork that the first guy never told me about, get approved for $5,000 on NET30 terms, and get the Wheel Saw shipped to our local Cat dealer.

About a week later, we pick up the wheel saw and bring it to the job site. I pick it up with our Case 95XT and immediately there are a few issues. 1) Having never used an attachment that required electronic controls before, I had no idea each skid steer manufacturer had their own unique electronic PIN hookup. They all have universal hydraulic adapters, why not have universal electronic controls. 2) The teeth, which lock the attachment onto the skid steer, don't fit in the attachment; they're about 1/4" too wide for the opening. Again, these are supposed to be universal on every skid steer, I'm still not sure why they didn't fit.

The first issue could likely be solved by moving the pins around. The 2nd issue can't be fixed. I can't exactly use a grinder on an attachment I don't own. So at this point I'm panicking. I already rented the concrete saw and breaker from Sunbelt rentals for close to $2,000/m. We already started prepping the road by breaking up and removing parts of the sidewalk where we need to put hand holes in, I already have the hand holes (boxes that you pull the conduit into) on-site, and we already have $3,000 worth of conduit sitting on the road. I'm in this for close to $8,000 at this point and we might not even be able to dig.

At this point I'm ready to go all-in, let's just get it done. The Case dealer told us that ordering a Wheel Saw through them would take 6-8 weeks. He knew for sure that that one would fit our Case skid steer, but we didn't have that kind of time at our disposal. We were supposed to be done with the entire project by the time we'd receive the saw. I then contacted some Bobcat dealers to see if they knew if their Wheel saw attachment would fit a Case. No one was really able to give me a straight answer. I was told that even if the electronic adapter didn't work, they sell a separate control box that would by-pass our skid-steer's electronics altogether. We were already about a month behind schedule at this point; fuck it, tell me where to sign.

After applying to numerous financial institutions, no one would give the company a loan for the attachment. When in doubt, turn to mom. My mom co-signed the loan and about a week later we now owned a Bobcat WS18 attachment. The attachment was around $20,000 and I think I paid around $1,000 upfront. Our payments are only around $795/m at 0% financing for 24 months. A far cry from the $2,800/m it would have cost to rent one.

The issues weren't over yet. No one has the electronic adapter in stock and they had to special order it and make one just for us. It was going to be another 10-14 days. We couldn't wait that long. The Bobcat dealer offered to let us rent a skid steer at a fair rate. At this point I had lost track of how much money we had invested into this project, we just needed to start cutting, so I take him up on the offer.

Finally Rolling

We started cutting the next day. Progress was very very slow in the beginning. We weren't aware that there was nearly 12in of concrete below the asphalt. There was a fair bit of a learning curve both with the wheel saw, finding the appropriate cutting teeth, and with our technique. We tried numerous things to get the saw to cut faster and eventually we determined the most efficient way to cut, do utility locates, and dig the holes for our hand holes. In the beginning it took us nearly 2 weeks just to do one intersection. In the past 2 weeks, we cut an entire block.

The city allowed us to use flowable fill (concrete with a very low PSI) to restore our trench since the road is being repaved afterwards.

Other Issues & PR

There are some other details that I skipped over above. We had numerous issues with the Case skid steer. Between the day we started breaking concrete and cutting to around the 2nd week on the project we had replaced just about every hydraulic hose there is on the machine.

Throughout the project we've had a few run-ins with property owners and commuters who have complained about the project. It's very hard not to lose motivation or get agitated when you've gone all-in on a project, who's only real goal is to improve the city, and then get numerous complaints because people need to wait an extra 2 minutes in slightly congested traffic during rush-hour or because you've dug up their sidewalk temporarily.

I've also found that no matter how well you fence off your holes and fresh concrete, there's always someone who will write their name in the concrete, step in it, or knock the barriers over.

We still have another week or two until we completely wrap up this project, so I may make another post after we try pulling cable through the conduit. That'll be the ultimate test of whether we had a good deployment design or not, as a few of the runs were ~500ft and others had a fair amount of high-angled bends.