|
March 8th, 1999
Alright people, on Friday I came home from my meeting with several Rogers
and Rogers @Home officials. Bill Lukewich, GM of Rogers Cable for the Greater
Toronto Area (my original contact); Vic Pollen, Director of Operations for
Rogers @Home; and Rob Boucher, Director of Network Operations for Rogers
@Home, all attended. I regret that President of Rogers @Home, Alek Krstajic
was not able to attend, but nevertheless, much progress was made without his
presence. I am very optimistic about the outcome of this meeting and believe
that a common ground between Rogers @Home management and subscribers can be
achieved. I will now touch upon each of the issues discussed and what the
response at the meeting was to each of those topics. Sorry that I didn't do
it any earlier, but as you can see, I didn't have time to write all this
right after the meeting ;). It is huge.
|
| 1. |
"the political relationship between Rogers and the @Home
organization"- Rogers asked for Trevor Fiatal to stop posting to the
rogerswave newsgroups because he was providing erroneous information, with
respect to Rogers's own network infrastructure. Although Mr. Fiatal is
extremely knowledgeable in his field of expertise, he does not have all the
information on the Rogers WAN. What Rogers will do (within the next
few months) is get a Rogers @Home official to post regularly to the newsgroups,
in order to clarify problems with the service. There is definitely a plan for
official two-way communication between users and management in store for us.
Rogers's take on Trevor (and anyone else working for @Home) posting to the
rogerswave newsgroups was that any information they provided was official and
representative of the @Home organization's views of Rogers and its service. If
any information turned out to be incorrect, both Rogers and @Home could
potentially be held liable. I understand their point of view on this situation
and am happy to see official representation in the newsgroups on the horizon.
|
| 2. |
"peering and routing (CANIX, UUNET, RNS, etc.), impact of high
latency, low reliability and jitter on network application performance, the
TCI @Home AUP"- There was quite a bit of emphasis put on these issues
at my meeting. As some of you may have noticed, RNS peering is now totally
intact. @Home->UUNET/Alter.net, Rogers->UUNET Canada and Rogers->CANIX
peering is still either poor or nonexistent. I had quite a bit to say on the
poor location of @Home->UUNET peering at PAIX- roughly 3700 miles away from
Toronto. I am awaiting an answer as to why this is the case. Rogers->UUNET
Canada peering is partly there, but still basically non-existant for
reaching most ISPs that are hosted by their backbone. This is a terrible
situation because if, say, I wanted to reach an ISP located 20-30 minutes away
from me, that uses UUNET Canada for its backbone connection, I will be routed
all the way to PAIX and back, right off my doorstep (e.g. www.communicopia.com,
which is located in Mississauga). Needless to say, this is just plain dumb. Not
only are my packets travelling through nearly 8000 miles of unnecessary
infrastructure, latency is automatically increased to just under 40
milliseconds for that distance, the reliability of any connection drops
exponentially from travelling through all the extra routers and the pipe
from us to PAIX is often full of "unnecessary" traffic. This is
something that has thoroughly baffled me from the outset. The participants
promised to find out for me what was happening on this front, so, God willing,
I'll have an answer soon. CANIX is another issue entirely. Apparently there
were legal and political barriers that prevented Rogers from entering the
"CANIX club", as Mr. Boucher referred to it as. The main political
obstacle was that any prospective interconnectees would require the unianimous
approval of all CANIX members, prior to becoming one themselves. This cleared
up at the end of 1997 and since then Rogers has been talking with them about
peering. I was told that there will be one DS3 (45 Mbps) circuit in place by
the end of the month and that some peering would be visible by April at the
earliest. In May, peering should be quite evident. Quite obviously, peering
in Canada is done very differently from that of our southern cousins. I am
digusted, as a Canadian, to hear of such barriers to enter simple agreements
in an industry that is so critical to our future as a country in the digital
age. I was told additionally that full BCIX and BCTel peering should be in
place within the next few weeks. I don't know anything about the situation
in BC, but I was notified of that anyway. TCI, their AUP, their policies,
their strategy and a lot of bad press about them, were all discussed at
length. I sincerely trust that Rogers will never achieve the level of
notoriety that TCI now has. There will also be no type of similar AUP
implemented as long as a dialogue between me and the participants exists.
|
| 3. |
"the expansion of backbone capacity to accomodate new users
"- I was told something interesting on this topic: circuits are now
ordered IN ADVANCE, as opposed to when congestion is first exhibited.
This is absolutely crucial in the operation of a successful network.
Remember the period from the beginning of November to the middle of December?
That was somewhat of a sick joke. I never thought it possible that I'd be
seeing 400-600 ms latency on cablemodem, due to extreme network saturation.
That's what happens when circuits are ordered in response to congestion, as
opposed to before it ever happens. Couple in a 45 day wait time (almost
exactly how long it took to get rid of the problem) for the actual
deployment of those new circuits and the addition of several hundred new
subscribers a week, and you've got a recipe for disaster. I know that Rogers
has learned its lesson from that first experience, but that other MSOs have
not. I don't think that we'll be seeing such high latency again. As always
I'll be keeping a constant eye on network performance to make sure this
doesn't happen again, as well.
|
| 4. |
"block sync, block sync, block sync..."- This is another
huge issue that is probably the largest impediment to the high performance of
cablemodem technology. There are many different factors that can affect the
loss of block sync, more than I mentioned in the past and didn't think much
of. Mr. Boucher mentioned that the reverse path (upstream) nodes were
segregated at the headend, so these problems are completely regional and
even neighbourhood-specific in most instances. Temperature is a biggie. The
expansion and contraction of coaxial cable encourages attenuation and other
big problems that will destroy the two-way nature of cablemodem technology.
Theft of cable is another big problem. The use of a single splitter or
amplifier will cause block sync difficulties a great deal of the time. Even
if one of your neighbours is using an old, noisy VCR or other appliance, like
a washing machine, they will contribute to loss of block sync. Some unlucky
neighbourhoods are also plagued by cracked foundations where the trunks are
placed. Obviously this results in block sync problems for the entire area.
Such problems are being addressed as I write this. I mentioned the Saint
Laurent, Brampton, Kitchener/Waterloo, Orleans (Ottawa) and Hunt Club (Ottawa)
areas. Again, not all the houses encompassed in a specific geographical
location will be affected by the constant loss of block sync. Certain
neighbourhoods and individuals are highly prone to this kind of problem. The
truth is that you should have your cablemodem on a completely dedicated cable
outlet and no splitters or amplifiers should be used at all in your house. I
know I don't use any and have lost block sync exactly five times briefly,
since I signed on with the service back at the beginning of October. I was
told additionally that there is a buffer of roughly two months when making
areas cablemodem ready. This can account for some areas seemingly being added
ahead of time, like Hunt Club. May 15th is a date I was given when a rebuild of
Toronto for cablemodem service will be completed. I stressed that even the
slightest block sync problems would kill any prospective voice service being
offered over cablemodem. Since there are big plans for voice-over-cable by
AT&T Canada, I think we can expect this problem to receive very high
priority.
|
| 5. |
"technical support- communication, communication, communication!!!
"- This problem is high up on the list of things to do for Rogers @Home.
Not only are more technical support people to be hired soon, but proactive/
reactive e-mail services will also be implemented over the next few months.
That means any people affected by block sync problems, downed infrastructure,
regular maintenance, server problems and other difficulties will be notified
immediately by e-mail- no tech support calls needed. Obviously this is not
the Holy Grail of customer support and many bugs will have to be worked out
of it when it actually comes into being, but I am very hopeful just the same.
I brought up several other communications issues, like their being aware of
server problems, locations susceptible to constant block sync loss, poor
network performance and other areas. I also advised there to be an amendment
to existing policy regarding the handling of block sync problems. NO
person at tech support should be telling subscribers to do anything via
software to combat block sync problems. Changes to the phone system, in order
to tier people into several different groups was also discussed. I believe
David Low mentioned something like this a week ago. I expressed my personal
extreme dissatisfaction with the e-mail portion of tech support, so hopefully
more (experienced) people will be hired to man that portion of customer support.
|
| 6. |
"the use of a network status page"- This has to be the
simplest single solution to easing subscriber frustration and the escalation
of already ungodly tech support hold times. The participants expressed some
interest in this idea. I strongly encouraged the implementation of this idea.
Meanwhile, I am working with a Rogers @Home subscriber in Ottawa on implementing
a status page, regardless of any official actions. Rest assured that this will
soon be taken care of, one way or the other.
|
| 7. |
"DOCSIS 1.2 standards and future subscriber payments regarding the
retail availability of devices based on such standards"- This is
something that the participants weren't aware that subscribers didn't have
a hold on. Back in the summer of 1998, Alek Krstajic sent an e-mail to all
Rogers @Home subscribers, explaining that modem rental fees would be waived
until DOCSIS-compliant cablemodems were available, by the way of retail
outlets for a reasonable cost. Newer subscribers, like me, were told this
when we signed on with the service. The 6 month deal was suggested in that
technical analysis incorrectly predicted that cablemodems would be available
in retail outlets by then. Rest assured that no Rogers @Home subscribers will
have to pay any additional fees until the DOCSIS 1.2 standards are adopted
and devices based on those standards are available in retail outlets for a
reasonable price. I strongly advised the participants to issue a press
release to clarify this situation. The six month deal is really confusing a
lot of people, including me.
|
| 8. |
"PST rebate in relation to credit"- Ah, the mosquito bite
of all the problems we face as subscribers :). Some people who have received
credit for constant loss of block sync and other downtime, have complained
about being billed for an inappropriate amount of provincial sales tax. These
amounts usually don't total any more than three bucks a month, but
nevertheless they grate on the nerves. This accounting discrepancy will be
corrected immediately.
|
| 9. |
"Zenith cablemodem users and their upgrade status"-
There's not much to say here. People are being upgraded as soon as possible.
Newmarket and the older areas have the bulk of these users, so they are being
addressed first. Basically if you've paid your cable bills and you ask (even
if you don't ask), you'll be upgraded. It's always better to ask, so that you
make sure you are upgraded as soon as possible.
|
| 10. |
"Rogers's plans for the Sprint to AT&T backbone crossover by
@Home in May"- The participants didn't have any information on this
topic. They did say that network management will be more efficient and the
technology used in AT&T is more advanced than Sprint's. While the current
@Home infrastructure is a chimera of many different technologies, much of it
is composed of incremental DS3 circuits. AT&T operates currently on an OC3
SONET and ATM over SONET infrastructure, with plans to operate at 622 Mbps in
only the next month or two- still before our May crossover. So not only is
there more bandwidth in store for @Home subscribers across North America,
but the superior technology used in AT&T backbone will allow for more
efficient transport.
|
| 11. |
"further upstream/downstream rate caps"- I strongly
advised the participants to NEVER curtail our existant upstream and
downstream rates without first lowering our monthly fees further, in
an appropriate ratio consistent with our loss of performance. I predict that,
perhaps, in a couple of years, we might see another tier of service offered
for a lower price. That base service would contain a 128 Kbps upstream rate.
Right now a lowered priced service- tiered or not tiered- simply isn't
profitable. I stressed the legal implications of such a move and the spineless
inaction of TCI @Home subscribers in Fremont, California to combat their
existing rate cap. Rest assured that I will not allow a further rate cap to
be imposed on Rogers @Home subscribers, (without monthly rates falling
appropriately) while I remain a subscriber of the service and continue an
open dialogue with the powers that be. I am confident that Rogers would
never pull this kind of crap anyway.
|
| 12. |
"MTU/RWIN tweaks that are responsible for poor performance
"- I, again, strongly advised the participants to bundle a type of
"speed boost" program with the @Home installer for subscribers
using the Windows platform. I'm not talking about the old speed boost prog
that Shaw brough out a while back, either. The speed boost prog would search
the registry for the MaxMTU, MaxMSS and DefaultReceiveWindow entries and
remove them. It would then add the keys "PMTUDiscovery and
PMTUBlackHoleDetect" and set them both to "1". I,
myself, use those exact settings and experience the maximum performance
possible (up to 2.9 Mbps sustained throughput with an average 2 ms first-hop
latency during optimal conditions). Many, many people complain of poor
throughput (under 20 KB/sec) without block sync problems in the newsgroups.
95% of these cases are due to incorrect settings in the OS, yet Rogers and the
@Home organizations are unfairly chastised in this situations. I really hope
this idea is put into practice, as it is very easy to do and will solve many
problems on the management and subscriber ends. Letting people know by e-mail
that this patch is available to existing subscribers is child's play.
|
| 13. |
"mail/news server ongoing issues"- This is something that
has improved greatly as of late. Mr. Low's presence in this forum has helped
clarify these problems with us and has allowed @Home to be immediately aware
of any server problems. The improvement of technical support will definitely
add to these benefits.
|
| 14. |
"Linux support and the general future plans for the service
"- I probably did most of the talking here :). The use of Linux by @Home
subscribers is growing rapidly, so I asked the participants to pay close
attention to this issue over the next few months. I personally feel that
official Linux support for the Rogers @Work service is a must. Tiering is
also on the horizon for subscribers after the DOCSIS 1.2 specification has
been adopted by all MSOs. Don't be surprised to see a 20 Mbps+ downstream
service offered for under $300 a month by the end of the year. It'll be
geared to businesses, of course, but this kind of thing will be available
for much cheaper prices in under 3 years. Telecommunication analysts
have predicted that bandwidth will become more than 10x cheaper in under 3
years from now. 24 channel DWDM (dense wavelength division multiplexing)
technology already exists commercially and basically increases bandwidth
on existing circuits by an entire 24x. I predict we'll be seeing 50 channel
systems commercially available by 2001. And that isn't even looking at
additional bandwidth amelioration technologies. That means both serious
performance (>10 Mbps) for the end user for the same prices that we
pay now and serious profits for the providers of those services- something
that no cablemodem-related firm has even seen any glimpse of yet. Right now
there are some serious problems with bandwidth abuse by people running servers
saturated with users, 24/7. If enough servers with constant usage are active
on a segment, their traffic can seriously hinder performance for everyone else
on that segment. I was quoted as to what Mr. Pollen referred to were "
record" figures of over 120 Gigabytes of upstream traffic in a one month
period by a particular Rogers @Home user. I recommended that anyone caught
advertising their services be immmediately warned that any subsequent
violation would result in their termination. Couple that with constant
upstream bandwidth usage and the violation is exponentially more serious.
I think that anyone making money off providing services, especially services
that use a significant amount of consistent bandwidth, shouldn't be a Rogers
@Home subscriber. I strongly encouraged the participants to make formal press
releases regaring some of these issues. I believe that official announcments
will be forthcoming.
|
|
In closing, I'd like to state that I fully favour cooperation with management,
rather than confrontation and I will maintain that stance for as long as I
remain a subscriber to the Rogers @Home broadband internet service. My next
meeting will take place sometime within the next two to three month period.
|
Regards,
Chris Weisdorf
On Behalf Of:
The Rogers @Home User's Association
|
