|
June 9th, 1999
Hello again, all. As many of you know, on Friday, June 4th, I met with several officials again,
from Rogers @Home senior management. Bill Lukewich, General Manager of Rogers Cablesystems for
the Greater Toronto Area (my original contact); Vic Pollen, Director of Operations for Rogers
@Home; Rob Boucher, Director of Network Operations for Rogers @Home; Susanna D'Arcy, GM of
Rogers Cablesystems for GTA West; and Stephen Haynes, Director of Customer Care for Rogers
@Home, all attended. Every attendee was issued an agenda for the meeting, which also included
ratings on all of the corresponding Rogers and @Home backbone routers for Ontario, along with
two newspaper articles written on MindSpring's customer service and the CRTC's open cable
ruling. The purpose of this second meeting was to address all of the outstanding issues and
obtain timelines for the resolutions of those issues. I believe this was achieved at the
latest meeting. I will now go over each of the discussion topics and include the responses
of senior management to each of those topics. Once again, sorry for not posting this any
earlier. I didn't get a chance to put this all together over the weekend. It's a great deal
larger than the report on my first meeting!
|
| 1. |
Official representation in the newsgroups- this is something that's coming soon.
Management never provided an actual timeline at the last meeting; they only referred to the
implementation date as sometime "within the next few months". This time I pressed
them for a timeline and they responded that July is the target, with August being the month
when we'll be seeing active participation from the representative.
|
| 2. |
Peering and routing issues- I first brought up the fact that there is a severe
lack of peering with Canadian service providers, i.e. AT&T Canada, MetroNet, Teleglobe,
UUNET Canada, Sprint Canada, Sympatico, ONet/CA*net and Toronto-area ISPs. In conjunction
with this, I also made sure to mention that inefficient routing through Mae-West (San Jose,
CA) and PAIX (Palo Alto, CA) results in poor performance and high latency to nodes located in
Ontario and mideastern Canada. At my first meeting, only UUNET peering at PAIX was a major
issue; unfortunately things since then have severely deteriorated. Basically now, if you want
to reach any ISP in Canada, other than the few that can be reached through RNS or PSINet
Canada/iStar, your packets will travel through either PAIX or Mae-West, in order to reach
their destinations. This is a major problem and leads to poor performance for all Rogers
@Home subscibers in Ontario, who are trying to reach domestic destination nodes. Mr. Boucher,
the Director of Network Operations, explained to me that this had to do with a capacity issue
on @Home's end and that it should be corrected soon. They were not able to provide a date, but
hopefully their prediction will prove to be correct. Up to this point, @Home still only peers
with UUNET on the west coast, so that also results in a performance hit for all @Home users on
the east coast. I asked what the status was on CANIX peering, as they promised at my previous
meeting that peering would be quite evident by the end of May. Again, the Director of Network
Operations was not able to provide any dates on peering, so this issue is still totally
unresolved. To be fair to management, though, there have been many problems at CANIX that
might have thwarted their peering efforts. Mr. Boucher told me that they're looking to first
get a DS3 (45 Mbps) to CANIX and expand that to an OC3 (155 Mbps) by fall. TORIX peering
(this point serves to interconnect all Toronto-based ISPs) happens to also be in the works.
Rogers is looking to deploy an OC3 to TORIX and expects quite a bit of traffic to travel over
it. A new peering agreement with UUNET Canada is another thing they're working on. The Director
of Network Operations stressed that after UUNET Canada and CANIX peering is in place, many others
with prospectives like AT&T Canada, Bell Canada and IBM Canada, will follow quickly thereafter.
They were not able to adequately explain why peering with affiliates AT&T Canada and MetroNet
has yet to be established. Hopefully this will change soon.
Another interesting bit of news is that we'll be seeing the implementation of a second RDC
(regional data center) in Ontario, sometime over the next couple of months. This will cut the
current level of local traffic almost in half and the performance increase should be clearly
evident. Creating servers accessible by region is something else that will be completed around
the time of this second RDC. The response times of all the RDC servers, such as mail and news
should be very speedy after these two changes. The last few things I spoke of under this current
topic, were the Shaw Fibrelink interconnect with Rogers, the Sprintlink interconnect with @Home,
the Rogers Ontario bordercore router (uplink 172.16.8.221) and some of the illogical routing on
@Home's end. The SFL interconnect is running much more smoothly, due to some very recent changes,
so that didn't need to be addressed. The Sprintlink interconnect in NJ, with @Home, has gotten
marginally better over the past two or three weeks. Whereas it used to drop at least 15% of all
packets during peak hours, now it drops roughly 10%- still completely unacceptable. Just a week
ago, that number was down to 6-7%, but it's now back up into the double digits. There's a huge
amount of room for improvement, although the problem has been addressed somewhat thus far. As a
great deal of our traffic travels across Sprint's circuits in the US, this problems affects most,
if not all, of us. Next, the ON bordercore router is still performing badly; earlier tonight I
logged average latency to be in the 100 ms range, although this device is located closeby in
the Ontario RDC. Obviously this needs to be rectified and I trust the problem is being worked
on. The Ontario bordercore router serves to interconnect with practically all Canadian providers,
so it's key that it performs well. The illogical routing I mentioned, has to do with @Home routing
traffic through east coast peering points, to get to west coast destinations and using west coast
interconnects to get to east coast destinations. This makes little sense to me and ends up costing
us all performancewise, so I made this fact clear at my meeting. I would expect to see some results
in this area by the end of the month.
|
| 3. |
IO capacity expansion vs. new users- luckily, this is something that hasn't been a thorn
in our sides for quite some time. At least the @Home interconnects on the east coast haven't been
too bad, Sprintlink notwithstanding. Before, seeing 200 millisecond latency to east coast routers
would be fairly commonplace, while now, seeing anything over 100 ms is a rare occurrance. The
Director of Network Operations told me that the Ontario RDC is now running Gigabit Ethernet, to
account for the current trafficload and that new HA (high availability) servers are being deployed
in the RDC, to further bear the brunt of the increasing network traffic. The HA servers are able
to practice active load balancing and other functions, that will contribute to the overall stability
of the Ontario RDC. Time will tell if this turns out to be in our favour.
|
| 4. |
Block sync issues- this is an ongoing issue that affects people in many different regions
across Ontario and BC. Work is continuing in the Kitchener-Waterloo and Brampton areas to tackle
these problems. My RHUA contacts in these various areas have been instrumental in getting the names
of affected subscribers, out to me and ultimately, contacted by Rogers personnel. Please, if you
are experiencing block sync difficulties, don't hesitate to contact your regional RHUA representative.
If you're in Toronto or Etobicoke, please contact me directly; Mississauga subscribers should contact
Wojtek Zlobicki; Brampton users, please get in touch with Adrian D. Henderson; Ottawa people, just
e-mail Buzz Haze; Kitchener-Waterloo subscribers, please get a hold of Scott Brown; London users
please provide Mike Haggerty your relevant info; and you BC users experiencing block sync problems,
just contact Ken Moren. Anyone else having these problems, please e-mail me directly. In addition
to some of these individuals taking part in newsgroup discussions, their contact info shall be
posted to the RHUA website over the next couple of days. I give you my personal assurances that
upon hearing of your block sync problems, your concerns will be forwarded directly to the General
Managers of your corresponding areas. You will be contacted and your problems dealt with on an
individual basis. I also recommend people asking for full credit on their monthly bills,
for every day that they experience the constant loss of block sync. The only way to do this is
to provide clear evidence of the problem, otherwise you probably won't get more than 50%
credit. I suggest logging ICMP queries to your gateway, for every day that the problem exists. I
advise pinging your gateway 200 times per query, with packets no larger than 100 bytes each (I
recommend 48 bytes of data per packet). I recommend using a prog like Netlab for this task. This
and other such progs can be easily found at shareware websites like TUCOWS. People experiencing
the constant loss of block sync, will see plenty of severe latency spikes, along with packet loss
that exceeds 2%, all to their first hop, or gateway. I strongly advise people to seek credit once
per month. Simply call or e-mail tech support your request. When they ask for evidence of your
problems, simply zip up the current month's ICMP gateway queries and send it away. You'll get your
credit, I guarantee it. If they give you any trouble, please let me know. Just for the record, I've
lost block sync less than ten times in all, since I first signed on to the service back in October.
I get an average 2 ms RTT (round trip time) to my gateway, drop perhaps 1 out of 500-800 packets and
I rarely see spikes past 10 ms. People with healthy physical connections should be seeing similar
numbers.
|
| 5. |
The state of technical support- this is a topic that probably all of you will know
something more about than me. I repeated your frustrations over the past few months at the long
hold times, non-responsive e-mail and often lack of knowledge of the personnel on current issues.
Stephen Haynes and Vic Pollen told me that they keep tabs on the average hold times and that they
were reduced dramatically to around 15-20 minutes after the beginning of May. As well, Mr. Haynes
stressed that all e-mail has been answered within a 24 hour period, also since the beginning
of May. I was notified that there was an average hold time of only one minute during the
Victoria Day Weekend. They do concede, however, that hold times often range from 30 to 60 minutes,
during peak hours, i.e. 7pm to 12am.
Stephen Haynes stated that we'll see some more big improvements, such as reduced hold times during
peak hours, around the end of this month and that they now train their support staff around the
clock. In addition to this, a service management center will be operational between the 15th and
30th of this month. This center will dramatically improve two-way communications between subscribers
and tech support, so that things like trouble tickets aren't closed without the user's knowldege.
As I said, over the past few months, I've had very limited communication with tech support, so I
need as much input as I can get on this subject, from all of you. Please let me know if the state
of tech support has improved as management claims, over the past five weeks. Remember, I'm only
interested in your experiences with tech support, particularly the e-mail aspect, since May 1st.
Believe me, I'm well aware of what happened before then.
|
| 6. |
Network status page- I suggested that this be implemented at my last meeting, but
unfortunately haven't heard a peep about it since then. Mr. Pollen said it will be done in a
July/August timeframe and that it'll either appear as a link off of the Buford portal, or end
up as a separate site, entirely. I stressed at my last meeting that such a site can be put
together for a low cost and within a few, short days. Because of that fact, I don't understand
why it's taking management many months to follow through on such a simple idea: something that
would benefit them, probably even more than us. I don't think I even need to explain how this
status page would be able to reduce a significant number of calls to tech support, during
outage periods. In any case, I expect that this status page will be well worth the wait.
|
| 7. |
DOCSIS modem standard/availability; coexistence with LANCity devices; notices sent to older
subscribers to clarify monthly pricing, until such devices become available for retail- I
lumped all of these together into one topic for simplicity, seeing as they're all related. Mr.
Pollen explained that Rogers will be phasing in the Terayon TeraPro cablemodems over the next
few months and that they'll coexist with the LANCity devices, without any problems. At some
point in the near future, there shall be as many as three different brands of cablemodems in
operation, in any one area. Management isn't anticipating any difficulties. At my last meeting,
I had asked that the modified monthly pricing scheme be relayed to subscribers that signed on
to Rogers @Home, after September of 1998. No clarification was ever given. Basically, the root
of the issue lies in the fact that many people think that they have only six months, from the
date of their first day of service, until they have to pay a monthly rental fee for their
cablemodem. This simply is not true, but has never been officially clarified by management.
Just for the record, no one will have to pay even one penny toward their cablemodem monthly
fees, until such a time that DOCSIS-compliant devices are widely available at Canadian
retail locations, for purchase. That's still a good six to eight months away, so you don't
have to worry about any extra fees until then. I hope that we'll see this official clarification
by the end of the month.
|
| 8. |
PST rebate- this issue was mentioned at my last meeting and was promptly resolved. It
had to do with people receiving monthly credit, but not seeing the proportional amount of tax credit
on their bills. I have heard absolutely no complaints from anyone since March.
|
| 9. |
Zenith users' status and Wave->@Home domain switchover- the former issue was resolved
a while back; I haven't heard anyone complain about not being able to get their old Zenith upgraded
to a LANCity. The latter issue has been somewhat of a problem in BC. People were issued some
strange e-mails regarding the switchover, as well as one or two e-mails that I would consider
as rude. The attendees expressed that every Wave subscriber was contacted numerous times, well
ahead of the potential switchover date, so that they would not be caught off guard when the
actual time came. Also, many Wave subscribers apparently resisted the switchover and as a result,
it was postponed by a while. I've been alerted that since the switchover, things have gone well.
People experiencing problems were contacted and had their concerns addressed, albeit taking quite
a bit of time in some cases, during peak hours. The switchover was definitely not without
incident. I don't think anything like this will every happen again. If it does, please make it
known to the RHUA immediately. Your concerns shall be relayed to management and you've got my
word on that.
|
| 10. |
Rogers's plans for the new AT&T backbone for @Home- Vic Pollen and Rob
Boucher said that this is all a matter of phasing in different regions, to accomodate their
individual bandwidth needs. So the less bandwidth your region has available right now, the
quicker you'll be connected to AT&T's backbone in the US. The attendees stressed that it's
all a matter of time. As it stands right now, there are roughly 600 nodes and 132 POPs in Ontario
that need to be phased in, so this can't happen overnight. Check out
http://work.home.net/whitepapers/backbone.html
for more information on the AT&T backbone. As I understand it, @Home's architecture will borrow
heavily from what you see there.
|
| 11. |
Further rate caps and forced proxy usage?- people keep on spreading totally unfounded
rumours about these two issues in the newsgroups, but you can rest assured that nothing of the sort
will ever be implemented with our service. Downstream rates will continue to be capped at 3 Mbps,
while upstream rates will stay at 400 Kbps. Upstream rates will NOT be capped any further,
as some of you may believe. Again, I give you my personal assurances that there will never be any
further rate caps to our existing service, without there first being a proportional reduction in our
monthly fees. There are no plans for either. As for forced proxy usage, there is a software component
of the newest @Home client release, which effectively "forces" proxy usage. This can easily
be avoided, either by using an older client version, hacking the existing version, or not using the
client at all. Let's face it- it's a Good Thing if the casual subscriber is using the proxy servers
because they end up saving all the more bandwidth for the rest of us. The choice to use the proxy
server is still there, only you have to possess a bit of technical know-how in order to be able to
circumvent it. I've been notified that port blocking, a la Sympatico HSE, will never find its way
into our service (well, other than the NetBIOS blocking that's already there). And speaking of
Sympatico HSE, I asked if anything similar to the Redback PPPoE (point-to-point protocol over
Ethernet) service will ever be instituted on Rogers @Home. I got a resounding answer of "NO",
so there's nothing to worry about on this front.
|
| 12. |
MTU/RWIN tweaks for Windows Oses- this is something that I thought would be a good idea
to see put into practice. Unfortunately, there are no plans for any such tweaks to be released. I
voiced my discord, but as it stands, you're all on your own. So those of you who at one time
tweaked your connections for dialup modems, please make sure to re-tweak your connections for
broadband services. One dead giveaway that your connection isn't tweaked properly, is low
throughput to all sites (<128 Kbps), even in the dead of night, coupled with no
loss of block sync. I recommend picking up an MTU-related program at a site such as TUCOWS
and trying out different settings. I promise to post some of my prior writings on cablemodem
tweaking, on the RHUA website. Please stay tuned.
|
| 13. |
Mail/news server problems- this is an interesting topic. At one time, the RDC servers
simply couldn't handle the amount of traffic pouring in because of software problems. This was
addressed efficiently by David Low and others at @Home, shortly after my first meeting.
Unfortunately, hardware-related problems started affecting the RDC servers around the beginning
of April. It took seven weeks until the news server was back up to par and performing admirably
again, but the other servers, notably the mail server, perform badly to this very day, during peak
hours. For instance, I logged an incredible 31% packet loss to the news server, only
earlier tonight! I've had to wait in excess of 30 seconds on some occasions, to retrieve only one
or two e-mails. The same box that acts as the Ontario RDC mail server, also happens to serve DHCP
requests. That means this piece of equipment is absolutely crucial to all of Ontario and
must be repaired as soon as possible. The RDC outage last night did nothing to help performance
in this area, I'm afraid. Luckily, the mail server is seemingly the only server that is affected
by this unreliability at this time. The proxy servers seem to be performing many times better than
they used to, since only about a couple of days ago. I expect the mail server to follow suit. Vic
Pollen and Rob Boucher volunteered that the new HA servers will help reduce the traffic load on
the current RDC servers.
|
| 14. |
Rogers @Work service deployment- I've had a number of people ask me when @Work will be
available in Canada. The attendees told me that this service will be a totally separate offering
from @Work in the US; it will even be called something completely different; July/August is the
rollout timeframe; it shall be cablemodem-based, just like Rogers @Home; it'll access the internet
via another network; and the retail cost is unknown at this time, but I think it's safe to say that
the basic service will cost more than $39.95 a month.
|
| 15. |
Recent Brantford Wave gateway (uplink 24.112.100.1) problems- this is an issue that crept
up within the past few weeks and was resolved just before my latest meeting. The gateway was
apparently misconfigured and was dropping anywhere from 50-88% of all ICMP packets. Average
throughput was greatly reduced to below 128 Kbps and it affected anyone in Brantford who used
that gateway. An incompatibility between the hardware used and a version of Cisco's IOS, was
the root of the problem. Like I said, all is well right now, with respect to this issue.
|
| 16. |
Recent changes in the Rogers @Home EUA- this is the last issue that was discussed at
my meeting. As you may or may not know, the Rogers @Home End User Agreement, which can be found at
http://rogers.home.com/CustomerSupport/EUA.html,
was altered at some point in May, I suspect the 19th. Of particular interest is section 7, subjection
(j), which states that "you will not use the Equipment or the Services to, directly or
indirectly: operate a server in connection with Rogers@Home or
the Services including but not limited to mail, news, file, gopher, telnet, chat, web, or host
configuration servers, multimedia streamers, or multi-user interactive forums." I made
it clear that this was not acceptable and the attendees clarified their stance for me. They
explained that the change was implemented to specifically catch the abusers of the service.
Casual servers apparently are allowed; this is defined as a non-advertised, low bandwidth
server, that is preferrably operated outside of peak hours. I've run servers on many occasions to
transfer some files with a few friends and there's nothing wrong with that. Unfortunately, there
are people out there who will not only run active servers 24 hours a day, 7 days a week
(obviously using a lot of bandwidth up in the process), but they'll get advertising revenue for
each login as well. I stressed at my last meeting that these people should not be using Rogers
@Home at all, as it is a residential service. Rogers @Home isn't a home T1 and it shouldn't be
treated as such. In the end, it's up to you how you actually use your server, but please try to
keep it among friends and be mindful of how much bandwidth you use. Remember, you'll be warned
if you use too much bandwidth within a given period. If you stay within the above definition of
a casual server, then you have absolutely nothing to worry about.
|
In conclusion, I would like to express my thanks to all those in the RHUA who have taken a lot of their
free time to forward the cause of our organization. Because of your effort, the RHUA has expanded, the
website's traffic is growing rapidly month by month and our collective voice is getting louder and louder.
I'm happy that an open channel of communication still exists between me and Rogers @Home senior management,
and I sincerely hope that the above issues which are unresolved, are addressed quickly and efficiently.
Yours Truly,
Christopher Weisdorf
On Behalf Of:
The Rogers @Home User's Association
|
