ping and tracert networking question

[techhelpbb]

Originally Posted by Ether

I thought I answered that:

The IP 3 hops away is probably at the network level near that ISP's perimeter.
So if the DNS doesn't turn out to be the issue you probably have lost your connection to them over that link.

If you have the information to access the equipment on your side you can probably query the status of the connectivity to the tower from the web interface. I think that unit supports SNMP as well so you could query the status of the link from that. They appear to have an MIB for that purpose available. Then you could work it out such that you can monitor the signal level and data transmission rates.

[Nirvash]

Just out of curiosity, what addresses are you pinging and what addresses are you tracerting too?

[nuttle]

This question leads to an iceberg answer, in that there's a lot to say. Here's the tip, so to speak:

> I can ping the tower but tracert does not work (or takes an inordinately long time).

The short answer is that you are most likely not pinging the tower, but something more local. The DNS servers are not likely to be local either, so DNS name resolution isn't likely to work either when you are having connectivity issues.


The analogy here is that to reach an endpoint (specified by IP address, possibly after the additional step of DNS name resolution from a textual name to an IP address that is an independent network operation that itself requires a healthy network connection), you have to traverse a network of roads. You can think of each intersection on this network of roads as having its own IP address. To get almost anywhere, the first few intersections are going to be the same, they take you from your home, out of your neighborhood, and onto a major road. Once you're on this road, chances are you are going to get where you are going, unless there is a problem toward the other end of the journey, when you are getting back onto local roads in the destination neighborhood. Or, if there's no one home at the address on the other end (the site or server is out but you can get to everything else).

There is a span of road that is sometimes out in your case, most likely the over-the-air link. If this happens, you can reach intersections or even end addresses that are local and do not require traversing the span where there's an outage, but can't get further.

Take a tracert when things are working, and let the IP addresses be resolved back into textual names (don't turn of DNS resolution when tracert for a working connection). These names may give you clues. Keep the list of names and IP addresses to some well-known site handy for comparison when things are out.

You will likely find that when you run a tracert that stops at some point, you can ping the addresses that it could reach, you just can't ping ones that are further away on the network. The first place you can reach when things are working but cannot when they are out is the far end of the link that is giving you problems. If you want, you can post details here and we can comment further.

Hope this helps!

[Ether]

Originally Posted by nuttle

The short answer is that you are most likely not pinging the tower, but something more local.

The tower is local. The AirOS PowerStation2 radio/antenna on my roof (mentioned in an earlier post) is aimed directly at the top of the tower, 1.2 miles away, line-of-sight.

3 hops: PC -> router -> radio -> tower.

BTW, I have ruled out the router. If I bypass the router and connect the PC directly to the radio when the problem occurs, it does not fix the problem. I do not have access to the radio, other than to cold-boot it by cycling the power (PoE), which I have tried, to no avail.

When the problem occurs, I can successfully ping the tower, but tracert to it times out (or takes an inordinately long time to complete).

This setup had been working almost flawlessly for 3 years. Problems started occurring occasionally about a year ago, and have become more frequent and longer in duration.

[nuttle]

Originally Posted by Ether

The tower is local. The AirOS PowerStation2 radio/antenna on my roof (mentioned in an earlier post) is aimed directly at the top of the tower, 1.2 miles away, line-of-sight.

3 hops: PC -> router -> radio -> tower.

I wouldn't classify a 1.2 mile over the air hop as local, particularly the over the air part -- but see below.

Originally Posted by Ether

When the problem occurs, I can successfully ping the tower, but tracert to it times out (or takes an inordinately long time to complete).

This is the part where I'd be inclined to measure twice. When you supply an IP address to ping, how do you know it is the address of something at the tower? It is possible the radio has its own address (cable modems do, for instance). If you have a list of IP addresses from a tracert taken when things are working, you can try pinging each of these addresses and should see very similar results as when running tracert, in either the case where things are working or not. In essence, tracert is automatically sending pings out is succession, one hop further each time.

Another thing you might try is opening a browser and typing the IP address you have for the tower and see if you can get to an admin interface. You should certainly be able to do this with the IP address of the router, to see how this works. Apologies if you've already tried this -- it is hard to find the right level of detail without being insulting and/or writing a whole lot here.

Do you have a list of names that you obtained when things were working (tracerout output but not using the option to suppress DNS translation)? It might be interesting to match this against the list of hops you supplied above and knowing just where things stop when they are not working would be very helpful. It could be further into the network on the other side of the radio link, but I'd suspect this link until I knew for sure it was good.

Are things worse when it is raining, at certain times of day, or any other pattern you've noticed?


Sorry for so many questions and so little actual help!


BTW, if you can get to the admin interface for the radio, you should check to be sure your firmware is current. Same goes for the router, but this isn't likely to be as important in this case.

[nuttle]

This may be useful:

Step 1 - Login to the web management
Open the web browser and type the default IP address of the AirOS powered device http://192.168.1.20 into the browser address field. You will be prompted to enter the default administrator login credentials:
User Name: ubnt
Password: ubnt
After successful administrator log on you will see the Main page of the web management interface.


There should be a way to see the signal strength -- if you had access to the antenna, this could help to make sure it is aligned optimally.

One final thing, it can be confusing to tell when tracert is first trying to ping an address vs. when it has done so and gotten a response, or has timed out. This is a good reason to use ping in isolation as a double check.

[Ether]

I have neither physical nor network access to the AirOS PS2 radio on my roof, other than to cold-boot it by cycling the power (PoE), which I have tried, to no avail. On a moonless night I can sort of see the signal strength LEDs with a pair of binoculars. I have not yet been able to observe them during an outage event.

Outage events are random and not correlated with weather conditions.


PC---(ethernet)--->router---(ethernet)--->radio---(wireless~1mi)--->local_tower---(microwave_beam~2mi)--->main_tower--->fiber_optic_backbone

ping/tracert script log when working correctly:

The current date is: Wed 11/20/2013

The current time is: 11:22:08.17

c:\> ping e.f.g.h

Pinging e.f.g.h with 32 bytes of data:

Reply from e.f.g.h: bytes=32 time=11ms TTL=62
Reply from e.f.g.h: bytes=32 time=6ms TTL=62
Reply from e.f.g.h: bytes=32 time=10ms TTL=62
Reply from e.f.g.h: bytes=32 time=5ms TTL=62

Ping statistics for e.f.g.h:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 11ms, Average = 8ms

The current time is: 11:22:11.45

    3.30 elapsed seconds for ping


c:\> tracert e.f.g.h

Tracing route to e.f.g.h over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.1.1  <-this is my router, hop #1
  2     2 ms     5 ms     1 ms  a.b.c.d     
  3    60 ms    86 ms    45 ms  e.f.g.h 

Trace complete.

The current time is: 11:22:13.76

    2.31 elapsed seconds for tracert

Press any key to continue . . .

ping/tracert script log when not working correctly:

The current date is: Sun 11/17/2013

The current time is: 22:05:52.32

c:\> ping e.f.g.h

Pinging e.f.g.h with 32 bytes of data:

Reply from e.f.g.h: bytes=32 time=7ms TTL=62
Reply from e.f.g.h: bytes=32 time=4ms TTL=62
Reply from e.f.g.h: bytes=32 time=4ms TTL=62
Reply from e.f.g.h: bytes=32 time=5ms TTL=62

Ping statistics for e.f.g.h:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 4ms, Maximum = 7ms, Average = 5ms

The current time is: 22:05:55.64

    3.24 elapsed seconds for ping


c:\> tracert e.f.g.h

Tracing route to e.f.g.h over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.1.1  <-this is my router
  2     2 ms     2 ms     1 ms  a.b.c.d      
  3     4 ms     6 ms     7 ms  e.f.g.h     

Trace complete.

The current time is: 22:06:57.79

   62.07 elapsed seconds for tracert   <-yikes!!!

Press any key to continue . . .

Here's where I need some networking help. Can the radio on my roof be in "bridge" mode? And if so, does that mean that IP address a.b.c.d in the above log files is not my radio, but something at the local tower or beyond?

I am planning to discuss the problem with my provider once I become a bit more knowledgeable so I know what questions to ask and what data to collect.


PS -

When internet is working, if I type 192.168.1.20 into my browser's address bar, it does not connect with anything - it just times out. If I ping it, it times out.

If I type 10.0.0.1 into my browser's address bar, I get an AirOS login splash screen, but I don't know if that's for my radio or the local tower or ...

[nuttle]

192.168.1.1 is your router -- you can get an admin screen, check firmware version, etc. by pointing a browser to this address, but this is not the problem and so there's no need to do anything with this. If you log in to the router, you should be able to see the "default gateway" IP address for the next hop, which would be handy to have -- though you probably already have it from the tracert output.

I'd guess 10.0.0.1 is your radio -- you could try logging in and looking around to confirm, unless your provider supplied the radio and has changed the default info. I found this utility you could run that would help to confirm this in any event: <http://www.ubnt.com/support/downloads/utilities>.

Is a.b.c.d in your output 10.0.0.1, or 10.b.c.d? Anything that starts with 192.168. or 10. is not exactly a real IP address, in that these are used all over the place. These addresses are private only to each of a great many different segments of the internet, around the edges. For example, 192.168.1.1 is used probably millions of places, so there's no issue with pasting numbers in these ranges here.

The tracert outputs are interesting. Do you see the 60 second delay even when you use the option to suppress DNS name translation? If not, this all makes sense. You really want to do a tracert to something like chiefdelphi.com when things are working and note the difference when things are not working. The idea is to look one hop past where things get when you experience an outage.

I'd have to investigate further, but your radio could have a sort of bridge mode. However, I think it is probably using a 10. address on the over-the-air side. It suspect that it has an address that starts with 192.168. as well. The utility above should help you find this, and hopefully you can log in via this address to get some more info.

[Ether]

192.168.1.1 is the LAN side of my router. See the annotation in the tracert logs I included in my previous post.

I have full physical and network access to the router. I do my own router setup. That's how I was able to change the DNS server as mentioned in an earlier post.

The default gateway at the PC is 192.168.1.1 (my router). The router is set up to get an internet gateway IP dynamically. When it does so, it always gets 10.0.0.1 (the a.b.c.d in the tracert logs).

As mentioned earlier, when the problem occurs it does not go away when I bypass the router and connect directly to the radio. So the router is not the problem.

As mentioned in previous posts, I do not have any access to the radio. 10.0.0.1 gives me an AirOS login splash screen, but I do not have password access, so I cannot log in, and I can't even tell for sure if it is the AirOS radio on my roof, or an AirOS product somewhere downstream (like perhaps the local tower).

As mentioned in a previous post, I will try the "-d" tracert option next time the problem occurs. That has not happened yet.

When things are not working, there is a 60-second tracert delay even when doing tracert 10.0.0.1 (2 hops). As mentioned in the previous post, I do not know for sure if that means the problem is at the radio, or if the radio is acting as a "bridge" so that its IP does not appear in the tracert (is this what would happen with radio in bridge mode?).

The AirOS PS2 supports bridge mode. I do not know (and have no way to determine) if it is setup for that mode though.

I will try the Ubiquiti "Discovery Tool" and see what happens and report back here.

[nuttle]

I suspect the tracert delay for 10.0.0.1 is simply trying to reverse lookup 10.0.0.1 (and other addresses along the way) to a DNS text name -- the servers to do this are on the other side of the outage and so the requests/retries to do this are timing out. The actual ping times being reported are much lower, sub-second.

Something like "tracert -d 198.101.239.6" (chiefdelphi.com) would be more interesting, since this would show the hop that it going out; when you run it when things are not out and when they are, you'll have some interesting data I think.

The whole IP address thing is at what is called "layer 3" in a stack of protocols. Bridging is "layer 2" but an outage in a layer 2 link will take out a (non-redundant) layer 3 link, so the first step is to work out where you lose layer 3 connectivity, then possibly there's a next step of diving into anything more complicated than a point-to-point link at layer 2.

[Ether]

Does anyone know: if the radio on my roof is in "bridge" mode, will I or will I not see an IP address for it when I do a tracert ?

[Mark McLeod]

A bridge still has a unique addressable IP and will show up as a node in a trace route.

[Ether]

Originally Posted by Mark McLeod

A bridge still has a unique addressable IP and will show up as a node in a trace route.

Thank you.

Based on that, would it be correct to conclude that the IP address on the ethernet side of my radio (the side my router is connected to) must be 10.0.0.1, since that is the very next tracert hop after 192.168.1.1, which I know is my router ?

Would it also be correct to conclude that the radio is likely the problem, and not DNS, since I get the abnormal delay when doing only a 2-hop tracert (pc-->router-->radio), and 10.0.0.1 would not be sent to a DNS server for reverse DNS lookup?

[Mark McLeod]

From your descriptions, I'd certainly suspect the radio as the weak link.
Your WISP should be able to check the radio connectivity and performance statistics, but when in doubt they usually just replace the unit.

[nuttle]

Could be the radio, could just be marginal alignment -- the antenna is highly directional. Regardless, if it isn't yours, you want to have the provider take a look. I highly doubt any provider would let you get access to the admin UI for a tower or any other part of their network, so the fact that you can bring up a login screen is another strong indication that you are seeing the rooftop radio at 10.0.0.1.

The whole DNS thing basically boils down to the fact that, unless you run your own private DNS server on your personal LAN, any DNS operation has to contact a server on the Internet (read on the other side of the air hops). So, when things are not well, this: 1) isn't going to work; 2) is going to incur a large delay as the system waits for the response that is never coming.

The radio could be doing a number of things: routing, bridging, NAT, etc. The main point here is things are going wrong inside your provider's network. If they let you access the admin UI for the radio, you could probably tell a whole lot more. If there's an option to use your own radio, it might not be the worst way to go. If someone comes out to service your radio, maybe they will leave you with the login info (doubtful, but one never knows). They should be able to check the signal strength from their network and may know your signal isn't great or be able to check when you are on the phone. If any of this doesn't make sense, let me know...


I assume you tried ubnt/ubnt to log into the radio? Did the utility tell you anything?


Here's something on layer 2 (bridge) tracert: <http://www.fragmentationneeded.net/2...raceroute.html>. A bridge will typically have a layer 3 IP address, but it doesn't follow that it is going to show up in tracert.


If you are using Windows, try the pathping command -- it is more informative than tracert and possibly somewhat less confusing.