EPB Device pushing bad DNS settings?

From: David White 
This is a bit OT, but I will ask it anyway. I went to a client's office
first thing this morning due to internet being "down". I quickly determined
it was a DNS issue. The client didn't have their login to their D-Link
router. I reset to defaults, rebuilt the network, and upgraded firmware.

Internet was working again for somefolks. 20 minutes later, some folks were
down. I took out my spare Linksys router, built the network on that,
plugged everything in.

Everything worked. 10-15 minutes later, some (but not all) users already
reporting issues again. I determined everytime a PC was "down" that for
some reason its primary DNS server was set to (which was
pushed to it by the router). That was a Cisco device (I checked) obviously
managed by EPB - basically the next hop from the client's router, although
the external IP address was correctly assigned to the client's router.

(The client's network was by the way)

I told the client even before I arrived that I had to be out by 10:00am
sharp. Under a time crunch, and nearing 10:30, I finally just went around
to eaxh machine and manually entered as the primary DNS server (and
yes, I assigned Google's DNS servers to both the d-link and then the spare
linksys router too).

But for some reason that stupid address kept getting assigned
via DHCP as the DNS server.

Ultimately,  this sounds like an issue EPB needs to deal with. Thoughts? Am
I missing something obvious?

Issues started happening a few days ago...

=============================================================== From: Dave Brockman ------------------------------------------------------ -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 What model Cisco router are we talking about? They usually prefer direct ONT ports for Internet access. Regards, dtb - -- "Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network." RFC 1925 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTrNRsAAoJEMP+wtEOVbcdBCYIAJmeMDTullrAXdz3/LKxXC5L r3d5CIq95zZYNrXdduNgYbfmB4S2xF/w0lXSs4S5B+fos9v5u77sV2Wdy2aaWK1b DYN6OL61eqm8CFbSXIXBVEhiawGBNNR/Q72/TNpsHWwMMheAi08ZiJGoTta0aDtk fYO5Ld07mNe4pA1oXU3hW6jGLXZZwUHf0BlRG6A6fXh/oYvs+/Z0/Iy2WBwwjVYj 43kfuZ0tx8nhYZll+hkSSyOSgmmZQ+Y3NM5iWuTfze0OXX01jkVh5PK0aPqqEbOY vLfvd8/+7LMx9tb4ErGnpIFw0ZQZo1lP/o6kM0Xb3C5a7rnZ5/7Op+6KuXkqF6k= =UwJf -----END PGP SIGNATURE-----

=============================================================== From: Andrew Rodgers ------------------------------------------------------ Your router should push any DHCP through the firewall, ever. You have a rogue device somewhere on that network.

=============================================================== From: Andrew Rodgers ------------------------------------------------------ *shouldn't

=============================================================== From: John Alcock ------------------------------------------------------ Evening, I am one of the engineers at EPB. With the exception of SmartHome subscribers, EPB does not control subscriber routers. The smarthome subscribers use a NetGear router. (I did not choose the router) There has been some odd issues with DNS issues lately. This has been due to a DNS DDOS attack that has been going on. Basically, infected subscriber machines have been DOS's the EPB caching servers. We have identified subscribers with infected machines as well as some subscriber routers that have an open DNS servers and having them fix the problem. EPB has also fired up more DNS resources while fixing this problem. Back to the issue at hand. I suspect one of a couple of things in your office. One - a rouge DHCP server in the network. This could be a malicious machine (Virus, etc). Two - Some one in the office has a wireless card in there machine and has bridged it with the network card. Either on purpose or by accident. Three - the router has been compromised. My suggestion would be to wireshark one of the machines getting the rogue dns server and get the mac address of the offending device. This should allow one to track down the dns/dhcp server causing the trouble and stomp it to death. John

=============================================================== From: Stephen Kraus ------------------------------------------------------ Yeah, it'd be really odd for DNS to be going bad unless something else is feeding bad DNS info to the network.

=============================================================== From: David White ------------------------------------------------------ This is helpful. Thanks. I suspected it could be a rogue device on the local network. I know one of the machines was pretty severely infected. When I left, I had only had Malwarebytes running for 5 minutes and it had already found a number of items. I will take a look at the model of the Cisco device, but I know it's hardware EPB installed. Probably not a router, but I did still find it odd that when I went to that IP address of 192.168.15 in a browser, it took me to a Cisco login page similar to what you'd find on managing a switch like the Cisco SG series. Anyway, I will focus my efforts on the local network. I know it wasn't a compromised router, since I swapped it out with one of my own and the symptoms were still there. 'Preciate it.

=============================================================== From: John Alcock ------------------------------------------------------ David, I know what it is. I will email you directly. John

=============================================================== From: John Alcock ------------------------------------------------------ Morning, So this issue is with EPB, but indirectly. I mention this since I know many of you have clients in the EPB area. EPB uses a separate router and switch for hosted pbx. This allows us to control phone quality end to end. In this particular case, the subscriber hooked their network to our switch essentially bridging the networks. One can imagine the issue with multiple dhcp servers on a single network. This can cause odd issues including this one. So, I stand corrected. We do use Cisco (early hosted pbx), and now the Ubquiti routers for hosted pbx. BTW: The ubquiti is basically a mini linux box running debian. We are doing some pretty cool things with this box. Check the box out at sometime. John

=============================================================== From: Andrew Rodgers ------------------------------------------------------ John, awesome that you guys are using the Edgerouter. I have experienced this before. Basically a VLAN cross-connect. Andrew Rodgers GIGTANK 2014 Technologist-in-Residence 256-508-7610

=============================================================== From: Dan Lyke ------------------------------------------------------ On Fri, 27 Jun 2014 09:24:13 -0400 John Alcock wrote: Aha! I knew that telephones were the devil's work and not to be trusted! And the only thing that you ever get on a wired telephone is solicitors. Dan

=============================================================== From: Steve McKnelly ------------------------------------------------------ LOL We just had our hosted PBX installed last August, with Cisco H/W, and it's now already the "early hosted pbx". :) Steve

=============================================================== From: David White ------------------------------------------------------ Yeah, John Alcock is awesome. #JustSaying In my original post, I didn't include the fact that the customer told me they had carpenters in the area a few days ago, which mysteriously was around the time their internet issues started happening, and that they "might have unplugged a few things and plugged them back in." I traced some of the wires by hand (including the uplink wire to the client's router), and didn't see any issues that jumped out at me, so I didn't think much of it. But that must have been what happened. On Sun, Jun 29, 2014 at 11:36 AM, Steve McKnelly wrote: