This type of issue can be incredibly annoying to deal with, because the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Hertzner says in the email that no response is necessary.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
>but we do expect you to check it and to resolve any potential issues.
That's the important part.
If they receive another one (or two, or a few) more abuse reports, they assume it is not fixed, and will expect a response then. Which ends up being annoying.
Well they are hetzner, they should understand the issue. I don't know if they would go through the hassle to verify by themselves by running a capture on a router leading to your server though...
I had a similar problem a good decade ago. Was running a game server for a while, and as it is with competitive games, some people get really angry when they lose. At some point I got DDoSed by a udp reflection attack, and as if that wasn't annoying enough, I got relayed an abuse complaint from a Brazilian ISP who claimed I was DoSing one of the servers in his network, which was in reality one of the zombies getting hit by the spoofed packets containing my server's address as the source. I tried to explain to them twice how this works, linked them two different articles about it, told them to look at the traffic to and from this server so that they would be able to verify the server is actually sending much more traffic in my direction, but no dice, they just sent and even angrier mail to hetzner. I quickly contacted hetzner after that incident telling them the story and they said it's fine and apparently fully understood the issue, which really shouldn't be surprising for an ISP, but the previous exchange with the other ISP made me question my sanity a bit.
This kinda thing is so rough, and I can really relate to the questioning of your own sanity part when everyone around you is insisting something works in a particular way that is just factually incorrect.
I got hired into a pretty old small technology company that has over a decade of tech debt, and "how the whole system works" is different depending on which engineer you're talking to. You have to do a context shift to a different engineering reality just to do basic improvements to the system. Lots and lots of built up confusion over years of incremental changes, some of which under pressure no doubt, some well intentioned half-refactors, some almost dead code...imagine your well established corporate morass and then give it a shoestring budget.
It's scrappy in its own way, but the threads where people advocate "don't worry about the tech debt, if the company succeeds they will have the budget to fix it" don't account for the middle ground of not having huge success but having enough success to continue indefinitely. I guess that could mean you could fix the problem over longer time spans, but people do t stay at orgs like this long enough for that to happen, because the job of fixing it is no fun and you can't just throw huge amounts of money at the problem.
> the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
Netflow data or equivalent? I'd assume any provider to have such records, at least in the short term. It can also be invaluable in debugging network problems post-hoc.
If there is technology and established protocols to prevent spoofing, but some ISPs refuse to follow these protocols, why should it be your burden to prove it wasn’t you?
Is it reasonable to allow people to get credit cards with your SSN, when it’s physically possible to confirm their identity when they present your SSN, but the bank is too lazy to do it, and we put it on you to show up and cancel the credit cards? And of course present 3rd party attestation that it wasn’t you who did this. Maybe even bring an alibi?
Some ISPs (often those of the "last-mile") allow outgoing packets whose source IP does not belong to their subnet. They have no rules in IPtables preventing packets that do not belong to the given subnet assigned to end customers. This is how spoofed packets enter Internet most of the time. The ISPs on upper tiers can not use such filters (even if they want to) because their networks are not strictly hierarchical like the networks of the "last-mile" ISPs and such filters will simply break the connectivity. The only way to significantly reduce spoofed packets is if all "last-mile" ISPs implement proper filtration.
>The legitimate answer would include some sort of real-world attestation about you from a trusted third party.
It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
>Maybe including a video call or something.
It's annoying to find someone at your provider who will take the time to do this. It's annoying to take my time to have to do this.
My point, overall, was that this is just a really annoying problem.
> It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
So it turns out at the network service level, anonymity has never been guaranteed. If I, as another chunk of the network, can't trust your chunk, it's going to get cut from accessing me.
There has to be some ability to establish baseline trust.
It's true due to the nature of what the network is.
In the abstract: if I own the infrastructure and someone uses that infrastructure to hurt someone, that someone who was hurt (or the parties who protect them) are going to come to me asking questions. If I just say "I don't know" and the law doesn't protect my willful ignorance, I'm at best enabling harm; I'm at worst socially or legally liable for negligence.
In the abstract, the systems of human governance recognize harm and seek to mitigate it.
So if I'm peered to a network using me as a bridge to do harm, I can't trust that network when the bad starts to outweigh the good. If I can't establish trust via human methods, I'm gonna cut that network off to protect myself.
(The Internet started as people who had working relationships with each other and grew out from there. Even though the web of connections is much larger and more indirect now, the whole thing is still at its core a human construct and beholden to human standards of conduct, because humans ultimately have their hands on the various plugs that are yankable).
OK, but in this specific example, what would you do in the shoes of Hetzner?
My understanding of the situation is, somebody in Network A is sending spoofed traffic to Network B. Hetzner receives abuse reports from Network B.
Should Hetzner either establish trust or cut off: Network A, Network B, or their customer?
Hetzner has or should have means to verify that their customer is not the one making port 22 requests. They are not the attacker. Network B is reporting the issue, they are also not the attacker. And Hetzner cannot identify Network A, at least not without Network B's cooperation. And even if Hetzner does identify and cut off Network A, the problem remains – Network A can still send spoofed traffic to Network B.
If they feel like it, they can reply to the abused party that they have misidentified the attacker (and why). It is up to the victim to then research further if they feel so inclined.
> The internet was broken 25 years ago and is still broken 25 years later. Spoofed source IP addresses should not still be a problem in 2024, but the larger internet community seems completely unwilling to enforce any kind of rules or baseline security that would make the internet safer for everyone.
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
The thing is, obviously, that the Internet isn't broken, it has incredible utility and reliability. If it was designed and operated to be perfect, then it would likely be massively broken quite often. It is the tolerance for mild brokenness that has contributed significantly to its robustness and utility.
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
Retaining functionality even in the face of mild-to-moderate borkedness is sorta the inciting goal for even making it in the first place, way back in the cold war days. Building on top of "How do we make a communications network that can handle a bunch of nukes" sets you up for a very resilient baseline :)
Spoofed and Randomized are not the same thing. Spoofing implies you are deliberately copying another machine’s MAC address in order to appear as that machine to the network.
Not especially, but most websites are protected by TLS, so the problem that DNS is insecure is less of a problem. It's mainly a coordination problem, you have up get a lot of people on board to design a new DNS-SECure, and then everyone would also have to adopt it. Which they did (create DNSSEC, that is), but it has not seen the desired adoption. The other one is DoH, DNS over https. It's not without issue either though. So there are efforts, it's just a hairy coordination problem.
* The integrity of registrar accounts that are the root of trust for most DNS zones (this was, last I checked, the overwhelming source of DNS corruption attacks),
* The security of one or more DNS lookups, depending (some CAs, like LetsEncrypt, do multi-perspective lookups), and
* The WebPKI Certificate Transparency system, which tracks the issuance of all certificates that Chrome and Mozilla will accept in a public ledger.
you can get certificates for an IP, but they're rare. How it generally works is the DNS server says Google.com is at w.x.y.z IP address, your browser talks to that, it gives you a certificate, (skipping a few cryptography steps for simplicity,) you computer checks the certificate coming from Google.com as being valid, without checking w.x.y.z, and then encrypts your connection and shows the green lock icon.
If the DNS server is bad, it'll return e.v.i.l as the IP, your browser will talk to that, but it can't give a certificate that your computer thinks is valid. so your protected from accidentally logging in to a fake bank website, but also you can't access the correct bank website, so there's still a denial of service problem.
The certificate authority (CA) that gives out the certificates has to verify you own the domain that you're asking for the certificate for. One method is to look up the IP, but as that's problematic if they get the wrong IP, they usually check that from multiple places all over the world.
It’s quite sad the only mail server out there which checks if you are allowed to use a email address is exchange. With all others you can set the from: header however you like.
Who cares whether it's the MTA that does it or a collection of daemons invoked by the MTA? Just get things configured correctly, and you should be gold.
Now as far as every other mail operator setting up their stuff right such that From spoofing is no longer feasible, well... Can't help ya there. I don't run my email to make money, so the incentive to adopt pathological configs for the sake of maximizing the number of users/Domains who can send from one IP ain't there.
Back in the day I would scan for DrDoS reflectors in a similar way, no hosting provider wants to get reports for port scanning so the source address of the scan would belong to an innocent cloud provider with a reputable IP that reflectors would happily send UDP replies to. The cloud provider would of course get a massive influx of complaints but you would just say that you aren't doing any scanning from your server (which they would verify) and they wouldn't shut your service off. The server sending out the spoofed scan packets is undetectable so you're able to scan the entire internet repeatedly without the typical abuse issues that come with it.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
This is nothing new. A few years back, I implemented a very basic firewall rule: if I received a TCP packet with SYN=1 and ACK=0 to destination port 22, the source IP would get blacklisted for a day. But then I started getting complaints about certain sites and services not working. It turned out that every few days, I'd receive such packets from IPs like 8.8.8.8 or 1.1.1.1, as well as from Steam, Roblox, Microsoft, and all kinds of popular servers—Facebook, Instagram, and various chat services. Of course, these were all spoofed packets, which eventually led me to adjust my firewall rules to require a bit more validation.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
> As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
Okay, looks like I will reply to a few of the comments to clarify things.
I’ll give a concrete, real example.
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
For your second scenario you should announce the /23 to both and each /24 to one of them. Usually you can also prepend your own AS, ISPs I've worked could also prepend for you with select communities.
I don't think your cases are good enough to allow anyone to spoof by default.
I said that we tried this.
They do not care about the announce. They care about what is injected into the routing table. We are announcing it but they see better path and drop us.
And they also said that it should work that way - just announce it somehow and it will work. Yes but no. It does not work.
In your first scenario, any connections established through the ISP-A's IP address would be routed back through the VPN connection that they came in on. If that server were to establish it's own connections to external resources, it would feasibly be able to use the 10g connection from ISP-B. It would not be able to dictate what source address was used with connections coming from ISP-B.
It could work the way OP described if they routed all outbound traffic via ISP-A regardless of source address, and ISP-A allowed spoofing. I think that's what they meant.
It is common practice for business subscribers (around UK) to get a /29
On the router we add a single /32 via the tunnel.
I think even the cheapest 100bucks business plans from many ISPs come with /28 or /29.
It is a complete waste because we had like 10 offices with 3-5 persons with laptops and NO servers. The common question from the ISPs is: Do you need some IPs? When we answer no, they give us /29.
>As a personal note, I know I’m a bit of an exception ...That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
"...and obviously, Pennywise, I must spoof ingress and egress..."
If it's your real IP, it's not spoofing, even if you send the packet through a different ISP than the one which gave you the IP. If you think about it: if you got an IP directly from ARIN you wouldn't have to send your packets through ARIN to make them legitimate.
Not really. Early IPv6 documentation kind of assumed that the vast address space would lead towards hierarchical addressing and that a multi-homed user would use addresses assigned by all of their ISPs, but at least in my experience, that doesn't really pan out --- if you have router advertisements from two different ISP prefixes, automatic configuration on common OSes (windows, linux, freebsd) will lead towards often sending traffic with ISP A through the router from ISP B, which doesn't really work well, especially if either or both ISPs run prefix filters. There's probably ways to make that style of multihoming work, but it's not fun.
Turns out, most multiphomed IPv6 users need provider indepdent addresses, just like with IPv4. And then you need to make sure your all your ISPs allow you to use all your prefixes. On the plus side, it's much more likely to get an IPv6 allocation that's contiguous and that you won't outgrow; so probably you only need one v6 prefix, and you may not need to change it as often as with v4.
The advantage of IPv6 is that can multiple addresses. This means that good way to organize network is to have machines use local provider addresses to access the Internet.
Then have ULA addresses for internal network. Those will be routed with tunnels and VPNs. That separates accessing the internet from internal network, and means that don't need to have routable address space.
The only people who would need own address space have data centers and routers.
Yeah, there are ways to make it work, for example by specifying source addresses or nets on the the routes. In openwrt it's a checkbox to tick on the upstream interfaces.
The “someone hates Tor relays” theory doesn’t sound worth the effort. This could be an entity running malicious relays, while also trying to unethically take down legitimate relays to increase the percentage of the network that they control.
This is almost certainly it. There’s a lot of head-sand-burying around here about just how easily an attacker with access to logs of a not-even-that-large segment of the nodes can gain visibility into individuals’ service access patterns.
Yeah. If you hate the tor network an easier thing to do is just to overwhelm it with traffic and degrade the service. Running some bittorrent downloads might be enough.
> Which means, if you just find one transit provider which doesn’t do BCP38 filtering… you can send IP packets tagged with any source IP you want! And unfortunately, even though the origins of BCP38 date back to 1998… there are still network providers 25 years later that don’t implement it.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
Cloudflare is probably enough. They already control enough ingress that their "checking the security of your connection" could actually mean something.
You'd have to find some way to make network providers care. Especially 'tier 1' transit providers and other networks of unusual size.
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
It seems like systems shouldn't report abuse (at least automatically) for single packet, no round trip, requests unless its reaching denial of service levels of traffic (and maybe these are). Like in particular for SSH there's no way thats even a valid connection attempt until some sort of handshake has occurred.
But since anyone can submit an abuse complaint, maybe server providers should actually check the abuse reports before triggering the "respond in 2 days or we suspend your server" or similar measure of their ToS.
I've had my main server thrown offline by a bogus abuse report claiming that they received an over 1Gbps DoS attack from my IP even though my server only has a 400 Mbps cap. Had a human actually read the report, they would've seen it was impossible and wouldn't have had to spend 2 days arguing with phone support on my holiday.
This is the IP version of SWATting, patent trolls, framing an innocent person, or using DMCA takedowns to remove the competition. It's basically weaponizing abuse-protection mechanisms to instead attack a target that is disliked. Interesting that the authorities can become a weak link here and be actively weaponized by unscrupulous actors to achieve their aims, but it's not really a new phenomena.
If they were smart and had their own relays it would also make them more likely to be selected proportional to how many other relays they took out. Looking at the number of relays and "bandwidth advertised" graphs on metrics.torproject.org it doesn't look like they've made much of a difference but it's interesting nonetheless.
To me, the worst part is that "Watchdog Cyber Defense," Spamhaus, Shadowserver, or some wannabe extortion artist like UCEPROTECT can submit millions of automated reports that hosts are de facto required to listen to lest their IP space be blacklisted.
There's no in-band solution to this problem, but out-of-band solutions might exist! For example: (1) Notify the destination ISP that you're receiving backscatter. (2) That ISP checks where the packets are coming from, and notifies that ISP. (3) Repeat step 2 until source is found. (4) Quarantine that part of the network until it behaves better.
People are sometimes shocked to learn that the Internet as a whole works because there is a subset of humanity that really, really likes overseeing the most over-the-top pipe game in existence.
But they're not arbitrarily-selected people: they're network administrators. Somebody spoofing IP addresses to the point of abuse reports is practically a personal insult to some of them. (Obligatory: https://xkcd.com/705/)
How difficult would it be to highjack this attack by sending these packages to everyone, so that providers like hetzner would get swamped with abuse emails? This way the attack would not work anymore. Either the honeypots would stop sending abuse emails, or the providers would filter those out.
Why not make ISPs responsible for blocking any such traffic. In the end it must originate from someone's network. And really they also should know who their peering partners are and what traffic should be allowed from there.
Internet where you send a packet over the wire and the network takes it and delivers it per RFC. Basically OG Internet. Network of networks of more or less trusted peers.
Or Internet where you need to requisition every connection/circuit be provisined before it is routed, which includes explaining why you need the service, and where any provider in the chain will deny you transit by default? You now must forge an intimate relationship with every middle box between you and the other endpoint. This process must be repeated by everyone on the network. Just operating as a middle box for someone else is now fraught with legal liability; as anything one of your transit's end up doing, you are now considered complicit in.
Both of these architectures of an Internet are equally valid and functional. The society that uses them however is completely different.
I prefer the former, warts and all, and lack of throat to throttle short of the asshat running the software on the other end, over the latter, because with the former at least, we're not creating power nexii to attract asshats to NetOps positions.
With the latter setup, sure, your spam problem has an ostensibly way higher barrier to entry in the form of having to create human trust networks, but the accretion of social power distinctly changes the culture of the net sector, attracting a type of personality that should never, ever be trusted to be given a yay/nay authority over other folks access to a network.
> don't see why verifying that an IP from your own subnet isn't claiming to be from outside it requires everything in your second paragraph.
You're looking at this as a collective update of firewall rules, and content to stop there. I'm more concerned about what that gesture turns into once it's significance percolates out to the public at large. Societies rearrange themselves around technical capabilities. Continue reasoning about how that constraint evolves into new obligations and legal precedents on the network operator, and you should eventually arrive at why I'm content to leave that particular bear unpoked.
What? This already exists and most ISPs already does it, bcp38.
They're only validating that the traffic that they originate use the IP addresses that they manage. So ISP that has an interface with 100.0.0.1/24 make sure that any ingress on that interface has source IP addresses in that range. If everyone does this spoofing becomes impossible and there's no cooperation or whatever you described required.
This is likely a very naive question, but how did the spoofer know his IP was participating as an internal Tor node? From what vantage point can that be seen? I imagine internal Tor nodes must know to connect to each other, so it must propagate through Tor. Is the attacker also a Tor node? Is it trivial to map all Tor hosts?
Tor has something called a consensus that lists all relays and their flags. Clients need this to know which relays to make a circuit with. For most clients, they select a relay labelled as a guard from this file which is where their traffic first enters Tor. Some countries realized they could just block all of these IP addresses and stop people using Tor, so there are unlisted guard nodes called bridges designed for censorship circumvention that you have to get by filling out a captcha or say sending an email.
gah, I remember once when I was working at a company, and we got an email complaining "stop hacking my systems!"
in the end, we had a load-balancer at .1 balancing a bunch of backend servers.
the complainer would have traffic to .1 that the load balancer would receive. Thing is, old or stale connections would drop out of the load balancer mapping table, and eventually the backend server connection would not get mapped, and the guy would get traffic direct from the backend server real ip address.
the traffic was actually generated by the customer, but these "unrelated" backend servers looked like they were hacking him.
They have posted several screenshots of discussions among people affected on various channels, including Mastodon and the official #tor-relays channel on IRC.
I don't understand what you're advocating for. Are you against Tor because there was a vulnerability (that has since been mitigated) which led to the deanonymization of users or are you against Tor because there was an illegal service that used it? By killing existing good relays you're making it easier for someone malicious to come in, add a bunch of relays to the network, and have those relays be more likely to be used. And it won't be a saintly do-gooder either, it'll probably be some guy trying to mitmproxy crypto exchanges to steal money.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.