This guys explanation of "quote stuffing" seems incorrect to me:
"What benefit could there be to whomever is generating these extremely high quote rates? After thoughtful analysis, we can only think of one. Competition between HFT systems today has reached the point where microseconds matter. Any edge one has to process information faster than a competitor makes all the difference in this game. If you could generate a large number of quotes that your competitors have to process, but you can ignore since you generated them, you gain valuable processing time."
I don't think this is correct, since it's not a trivial matter to ignore your own quotes. Say there are two orders about to come in, and you place a third. The market data feed then tells you [order 1, order 2, order 3]. It does not tell you which order is yours. So you need to separately run code which remembers when you placed orders, and then guess that "look, order 1 and order 2 match the order I placed, I guess order 1 is mine, lets ignore it when doing the rest of our calculations". That's not fast code.
Besides, if your broker thinks you are DOSing the markets (as well as the broker's system), they will shut you down FAST.
My "stupidity, not malice" explanation: novices didn't throttle the oscillations in their code and feedback loops developed. Some were self-loops - basically, you just don't have the self-order detection code I described above, and try to trade based on your own actions. Some were loops between traders - trader 1 places an order, trader 2 cancels in response, trader 1 cancels in response to 2, trader 2 places an order in response to the cancel, etc. The band-saw patterns look like bidding wars on wildly undervalued stocks (e.g., one system places an order for accenture at 0.05, another at 0.06, another at 0.07, etc), which is actually a very good thing since it pushes prices back up to their proper levels.
Technically speaking, it may not be a trivial matter to ignore your own quotes, but it's not very difficult, either - if it takes M operations to handle your typical fully processed quote, and N operations to hash the quote and compare it to a table of quotes that you've delivered, then as long as M > N you've got some room for manipulation as long as you're able to pump enough bullshit into the system that everyone else has to deal with; I assume that in these situations M is usually substantially greater than N, probably on the order of at least 10:1 unless the quote is discarded as obviously irrelevant. It's just awful hard to imagine that the amount of processing done per quote in a HFT system can be less than a few hash table lookups...
I'd imagine the real difficulty would be making sure that the quotes you're putting in are a) relevant enough not to be immediately and trivially screened out by your competitors, b) not in danger of actually getting acted upon, and c) seemingly legitimate even in the face of after-the-fact scrutiny so that you avoid getting butt raped by the SEC for manipulation. a) and b) would likely be pretty easy to satisfy if you didn't have to worry about c), so...I don't know, this doesn't seem terribly implausible to me.
Then again, I don't know much about HFT, there may be factors in play that would make it a lot trickier to manipulate things in this way. And I'm certainly not convinced that this is what was behind the crash - "stupidity, not malice" is a good rule of thumb, and I'd have to see more evidence to assume that someone deliberately manipulated the markets here.
If you are doing high frequency trading, processing a quote isn't that hard either. It can't be, or you'd never respond fast enough. I suppose the best way to do it would be to ignore quotes X distance from the book, and then do your DOS X+0.01 from the book, though this might not be compatible with your regular strategy. Whatever DOS you want to do has to be compatible with your already existing strategy. I'm not saying it's 100% impossible, only that it isn't easy.
Additionally, your broker starts getting annoyed at you when your fill rate drops below 0.5% or so. If you DOSed the market, you wouldn't even be in that ballpark.
FWIW, looking at those graphs, it does appear that a lot of the extraneous action was happening right on the edge of the inside bid/ask.
Additionally, your broker starts getting annoyed at you when your fill rate drops below 0.5% or so. If you DOSed the market, you wouldn't even be in that ballpark.
That's definitely true, and I'm sure it would be a serious problem with doing anything of this nature.
Is there always a middleman that cares about what quotes you're pushing through, though, or do some people have closer connections that wouldn't be watched very closely?
Some larger funds become a broker/dealer, so they can cut their brokers out of the loop. I really doubt that anyone big enough to do that would be willing to risk their business on a DOS attack, however.
I fully agree that it's possible to do. I just find other explanations, like badly written algorithms, much more plausible.
Do you have a technical background in trading/exchanges? (FYI, I have a background in trading, but not HFT or connecting to exchanges) I think we should defer somewhat to the people who provide exchange data professionally.
Don't you think the people who work with this data (and its consumers) professionally know what's fast enough to run and what's not?
Don't you think they know what behavior gets you shut down, how quickly you get shut down (and therefore how long you can game the system), and what doesn't?
I'm not saying you're explanation isn't possible/plausible, or that you're skepticism is unfounded, but I've seen you post several times about Finance/HFT and I haven't usually found your commments/assertions were very substantive.
Thanks, I didn't know about his background (I had checked his profile/web pages but didn't see anything about the HFT stuff). The hilarious (perhaps stubborn) thing is that I still don't weight his information very highly.
Nanex is a data feed company that was started by Eric Scott Hunsader who is one bad ass programmer and, based on my limited interactions with him when I used to program against a previous API he designed, a decent person. He was able to deliver results back then that other people couldn't dream of, and Nanex (called GenerationV at the time I think) was his "build it all again from scratch based on what you've learned to date" data feed engine. It's interesting to see the name come up here. How big of a player are these guys?
They're not, really. Not on the HFT landscape, anyway. It's tempting to say they're being slept on, but the cream rises to the top VERY fast in HFT--there have been multiple instances of products brought to market and the company being purchased outright by the first would-be customer (although this is much, much more frequent on the networking and hardware sides).
I think the analysis and borderline-accusation of what is termed "insincere quoting" by the exchanges is... laughable. It doesn't speak well to the authors' knowledge of their industry. I've had an exchange disconnect my session for sending 1000+ orders per second by myself, and I would have been quite happy to execute any of them (or else they wouldn't have been sent). To be honest, I'm surprised 5000 is the biggest number they could find. I would have guess an order of magnitude higher.
What happened on 5/6 was that the NYSE finally learned that they aren't the boss in the equities markets anymore. Trying to slow down trading without regard for the other routing destinations was nothing but quaint.
You know those videos you see of suburban shoppers at 6:00a on Black Friday, rushing through the doors to get one of the three available Xbox 360s or whatever? The NYSE was the hapless security guard whose watch was slow telling everyone the store wasn't open yet.
I wrote too hastily when I said Hunsader was producing results at the time that others couldn't dream of; that was a sloppy exaggeration. But his work was very good.
Short conclusion: "In summary, quotes from NYSE began to queue, but because they were time stamped after exiting the queue, the delay was undetectable to systems processing those quotes. On 05/06/2010 the delay was enough to cause the NYSE bid to be just slightly higher than the lowest offer price from competing exchanges, but small enough that is was difficult to detect. This caused sell order flow to route to NYSE -- thus removing any buying power that existed on other exchanges. When these sell orders arrived at NYSE, the actual bid price was lower because new lower quotes were still waiting to exit a queue for dissemination."
"This situation led to orders executing against whatever buy orders existed in the NYSE designated market maker (DMM) order book."
This is actually the most cogent explanation I've yet heard.
When you can see the spread like that then something has gone terribly wrong. As I read it, that spread is between NYSE on the bottom and the rest of the market on the top.
So the NYSE has been blaming the other exchanges for their behavior, when this was really a NYSE problem.
However, it was only a NYSE problem because they were effectively being DOS'd by some of the other exchanges.
On the subject of HFT systems, we were shocked to find
cases where one exchange was sending an extremely high
number of quotes for one stock in a single second --
as high as 5,000 quotes in 1 second!
So the NYSE was right to blame the other exchanges, but they were right for the wrong reasons.
I don't think they mean that one exchange was DoSing another.
Quotes within a single exchange are disseminated immediately to all interested parties -- if one party was spamming thousands of quotes at a stock, that stream would be repeated for everyone with a feed from that exchange. There's no way to tell which party is sending it, so the only thing you can measure is the quantity being sent by the exchange. Thus it is technically accurate to say that "one exchange was sending an extremely high number of quotes". It's in the exchange's best interest to disseminate quotes as quickly as possible in order to attract order flow and thus collect fees.
On the other hand, inter-exchange feeds are federally mandated and used primarily to comply with Regulation NMS rules that prevent trades from executing at a price inferior to the NBBO (national best bid/offer). In other words, if I send a buy order to NYSE but NASDAQ has a lower price, NYSE is required to either reject the order or route it to NASDAQ for a fee (it's up to the customer which method is used). It's not necessary to send 5000 quotes per second to another exchange because the primary motivation is compliance. Generally, institutions will send their orders directly to the exchange with the best price in order to avoid the extra fee for having it routed by the exchange.
I may be way off on the inter-exchange feeds as I am not a professional, but there are undoubtedly many here who are so please correct me.
Can anyone say why this momentary "flash crash" was a big deal? Who cares if people or computers trade securities under market value for a short amount of time? It only affects the idiot that is selling them at that undervalued price. The value of the underlying company is unaffected and market forces will always quickly correct it. The people or firms that sold the securities at those prices are probably either fired or lost their own money and they chance it happens again is unlikely. Use buy and sell limit orders, you should be already.
Who cares if people or computers trade securities under market value for a short amount of time? It only affects the idiot that is selling them at that undervalued price. The value of the underlying company is unaffected and market forces will always quickly correct it.
Well, the stock price is the market value - if it's not then what is? Detecting errors as opposed to genuine changes in value is a non-trivial problem.
Use buy and sell limit orders, you should be already.
The flash crash happened so quickly that a lot of limit orders weren't executed, because the price dropped past the limits too fast.
If you had an order to sell something at $20, and the price dropped from $25 to $1 in seconds then there's a good chance there were no buyers for you at $20, but instead your trade might have been executed at $1. That makes a pretty big difference.
A limit order sell at $20.00 will never be executed at $1. It will never even be executed at $19.99. It will simply go unfilled until someone is willing to sell at $20.00 or higher.
These kinds of writings are why I read HN. The article is pretty technical, well written, and removes any political BS that you would most definitely hear on TV.
Notice that "circuit breakers" wasn't offered up as a solution to the problem.
While time stamping solves the problem of inconsistent quotes, it does not do anything against bogus trades or somebody accidentally adding an extra 0 when typing in their data.
A circuit breaker would prevent the absolute free-fall that happened during the flash crash. At that stage, nobody knew what the "real" price of a stock was and everybody was guessing, the algos could not handle the situation. If a circuit breaker was triggered, it would lead to a trading halt on the stock and an auction to set the price.
Time stamping quotes solves system errors, but not mistrades or market panic.
i don't know what all the big deal is about this 'flash crash'. they happen all the time in the futures markets. they are the risk premium for strategies that make a market automatically.
this is not a threat to the economy if that's what the concern is. it's only a 'threat' in conjunction with a perceived fear. in this particular case, it had to do with greece. in the 87 crash's case, it had to do with a collapse in the bond markets.
all high-frequency trading or program trades do is quicken the whole process. one can implement safeguards to make the whole thing slower -- or you can let the market participants take care of themselves and/or fix the underlying problems and ignore mass hysteria which will reset as soon as the absence of a rational reason for a correction is made clear.
i feel like the 1987+ 'crashes' are very different from the 1929 crash, etc. they are not in themselves a problem. so far they've always popped back.
which is not to say that our economy has been in good shape since 1987. but that there are deeper issues -- having very little to do with hft.
"What benefit could there be to whomever is generating these extremely high quote rates? After thoughtful analysis, we can only think of one. Competition between HFT systems today has reached the point where microseconds matter. Any edge one has to process information faster than a competitor makes all the difference in this game. If you could generate a large number of quotes that your competitors have to process, but you can ignore since you generated them, you gain valuable processing time."
I don't think this is correct, since it's not a trivial matter to ignore your own quotes. Say there are two orders about to come in, and you place a third. The market data feed then tells you [order 1, order 2, order 3]. It does not tell you which order is yours. So you need to separately run code which remembers when you placed orders, and then guess that "look, order 1 and order 2 match the order I placed, I guess order 1 is mine, lets ignore it when doing the rest of our calculations". That's not fast code.
Besides, if your broker thinks you are DOSing the markets (as well as the broker's system), they will shut you down FAST.
My "stupidity, not malice" explanation: novices didn't throttle the oscillations in their code and feedback loops developed. Some were self-loops - basically, you just don't have the self-order detection code I described above, and try to trade based on your own actions. Some were loops between traders - trader 1 places an order, trader 2 cancels in response, trader 1 cancels in response to 2, trader 2 places an order in response to the cancel, etc. The band-saw patterns look like bidding wars on wildly undervalued stocks (e.g., one system places an order for accenture at 0.05, another at 0.06, another at 0.07, etc), which is actually a very good thing since it pushes prices back up to their proper levels.