I think one of the great ironies is that psychology is one of the hardest sciences but is treated so soft. I say this holding a degree in physics! (undergrad physics, grad CS/ML)
By this I mean that to make confident predictions, you need some serious statistics, but psych is one of the least math heavy sciences (thankfully they recently learned about Bayes and there's a revolution going on). Unlike physics or chemistry, you have so little control over your experiments.
There's also the problem of measurements. We stress in experimental physics that you can only measure things by proxy. This is like you measure distance by using a ruler, and you're not really measuring "a meter" but the ruler's approximation of a meter. This is why we care so much about calibration and uncertainty, making multiple measurements with different measuring devices (gets stats on that class of device) and from different measuring techniques (e.g. ruler, laser range finder, etc). But psych? What the fuck does it even mean "to measure attention"?! It's hard enough dealing with the fact that "a meter" is "a construct" but in psych your concepts are much less well defined (i.e. higher uncertainty). And then everything is just empirical?! No causal system even (barely) attempted?! (In case you've ever wondered, this is a glimpse of why physicists struggle in ML. Not because the work, but accepting the results. See also Dyson and von Neumann's Elephant)
I've jokingly likened psych to alchemy, meaning proto-chemistry -- chemistry prior to the atomic model (chemistry is "the study of electrons") -- or to astrology (astronomy pre-Kepler, not astrology we see today). I do think that's where the field is at, because there is no fundamental laws. That doesn't mean it isn't useful. Copernicus, Brahe, Galileo (same time as Kepler; they fought), and many others did amazing work and are essential figures to astronomy and astrophysics today. But psych is in an interesting boat. There are many tools at their disposal that could really help them make major strides towards determining these "laws". But it'll take a serious revolution and some major push to have some extremely tough math chops to get there. It likely won't come from ML (who suffers similar issues of rigor), but maybe from neuroscience or plain old stats (econ surprisingly contributes, more to sociology though). My worry is that the slop has too much momentum and that criticism will be dismissed because it is viewed as saying that the researchers are lazy, dumb, or incompetent rather than the monumental difficulties that are natural to the field (though both may be true, and one can cause the other). But I do hope to see it. Especially as someone in ML. We can really see the need to pin down these concepts such as cognition, consciousness, intelligence, reasoning, emotions, desire, thinking, will, and so on. These are not remotely easy problems to solve. But it is easy to convince yourself that you do understand, as long as you stop asking why after a certain point.
And I do hope these conversations continue. Light is the best disinfectant. Science is about seeking truth, not answers. That often requires a lot of nuance, unfortunately. I know it will cause some to distrust science more, but I have the feeling they were already looking for reasons to.
As someone who did statistics and psychology, I'm very surprised by this take, for a few reasons:
1. Many of the early pioneers in statistics were psychologists.
2. The econ x psych connection is strong (eg econometrics and psychometrics share a lot in common and know of each other)
3. Many of the people I see with math chops trying to do psychology are bad at the philosophy side (eg what is a construct; how do constructs like intelligence get established)
As in many fields, the strength of statistical practices continually improve. And the parent comment has it right about the difficulty. In physics its much easier to ensure your sample is representative (heterogeneity is huge), and you have no way of ensuring that last sample of 100 participants have the same characteristics as your next sample of 100.
I'm sorry, maybe a didn't communicate clearly. SubiculumCode commented the main part of what I wanted to convey, so I won't repeat.
1. Yes! But that's doesn't exactly change things, in fact, it's part of my point. A big part of why this happened (and still does!) is due to the inherent difficulties and the lack of existing tools. If you ever get a chance, go look at a university physics lab. Even Columbia's nuclear reactors (fusion or fission!) and I think many will be surprised how "janky" it looks. It's because they build the tools along the way, not because lack of monetary resources (well... That too...) but because the tools don't exist!
My critique about the psych field is that this is not more embraced. You have to embrace the fuzziness! The uncertainty. But the field is dominated by people publishing studies that use very simple statistical models, low sample sizes, and put a lot of faith in unreliable metrics with arbitrary cutoffs (most well known being the p-value). Many people will graduate grad school without a strong background in statistics and calculus (it's also easier to think this is stronger than it is. And of course, there are also plenty who would be indistinguishable from mathematicians. But on average?). There are rockstars in every field, even when not recognized as rockstars. But it matters who the field follows.
And I must be absolutely clear, this is not to say that work and those results are useless. Utility and confidence are orthogonal. You might need 5 sigma confidence verified by multiple teams and replicated on different machines to declare discovery of a particle but before that there's many works published with only a few sigma and plenty of purely theoretical works. (Note: in physics replication is highly valued. Most work is not novel and it is easy to climb the academic ladder without leading novel research. This is a whole other conversation though) This is why I discussed Copernicus and Brahe but would not call them astronomers. That's not devaluing them, but rather nothing a categorical difference due to the paradigm shift Kepler caused. Mind you, chemistry even later!
2. I specifically mention economists (my partner is one). I could highlight them more but I feel this would only add to confusion. I believe those close to the details will have no doubt to their role. I don't want to detract from their contribution but I also don't want to convolute my message which is already difficult to accurately convey.
3. I think this is orthogonal. I'm happy to bash on other fields if that makes my comment feel less like an attack rather than a critique (or wakeup call). I'm highly critical of my own community (ML) and believe it is important that we all are *most* critical of our own tribes, because even if we don't dictate where the ship goes we're not far removed from those that do. I'll rant all day if you want (or check my comment (recent) history if you want to know if I'm honest about this). I'll happily critique physicists who can solve high order PDEs in their sleep but struggle with for loops. Or the "retired engineer" trope that every physicist knows and most have experienced.
But it is hard to be critical while not offending (there's a few comments about this too). Maybe you disagree with my critique but I hope you can reread the end of comment as hopeful and encouraging. I want psychology to be taken more seriously. But if the field is unable to recognize why the other fields are dismissive of them, then this won't happen. Sure, there's silly reasons that aren't reasonable, but that doesn't mean there's no reason.
It is a matter of fact that the (statistical) confidence of studies in psychology is much lower than those of the "hard" sciences (physics, chemistry, and yes, even biology (the last part is a joke. Read how you will)). In part this is due to the studies and researchers themselves, but the truth is that the biggest contributing factor is the nature of the topic. That is not an easy thing to deal with and I have a lot of sympathy for it. But how to handle it is up to the field.
Not all psych is as jurasic as you describe. For example cofnitive psychology has better theorios with more predictive power than personal psychology that is often picked at. Sure journals are flooded with underpowered studies and studies with very little links to theory, and there is still massive gaps in scientific knowledge but core consturcts are solid.
Certainly. It's difficult to talk in general because there's always exceptions. I also think it's very easy to misunderstand my comment. I didn't think psych is useless. But science isn't so much about having the right answer as your confidence in your model of "the answer". It's not binary. My point is that when working in a field where it's very difficult to have high confidence to normalize it and become over confident in results because everyone else is working with similar levels. I have (and made) similar criticisms of my own field, ML.
> core consturcts are solid.
- which concepts?
- how solid?
- how do you know they're solid?
You don't have to answer, this isn't a challenge. But these are questions every good scientist should be constantly asking about their own work and their own field. That's the "trust but verify" part. It's why every scientist should constantly challenge authority. Because replication is the foundation of science and you don't get that without the skepticism.
Lots of great points. I would start with semiotics, including during the problem definition phase, otherwise you could easily end up lost in language without the slightest clue of the predicament you're in.
Epistemology is also useful, because it might allow one to wonder if the problem space is non-deterministic (or not discoverable as).
Psychology is not inherently treated as soft, it's jusst that its human element attracts intuitive people much more than rational ones. If nore rationally minded people took up the study and research of psychology fields, more hard stuff would come to the front, although soft stuff is hardly behind in intelligence.
The parallels to ML you drew are on point. ML has this tendency to oversimplify complex phenomena with a easy to produce datasets, because that's what ML folks do, they find a smart and easy way to create a dataset and then they focus on the models. But this falls apart pretty quickly when you go into societal problems, such as hate speech or misinformation. Maybe there it would be nice to have some rigor and theory behind the dataset instead of just winging it. I am working on societal biases in NLP and I feel confident that majority of the datasets used have practically no validity.
I love ML. I came over from physics because it is so cool. But what I found odd is that few people were as interested in the math as I was, and even moreso were dismissive of the utility of the math (even when demonstrated!). It's gotten less mathy by the year.
My criticisms of ML aren't out of hate, but actually love. We've done great work and you need to be excited to do research -- and sometimes blind and sometimes going just on faith --, because it is grueling dealing with so much failure. Because it's so easy to mistake failure for success and success for failure. It's a worry that we'll be overtaken by conmen, that I have serious concerns that we are not moving towards building AGI. But it's difficult to criticize and receive criticism (many physics groups specifically train students to take it and deal with this seemingly harsh language. To separate your internal value from your idea). So I want to be clear that my criticism (which you can see in many of my posts) is not a call to stop ML or even slow it down, but a desire to sail in a different direction (or even allow others to sail in those directions as opposed to being on one big ship).
I do know there are others in psych with similar stories and beliefs. But there's a whole conversation about the structure of academia if we're going to discuss how to stop building mega ships and allow people to truly explore (there will always be a big ship, and there always should! But it should never prevent those from venturing out to explore the unknown. (Obviously I'm a fan of
Antoine de Saint-Exupéry lol ))
A tangent but if you're frustrated with the dismissal of math in ML I think you would probably enjoy diving in the Reinforcement Learning subfield; although some people tend to call it experimental math ;]
ML applied to robotics/embodiment is another fun topic, where physics is very relevant in every step of it. A bit harder to dataset-hack when a physical system is crumbling in front of you.
Oh I've found some areas. I'm particularly fond of explicit density models and normalizing flows. I do like RL too and was very surprised to hear that the math was "confusing" when it just seemed like weird notation. But it's hard to get works through review because even if you get SOTA for your architecture some asshat will ask you why you aren't better than the architectures being worked on by tens of thousands of people and with compute budgets 1000x yours. Though it's better if you find a very small niche so the reviews you get are more likely to actually know the bare basics of the topic.
> I do think that's where the field is at, because there is no fundamental laws
I think that there are some fundamental laws, which are based on perceptions and their interplay. Speaking very briefly, there are five classes of perceptions: emotions, wishes, thoughts, beliefs, and body sensations. The division of perceptions into these classes is not a result of purely intellectual exercise or idle theorizing. If one starts carefully and diligently observe contents of their mind, these contents will delaminate into such classes naturally. Try that yourself and you will see it as a fact.
Further introspection and assessment of arising perceptions would reveal some interesting patterns: there are two mutually incompatible kinds of emotions, and two mutually incompatible kinds of wishes, and so on.
One could make observations about the interplay of these perceptions and their dynamic. For example, if someone in some specific situation experiences an emotion X and a wish A (with some specific qualities), they can either realize that wish or choose not to do so. Each choice will lead to some changes in the contents of the mind: emotion X is replaced by emotion Y, and/or wish A is changed into wish B, and so on. Gather enough observations of that kind, and you could eventually formulate some hypotheses about possible general laws of perceptions (e.g. make a prediction that emotion X will change to emotion Y in specific set of circumstances).
These hypotheses could be verified by training several people to observe the same five classes of perceptions in the same manner. Arrange various test events for them, record their choices and outcomes of these choices (described in the same language of five classes of perceptions).
If most of them report more or less the same subjective outcomes (without being told about hypothesis and predictions, of course), that's the first step of verification for a possible general law of perceptions.
The second step of verification would be to apply brain imaging to those trained people, allowing us to map emotions X,Y,Z to some distinct patterns of the brain activity. After that do the same experiments with people who are untrained: arrange same test events for them while recording their brain activity. If changes in their brain patterns match for emotion X changed to emotion Y, that would be an objective confirmation of hypothesis formulated earlier.
By this I mean that to make confident predictions, you need some serious statistics, but psych is one of the least math heavy sciences (thankfully they recently learned about Bayes and there's a revolution going on). Unlike physics or chemistry, you have so little control over your experiments.
There's also the problem of measurements. We stress in experimental physics that you can only measure things by proxy. This is like you measure distance by using a ruler, and you're not really measuring "a meter" but the ruler's approximation of a meter. This is why we care so much about calibration and uncertainty, making multiple measurements with different measuring devices (gets stats on that class of device) and from different measuring techniques (e.g. ruler, laser range finder, etc). But psych? What the fuck does it even mean "to measure attention"?! It's hard enough dealing with the fact that "a meter" is "a construct" but in psych your concepts are much less well defined (i.e. higher uncertainty). And then everything is just empirical?! No causal system even (barely) attempted?! (In case you've ever wondered, this is a glimpse of why physicists struggle in ML. Not because the work, but accepting the results. See also Dyson and von Neumann's Elephant)
I've jokingly likened psych to alchemy, meaning proto-chemistry -- chemistry prior to the atomic model (chemistry is "the study of electrons") -- or to astrology (astronomy pre-Kepler, not astrology we see today). I do think that's where the field is at, because there is no fundamental laws. That doesn't mean it isn't useful. Copernicus, Brahe, Galileo (same time as Kepler; they fought), and many others did amazing work and are essential figures to astronomy and astrophysics today. But psych is in an interesting boat. There are many tools at their disposal that could really help them make major strides towards determining these "laws". But it'll take a serious revolution and some major push to have some extremely tough math chops to get there. It likely won't come from ML (who suffers similar issues of rigor), but maybe from neuroscience or plain old stats (econ surprisingly contributes, more to sociology though). My worry is that the slop has too much momentum and that criticism will be dismissed because it is viewed as saying that the researchers are lazy, dumb, or incompetent rather than the monumental difficulties that are natural to the field (though both may be true, and one can cause the other). But I do hope to see it. Especially as someone in ML. We can really see the need to pin down these concepts such as cognition, consciousness, intelligence, reasoning, emotions, desire, thinking, will, and so on. These are not remotely easy problems to solve. But it is easy to convince yourself that you do understand, as long as you stop asking why after a certain point.
And I do hope these conversations continue. Light is the best disinfectant. Science is about seeking truth, not answers. That often requires a lot of nuance, unfortunately. I know it will cause some to distrust science more, but I have the feeling they were already looking for reasons to.