I work on security at PostHog. We resolved these SSRF findings back in October 2024 when this report was responsibly disclosed to us. I'm currently gathering the relevant PRs so that we can share them here. We're also working on some architectural improvements around egress, namely using smokescreen, to better protect against this class of issue.
Here's the PR[0] that resolved the SSRF issue. This fix was shipped within 24 hours of receiving the initial report.
It's worth noting that at the time of this report, this only affected PostHog's single tenant hobby deployment (i.e. our self hosted version). Our Cloud deployment used our Rust service for sending webhooks, which has had SSRF protection since May 2024[1].
Since this report we've evolved our Cloud architecture significantly, and we have similar IP-based filtering throughout our backend services.
> The Deception of the Challenged Representations and Unlawful Marketing &
Sale of the Products. The Challenged Representations misled reasonable consumers into believing the Products possessed certain AI qualities, capabilities, and features, they simply do not have. As a result, Apple charged consumers for Products they would not have purchased, or at least not at its
premium price, had the advertising been honest. Beyond exploiting unsuspecting consumers, Apple also gained an unfair advantage over competitors in the market who do not tout non-existent AI features, or who actually deliver them as promised.
I utilized SSE when building automatic restart functionality[0] into Doppler's CLI. Our api server would send down an event whenever an application's secrets changed. The CLI would then fetch the latest secrets to inject into the application process. (I opted not to directly send the changed secrets via SSE as that would necessitate rechecking the access token that was used to establish the connection, lest we send changed secrets to a recently deauthorized client). I chose SSE over websockets because the latter required pulling in additional dependencies into our Golang application, and we truly only needed server->client communication.
One issue we ran into that hasn't been discussed is HTTP timeouts. Some load balancers close an HTTP connection after a certain timeout (e.g. 1 hour) to prevent connection exhaustion. You can usually extend this timeout, but it has to be explicitly configured. We also found that our server had to send intermittent "ping" events to prevent either Cloudflare or Google Cloud Load Balancing from closing the connection, though I don't remember how frequently these were sent. Otherwise, SSE worked great for our use case.
Generally you're going to want to send ping events pretty regularly (I'd default to every 15-30 seconds depending on application) whether you're using SSE, WebSockets, or something else. Otherwise if the server crashes the client might not know the connection is no longer live.
The way I've implemented SSE is to make use of the fact it can also act like HTTP long-polling when the GET request is initially opened. The SSE events can be given timestamps or UUIDs and then subsequent requests can include the last received ID or the time of the last received event, and request the SSE endpoint replay events up until the current time.
You could also add a ping with a client-requestable interval, e.g. 30 seconds (for foreground app) and 5 minutes or never (for backgrounded app), so the TCP connection is less frequently going to cause wake events when the device is idle. As client, you can close and reopen your connection when you choose, if you think the TCP connection is dead on the other side or you want to reopen it with a new ping interval.
Tradeoff of `?lastEventId=` - your SSE serving thing needs to keep a bit of state, like having a circular buffer of up to X hours worth of events. Depending on what you're doing, that may scale badly - like if your SSE endpoint is multiple processes behind a round-robin load balancer... But that's a problem outside of whether you're choosing to use SSE, websockets or something else.
To be honest, if you're worrying about mobile drain, the most battery efficient thing I think anyone can do is admit defeat and use one of the vendor locked-in things like firebase (GCM?) or apple's equivalent notification things: they are using protocols which are more lightweight than HTTP (last I checked they use XMPP same as whatsapp?), can punch through firewalls fairly reliably, batch notifications from many apps together so as to not wake devices too regularly, etc etc...
Having every app keep their own individual connections open to receive live events from their own APIs sucks battery in general, regardless of SSE or websockets being used.
I also used SSE 6 or so years ago, and had the same issue with out load balancer; a bit hacky but what I did was to set a timer that would send a single colon character (which is the comment delimiter IIRC) periodically to the client. Is that what you meant by “ping”?
> The Secure Enclave randomizes the data volume’s encryption keys on every reboot and does not persist these random keys, ensuring that data written to the data volume cannot be retained across reboot. In other words, there is an enforceable guarantee that the data volume is cryptographically erased every time the PCC node’s Secure Enclave Processor reboots.
Intel and AMD server processors can use DRTM late launch for fast attested restart, https://www.semanticscholar.org/paper/An-Execution-Infrastru.... If future Apple Silicon processors can support late launch, then PCC nodes can reduce intermingling of data from multiple customer transactions.
> The server can't afford
What reboot frequency is affordable for PCC nodes?
I wonder what impact this will have on Mozilla's OpenSSH configuration guide[0], which currently specifies `chacha20-poly1305@openssh.com` as its primary cipher. Should that be dropped to rely solely on AES ciphers?
I'm the CTO of a popular Secrets Management platform. It's fair to say that I personally have a lot of experience with secrets and requirements around them, based on conversations with customers.
The primary missing feature here seems to be multiline support. That's super common for keys, certificates, and other JSON configuration. Based on the Node PR, they appear open to adding that later (big +1 on shipping incrementally).
The other missing feature that folks tend to heavily rely on is variable expansion. For that to truly work well, I recommend using a holistic platform like Doppler. That allows for expansion/referencing across environments and projects, like when you have multiple independent services that need access to the same set of secrets (e.g. database creds, error reporting tool, stripe key, etc). You can then update the secret once and have the change propagate to all the places it's used.
Lastly, I'd be remiss if I didn't mention the Doppler CLI and our own fairly unique support for .env files. We've traditionally taken a dim view of .env files because they represent a static, long-lived collection of sensitive information that lives offline. Often, these get checked into a git repo. This is probably fine for personal projects, but a major issue for companies and the security aware. However, .env files are a pseudo standard and folks want a way of continuing to consume their secrets via them. Our CLI's approach is to mount a named pipe that we can write secrets to when a reader attaches. That allows us to limit the amount of times the "file" can be read (e.g. once), it guarantees that the file's contents are unavailable once the application process dies, and it uses the same open/read interface as a standard file.
In all, this is an exciting development for Node. I'm glad to see more standard features make it into core and hope that multiline support is a fast follow.
When this deal was originally announced, Adobe's stock took a ~10% hit. Now that the deal is getting blocked, their stock is again taking a hit? I certainly can't claim to understand the public markets.
The first hit was from investors that didn't like the deal... the second is from investors that liked it. That the former won't come back makes sense and so does the market. Well at least in this case.
You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.
At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.
At Doppler we have a strong culture of engineering excellence, thorough PR review, security, automation/CI, and helping each other succeed. We give engineers 30% time to work on things they think are important and are remote first (but also have an office in SF). Our engineering team hangs out in Discord all day to make pair programming/debugging easy. It helps that our CEO used to be an engineer.
Our process never includes ghosting anybody, even when we pass on a candidate, so something must have gone very wrong here. Could you email me at thomas@[company domain]? I'll look into this further.
Co-founder/CTO of Doppler here. I'd agree that Doppler currently focuses very heavily on UX. We aim to fit into developer workflows, regardless of where devs run their code. This includes local development (macOS/Windows/Linux), CI/CD, Kubernetes, Vercel, and just about anywhere else. We've found that security tools that are difficult to use end up being worked around by developers, which ultimately decreases an org's security posture (see HashiCorp Vault). This does come with some tradeoffs- specifically that you must trust Doppler with your secrets, given our tokenization model[0].
We do recognize that the current security tradeoffs of Doppler aren't going to satisfy everyone. For Enterprises, we offer Enterprise Key Management (EKM), which allows orgs to encrypt their secrets using a cloud KMS. Of course, this still doesn't satisfy every concern. And so, for customers requiring additional security guarantees, stay tuned!
I wonder why is it difficult to add an extra password on top of the access key.
What I mean is you can have another password or hash (separate to normal auth pass) which is only used to encrypt at browser and decrypt at the time of usage(through env load packages or your custom doppler command. This extra password is never sent or stored at doppler server.
