Suppose that I have four nodes, each with 128 GB of RAM, can I create like a 64 GB RAM disk on each node, and then use it as a Ceph cache pool (akin to ZFS ARC, but not because Ceph doesn’t have ARC like ZFS does)?
I have no idea, but per their docs Cache tiering has been deprecated so not sure how useful it would be over time Cache Tiering — Ceph Documentation
This talks about a cache pool.
From the docs;
“Use a pool of fast storage devices (probably SSDs) and use it as a cache for an existing slower and larger pool.
…
Read-only pool, weak consistency
We have an existing data pool and add one or more read-only cache pools. We copy data to the cache pool(s) on read. Writes are forwarded to the original data pool. Stale data is expired from the cache pools based on the defined policy.
This is likely only useful for specific applications with specific data access patterns. It may be a match for rgw, for example.”
Using tiering might be deprecated due to the lack of a maintainer, but I wonder if there might still be a way to still use this kind of a cache pool.
If you can map a ram disk to each host, it might work, but probably not supported. Sounds like ceph is doing things similar to how vSan works, high speed ssd going to slower speed spinning. But even that is losing favor for vSan Express using nvme. Longhorn pretty much requires ssd or nvme to keep things reasonable, and at that I haven’t tested it yet as other things are getting in the way of testing Harvester (Longhorn storage on single consumer nvme).
1 Like
That’s what I was thinking as well, but what I wasn’t as sure how to get it work with the slower HDD based Ceph pool.
I found an old thread that talked a little bit about it, but from there, they explicit mentioned that they use Linux’s native caching to deal with that, rather than an explicit ARC or ARC-like set up.
The deprecated (due to a lack of a maintainer) cache tiering was probably the closest thing to it (if you set it as a read-only cache), but my thinking was that if I have a four node, six 3.5” HDD/OSDs per node and it has to hit the drives each time there is a request for data rather than having something akin to the ZFS ARC, then it can have significant adverse performance impacts.
So my thought was that before I go spending money on a new-to-me system, that I should probably do some research to find out how I can get like a “Ceph ARC” going, that way I don’t end up wasting my money on something that wouldn’t work because I didn’t study/research this prior to buying the new-to-me system.