How does that work for non-profit web crawls like the Common Crawl Foundation?

Common Crawl is text-only.

How is this different from x86_64 and aarch64? Or even the various Alpha and Mips64 chips.

For example x86_64 has v1, v2, v3 and v4 baselines, this tells you which instructions they support (e.g. v4 has AVX-512, v3 has AVX2, etc.).

RISC-V RVA22 and RVA23 aren't too different in this regard. Each one prescribes which extensions must be supported by the processor. I saw RV64GC mentioned, this is just a shortening of RV64IMAFDC, so I for baseline instructions, M for multiplication and division, A for atomic, F for floating point, D for double precision floating point and C for compressed instructions.

You can have a baseline E profile instead of I (less registers, some other features stripped), but I don't think we will ever see manufactured RV64E core, trough RV32EC cores exist.

The E extension will probably only ever appear in softcores (FPGA) to reduce gate count.

CH32V003 has RV32EC core, for RV64, yes, there is no point.

Huh, where does the “G” come from in RV64GC?

It is a shorthand for "IMAFD" for RV64.

Another aspect of the x86_64 architectural swamp is that Intel deliberately makes their optimized math libraries fail if run on a chip that is not 'INTEL INSIDE'. That block totally ignores cpu feature bits.

I've always wondered if virtualization software vendors made everything claim to be INTEL INSIDE, just to avoid this problem.

I know the RPCS3 (PS3 emulator) encountered some differences in a floating point instruction where Intel and AMD had different precision, causing de-synchronizations in LittleBigPlanet multiplayer.

Could be down to this.

Are you talking about the 8087 transcendental instructions? Which are well known and well-written software doesn't have a problem with it.

Nope, there are subtle differences between Intel and AMD that make it a Bad Idea to fake the vendor.

There are? We aren't talking about feature bits, and the only known software that checks the vendor are Intel's math libraries.

They are minor but enough to complicate things. For example Intel rejects the SYSCALL instruction outside 64-bit mode.

Huh. A quick google says that's a feature bit and there's also an OS call to turn it on.

David recently announced he was leaving MLCommons, so maybe we'll get him back as an industry analyst.

Common Crawl has very little content from Reddit and 4chan -- both blocked us years ago.

Common Crawl is about 1 petabyte per year, compressed. Uncompressed is 4X.

Both IA and CCF are private foundations and accept donations.

> It's a shame that CCBot is caught in the cross fire, but that's life.

We're used to it. Sadly.

RIP to a real one. Have you considered using one billion Chinese residential ips to address your problem? I’ve heard one billion Chinese residential ips really does the trick here.

The author blocked CCBot even though CCBot isn't part of the high traffic problem -- apparently he trusted Cloudflare labeling us as an "AI Bot".

That’s already a big business!