Table of Contents

• BHyVe & jails DHCP/NAT configuration
  • Setup
  • Initial system configuration
    • DHCP Configuration
    • vm-bhyve switch Creation
    • pf Configuration
• tl;dr

Setup

This is how I configured dhcp and nat for my jails and bhyve vm’s on FreeBSD.

I’m assuming that jailer is used for handling jails, it’s a nice little utility and just writes and parses /usr/local/jails or wherever you prefer having your jails. The reason for this is assumption is that jailer wants to create a bridge0 interface when auto configuring dhcp, so lets just let it do that.

Then when we use vm-bhyve we are suggested by its manual to use dnsmasq but theres no need for this as we already have dhcpd configured for our jails and can piggyback off that by attaching to bridge0.

Initial system configuration

pkg install dhcpd
pkg install vm-bhyve
git clone https://github.com/illuria/jailer

DHCP Configuration

Jailer will create something like this for you when you init dhcp

# cat /usr/local/etc/dhcpd.conf
subnet 10.0.0.0 netmask 255.255.255.0 {
range 10.0.0.2 10.0.0.254;
option subnet-mask 255.255.255.0;
option routers 10.0.0.1;
}

This configuration allows our jails to obtain an IP address from the DHCP server, which in this case is set to 10.0.0.1.

vm-bhyve switch Creation

To create a new vm-bhyve switch, run the following command: vm switch create -t manual -b bridge0 NAT

This will create a new VM switch called NAT. The -t manual flag tells vm-bhyve that we know what we’re doing and the -b bridge0 flag let’s use choose our own existing bridge instead of having a new one created.

pf Configuration

echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf
service sysctl restart

This enables IP forwarding on our system.

Edit the /etc/pf.conf file:

# Allow traffic from any IP address to our dhcp range
nat on $default_interface from 10.0.0.0/24 to any -> ($default_interface)

set skip on bridge0
pass in quick on 10.0.0.0/24 to any

Make sure the nat on line isn’t after your defaults block in; pass out. reload pf with pfctl -f /etc/pf.conf

tl;dr

pkg install vm-bhyve dhcpd
sysrc gateway_enable="YES"
sysrc pf_enable="YES"
sysrc vm_enable="YES"
echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf
service sysctl restart
echo "nat on $default_interface from {10.0.0.0/24} to any ->
default_interface=$(route get default | grep interface | cut -w -f 3)
($default_interface)" >> /etc/pf.conf
echo "set skip on bridge0" >> /etc/pf.conf
echo "pass in quick on 10.0.0.0/24 to any" >> /etc/pf.conf
pfctl -f /etc/pf.conf
git clone https://github.com/illuria/jailer
jailer init
jailer init bridge
jailer init dhcp
cd /usr/local/vm
vm init
vm switch create -t manual -b bridge0 NAT
echo 'network0_type="virtio-net"' >> /usr/local/vm/guest/guest.conf
echo 'network0_switch="NAT"' >> /usr/local/vm/guest/guest.conf

Table of Contents

• GSoC status update #5 (cpy, ccpy, lcpy)
• Whats been going on the last few weeks
  • memccpy
  • strlcpy
  • strlcat
  • memcpy
  • Benchmarks
• References

Whats been going on the last few weeks

So the last few weeks I’ve been working on the *cpy functions. I gave strcspn a try but it’s really tricky and I decided to save it to last and perhaps continue with it past the GSoC deadline if needed.

I started with memccpy and let it be the base for strlcpy. I also noticed that stpncpy is quite similar but I haven’t gotten around to writing it yet although the already done functions will serve as a rough base for it.

memccpy

It turned out to be rather tricky to get this one right, a classic case of the Ninety–ninety rule. It went smooth at first but ironing out little bugs as being off by one here and there really took a while. Having two debuggers open side by side and comparing register values from the amd64 variant to my aarch64 port was really useful. To complicate things further I found a bug in the existing memccpy implementation which fuz’ took care of. [1] This first resulted in a small degradation in performance but after some rework it resulted in an improvement over the original faulty code! :-)

Although this meant that I rewrote parts of memccpy 3 times in order to improve perfomance. In the end it turned out quite nice, beating the Scalar performance by over 1000% for longer strings across all processors I was able to benchmark on. Sadly it turned out to be a whole 20% slower on short strings on the Neoverse N1 chip.

strlcpy

very similar to memccpy except that we return the length of the src string instead of a pointer. This means that we essentially run strlen at the same time. This leads to some nice multi-tasking, an idea I had was to use strlen implementation which was fast for short to medium strings when the whole source string wasn’t copied. My reasoning is that its more common that we generally reach a limit near the end of a string rather than near the beginning.

strlcpy is presented in this paper [2] and recently the linux kernel has been moving away from strlcpy in favor of strscpy. [3] There surely is a meme out there for the naming of each new variant of the string functions to make it more “safe”. :-)

strlcat

strlcat is rather simple to implement using existing string function. Take the one from FreeBSD’s libc. [4]

size_t
strlcat(char *restrict dst, const char *restrict src, size_t dstsize)
{
    char *loc = __memchr(dst, '\0', dstsize);

    if (loc != NULL) {
        size_t dstlen = (size_t)(loc - dst);

        return (dstlen + __strlcpy(loc, src, dstsize - dstlen));
    } else
        return (dstsize + strlen(src));
}

Now the only issue is that I haven’t written a SIMD memchr as there is also a well optimized variant in the arm-optimized-routines repository in /contrib. The issue is that I should not touch anything in /contrib as it just pulled from a git repo. And second is that arm labels their memchar _memchraarch64 and then the Makefile does a little rewrite.

.for FUNC in ${AARCH64_STRING_FUNCS}
.if !exists(${FUNC}.S)
${FUNC}.S:
    printf '/* %sgenerated by libc/aarch64/string/Makefile.inc */\n' @ > ${.TARGET}
    printf '#define __%s_aarch64 %s\n' ${FUNC} ${FUNC} >> ${.TARGET}
    printf '#include "aarch64/%s.S"\n' ${FUNC} >> ${.TARGET}
CLEANFILES+=    ${FUNC}.S
.endif

The reason why we can’t just call memchr is that libc functions must be designed such that they do not appear to call other libc functions in case the user overrides any of them. And calling something __something is a rather simple fix. :-)

So the solution is to create a little wrapper for memchr, which naturally brings us to memcpy!

memcpy

I noticed that the memcpy that was currently included in libc wasn’t SIMD optimized despite there being an ASIMD memcpy variant in /contrib. The reason is that compared to all the other string functions then arm has provided a Scalar memcpy and a SIMD memcpy. This is probably due to the string functions being grandfathered in from an older optimized routines repository. Anyway, ASIMD is a part of the base ISA of Aarch64 so as long as it’s not slower then theres little incentive to not use it. The fix was simple.

diff --git a/lib/libc/aarch64/string/memcpy.S b/lib/libc/aarch64/string/memcpy.S
--- a/lib/libc/aarch64/string/memcpy.S
+++ b/lib/libc/aarch64/string/memcpy.S
@@ -1,6 +1,6 @@
-#define    __memcpy_aarch64    memcpy
-#define    __memmove_aarch64   memmove
-#include "aarch64/memcpy.S"
+#define    __memcpy_aarch64_simd   memcpy
+#define    __memmove_aarch64_simd  memmove
+#include "aarch64/memcpy-advsimd.S"

coming week

GSoC is slowly nearing its end, there’s just over 2 weeks left of the standard (12 week) coding period. These final weeks will consist of me tying up some loose ends such as missing bcmp ifdef’s for memcp which bundles both bcmp and memcmp into the same file and finishing the implementation the fancy algorithm for strcspn.

Benchmarks

There’s quite a few benchmarks so it’s better to visit the DR for each of them instead.

memcpy
memccpy
strlcpy

References

[1]: https://reviews.freebsd.org/D46052
[2]: http://www.usenix.org/publications/library/proceedings/usenix99/full_papers/millert/millert.pdf
[3]: https://lwn.net/Articles/905777/
[4]: https://cgit.freebsd.org/src/tree/lib/libc/amd64/string/strlcat.c

Table of Contents

• GSoC status update #4 (A bit of everything)
• Whats been going on this week
  • strncmp pains
  • Next function, strcspn
    • strcspn tomfoolery
  • Performance analysis
    • OSACA results
• References

Whats been going on this week

This week I did some finishing touches on strncmp, it’s essentially the same as strcmp except with special handling for the limit and breaks the main loop when theres less than 32 bytes left of the limit.

I also got some good code review on str(n)cmp and applied it which resulted in a few % performance improvement. [1]

strncmp pains

I thought I could do a clever solution for handling strings near a page boundary for short strings (less than 16 bytes) for strncmp but I was mistaken. After updating the test suite to have buffers placed right at the end of a page with DEATH waiting across the boundary I noticed that it failed. So I had to revert to the more complicated handling and make it even more complicated as simply checking for null bytes isnt enough I also need to insert a fake null byte wherever the limit is. I solved the first part like this and the latter part is a problem for tomorrow morning. :-)

@@ -103,24 +103,56 @@ ENTRY(strncmp)
    .p2align 4
 .Llt16:
-   tbz w3, #PAGE_SHIFT, 0f
+   /*
+    * Check if either string is located at end of page to avoid crossing
+    * into unmapped page. If so, we load 16 bytes from the nearest
+    * alignment boundary and shift based on the offset.
+    */
+   tbz w3, #PAGE_SHIFT, 2f

    ldr q0, [x8]            // load aligned head
    ldr q1, [x10]

+   lsl x14, x9, #2
+   lsl x15, x11, #2
+   lsl x3, x13, x14            // string head
+   lsl x4, x13, x15
+
+   cmeq    v5.16b, v0.16b, #0
+   cmeq    v6.16b, v1.16b, #0
+
+   shrn    v5.8b, v5.8h, #4
+   shrn    v6.8b, v6.8h, #4
+   fmov    x5, d5
+   fmov    x6, d6
+
    adrp    x14, shift_data
    add x14, x14, :lo12:shift_data

    /* heads may cross page boundary, avoid unmapped loads */
+   tst x5, x3
+   b.eq    0f
+
    ldr q4, [x14, x9]           // load permutation table
    tbl v0.16b, {v0.16b}, v4.16b
+
+   b   1f
+   .p2align 4
+0:
+   ldr q0, [x0]            // load true head
+1:
+   tst x6, x4
+   b.eq    0f
+
    ldr q4, [x14, x11]
    tbl v4.16b, {v1.16b}, v4.16b
+
    b 1f

    .p2align 4
-0:
+2:
    ldr q0, [x0]            // load true heads
+0:
    ldr q4, [x1]
 1:

Next function, strcspn

It’s also time to port a new function, I was choosing between memccpy and strcspn. I chose to start exploring strcspn and it has some really interesting tricks required to get it fast. On x86 when SSE4.2 is available we can use the amazing pcmpistri instruction to compare a vector register with a set without overreading past a null byte. [2]

On Aarch64 there is no equivalent, although SVE2 has the MATCH instruction, but thats to no avail for us. The Graviton 3 CPU that I’ve been benchmarking on has SVE support but not SVE2. And FreeBSD is just about to get SVE support, it’s about to land in -CURRENT any second now IIRC.

strcspn tomfoolery

So checking with SIMD whether a byte is present in a set is not a completely new problem. Although I haven’t seen any such algorithm implemented for Aarch64 then there is one for x86 developed by Wojciech Muła and Geoff Langdale. [3]

I won’t go into detail just yet, it’s quite a lot to wrap my head around but the article linked is a great resource. A version of this algorithm is used by the Intel Hyperscan project. [4][5]

An interesting suggestion fuz had was to duplicate a byte in the set and iterative over the string during the setup phase. In the case of a match of a early set member then this approach would be greatly beneficial.

This as the LUT for the algorithm can be done using scalar registers and the check by ld1r, cmeq could be done in vector registers. Fully utilizing the processors different pipelines.

Another useful article is [6] by Harold Aptroot but the lack of the extremely versatile but tricky gf2p8affineqb Galois Field Affine Transformation Instruction on Aarch64 will need to be figured out.

A list of unexcepted uses for the Galois Field Affine Transformation Instruction is available here [7]

Performance analysis

I was also introduced by my mentor to some really cool projects which simulate different computer architectures and predicts the throughput of basic blocks. The first one is uiCA (uops.info Code Analyzer) [8] developed by researchers at Saarland University. The second one is OSACA (Open Source Architecture Code Analyzer) [9] which supports both x86 and Aarch64. It’s developed by RRZE-HPC at the Erlangen National High Performance Computing Center. OSACA also has integration with Compiler Explorer.

These tools are a clear step up from SimpleScalar that I used in university for our computer architecture course. [10] Simplescalar is legacy research software. I suspect it might soon be swapped out in our curriculum, that course has been getting serious upgrades the last few years.

For my strncmp implementation OSACA produces the following:

OSACA results

Open Source Architecture Code Analyzer (OSACA) - 0.5.2
Analyzed file:      /app/example.asm
Architecture:       A64FX
Timestamp:          2024-07-14 17:00:03

-------------------------- WARNING: No micro-architecture was specified -------------------------
         A default uarch for this particular ISA was used. Specify the uarch with --arch.
         See --help for more information.
-------------------------------------------------------------------------------------------------
----------------- WARNING: You are analyzing a large amount of instruction forms ----------------
         Analysis across loops/block boundaries often do not make much sense.
         Specify the kernel length with --length. See --help for more information.
         If this is intentional, you can safely ignore this message.
-------------------------------------------------------------------------------------------------

 P - Throughput of LOAD operation can be hidden behind a past or future STORE instruction
 * - Instruction micro-ops not bound to a port
 X - No throughput/latency information for this instruction in data file

Combined Analysis Report
------------------------
                                      Port pressure in cycles                                       
     |  0   - 0DV  |  1   |  2   |  3   |   4   |  5   -  5D  |  6   -  6D  |  7   ||  CP  | LCD  |
---------------------------------------------------------------------------------------------------
   1 |             |      |      |      |       |             |             |      ||      |      |   strncmp:
   2 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X bic x8, x0, #0xf   // x8 is x0 but aligned to the boundary
   3 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   and x9, x0, #0xf   // x9 is the offset
   4 |             |      |      |      |       |             |             |      ||      |      | X bic x10, x1, #0xf   // x10 is x1 but aligned to the boundary
   5 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   and x11, x1, #0xf   // x11 is the offset
   7 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   subs x2, x2, #1
   8 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.mi .Lempty
  10 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   mov x13, #-1    // save constant for later
  12 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x3, x0, #16   // end of head
  13 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x4, x1, #16
  14 |             |      |      |      |       |             |             |      ||      |      | X eor x3, x3, x0
  15 |             |      |      |      |       |             |             |      ||      |      | X eor x4, x4, x1   // bits that changed
  16 |             |      |      | 0.50 | 0.500 | 0.00        | 0.00        |      ||      |      |   orr x3, x3, x4   // in either str1 or str2
  17 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2,#16
  18 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.lt .Llt16
  19 |             |      |      |      |       |             |             |      ||      |      | X tbz w3, #4096, .Lbegin
  21 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8]   // load aligned head
  22 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x10]
  24 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x14, x9, #2
  25 |             |      |      |      |       |             |             |      ||      |      | X lsl x3, x13, x14   // string head
  26 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x15, x11, #2
  27 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x15
  29 |             |      |      |      |       |             |             |      ||      |      | X cmeq v5.16b, v0.16b, #0
  30 |             |      |      |      |       |             |             |      ||      |      | X cmeq v6.16b, v1.16b, #0
  32 |             |      |      |      |       |             |             |      ||      |      | X shrn v5.8b, v5.8h, #4
  33 |             |      |      |      |       |             |             |      ||      |      | X shrn v6.8b, v6.8h, #4
  34 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d5
  35 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d6
  37 |             |      |      |      |       |             |             |      ||      |      | X adrp x14, shift_data
  38 |             |      |      |      |       |             |             |      ||      |      | X add x14, x14, :lo12:shift_data
  40 |             |      |      |      |       |             |             |      ||      |      | X tst x5, x3
  41 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.eq zero
  43 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x14, x9]   // load permutation table
  44 |             |      |      |      |       |             |             |      ||      |      | X tbl v0.16b, {v0.16b}, v4.16b
  46 |             |      |      |      |       |             |             | 1.00 ||      |      |   b one
  47 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
  48 |             |      |      |      |       |             |             |      ||      |      |   zero:
  49 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x0]   // load true head
  50 |             |      |      |      |       |             |             |      ||      |      |   one:
  51 |             |      |      |      |       |             |             |      ||      |      | X tst x6, x4
  52 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.eq zeroo
  54 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x14, x11]
  55 |             |      |      |      |       |             |             |      ||      |      | X tbl v4.16b, {v1.16b}, v4.16b
  57 |             |      |      |      |       |             |             | 1.00 ||      |      |   b onee
  59 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
  60 |             |      |      |      |       |             |             |      ||      |      |   .Lbegin:
  61 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x0]   // load true heads
  62 |             |      |      |      |       |             |             |      ||      |      |   zeroo:
  63 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x1]
  64 |             |      |      |      |       |             |             |      ||      |      |   onee:
  65 |             |      |      |      |       |             |             |      ||      |      | X cmeq v2.16b, v0.16b, #0  // NUL byte present?
  66 |             |      |      |      |       |             |             |      ||      |      | X cmeq v4.16b, v0.16b, v4.16b  // which bytes match?
  68 |             |      |      |      |       |             |             |      ||      |      | X orn v2.16b, v2.16b, v4.16b  // mismatch or NUL byte?
  70 |             |      |      |      |       |             |             |      ||      |      | X shrn v2.8b, v2.8h, #4
  71 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d2
  73 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lhead_mismatch
  74 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x8, #16]   // load second chunk
  75 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q3, [x10, #16]
  77 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x2, x2, x11
  78 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x2, x2, #16   // account for length of RSI chunk?
  80 |             |      |      |      |       |             |             |      ||      |      | X subs x9, x9, x11
  81 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.lt .Lswapped   // if not swap operands
  82 |             |      |      |      |       |             |             | 1.00 ||      |      |   b .Lnormal
  84 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
  85 |             |      |      |      |       |             |             |      ||      |      |   .Llt16:
  87 |             |      |      |      |       |             |             |      ||      |      | X tbz w3, #4096, two
  89 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8]   // load aligned head
  90 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x10]
  92 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x14, x9, #2
  93 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x15, x11, #2
  94 |             |      |      |      |       |             |             |      ||      |      | X lsl x3, x13, x14   // string head
  95 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x15
  97 |             |      |      |      |       |             |             |      ||      |      | X cmeq v5.16b, v0.16b, #0
  98 |             |      |      |      |       |             |             |      ||      |      | X cmeq v6.16b, v1.16b, #0
 100 |             |      |      |      |       |             |             |      ||      |      | X shrn v5.8b, v5.8h, #4
 101 |             |      |      |      |       |             |             |      ||      |      | X shrn v6.8b, v6.8h, #4
 102 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d5
 103 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d6
 105 |             |      |      |      |       |             |             |      ||      |      | X adrp x14, shift_data
 106 |             |      |      |      |       |             |             |      ||      |      | X add x14, x14, :lo12:shift_data
 108 |             |      |      |      |       |             |             |      ||      |      | X tst x5, x3
 109 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.eq zerooo
 111 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x14, x9]   // load permutation table
 112 |             |      |      |      |       |             |             |      ||      |      | X tbl v0.16b, {v0.16b}, v4.16b
 114 |             |      |      |      |       |             |             | 1.00 ||      |      |   b oneee
 115 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 116 |             |      |      |      |       |             |             |      ||      |      |   zerooo:
 117 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x0]   // load true head
 118 |             |      |      |      |       |             |             |      ||      |      |   oneee:
 119 |             |      |      |      |       |             |             |      ||      |      | X tst x6, x4
 120 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.eq noll
 122 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x14, x11]
 123 |             |      |      |      |       |             |             |      ||      |      | X tbl v4.16b, {v1.16b}, v4.16b
 125 |             |      |      |      |       |             |             | 1.00 ||      |      |   b ett
 127 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 128 |             |      |      |      |       |             |             |      ||      |      |   two:
 129 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x0]   // load true heads
 130 |             |      |      |      |       |             |             |      ||      |      |   noll:
 131 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q4, [x1]
 132 |             |      |      |      |       |             |             |      ||      |      |   ett:
 134 |             |      |      |      |       |             |             |      ||      |      | X cmeq v2.16b, v0.16b, #0  // NUL byte present?
 135 |             |      |      |      |       |             |             |      ||      |      | X cmeq v4.16b, v0.16b, v4.16b  // which bytes match?
 137 |             |      |      |      |       |             |             |      ||      |      | X bic v2.16b, v4.16b, v2.16b  // match and not NUL byte
 139 |             |      |      |      |       |             |             |      ||      |      | X shrn v2.8b, v2.8h, #4
 140 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d2
 141 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x4, x2, #2
 142 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 143 |             |      |      |      |       |             |             |      ||      |      | X orn x5, x4, x5   // mismatch or NUL byte?
 145 |             |      |      |      |       |             |             |      ||      |      |   .Lhead_mismatch:
 146 |             |      |      |      |       |             |             |      ||      |      | X rbit x3, x5
 147 |             |      |      |      |       |             |             |      ||      |      | X clz x3, x3    // index of mismatch
 148 |             |      |      |      |       |             |             |      ||      |      | X lsr x3, x3, #2
 149 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w4, [x0, x3]
 150 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w5, [x1, x3]
 151 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub w0, w4, w5
 152 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 154 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 155 |             |      |      |      |       |             |             |      ||      |      |   .Lnormal:
 156 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x12, x10, x9
 157 |             |      |      | 0.75 | 0.750 | 0.25   0.50 | 0.25   0.50 |      ||      |      |   ldr q0, [x12, #16]!
 158 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x10, x10, x8
 159 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x11, x10, x9
 161 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v3.16b, #0  // NUL present?
 162 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b  // Mismatch between chunks?
 163 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 164 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 165 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 166 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 168 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x8, x8, #32   // advance to next iteration
 170 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x4, x2, #2
 171 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 172 |             |      |      | 0.50 | 0.500 | 0.00        | 0.00        |      ||      |      |   orr x3, x6, x4   // introduce a null byte match
 173 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #16    // does the buffer end within x2
 174 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   csel x6, x3, x6, lt
 175 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound2   // NUL or end of buffer found?
 176 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 177 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch2
 178 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x2, x2, #16
 179 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #32    // end of buffer within first main loop iteration?
 180 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.lt .Ltail
 182 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 183 |             |      |      |      |       |             |             |      ||      |      |   nada:
 184 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8, x11]
 185 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x8, x10]
 186 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x8]
 188 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0  // end of string?
 189 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b  // do the chunks match?
 191 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 192 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 193 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 194 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 195 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound
 196 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5    // any mismatches?
 197 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch
 199 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x8, x8, #16
 202 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8, x11]
 203 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x8, x10]
 204 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x8]
 206 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x8, x8, #16
 207 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0
 208 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 210 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 211 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 212 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 213 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 214 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound2
 215 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 216 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch2
 217 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x2, x2, #32
 218 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #32    // end of buffer within next iteration
 219 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.ge nada    // if yes, process tail
 222 |             |      |      |      |       |             |             |      ||      |      |   .Ltail:
 223 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8, x11]
 224 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x8, x10]
 225 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x8]
 227 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0  // end of string?
 228 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b  // do the chunks match?
 230 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 231 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 232 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 233 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 237 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x4, x2, #2
 238 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 239 |             |      |      | 0.50 | 0.500 | 0.00        | 0.00        |      ||      |      |   orr x3, x6, x4   // introduce a null byte match
 240 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #16    // does the buffer end within x2
 241 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   csel x6, x3, x6, lt
 243 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound   // NUL or end of string found
 244 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 245 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch
 247 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x8, x8, #16
 249 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8, x11]
 250 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x8, x10]
 251 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x8]
 253 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x8, x8, #16
 254 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0
 255 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 257 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 258 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 259 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 260 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 262 |             |      |      |      |       |             |             |      ||      |      | X ubfiz x4, x2, #2, #4 // (x2 - 16) << 2
 263 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4   // take first half into account
 264 |             |      |      | 0.50 | 0.500 | 0.00        | 0.00        |      ||      |      |   orr x6, x6, x4   // introduce a null byte match
 266 |             |      |      |      |       |             |             |      ||      |      |   .Lnulfound2:
 267 |             |      |      | 0.50 | 0.500 |             |             |      ||  1.0 |  1.0 |   sub x8, x8, #16
 269 |             |      |      |      |       |             |             |      ||      |      |   .Lnulfound:
 270 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   mov x4, x6
 272 |             |      |      |      |       |             |             |      ||      |      | X ubfiz x7, x9, #2, #4
 273 |             |      |      |      |       |             |             |      ||      |      | X lsl x6, x6, x7   // adjust NUL mask to indices
 275 |             |      |      |      |       |             |             |      ||      |      | X orn x5, x6, x5
 276 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch
 278 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x8, x9]
 279 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x8, x10]
 281 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v0.16b, v1.16b
 282 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 283 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d1
 285 |             |      |      |      |       |             |             |      ||      |      | X orn x5, x4, x5
 287 |             |      |      |      |       |             |             |      ||      |      | X rbit x3, x5
 288 |             |      |      |      |       |             |             |      ||      |      | X clz x3, x3
 289 |             |      |      |      |       |             |             |      ||      |      | X lsr x5, x3, #2
 291 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x10, x10, x8   // restore x10 pointer
 292 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x8, x8, x9   // point to corresponding chunk in x0
 294 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w4, [x8, x5]
 295 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w5, [x10, x5]
 296 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub w0, w4, w5
 297 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 299 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 300 |             |      |      |      |       |             |             |      ||      |      |   .Lmismatch2:
 301 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x8, x8, #16   // roll back second increment
 302 |             |      |      |      |       |             |             |      ||      |      |   .Lmismatch:
 303 |             |      |      |      |       |             |             |      ||      |      | X rbit x3, x5
 304 |             |      |      |      |       |             |             |      ||      |      | X clz x3, x3    // index of mismatch
 305 |             |      |      |      |       |             |             |      ||      |      | X lsr x3, x3, #2
 306 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x11, x8, x11
 308 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w4, [x8, x3]
 309 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w5, [x11, x3]
 310 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub w0, w4, w5   // difference of the mismatching chars
 311 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 313 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 314 |             |      |      |      |       |             |             |      ||      |      |   .Lswapped:
 315 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x12, x8, x9
 316 |             |      |      | 0.75 | 0.750 | 0.25   0.50 | 0.25   0.50 |      ||      |      |   ldr q0, [x12, #16]!
 317 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x8, x8, x10
 318 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x11, x8, x9
 319 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x2,x2,x9
 320 |             |      |      |      |       |             |             |      ||      |      | X neg x9, x9
 322 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v2.16b, #0
 323 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v3.16b
 324 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 325 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 326 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 327 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 329 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x10, x10, #32
 331 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x4, x2, #2
 332 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 333 |             |      |      | 0.38 | 0.370 | 0.13        | 0.12        |      ||      |      |   orr x3,x6,x4   // introduce a null byte match
 334 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2,#16
 335 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   csel x6, x3, x6, lt
 336 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound2s
 337 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 338 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch2s
 340 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x2, x2, #16
 341 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #32
 342 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.lt .Ltails
 344 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 345 |             |      |      |      |       |             |             |      ||      |      |   nein:
 346 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x10, x11]
 347 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x10, x8]
 348 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x10]
 350 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0
 351 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 353 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 354 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 355 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 356 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 357 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfounds
 358 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 359 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatchs
 361 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x10, x10, #16
 364 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x10, x11]
 365 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x10, x8]
 366 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q2, [x10]
 368 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |  1.0 |   add x10, x10, #16
 369 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0
 370 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 372 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 373 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 374 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 375 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 376 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfound2s
 377 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 378 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatch2s
 379 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x2, x2, #32
 380 |             |      |      | 0.25 | 0.750 |             |             |      ||      |      |   cmp x2, #32
 381 |             |      |      |      |       |             |             | 1.00 ||      |      |   b.ge nein
 383 |             |      |      |      |       |             |             |      ||      |      |   .Ltails:
 384 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x10, x11]
 385 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q1, [x10, x8]
 386 |             |      |      |      |       | 0.51   0.50 | 0.49   0.50 |      ||      |      |   ldr q2, [x10]
 388 |             |      |      |      |       |             |             |      ||      |      | X cmeq v1.16b, v1.16b, #0
 389 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 391 |             |      |      |      |       |             |             |      ||      |      | X shrn v1.8b, v1.8h, #4
 392 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 393 |             |      |      |      |       |             |             |      ||      |      | X fmov x6, d1
 394 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 397 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   lsl x4, x2, #2
 398 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 399 |             |      |      | 0.50 | 0.000 | 0.24        | 0.26        |      ||      |      |   orr x3, x6, x4    // introduce a null byte match
 400 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cmp x2, #16
 401 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   csel x6, x3, x6, lt
 403 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x6, .Lnulfounds
 404 |             |      |      |      |       |             |             |      ||      |      | X mvn x5, x5
 405 |             |      |      | 0.74 | 0.260 |             |             |      ||      |      |   cbnz x5, .Lmismatchs
 407 | 0.50        |      | 0.50 | 0.00 | -0.01 |             |             |      ||  1.0 |  1.0 |   add x10, x10, #16
 409 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x10, x11]
 410 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||  5.0 |      |   ldr q1, [x10, x8]
 411 |             |      |      |      |       | 0.51   0.50 | 0.49   0.50 |      ||      |      |   ldr q2, [x10]
 413 | 0.50        |      | 0.50 | 0.00 | -0.01 |             |             |      ||      |  1.0 |   add x10, x10, #16
 414 |             |      |      |      |       |             |             |      ||  0.0 |      | X cmeq v1.16b, v1.16b, #0
 415 |             |      |      |      |       |             |             |      ||      |      | X cmeq v0.16b, v0.16b, v2.16b
 417 |             |      |      |      |       |             |             |      ||  0.0 |      | X shrn v1.8b, v1.8h, #4
 418 |             |      |      |      |       |             |             |      ||      |      | X shrn v0.8b, v0.8h, #4
 419 |             |      |      |      |       |             |             |      ||  0.0 |      | X fmov x6, d1
 420 |             |      |      |      |       |             |             |      ||      |      | X fmov x5, d0
 422 |             |      |      |      |       |             |             |      ||      |      | X ubfiz x4, x2, #2, #4
 423 |             |      |      |      |       |             |             |      ||      |      | X lsl x4, x13, x4
 424 |             |      |      | 0.00 | 0.510 | 0.23        | 0.26        |      ||  1.0 |      |   orr x6, x6, x4    // introduce a null byte match
 426 |             |      |      |      |       |             |             |      ||      |      |   .Lnulfound2s:
 427 |             |      |      | 0.50 | 0.500 |             |             |      ||      |  1.0 |   sub x10, x10, #16
 428 |             |      |      |      |       |             |             |      ||      |      |   .Lnulfounds:
 429 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||  1.0 |      |   mov x4, x6
 431 |             |      |      |      |       |             |             |      ||      |      | X ubfiz x7, x9, #2, #4
 432 |             |      |      |      |       |             |             |      ||      |      | X lsl x6, x6, x7
 434 |             |      |      |      |       |             |             |      ||      |      | X orn x5, x6, x5
 436 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   cbnz x5, .Lmismatchs
 438 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldr q0, [x10, x9]
 439 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |  5.0 |   ldr q1, [x10, x8]
 441 |             |      |      |      |       |             |             |      ||      |  0.0 | X cmeq v1.16b, v0.16b, v1.16b
 442 |             |      |      |      |       |             |             |      ||      |  0.0 | X shrn v1.8b, v1.8h, #4
 443 |             |      |      |      |       |             |             |      ||      |  0.0 | X fmov x5, d1
 445 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X orn x5, x4, x5
 447 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X rbit x3, x5
 448 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X clz x3, x3
 449 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X lsr x5, x3, #2
 451 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x11, x10, x8
 452 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x10, x10, x9
 454 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |      |   ldrb w4, [x10, x5]
 455 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||  5.0 |  5.0 |   ldrb w5, [x11, x5]
 456 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub w0, w5, w4
 457 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 459 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 460 |             |      |      |      |       |             |             |      ||      |      |   .Lmismatch2s:
 461 |             |      |      | 0.50 | 0.500 |             |             |      ||      |      |   sub x10, x10, #16
 462 |             |      |      |      |       |             |             |      ||      |      |   .Lmismatchs:
 463 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X rbit x3, x5
 464 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X clz x3, x3
 465 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X lsr x3, x3, #2
 466 | 0.50        |      | 0.50 | 0.00 | 0.000 |             |             |      ||      |      |   add x11, x10, x11
 468 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||  5.0 |      |   ldrb w4, [x10, x3]
 469 |             |      |      |      |       | 0.50   0.50 | 0.50   0.50 |      ||      |  5.0 |   ldrb w5, [x11, x3]
 470 |             |      |      | 0.50 | 0.500 |             |             |      ||  1.0 |  1.0 |   sub w0, w5, w4
 471 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 473 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 474 |             |      |      |      |       |             |             |      ||      |      |   .Lempty:
 475 |             |      |      |      |       |             |             |      ||  0.0 |  0.0 | X eor x0, x0, x0
 476 |             |      |      |      |       | 0.50        | 0.50        |      ||      |      |   ret
 478 |             |      |      |      |       |             |             |      ||      |      |   .section .rodata
 479 |             |      |      |      |       |             |             |      ||      |      |   .p2align 4
 480 |             |      |      |      |       |             |             |      ||      |      |   shift_data:
 481 |             |      |      |      |       |             |             |      ||      |      |   .byte 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
 482 |             |      |      |      |       |             |             |      ||      |      |   .fill 16, 1, -1
 483 |             |      |      |      |       |             |             |      ||      |      |   .size shift_data, .-shift_data

       18.0                 18.0   29.9   29.87   31.1   28.0   31.1   28.0   16.0    29.0   29.0  

Loop-Carried Dependencies Analysis Report
-----------------------------------------
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 19.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 19.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 15.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 451, 455, 463, 464, 465, 468, 470, 475]
   2 | 15.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 451, 455, 463, 464, 465, 469, 470, 475]
   2 | 11.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 317, 451, 466, 469, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 20.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 451, 455, 463, 464, 465, 468, 470, 475]
   2 | 20.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 451, 455, 463, 464, 465, 469, 470, 475]
   2 | 16.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 451, 466, 469, 470, 475]
   2 | 17.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 452, 461, 466, 469, 470, 475]
   2 | 16.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 158, 291, 329, 361, 368, 407, 413, 427, 452, 461, 468, 470, 475]
   2 | 26.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 26.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 24.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 20.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 451, 455, 463, 464, 465, 468, 470, 475]
   2 | 20.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 451, 455, 463, 464, 465, 469, 470, 475]
   2 | 16.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 317, 451, 466, 469, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 29.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 25.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 451, 455, 463, 464, 465, 468, 470, 475]
   2 | 25.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 451, 455, 463, 464, 465, 469, 470, 475]
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 451, 466, 469, 470, 475]
   2 | 22.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 452, 461, 466, 469, 470, 475]
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 291, 329, 361, 368, 407, 413, 427, 452, 461, 468, 470, 475]
   2 | 27.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 27.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 25.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   2 | 25.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 451, 455, 463, 464, 465, 468, 470, 475]
   2 | 21.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 451, 455, 463, 464, 465, 469, 470, 475]
   2 | 17.0 | bic       x8, x0, #0xf              // x8 is x0 but aligned to the boundary| [2, 168, 199, 206, 247, 253, 267, 292, 301, 317, 451, 466, 469, 470, 475]
   3 | 22.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   3 | 22.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 410, 414, 417, 419, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   3 | 20.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   3 | 20.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 439, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   3 | 16.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 451, 455, 463, 464, 465, 468, 470, 475]
   3 | 16.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 451, 455, 463, 464, 465, 469, 470, 475]
   3 | 12.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 292, 301, 317, 451, 466, 469, 470, 475]
   3 | 17.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 319, 340, 379, 422, 423, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   3 | 17.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 319, 340, 379, 422, 423, 424, 429, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   3 | 17.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 320, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 468, 470, 475]
   3 | 17.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 320, 438, 441, 442, 443, 445, 447, 448, 449, 455, 463, 464, 465, 469, 470, 475]
   3 | 10.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 320, 452, 461, 466, 469, 470, 475]
   3 |  9.0 | and       x9, x0, #0xf              // x9 is the offset| [3, 80, 320, 452, 461, 468, 470, 475]
   7 |  8.0 | subs      x2, x2, #1                     | [7, 77, 78, 178, 217, 319, 340, 379]

It’s also accessible here if you want to play around with it.

References

[1]: https://reviews.freebsd.org/D45943
[2]: https://cgit.freebsd.org/src/tree/lib/libc/amd64/string/strcspn.S#n228
[3]: http://0x80.pl/articles/simd-byte-lookup.html
[4]: https://twitter.com/geofflangdale/status/1053227022795722752
[5]: https://github.com/intel/hyperscan/blob/master/src/nfa/truffle.c
[6]: https://bitmath.blogspot.com/2023/04/not-transposing-16x16-bitmatrix.html
[7]: https://gist.github.com/animetosho/d3ca95da2131b5813e16b5bb1b137ca0
[8]: https://uica.uops.info/
[9]: https://github.com/RRZE-HPC/OSACA
[10]: https://pages.cs.wisc.edu/~mscalar/simplescalar.html

Table of Contents

• GSoC status update #3 (strcmp)
  • What is strcmp even?
  • Scalar compiler output
  • amd64 SIMD implementation
• Translating to Aarch64
  • Stay within your page
• Benchmarks
• References

What is strcmp even?

The man page on FreeBSD has the following to say:

STRCMP(3)              FreeBSD Library Functions Manual              STRCMP(3)

NAME
     strcmp, strncmp – compare strings

LIBRARY
     Standard C Library (libc, -lc)

SYNOPSIS
     #include <string.h>

     int
     strcmp(const char *s1, const char *s2);

     int
     strncmp(const char *s1, const char *s2, size_t len);

DESCRIPTION
     The strcmp() and strncmp() functions lexicographically compare the null-
     terminated strings s1 and s2.

     The strncmp() function compares not more than len characters.  Because
     strncmp() is designed for comparing strings rather than binary data,
     characters that appear after a ‘\0’ character are not compared.

RETURN VALUES
     The strcmp() and strncmp() functions return an integer greater than,
     equal to, or less than 0, according as the string s1 is greater than,
     equal to, or less than the string s2.  The comparison is done using
     unsigned characters, so that ‘\200’ is greater than ‘\0’.

SEE ALSO
     bcmp(3), memcmp(3), strcasecmp(3), strcoll(3), strxfrm(3), wcscmp(3)

STANDARDS
     The strcmp() and strncmp() functions conform to ISO/IEC 9899:1990
     (“ISO C90”).

HISTORY
     The strcmp() function first appeared in the Programmer's Workbench
     (PWB/UNIX) and was ported to Version 7 AT&T UNIX; strncmp() first
     appeared in Version 7 AT&T UNIX.

FreeBSD 15.0-CURRENT             April 3, 2022            FreeBSD 15.0-CURRENT

Scalar compiler output

amd64 SIMD implementation

This is a longest one of the bunch, but its split into two separate code paths which makes the code long but the algorithm ~1/2 of the total LoC.

ARCHENTRY(strcmp, baseline)
    /* check if either string crosses a page in the head */
    lea     15(%rdi), %r8d  # end of head
    lea     15(%rsi), %r9d
    mov     %edi, %eax
    mov     %esi, %edx
    xor     %edi, %r8d  # bits that changed between first and last byte
    xor     %esi, %r9d
    and     $~0xf, %rdi # align heads to 16 bytes
    and     $~0xf, %rsi
    or      %r8d, %r9d  # in either RSI or RDI
    and     $0xf, %eax  # offset from alignment
    and     $0xf, %edx
    pxor        %xmm1, %xmm1
    test        $PAGE_SIZE, %r9d # did the page change?
    jz      0f      # if not, take fast path

    /* heads may cross page boundary, avoid unmapped loads */
    movdqa      (%rdi), %xmm0   # load aligned heads
    movdqa      (%rsi), %xmm2
    mov     $-1, %r8d
    mov     $-1, %r9d
    mov     %eax, %ecx
    shl     %cl, %r8d   # string head in XMM0
    mov     %edx, %ecx
    shl     %cl, %r9d   # string head in XMM2
    movdqa      %xmm0, -40(%rsp) # stash copies of the heads on the stack
    movdqa      %xmm2, -24(%rsp)
    pcmpeqb     %xmm1, %xmm0
    pcmpeqb     %xmm1, %xmm2
    pmovmskb    %xmm0, %r10d
    pmovmskb    %xmm2, %r11d
    test        %r8d, %r10d # NUL byte present in first string?
    lea     -40(%rsp), %r8
    cmovz       %rdi, %r8
    test        %r9d, %r11d # NUL byte present in second string?
    lea     -24(%rsp), %r9
    cmovz       %rsi, %r9
    movdqu      (%r8, %rax, 1), %xmm0 # load true (or fake) heads
    movdqu      (%r9, %rdx, 1), %xmm4
    jmp     1f

0:  movdqu      (%rdi, %rax, 1), %xmm0 # load true heads
    movdqu      (%rsi, %rdx, 1), %xmm4
1:  pxor        %xmm2, %xmm2
    pcmpeqb     %xmm0, %xmm2    # NUL byte present?
    pcmpeqb     %xmm0, %xmm4    # which bytes match?
    pandn       %xmm4, %xmm2    # match and not NUL byte?
    pmovmskb    %xmm2, %r9d
    xor     $0xffff, %r9d   # mismatch or NUL byte?
    jnz     .Lhead_mismatch

    /* load head and second chunk */
    movdqa      16(%rdi), %xmm2 # load second chunks
    movdqa      16(%rsi), %xmm3
    sub     %rdx, %rax  # is a&0xf >= b&0xf?
    jb      .Lswapped   # if not, proceed with swapped operands

    neg     %rax
    movdqu      16(%rsi, %rax, 1), %xmm0
    sub     %rdi, %rsi  # express RSI as distance from RDI
    lea     (%rsi, %rax, 1), %rdx # point RDX to offset in second string
    neg     %rax
    pcmpeqb     %xmm3, %xmm1    # ... corresponding to RDI
    pcmpeqb     %xmm2, %xmm0
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    add     $16, %rdi
    test        %r8d, %r8d
    jnz     .Lnul_found
    xor     $0xffff, %r9d
    jnz     .Lmismatch
    add     $16, %rdi   # advance aligned pointers

    /*
     * During the main loop, the layout of the two strings is something like:
     *
     *          v ------1------ v ------2------ v
     *     RDI:    AAAAAAAAAAAAABBBBBBBBBBBBBBBB...
     *     RSI: AAAAAAAAAAAAABBBBBBBBBBBBBBBBCCC...
     *
     * where v indicates the alignment boundaries and corresponding chunks
     * of the strings have the same letters.  Chunk A has been checked in
     * the previous iteration.  This iteration, we first check that string
     * RSI doesn't end within region 2, then we compare chunk B between the
     * two strings.  As RSI is known not to hold a NUL byte in regsions 1
     * and 2 at this point, this also ensures that RDI has not ended yet.
     */
    ALIGN_TEXT
0:  movdqu      (%rdi, %rdx, 1), %xmm0 # chunk of 2nd string corresponding to RDI?
    pxor        %xmm1, %xmm1
    pcmpeqb     (%rdi, %rsi, 1), %xmm1 # end of string in RSI?
    pcmpeqb     (%rdi), %xmm0   # where do the chunks match?
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    test        %r8d, %r8d
    jnz     .Lnul_found
    xor     $0xffff, %r9d   # any mismatches?
    jnz     .Lmismatch

    /* main loop unrolled twice */
    movdqu      16(%rdi, %rdx, 1), %xmm0 # chunk of 2nd string corresponding to RDI?
    pxor        %xmm1, %xmm1
    pcmpeqb     16(%rdi, %rsi, 1), %xmm1 # end of string in RSI?
    pcmpeqb     16(%rdi), %xmm0 # where do the chunks match?
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    add     $32, %rdi
    test        %r8d, %r8d
    jnz     .Lnul_found2
    xor     $0xffff, %r9d   # any mismatches?
    jz      0b

    sub     $16, %rdi   # roll back second increment

    /* a mismatch has been found between RDX and RSI */
.Lmismatch:
    tzcnt       %r9d, %r9d  # where is the mismatch?
    add     %rdi, %rdx  # turn RDX from offset to pointer
    movzbl      (%rdx, %r9, 1), %ecx
    movzbl      (%rdi, %r9, 1), %eax
    sub     %ecx, %eax  # difference of the mismatching chars
    ret

    /* mismatch in true heads */
.Lhead_mismatch:
    tzcnt       %r9d, %r9d  # where is the mismatch?
    add     %rax, %rdi  # return to true heads
    add     %rdx, %rsi
    movzbl      (%rdi, %r9, 1), %eax # mismatching characters
    movzbl      (%rsi, %r9, 1), %ecx
    sub     %ecx, %eax
    ret

.Lnul_found2:
    sub     $16, %rdi   # roll back second increment

    /* a NUL has been found in RSI */
.Lnul_found:
    mov     %eax, %ecx
    mov     %r8d, %r10d
    shl     %cl, %r8w   # adjust NUL mask to positions in RDI/RDX
    xor     $0xffff, %r9d   # mask of mismatches
    or      %r8d, %r9d  # NUL bytes also count as mismatches
    jnz     .Lmismatch

    /*
     * (RDI) == (RSI) and NUL is past the string.
     * Compare (RSI) with the corresponding part
     * of the other string until the NUL byte.
     */
    movdqu      (%rdi, %rax, 1), %xmm0
    pcmpeqb     (%rdi, %rsi, 1), %xmm0
    add     %rdi, %rsi  # restore RSI pointer
    add     %rax, %rdi  # point RDI to chunk corresponding to (RSI)
    pmovmskb    %xmm0, %ecx # mask of matches
    not     %ecx        # mask of mismatches
    or      %r10d, %ecx # mask of mismatches or NUL bytes
    tzcnt       %ecx, %ecx  # location of first mismatch
    movzbl      (%rdi, %rcx, 1), %eax
    movzbl      (%rsi, %rcx, 1), %ecx
    sub     %ecx, %eax
    ret

    /*
     * If (a&0xf) < (b&0xf), we do the same thing but with swapped
     * operands.  I found that this performs slightly better than
     * using conditional moves to do the swap branchless.
     */
.Lswapped:
    movdqu      16(%rdi, %rax, 1), %xmm0
    sub     %rsi, %rdi  # express RDI as distance from RSI
    lea     (%rdi, %rax, 1), %rdx # point RDX to offset in RDI corresponding to RSI
    neg     %rax        # make difference positive
    pcmpeqb     %xmm2, %xmm1
    pcmpeqb     %xmm3, %xmm0
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    add     $16, %rsi   # advance aligned pointers
    test        %r8d, %r8d
    jnz     .Lnul_founds
    xor     $0xffff, %r9d
    jnz     .Lmismatchs
    add     $16, %rsi

    /*
     * During the main loop, the layout of the two strings is something like:
     *
     *          v ------1------ v ------2------ v
     *     RDI:    AAAAAAAAAAAAABBBBBBBBBBBBBBBB...
     *     RSI: AAAAAAAAAAAAABBBBBBBBBBBBBBBBCCC...
     *
     * where v indicates the alignment boundaries and corresponding chunks
     * of the strings have the same letters.  Chunk A has been checked in
     * the previous iteration.  This iteration, we first check that string
     * RSI doesn't end within region 2, then we compare chunk B between the
     * two strings.  As RSI is known not to hold a NUL byte in regsions 1
     * and 2 at this point, this also ensures that RDI has not ended yet.
     */
    ALIGN_TEXT
0:  movdqu      (%rsi, %rdx, 1), %xmm0 # chunk of 2nd string corresponding to RDI?
    pxor        %xmm1, %xmm1
    pcmpeqb     (%rsi, %rdi, 1), %xmm1 # end of string in RSI?
    pcmpeqb     (%rsi), %xmm0   # where do the chunks match?
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    test        %r8d, %r8d
    jnz     .Lnul_founds
    xor     $0xffff, %r9d   # any mismatches?
    jnz     .Lmismatchs

    /* main loop unrolled twice */
    movdqu      16(%rsi, %rdx, 1), %xmm0 # chunk of 2nd string corresponding to RDI?
    pxor        %xmm1, %xmm1
    pcmpeqb     16(%rsi, %rdi, 1), %xmm1 # end of string in RSI?
    pcmpeqb     16(%rsi), %xmm0 # where do the chunks match?
    pmovmskb    %xmm1, %r8d
    pmovmskb    %xmm0, %r9d
    add     $32, %rsi
    test        %r8d, %r8d
    jnz     .Lnul_found2s
    xor     $0xffff, %r9d   # any mismatches?
    jz      0b

    sub     $16, %rsi   # roll back second increment

    /* a mismatch has been found between RDX and RDI */
.Lmismatchs:
    tzcnt       %r9d, %r9d  # where is the mismatch?
    add     %rsi, %rdx  # turn RDX from offset to pointer
    movzbl      (%rdx, %r9, 1), %eax
    movzbl      (%rsi, %r9, 1), %ecx
    sub     %ecx, %eax  # difference of the mismatching chars
    ret

.Lnul_found2s:
    sub     $16, %rsi   # roll back second increment

    /* a NUL has been found in RSI */
.Lnul_founds:
    mov     %eax, %ecx
    mov     %r8d, %r10d
    shl     %cl, %r8w   # adjust NUL mask to positions in RDI/RDX
    xor     $0xffff, %r9d   # mask of mismatches
    or      %r8d, %r9d  # NUL bytes also count as mismatches
    jnz     .Lmismatchs

    /*
     * (RDI) == (RSI) and NUL is past the string.
     * Compare (RSI) with the corresponding part
     * of the other string until the NUL byte.
     */
    movdqu      (%rsi, %rax, 1), %xmm0
    pcmpeqb     (%rsi, %rdi, 1), %xmm0
    add     %rsi, %rdi  # restore RDI pointer
    add     %rax, %rsi  # point RSI to chunk corresponding to (RDI)
    pmovmskb    %xmm0, %ecx # mask of matches
    not     %ecx        # mask of mismatches
    or      %r10d, %ecx # mask of mismatches or NUL bytes
    tzcnt       %ecx, %ecx  # location of first mismatch
    movzbl      (%rdi, %rcx, 1), %eax
    movzbl      (%rsi, %rcx, 1), %ecx
    sub     %ecx, %eax
    ret
ARCHEND(strcmp, baseline)

fuz described this the following way:

The basic idea is to process the bulk of the string in aligned blocks of 16 bytes such that one string runs ahead and the other runs behind. The string that runs ahead is checked for NUL bytes, the one that runs behind is compared with the corresponding chunk of the string that runs ahead. This trades an extra load per iteration for the very complicated block-reassembly needed in the other implementations (bionic, glibc). On the flip side, we need two code paths depending on the relative alignment of the two buffers.

The initial part of the string is compared directly if it is known not to cross a page boundary. Otherwise, a complex slow path to avoid crossing into unmapped memory commences. [1]

Translating to Aarch64

As we’ve been gotten somewhat comfortable with porting the common instructions like PMOVMSKB we can do a rather simple 1:1 mapping of the amd64 code and optimize it afterwards. The only gripe is the Swapping based on the two input strings relative offset to the nearest 16 byte boundary.

ENTRY(_strcmp)

    bic x8, x0, #0xf            // x8 is x0 but aligned to the boundary
    and x9, x0, #0xf            // x9 is the offset
    bic x10, x1, #0xf           // x10 is x1 but aligned to the boundary
    and x11, x1, #0xf           // x11 is the offset

    /*
     * Check if either string is located at end of page to avoid crossing
     * into unmapped page. If so, we load 16 bytes from the nearest
     * alignment boundary and shift based on the offset.
     */

    add x3, x0, #16         // end of head
    add x4, x1, #16
    eor x3, x3, x0
    eor x4, x4, x1              // bits that changed
    orr x3, x3, x4              // in either str1 or str2
    tst w3, #PAGE_SIZE
    b.eq    .Lbegin

    ldr q0, [x8]                // load aligned head
    ldr q2, [x10]

    mov x3, #0xffffffffffffffff
    mov x4, #0xffffffffffffffff
    lsl x14, x9, #2
    lsl x3, x3, x14         // string head
    lsl x15, x11, #2
    lsl x4, x4, x15

    cmeq    v0.16b, v0.16b, #0
    cmeq    v2.16b, v2.16b, #0

    shrn    v0.8b, v0.8h, #4
    shrn    v2.8b, v2.8h, #4
    fmov    x5, d0
    fmov    x6, d2

    /* heads may cross page boundary, avoid unmapped loads */
    tst x5, x3
    b.eq    0f
    adrp    x2, shift_data
    add x2, x2, :lo12:shift_data
    add x2, x2, x9
    ldr q0, [x8]
    ldr q4, [x2]                // load permutation table
    tbl v0.16b, {v0.16b}, v4.16b
    b       1f
    .p2align 4
0:
    ldr q0, [x0]                // load true head
1:
    tst x6, x4
    b.eq    0f

    adrp    x2, shift_data
    add x2, x2, :lo12:shift_data
    add x2, x2, x11
    ldr q1, [x10]
    ldr q4, [x2]
    tbl v4.16b, {v1.16b}, v4.16b

    b 1f

    .p2align 4
.Lbegin:
    ldr q0, [x0]                // load true heads
0:
    ldr q4, [x1]
1:

    cmeq    v2.16b, v0.16b, #0      // NUL byte present?
    cmeq    v4.16b, v0.16b, v4.16b  // which bytes match?

    not v2.16b, v2.16b
    and v2.16b, v2.16b, v4.16b  // match and not NUL byte

    shrn    v2.8b, v2.8h, #4
    fmov    x5, d2
    mvn x5, x5                  // mismatch or NUL byte?

    cbnz    x5, .Lhead_mismatch

    ldr q2, [x8, #16]           // load second chunk
    ldr q3, [x10, #16]
    subs    x9, x9, x11         // is a&0xf >= b&0xf
    b.lt    .Lswapped               // if not swap operands
    neg x9, x9
    add x12, x10, x9
    ldr q0, [x12, #16]!
    sub x10, x10, x8
    add x11, x10, x9            // point x11 to offset in second string
    neg x9, x9

    cmeq    v1.16b, v3.16b, #0
    cmeq    v0.16b, v0.16b, v2.16b
    add x8, x8, #16
    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfound
    mvn x5, x5
    cbnz    x5, .Lmismatch
    add x8, x8, #16         // advance aligned pointers

    /*
     * During the main loop, the layout of the two strings is something like:
     *
     *          v ------1------ v ------2------ v
     *      X0:    AAAAAAAAAAAAABBBBBBBBBBBBBBBB...
     *      X1: AAAAAAAAAAAAABBBBBBBBBBBBBBBBCCC...
     *
     * where v indicates the alignment boundaries and corresponding chunks
     * of the strings have the same letters.  Chunk A has been checked in
     * the previous iteration.  This iteration, we first check that string
     * X1 doesn't end within region 2, then we compare chunk B between the
     * two strings.  As X1 is known not to hold a NUL byte in regions 1
     * and 2 at this point, this also ensures that x0 has not ended yet.
     */
    .p2align 4
0:
    ldr q0, [x8, x11]
    ldr q1, [x8, x9]
    ldr q2, [x8]

    cmeq    v1.16b, v1.16b, #0      // end of string?
    cmeq    v0.16b, v0.16b, v2.16b  // do the chunks match?

    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfound
    mvn x5, x5                  // any mismatches?
    cbnz    x5, .Lmismatch

    add x8, x8, #16

    ldr q0, [x8, x11]
    ldr q1, [x8, x9]
    ldr q2, [x8]

    add x8, x8, #16
    cmeq    v1.16b, v1.16b, #0
    cmeq    v0.16b, v0.16b, v2.16b

    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfound2
    mvn x5, x5
    cbz x5, 0b

    sub x8, x8, #16         // roll back second increment
.Lmismatch:
    rbit    x2, x5
    clz x2, x2                  // index of mismatch
    lsr x2, x2, #2
    add x11, x8, x11

    ldrb    w4, [x8, x2]
    ldrb    w5, [x11, x2]
    sub w0, w4, w5              // difference of the mismatching chars
    ret

.Lnulfound2:
    sub x8, x8, #16

.Lnulfound:
    mov x7, x9
    mov x4, x6

    and x7, x7, #0xf
    lsl x7, x7, #2
    lsl x6, x6, x7              // adjust NUL mask to indices
    mvn x5, x5
    orr x5, x5, x6              // NUL bytes also count as mismatches
    cbnz    x5, .Lmismatch

    /*
     * (x0) == (x1) and NUL is past the string.
     * Compare (x1) with the corresponding part
     * of the other string until the NUL byte.
     */
    ldr q0, [x8, x9]
    ldr q1, [x8, x10]

    cmeq    v1.16b, v0.16b, v1.16b
    shrn    v1.8b, v1.8h, #4
    fmov    x5, d1

    mvn x5, x5
    orr x5, x5, x4

    rbit    x2, x5
    clz x2, x2
    lsr x5, x2, #2

    add x10, x10, x8            // restore x10 pointer
    add x8, x8, x9              // point to corresponding chunk in x0

    ldrb    w4, [x8, x5]
    ldrb    w5, [x10, x5]
    sub w0, w4, w5
    ret

.Lhead_mismatch:
    rbit    x2, x5
    clz x2, x2                  // index of mismatch
    lsr x2, x2, #2
    ldrb    w4, [x0, x2]
    ldrb    w5, [x1, x2]
    sub w0, w4, w5
    ret

    /*
     * If (a&0xf) < (b&0xf), we do the same thing but with swapped
     * operands.  I found that this performs slightly better than
     * using conditional moves to do the swap branchless.
     */
.Lswapped:
    add x12, x8, x9
    ldr q0, [x12, #16]!
    sub x8, x8, x10
    add x11, x8, x9
    neg x9, x9

    cmeq    v1.16b, v2.16b, #0
    cmeq    v0.16b, v0.16b, v3.16b
    add x10, x10, #16
    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfounds
    mvn x5, x5
    cbnz    x5, .Lmismatchs
    add x10, x10, #16

    /*
     * During the main loop, the layout of the two strings is something like:
     *
     *          v ------1------ v ------2------ v
     *      X1:    AAAAAAAAAAAAABBBBBBBBBBBBBBBB...
     *      X0: AAAAAAAAAAAAABBBBBBBBBBBBBBBBCCC...
     *
     * where v indicates the alignment boundaries and corresponding chunks
     * of the strings have the same letters.  Chunk A has been checked in
     * the previous iteration.  This iteration, we first check that string
     * X0 doesn't end within region 2, then we compare chunk B between the
     * two strings.  As X0 is known not to hold a NUL byte in regions 1
     * and 2 at this point, this also ensures that X1 has not ended yet.
     */
    .p2align 4
0:
    ldr q0, [x10, x11]
    ldr q1, [x10, x8]
    ldr q2, [x10]

    cmeq    v1.16b, v1.16b, #0
    cmeq    v0.16b, v0.16b, v2.16b

    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfounds
    mvn x5, x5
    cbnz    x5, .Lmismatchs

    add x10, x10, #16

    ldr q0, [x10, x11]
    ldr q1, [x10, x8]
    ldr q2, [x10]

    add x10, x10, #16
    cmeq    v1.16b, v1.16b, #0
    cmeq    v0.16b, v0.16b, v2.16b

    shrn    v1.8b, v1.8h, #4
    fmov    x6, d1
    shrn    v0.8b, v0.8h, #4
    fmov    x5, d0
    cbnz    x6, .Lnulfound2s
    mvn x5, x5
    cbz x5, 0b

    sub x10, x10, #16

.Lmismatchs:
    rbit    x2, x5
    clz x2, x2
    lsr x2, x2, #2
    add x11, x10, x11

    ldrb    w4, [x10, x2]
    ldrb    w5, [x11, x2]
    sub w0, w5, w4
    ret

.Lnulfound2s:
    sub x10, x10, #16
.Lnulfounds:
    mov x7, x9
    mov x4, x6

    and x7, x7, #0xf
    lsl x7, x7, #2
    lsl x6, x6, x7
    mvn x5, x5
    orr x5, x5, x6
    cbnz    x5, .Lmismatchs

    ldr q0, [x10, x9]
    ldr q1, [x10, x8]

    cmeq    v1.16b, v0.16b, v1.16b
    shrn    v1.8b, v1.8h, #4
    fmov    x5, d1

    mvn x5, x5
    orr x5, x5, x4

    rbit    x2, x5
    clz x2, x2
    lsr x5, x2, #2

    add x11, x10, x8
    add x10, x10, x9

    ldrb    w4, [x10, x5]
    ldrb    w5, [x11, x5]
    sub w0, w5, w4
    ret

END(_strcmp)

    .section .rodata
    .p2align 4
shift_data:
    .byte 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15

strcmp is implemented the following way

• Check if the page boundary is within 16 bytes, if so we enter a complex path similarly to how we implemented memcmp to avoid crossing into unmapped memory. But we simply need to check if theres any null bytes present in either of the strings. Otherwise we compare the true heads directly, i.e. we load 16 bytes from the source addresses for string 1 and 2.

• When loading the true heads we first check for the presence of null bytes and then if theres any mismatch between the strings. The result of those two operations are then compared and whichever occured first gives us the result.

• If no match was found we start setting up the loop by loading the second chunk from aligned addresses, this means that is the worst case where a string was only 1 byte from the nearest boundary 15 bytes will be reevaluated in this pass.

• The algorithm for the setup is that based on which of the two source buffers has the greatest relative alignment we swap the operands to have one buffer run one chunk ahead checking for null bytes.

• Then the main loop is entered and is unrolled twice, checking for null bytes first and then checking for mismatches. It is described well in the block comment in the code

/*
 * During the main loop, the layout of the two strings is something like:
 *
 *          v ------1------ v ------2------ v
 *     RDI:    AAAAAAAAAAAAABBBBBBBBBBBBBBBB...
 *     RSI: AAAAAAAAAAAAABBBBBBBBBBBBBBBBCCC...
 *
 * where v indicates the alignment boundaries and corresponding chunks
 * of the strings have the same letters.  Chunk A has been checked in
 * the previous iteration.  This iteration, we first check that string
 * RSI doesn't end within region 2, then we compare chunk B between the
 * two strings.  As RSI is known not to hold a NUL byte in regions 1
 * and 2 at this point, this also ensures that RDI has not ended yet.
 */

• Then in the case of a null byte found in the string running ahead we find its offset and shift according to the other strings relative offset and compare to the mismatch. So that we catch mismatches in the strings running behind the string being checked for presence of null bytes. (Here we need to shift the offset left by two as PMOVMSKB results in a 16 bit result whereas cmeq followed by shrn results in a 64 bit result).

Stay within your page

Similarly to memcmp we also need to make sure we dont read into unmapped pages. This can be achieved in a similar way by checking that there are no null bytes in the bytes leading up to a page boundary, if there is no page boundary within the first few bytes we can go on as usual.

Improvements

The main loop for checking mismatches and occurrences of null bytes is currently implemented using a cmeq followed by shrn similarly to how I’ve done it in strlen and memcmp. A downside of shrn is that it commonly only utilizes vector pipelines V13 (1 and 3). I’ve experimented with using umaxp to overcome this which utilizes vector pipeline V (all the pipelines). I realized that preloading a vector register with 0, 1, 2, ..., 15, 16 and or’ing it with the result from cmeq followed with uminv will immediately produce the index of the first mismatch. Although uminv is also bound to V13 we can reduce the scalar instruction count for the rbit, clz, lsr dance that we have perfomed previously.

Note: The pipelines utilized by specific instructions varies between processors but comparing the available Software Optimization Guides [2] from ARM has shown that this is a common implementation.

This could be implemented the following way:

cmeq       v0.16B,v0.16B,#0     // v0 <- 0xFFFF00FFFFFF
ldr        q1,shift_data
orr        v0.16B,v0.16B,v1.16B // v0 <- 0xFFFF07FFFFFF
uminv      b0,v0.16B            // b0 <- 0x07
fmov       w2,s0
ldrb       w4,[x0, x2]
ldrb       w5,[x1, x2]
sub        x0,x4,x5
ret

Benchmarks

os: FreeBSD
arch: arm64
cpu: ARM Cortex-A76 r4p1
                     │  strcmpARM  │             strcmpSIMD              │
                     │   sec/op    │   sec/op     vs base                │
StrcmpShortAligned     137.6µ ± 1%   113.8µ ± 0%  -17.35% (p=0.000 n=20)
StrcmpMidAligned       37.54µ ± 2%   38.93µ ± 0%   +3.69% (p=0.000 n=20)
StrcmpLongAligned      17.65µ ± 0%   14.67µ ± 0%  -16.89% (p=0.000 n=20)
StrcmpShortUnaligned   183.7µ ± 1%   125.2µ ± 0%  -31.83% (p=0.000 n=20)
StrcmpMidUnaligned     51.74µ ± 0%   38.69µ ± 2%  -25.23% (p=0.000 n=20)
StrcmpLongUnaligned    16.20µ ± 0%   16.12µ ± 0%   -0.50% (p=0.000 n=20)
StrcmpShortQsort       1.511m ± 0%   1.450m ± 0%   -4.05% (p=0.000 n=20)
StrcmpMidQsort         354.1µ ± 0%   345.1µ ± 0%   -2.56% (p=0.000 n=20)
geomean                96.49µ        84.24µ       -12.69%

                     │  strcmpARM   │              strcmpSIMD               │
                     │     B/s      │      B/s       vs base                │
StrcmpShortAligned     866.2Mi ± 1%   1048.0Mi ± 0%  +20.99% (p=0.000 n=20)
StrcmpMidAligned       3.101Gi ± 1%    2.991Gi ± 0%   -3.56% (p=0.000 n=20)
StrcmpLongAligned      6.597Gi ± 0%    7.938Gi ± 0%  +20.33% (p=0.000 n=20)
StrcmpShortUnaligned   649.0Mi ± 1%    952.1Mi ± 0%  +46.70% (p=0.000 n=20)
StrcmpMidUnaligned     2.250Gi ± 0%    3.009Gi ± 2%  +33.74% (p=0.000 n=20)
StrcmpLongUnaligned    7.186Gi ± 0%    7.222Gi ± 0%   +0.50% (p=0.000 n=20)
StrcmpShortQsort       78.89Mi ± 0%    82.22Mi ± 0%   +4.22% (p=0.000 n=20)
StrcmpMidQsort         336.6Mi ± 0%    345.5Mi ± 0%   +2.62% (p=0.000 n=20)
geomean                1.207Gi         1.382Gi       +14.53%

os: FreeBSD
arch: arm64
cpu: ARM Neoverse-V1 r1p1
                     │  strcmpARM   │             strcmpSIMD              │
                     │    sec/op    │   sec/op     vs base                │
StrcmpShortAligned      85.63µ ± 1%   72.88µ ± 1%  -14.88% (p=0.000 n=20)
StrcmpMidAligned        21.59µ ± 5%   17.51µ ± 2%  -18.88% (p=0.000 n=20)
StrcmpLongAligned      13.457µ ± 8%   8.315µ ± 3%  -38.21% (p=0.000 n=20)
StrcmpShortUnaligned   135.96µ ± 0%   85.68µ ± 0%  -36.98% (p=0.000 n=20)
StrcmpMidUnaligned      30.31µ ± 1%   18.64µ ± 1%  -38.49% (p=0.000 n=20)
StrcmpLongUnaligned    13.649µ ± 1%   8.442µ ± 2%  -38.15% (p=0.000 n=20)
StrcmpShortQsort       1173.0µ ± 0%   984.4µ ± 0%  -16.08% (p=0.000 n=20)
StrcmpMidQsort          263.1µ ± 0%   227.8µ ± 0%  -13.38% (p=0.000 n=20)
geomean                 67.51µ        48.79µ       -27.74%

                     │  strcmpARM   │              strcmpSIMD               │
                     │     B/s      │      B/s       vs base                │
StrcmpShortAligned     1.360Gi ± 1%    1.597Gi ± 1%  +17.49% (p=0.000 n=20)
StrcmpMidAligned       5.393Gi ± 5%    6.648Gi ± 2%  +23.28% (p=0.000 n=20)
StrcmpLongAligned      8.651Gi ± 9%   14.001Gi ± 3%  +61.83% (p=0.000 n=20)
StrcmpShortUnaligned   876.8Mi ± 0%   1391.3Mi ± 0%  +58.69% (p=0.000 n=20)
StrcmpMidUnaligned     3.841Gi ± 1%    6.245Gi ± 1%  +62.58% (p=0.000 n=20)
StrcmpLongUnaligned    8.529Gi ± 1%   13.791Gi ± 2%  +61.69% (p=0.000 n=20)
StrcmpShortQsort       101.6Mi ± 0%    121.1Mi ± 0%  +19.16% (p=0.000 n=20)
StrcmpMidQsort         453.2Mi ± 0%    523.2Mi ± 0%  +15.45% (p=0.000 n=20)
geomean                1.724Gi         2.386Gi       +38.39%

References

[1]: https://reviews.freebsd.org/D41971
[2]: https://developer.arm.com/documentation/109898/latest/

Table of Contents

• GSoC status update #2 (memcmp)
  • What is memcmp even?
    • Here be dragons
  • Scalar compiler output
  • amd64 SIMD implementation
• Translating to Aarch64
  • Stay within your page
  • wide shifts
• Benchmarks
• References

What is memcmp even?

The man page on FreeBSD has the following to say:

MEMCMP(3)              FreeBSD Library Functions Manual              MEMCMP(3)

NAME
     memcmp – compare byte string

LIBRARY
     Standard C Library (libc, -lc)

SYNOPSIS
     #include <string.h>

     int
     memcmp(const void *b1, const void *b2, size_t len);

DESCRIPTION
     The memcmp() function compares byte string b1 against byte string b2.
     Both strings are assumed to be len bytes long.

RETURN VALUES
     The memcmp() function returns zero if the two strings are identical,
     otherwise returns the difference between the first two differing bytes
     (treated as unsigned char values, so that ‘\200’ is greater than ‘\0’,
     for example).  Zero-length strings are always identical.

SEE ALSO
     bcmp(3), strcasecmp(3), strcmp(3), strcoll(3), strxfrm(3),
     timingsafe_memcmp(3), wmemcmp(3)

STANDARDS
     The memcmp() function conforms to ISO/IEC 9899:1990 (“ISO C90”).

FreeBSD 15.0-CURRENT            August 15, 2016           FreeBSD 15.0-CURRENT

Now when starting off coding anything I like to begin with a minimum example, a PoC (Proof of Concept). So for these string functions I first make a small wrapper program to either call the libc function or my assembly function. Then I compare the output so get some quick turnaround times before integrating it into libc and running the proper unit tests.
So I did just that, and to my surprise libc wasn’t returning the expected values.

Here be dragons

So when I run my little wrapper program with the following:

#include <stdio.h>
#include <string.h>

int main() {
    printf("%i\n", memcmp("\200", "\0", 1));
}

It returns 1, thats odd. However, ISO/IEC 9899:1990 only specifies that a number greater than, equal to, or less than zero shall be returned. So going by the standard it’s a valid return value. But it doesn’t match up with what the FreeBSD man page states.

But if we edit our program a bit like the following we notice something interesting.

#include <stdio.h>
#include <string.h>

int main() {
  printf("%i\n", memcmp((char[] ) {"\200"}, "\0", 1));
}

It returns 128, as expected!

The reason for this is that on Aarch64 we have inherited string functions from the Arm Optimized Routines. These functions aren’t originally written with FreeBSD in mind and this has simply gone unnoticed. If we instead glance over at the manpage for memcmp on glibc we see the following regarding its return value:

RETURN VALUE 

    The memcmp() function returns an integer less than, equal to, or greater than
    zero if the first n bytes of s1 is found, respectively, to be less than, to
    match, or be greater than the first n bytes of s2.

    For a nonzero return value, the sign is determined by the sign of the difference
    between the first pair of bytes (interpreted as unsigned char) that differ in s1
    and s2.

    If n is zero, the return value is zero.

I strongly suspect that nobody is checking the return value beyond <, =, or >, except for the units tests… Which do fail on FreeBSD!

I generated this result page by using the kyua command kyua report-html -r ${TEST}.db, everything you need to know about the FreeBSD test suite is available on the FreeBSD wiki.

Finally, these inherited Arm Optimized Routines are located in /usr/src/contrib/arm-optimized-routines and plugged in to the string functions Makefile.inc like the following:

AARCH64_STRING_FUNCS= \
    memchr \
    memcmp \
    memcpy \
    memmove \
    memrchr \
    memset \
    stpcpy \
    strchr \
    strchrnul \
    strcmp \
    strcpy \
    strlen \
    strncmp \
    strnlen \
    strrchr

#
# Add the above functions. Generate an asm file that includes the needed
# Arm Optimized Routines file defining the function name to the libc name.
# Some file need multiple macros defined or a weak symbol added we can
# override the generated file in these cases.
#
.for FUNC in ${AARCH64_STRING_FUNCS}
.if !exists(${FUNC}.S)
${FUNC}.S:
    printf '/* %sgenerated by libc/aarch64/string/Makefile.inc */\n' @ > ${.TARGET}
    printf '#define __%s_aarch64 %s\n' ${FUNC} ${FUNC} >> ${.TARGET}
    printf '#include "aarch64/%s.S"\n' ${FUNC} >> ${.TARGET}
CLEANFILES+=    ${FUNC}.S
.endif

MDSRCS+=    ${FUNC}.S
CFLAGS.${FUNC}.S+=-I${SRCTOP}/contrib/arm-optimized-routines/string
.endfor

There we can swap out the faulty one for a compliant compiler generated implementation, but lets get going with porting ours!

@@ -15,11 +15,12 @@ AARCH64_STRING_FUNCS= \
        strchrnul \
        strcmp \
        strcpy \
-       memcmp \
        strncmp \
        strnlen \
        strrchr

+MDSRCS+= \
+       memcmp.S
 #
 # Add the above functions. Generate an asm file that includes the needed
 # Arm Optimized Routines file defining the function name to the libc name.

These functions already have ARM NEON enchancements from the Arm Optimized Routines

    memchr             |   x
    memcmp             |   x
    memcpy             |   SWAR
    memmove            |   SWAR
    memrchr            |   x
    memset             |   x
    stpcpy             |   x
    strchr             |   x
    strchrnul          |   x
    strcmp             |   SWAR
    strcpy             |   x
    strlen             |   x
    strncmp            |   SWAR
    strnlen            |   x
    strrchr            |   x

Now I wont be touching memcpy as its implementation could benefit from another design [1]. But if we ignore the functions with pre-existing implementations by ARM we are left with the following list. But lets finish off memcmp first because we already got started on it and maybe we can beat Arm at their own game.

           FUNCTION          AMD64
           bcmp              S1
           index             S1
           memccpy           S1
           rindex            S1
           stpncpy           S1
           strcat            S1
           strcmp            S1
           strcspn           S2
           strlcat           S1
           strlcpy           S1
           strncat           S1
           strncmp           S1
           strncpy           S1
           strpbrk           S2
           strsep            S2
           strspn            S2
           timingsafe_bcmp   S1

Scalar compiler output

After the above small hiccup I took a look at the .c code for memcmp and its generated assembly. Nothing particularly notable here

amd64 SIMD implementation

Now this is a long one

ARCHENTRY(memcmp, baseline)
    cmp     $32, %rdx       # enough to permit use of the long kernel?
    ja      .Llong

    test        %rdx, %rdx      # zero bytes buffer?
    je      .L0

    /*
     * Compare strings of 1--32 bytes.  We want to do this by
     * loading into two xmm registers and then comparing.  To avoid
     * crossing into unmapped pages, we either load 32 bytes from
     * the start of the buffer or 32 bytes before its end, depending
     * on whether there is a page boundary between the overread area
     * or not.
     */

    /* check for page boundaries overreads */
    lea     31(%rdi), %eax      # end of overread
    lea     31(%rsi), %r8d
    lea     -1(%rdi, %rdx, 1), %ecx # last character in buffer
    lea     -1(%rsi, %rdx, 1), %r9d
    xor     %ecx, %eax
    xor     %r9d, %r8d
    test        $PAGE_SIZE, %eax    # are they on different pages?
    jz      0f

    /* fix up rdi */
    movdqu      -32(%rdi, %rdx, 1), %xmm0
    movdqu      -16(%rdi, %rdx, 1), %xmm1
    lea     -8(%rsp), %rdi      # end of replacement buffer
    sub     %rdx, %rdi      # start of replacement buffer
    movdqa      %xmm0, -40(%rsp)    # copy to replacement buffer
    movdqa      %xmm1, -24(%rsp)

0:  test        $PAGE_SIZE, %r8d
    jz      0f

    /* fix up rsi */
    movdqu      -32(%rsi, %rdx, 1), %xmm0
    movdqu      -16(%rsi, %rdx, 1), %xmm1
    lea     -40(%rsp), %rsi     # end of replacement buffer
    sub     %rdx, %rsi      # start of replacement buffer
    movdqa      %xmm0, -72(%rsp)    # copy to replacement buffer
    movdqa      %xmm1, -56(%rsp)

    /* load data and compare properly */
0:  movdqu      16(%rdi), %xmm1
    movdqu      16(%rsi), %xmm3
    movdqu      (%rdi), %xmm0
    movdqu      (%rsi), %xmm2
    mov     %edx, %ecx
    mov     $-1, %edx
    shl     %cl, %rdx       # ones where the buffer is not
    pcmpeqb     %xmm3, %xmm1
    pcmpeqb     %xmm2, %xmm0
    pmovmskb    %xmm1, %ecx
    pmovmskb    %xmm0, %eax
    shl     $16, %ecx
    or      %ecx, %eax      # ones where the buffers match
    or      %edx, %eax      # including where the buffer is not
    not     %eax            # ones where there is a mismatch
#ifndef BCMP
    bsf     %eax, %edx      # location of the first mismatch
    cmovz       %eax, %edx      # including if there is no mismatch
    movzbl      (%rdi, %rdx, 1), %eax   # mismatching bytes
    movzbl      (%rsi, %rdx, 1), %edx
    sub     %edx, %eax
#endif
    ret

    /* empty input */
.L0:    xor     %eax, %eax
    ret

    /* compare 33+ bytes */
    ALIGN_TEXT
.Llong: movdqu      (%rdi), %xmm0       # load head
    movdqu      (%rsi), %xmm2
    mov     %rdi, %rcx
    sub     %rdi, %rsi      # express rsi as distance from rdi
    and     $~0xf, %rdi     # align rdi to 16 bytes
    movdqu      16(%rsi, %rdi, 1), %xmm1
    pcmpeqb     16(%rdi), %xmm1     # compare second half of this iteration
    add     %rcx, %rdx      # pointer to last byte in buffer
    jc      .Loverflow      # did this overflow?
0:  pcmpeqb     %xmm2, %xmm0
    pmovmskb    %xmm0, %eax
    xor     $0xffff, %eax       # any mismatch?
    jne     .Lmismatch_head
    add     $64, %rdi       # advance to next iteration
    jmp     1f          # and get going with the loop

    /*
     * If we got here, a buffer length was passed to memcmp(a, b, len)
     * such that a + len < a.  While this sort of usage is illegal,
     * it is plausible that a caller tries to do something like
     * memcmp(a, b, SIZE_MAX) if a and b are known to differ, intending
     * for memcmp() to stop comparing at the first mismatch.  This
     * behaviour is not guaranteed by any version of ISO/IEC 9899,
     * but usually works out in practice.  Let's try to make this
     * case work by comparing until the end of the address space.
     */
.Loverflow:
    mov     $-1, %rdx       # compare until the end of memory
    jmp     0b

    /* process buffer 32 bytes at a time */
    ALIGN_TEXT
0:  movdqu      -32(%rsi, %rdi, 1), %xmm0
    movdqu      -16(%rsi, %rdi, 1), %xmm1
    pcmpeqb     -32(%rdi), %xmm0
    pcmpeqb     -16(%rdi), %xmm1
    add     $32, %rdi       # advance to next iteration
1:  pand        %xmm0, %xmm1        # 0xff where both halves matched
    pmovmskb    %xmm1, %eax
    cmp     $0xffff, %eax       # all bytes matched?
    jne     .Lmismatch
    cmp     %rdx, %rdi      # end of buffer reached?
    jb      0b

    /* less than 32 bytes left to compare */
    movdqu      -16(%rdx), %xmm1    # load 32 byte tail through end pointer
    movdqu      -16(%rdx, %rsi, 1), %xmm3
    movdqu      -32(%rdx), %xmm0
    movdqu      -32(%rdx, %rsi, 1), %xmm2
    pcmpeqb     %xmm3, %xmm1
    pcmpeqb     %xmm2, %xmm0
    pmovmskb    %xmm1, %ecx
    pmovmskb    %xmm0, %eax
    shl     $16, %ecx
    or      %ecx, %eax      # ones where the buffers match
    not     %eax            # ones where there is a mismatch
#ifndef BCMP
    bsf     %eax, %ecx      # location of the first mismatch
    cmovz       %eax, %ecx      # including if there is no mismatch
    add     %rcx, %rdx      # pointer to potential mismatch
    movzbl      -32(%rdx), %eax     # mismatching bytes
    movzbl      -32(%rdx, %rsi, 1), %edx
    sub     %edx, %eax
#endif
    ret

#ifdef BCMP
.Lmismatch:
    mov     $1, %eax
.Lmismatch_head:
    ret
#else /* memcmp */
.Lmismatch_head:
    tzcnt       %eax, %eax      # location of mismatch
    add     %rax, %rcx      # pointer to mismatch
    movzbl      (%rcx), %eax        # mismatching bytes
    movzbl      (%rcx, %rsi, 1), %ecx
    sub     %ecx, %eax
    ret

.Lmismatch:
    movdqu      -48(%rsi, %rdi, 1), %xmm1
    pcmpeqb     -48(%rdi), %xmm1    # reconstruct xmm1 before PAND
    pmovmskb    %xmm0, %eax     # mismatches in first 16 bytes
    pmovmskb    %xmm1, %edx     # mismatches in second 16 bytes
    shl     $16, %edx
    or      %edx, %eax      # mismatches in both
    not     %eax            # matches in both
    tzcnt       %eax, %eax      # location of mismatch
    add     %rax, %rdi      # pointer to mismatch
    movzbl      -64(%rdi), %eax     # mismatching bytes
    movzbl      -64(%rdi, %rsi, 1), %ecx
    sub     %ecx, %eax
    ret
#endif
ARCHEND(memcmp, baseline)

The following table might be helpful when reading unless the x86 registers are already ingrained in your head. :-)

Luckily this code is well commented, but in short it does the following:

• Checks if we are checking more than 32 bytes

• If not it checks if the bytes we are about to check will cross a page boundary as we check 32 bytes simultaneously. For example if we are to check 20 bytes but the page boundary is at 25 bytes from the beginning of one of our inputs.

• If the above is the case we copy the data to a replacement/bounce buffer and override rdi/rsi.

• If no funny business is required we simply start comparing data in a similar way as to strlen except now we have a mask where we check if everything is matching or not. So we look for the occurence of the first 00 rather than FF as we did in `strlen.

• Then once we do hit we load the word at the offset from our base address and subtract them before we return

Translating to Aarch64

To begin with we ignore the special case regarding page crossing and save it for last, like a dessert. We start loading registers pairwise, so 32 bytes are immediately loaded. Then we start comparing using cmeq like we did with strlen except that we compared with #0 there.

then we need to reduce the width so the result can fit in a GPR, we use shrn for that, just as we did before.

But for matches the result is FF, so now the register is all FF’s if they were identical and if they were identical we want to keep looping. But the problem is that we can’t do a floating point compare with FF as its NaN, therefore we move it to a GPR and invert it. Inverting it also allows us to count the zeroes and if we have 32 zeroes (FF’s before inverting) we had no matches and we can keep looking! To make sure we dont look for too long we do a comparison with the max limit first, if we are past our limit and no matches were found then we simply return 0.

.Lbegin:
    mov x8,x0           // store base address for later
    mov x9,x1

    ldp q0,q1,[x0]      //load 32 bytes of b1 into vector registers
    ldp q2,q3,[x1]

    cmeq    v0.16b,v0.16b,v2.16b    // do compare between 0-15 b1 vs b2
    shrn    v0.8b,v0.8h,#4      // shift them right to fit in x1
    cmeq    v1.16b,v1.16b,v3.16b    // do compare between 16-31 b1 vs b2
    shrn    v1.8b,v1.8h,#4

    fmov    x1,d0
    fmov    x3,d1

    mvn x0,x1           // invert to use clz
    cbz x0,0f
    rbit    x0,x0
    clz x0,x0           // if this is zero check bytes 16..32
    b   1f
    
0:  
    rbit    x1,x3
    mvn x1,x1
    clz x0,x1
    add x0,x0,#64
1:  
    lsr x0,x0,#2
    cmp x0,limit        // if we have more matches than the limit return 0
    b.ge    .Lnone  
    cmp x0,#32          // x0 == 32 if no hit (32 0's)
    b.eq    .Lloop
2:  
    ldrb    w4,[x8,x0]      // gonna need to edit this based on align offset
    ldrb    w5,[x9,x0]  
    subs    w0,w4,w5        // get the byte difference
    ret

Now this approach works for up to 32 byte comparisons but we want to turn it into a loop, its quite straight forward with the way the code was written. Simpy make the loads increment the pointer and subtract from the limit.

    subs    limit,limit,#32
    b.lt    .Lnone          //we could do special handling here
    ldp q0,q1,[x8,32]!      //load 32 bytes of b1 into vector registers
    ldp q2,q3,[x9,32]!

Now the problem with this approach is that the loop condition is buried deep within a code path so even with speculative execution etc it will struggle.

Now we could try doing something like this:

.Lloop:
    subs    limit,limit,#32
    b.lt    .Lnone          //we could do special handling here
    ldp     q0,q1,[x8,32]!      //load 32 bytes of b1 into vector registers
    ldp     q2,q3,[x9,32]!
    
    cmeq    v0.16b,v0.16b,v2.16b    // do compare between 0-15 b1 vs b2
    cmeq    v1.16b,v1.16b,v3.16b    // do compare between 16-32 b1 vs b2

    umin    v2.16b,v0.16b,v1.16b
    umin    v2.16b,v2.16b,v2.16b    // separate lowest value    
    NOT     v2.16b,v2.16b           // invert it (FF -> we want to loop)    
    shrn    v2.8b,v2.8h,#4
    fcmp    d2,#0.0
    b.eq    .Lloop

But benchmarks showed that this resulted in better performance but I tried a few other ways to tighten the loop that turned out to be even faster.

.Lloop:
    subs    limit,limit,#32
    b.lt    .Lnone          //we could do special handling here
    ldp q0,q1,[x8,32]!      //load 32 bytes of b1 into vector registers
    ldp q2,q3,[x9,32]!
    
    eor v4.16b,v0.16b,v2.16b    // v0 = b1(16-32) XOR b2(16-32)
    eor v5.16b,v1.16b,v3.16b    // v1 = b1(32-48) XOR b2(32-48)
    umaxp   v4.16b,v4.16b,v5.16b    // v0 = max(v0,v1)
    umaxp   v4.16b,v4.16b,v4.16b    // v0 = max(v0,v0)
    fmov    tmp, d4
    cbz tmp, .Lloop

    cmeq    v0.16b,v0.16b,v2.16b    // do compare between 0-15 b1 vs b2  
    cmeq    v1.16b,v1.16b,v3.16b    // do compare between 16-32 b1 vs b2

Despite not reusing the result from cmeq this resulted in a few percent better perfomance across several Arm64 cores. This could be attributed to cmeq and shrn utilizing fewer pipelines than eor and umaxp. This is shown for the Neoverse V2 core the the optimization guide. [3]

A nice thing I realized is that this quick check is also applicable for the first 32 byte iteration aswell. Adding it there resulted in almost a 50% improvement for short string using the now familiar suite of benchmarks.

Stay within your page

Now to the dessert, the crossing into unmapped pages.

We put our memcmp through the unit tests as previously described and we get the following.

One failed! But why? And it’s not even a test of memcmp!

If we look into the log we notice the following

Process with PID 9597 exited with signal 11 and dumped core; attempting to gather stack trace
[New LWP 120611]
Core was generated by `/usr/obj/home/getz/gsoc/freebsd-src/arm64.aarch64/lib/libc/tests/string/checkdir'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  memcmp () at /home/getz/gsoc/freebsd-src/lib/libc/aarch64/string/memcmp.S:78
78    ldp q2,q3,[x1] //load 32 bytes of b2 into vector registers

So it’s a SIGSEV due to a address not being mapped, i.e. an unmapped page.

If we now run it through gdb (or lldb) we see the following values for x0 and x1 right before the attempted load.

>>> p/tz  $x0
$2 = 0x0000000041035000
>>> p/tz $x1
$3 = 0x0000000041036fff

To handle this we can first make check if we are allowed to check more than 32 bytes (limit > 32). If not we can check how many bytes are left to the next page crossing. If its less than 32 bytes we can load our 32 bytes from the buffers pointer minus the offset. Then after its loaded we can shift the entire vector register to get rid of the unwanted bytes.

On SSE there is a PSHUFB instruction that does this and luckily there is a more or less equivalent instruction for Aarch64 TBL. We need to use TBL as the other shift instructions like EXT and USHL either dont work with variable shifts or across entire vector registers. USHL would shift each byte a certain amount depending on the arrangement specifier (the .16b piece you often see).

The numbers followed by a letter for the vector registers are the arrangement specifiers is the number and size of the elements in the vector. A simple way to keep track of them is by thinking:

• B = Bytes (8bit)

• H = Halfwords (16bit)

• S = Single words (32bit)

• D = Double words (64bit)

courtesy of [4]

So for checking if we are about to cross a page we can do the following

cmp     x2,#32
b.hi    .Lbegin    // Jump to regular loop if we are allowed to check more than 32 bytes
add     x3,x8,#32
add     x4,x9,#32
eor     x3,x3,x8
eor     x4,x4,x9
tst     w3,#PAGE_SIZE
b.ne    .Lspecialhandling
tst     w4,#PAGE_SIZE
b.ne    .Lspecialhandling

Then theres also a piece that was overlooked in the main loop. As neither buf1 or buf2 is aligned there is a chance that we may overread when theres less than 32 bytes left to compare. This is easily solved by adding a new branch where we previously checked if the limit had gone down to 0.

.Lloop:
    subs    x2,x2,#32
    b.le    .Lnone
    cmp x2,#32
    b.le    .Llast32
    ldp q0,q1,[x8,32]!      // load 32 bytes to vector registers
    ldp q2,q3,[x9,32]!

    eor v4.16b,v0.16b,v2.16b
    eor v5.16b,v1.16b,v3.16b
    umaxp   v4.16b,v4.16b,v5.16b
    umaxp   v4.16b,v4.16b,v4.16b
    fmov    x6,d4
    cbz x6,.Lloop
    b   .Lmatch

    /* If 32 bytes left to compare only load 32 bytes from x8 - limit to
     * avoid overread */
.Llast32:
    mov x3,#32
    sub x3,x3,x2        // x3 = 32 - x2
    add x2,x2,x3        // add the amount we shifted to limit
    sub x8,x8,x3
    sub x9,x9,x3

    ldp q0,q1,[x8,#32]! // load 32 bytes to vector registers
    ldp q2,q3,[x9,#32]!

.Lmatch:
    cmeq    v0.16b,v0.16b,v2.16b    // compare between 0-15 b1 vs b2
    cmeq    v1.16b,v1.16b,v3.16b    // compare between 16-31 b1 vs b2

wide shifts

So now the only piece left in the puzzle is to shift the vector register for the case when we have 32 bytes or less to compare and they are located near the end of a page.
To utilize TBL here we need to generate a suitable permutation mask, to do this we can take the identity permutation 0, 1, 2, ..., 15, 16, 17, ..., 31 and add to each byte to rotate it, or load from it with a desired offset. This can table can easily be prepared in a data segment.

    .section .rodata
    .p2align 4
shift_table:
    .byte 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
    .byte 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

USHR/USHL could also work but as it can only shift 64 bit words it would only be useful for very short strings unless there some nice way to utilize it here.

The TBL/TBL instructions are rather tricky to figure out how they work based on the reference manual but ARM provides a graphical representation of how they operate which cleared some things out for me.

We simply check if the 32nd byte from our buffers starting address is located on another page as defined by the $PAGE_SIZE preprocessor directive defined in sys/arm64/include/param.h. Then we shift and load by the appropriate amount.

So the check and fix can be implemented like the following:

ENTRY(_memcmp)

    mov x8, x0
    mov x9, x1
    cbz x2, .Lnone      // 0 length

    /*
     * Check if buffer is located at end of page to avoid crossing
     * into unmapped page. If so, we load 32 bytes from the end of the
     * limit and check the other buffer.
     */

    cmp x2, #32
    b.hi    .Lbegin
    add x3, x8, #32
    add x4, x9, #32
    eor x3, x3, x8
    eor x4, x4, x9

    tst w3, #PAGE_SIZE
    b.eq    0f

    mov x3, #32
    sub x3, x3, x2
    sub x8, x8, x3

    /*
     * We perform a variable shift in the vector register using TBL,
     * a suitable permutation is generated by loading a table of bytes
     * with a desired offset.
     */

    adrp    x0, shift_table
    add x0, x0, :lo12:shift_table
    add x0, x0, x3
    ldp q0, q1, [x8]
    ldp q4, q5, [x0]        // load permutation table
    tbl v0.16b, {v0.16b, v1.16b}, v4.16b
    tbl v1.16b, {v0.16b, v1.16b}, v5.16b
    add x8, x8, x3      // reset pointer to beginning of src
    b   1f

0:
    ldp q0, q1, [x8]

1:
    tst w4, #PAGE_SIZE
    b.eq    0f

    mov x3, #32
    sub x3, x3, x2
    sub x9, x9, x3

    ldp q2, q3, [x9]
    adrp    x0, shift_table
    add x0, x0, :lo12:shift_table
    add x0, x0, x3
    ldp q4, q5, [x0]
    tbl v2.16b, {v2.16b, v3.16b}, v4.16b
    tbl v3.16b, {v2.16b, v3.16b}, v5.16b
    add x9, x9, x3
    b   1f

    /*
     * Compare strings of 1--32 bytes.  We do this by loading into two
     * vector registers and then doing a quick compare with XOR,  UMAXP
     * do determine if the first 32 bytes all match.
     */
.Lbegin:
    ldp q0, q1, [x8]
0:
    ldp q2, q3, [x9]
1:

    /*
     * REST OF CODE HERE
     */

END(_memcmp)

    .section .rodata
    .p2align 4
shift_table:
    .byte 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
    .byte 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

Benchmarks

Collin Percival was kind enough to give me a voucher for AWS credits so now I can also benchmark on a more powerful ARM core which might be more common to find running a FreeBSD system.

os: FreeBSD
arch: arm64
cpu: ARM Neoverse-V1 r1p1
            │ memcmpScalar │              memcmpARM              │             memcmpSIMD              │
            │    sec/op    │   sec/op     vs base                │   sec/op     vs base                │
MemcmpShort   139.41µ ± 2%   63.96µ ± 1%  -54.12% (p=0.000 n=20)   25.14µ ± 1%  -81.97% (p=0.000 n=20)
MemcmpMid      93.38µ ± 3%   12.09µ ± 1%  -87.05% (p=0.000 n=20)   10.91µ ± 1%  -88.31% (p=0.000 n=20)
MemcmpLong    86.722µ ± 7%   4.648µ ± 1%  -94.64% (p=0.000 n=20)   4.931µ ± 0%  -94.31% (p=0.000 n=20)
geomean        104.1µ        15.32µ       -85.29%                  11.06µ       -89.38%

            │ memcmpScalar │                memcmpARM                │               memcmpSIMD                │
            │     B/s      │      B/s       vs base                  │      B/s       vs base                  │
MemcmpShort   855.1Mi ± 2%   1864.0Mi ± 1%   +117.99% (p=0.000 n=20)   4742.2Mi ± 1%   +454.58% (p=0.000 n=20)
MemcmpMid     1.247Gi ± 3%    9.629Gi ± 1%   +671.89% (p=0.000 n=20)   10.668Gi ± 1%   +755.20% (p=0.000 n=20)
MemcmpLong    1.342Gi ± 6%   25.048Gi ± 1%  +1765.92% (p=0.000 n=20)   23.608Gi ± 0%  +1658.68% (p=0.000 n=20)
geomean       1.118Gi         7.600Gi        +579.66%                   10.53Gi        +841.33%

os: FreeBSD
arch: arm64
cpu: ARM Cortex-A76 r4p1
            │ memcmpScalar  │              memcmpARM               │             memcmpSIMD              │
            │    sec/op     │    sec/op     vs base                │   sec/op     vs base                │
MemcmpShort    183.12µ ± 0%   101.29µ ± 0%  -44.68% (p=0.000 n=20)   39.96µ ± 0%  -78.18% (p=0.000 n=20)
MemcmpMid      129.12µ ± 0%    24.55µ ± 1%  -80.99% (p=0.000 n=20)   21.48µ ± 0%  -83.37% (p=0.000 n=20)
MemcmpLong    111.374µ ± 0%    6.288µ ± 0%  -94.35% (p=0.000 n=20)   7.593µ ± 0%  -93.18% (p=0.000 n=20)
geomean         138.1µ         25.01µ       -81.89%                  18.68µ       -86.47%

            │ memcmpScalar │                memcmpARM                │               memcmpSIMD                │
            │     B/s      │      B/s       vs base                  │      B/s       vs base                  │
MemcmpShort   651.0Mi ± 0%   1176.9Mi ± 0%    +80.78% (p=0.000 n=20)   2983.1Mi ± 0%   +358.23% (p=0.000 n=20)
MemcmpMid     923.3Mi ± 0%   4856.0Mi ± 1%   +425.95% (p=0.000 n=20)   5551.0Mi ± 0%   +501.23% (p=0.000 n=20)
MemcmpLong    1.045Gi ± 0%   18.514Gi ± 0%  +1671.24% (p=0.000 n=20)   15.332Gi ± 0%  +1366.84% (p=0.000 n=20)
geomean       863.3Mi         4.656Gi        +452.24%                   6.233Gi        +639.33%

References

[1]: https://research.google/pubs/automemcpy-a-framework-for-automatic-generation-of-fundamental-memory-operations/
[2]: https://www.cs.uaf.edu/2017/fall/cs301/lecture/09_11_registers.html
[3]: https://developer.arm.com/documentation/109898/latest/
[4]: https://stackoverflow.com/a/73293215/17418180

Table of Contents

• Hello Beastie and Google Summer of Code!
  • Background
  • Why write assembly
    • Architectural levels
  • amd64 strlen implementation
    • Substituting missing instructions
  • strlen PoC
    • Simple strlen
    • Naïve implementation
    • Improved implementation
    • FCMP to avoid GPR move
  • libc integration
  • Tests
  • Benchmarks
  • What’s next
  • References
• Bonus: Hello World in Aarch64 Assembly

I have been admitted to be a part of Google Summer of Code (GSoC) 2024! [1]

Participating in GSoC has been on my radar for many years but I always thought, oh I’ll do it next summer, well that summer is now! :D

Background

The admission process to GSoC is as follows: Organizations publish some suggested projects with possible mentors, students get in touch with said mentors or come up with their own project and convince someone to mentor them. Then voting takes place and the top N applicants for the organizations are accepted where N is the numbers of slots Google has allocated for that organization. So I found a project that I thought sounded interesting, got in touch with the mentor and sent in a proposal [2] through google’s portal, the project was posted on the FreeBSD GSoC ideas page [3]. The project is to port SIMD enhanced string functions in libc from amd64 (x86_64) to arm64 (Aarch64).

A great incentive to the program is that you get the chance to be mentored by member(s) of the community. I have the pleasure to be mentored by two wonderful people, Robert Clausecker <fuz> and Ed Maste <emaste>. Another GSoC contributor is porting the same algorithms to RISC-V, he is basing his implementation on the base RISC-V ISA using SIMD Within A Register (SWAR) techniques, and thus not having any dependency on processor extensions such as the recently ratified 1.0 RISC-V Vector Extension for which almost no hardware is currently available (and no hardware running FreeBSD). His blog documenting his adventures is available at [4].

As for me, I’ve been using FreeBSD for a few years now, before that I used Linux but after frustration over lack of good documentation and a fragmented system I switched and haven’t looked back. I still enjoy the linux kernel but userland and distros is something I just want out of my way.

But this is not an article about why FreeBSD is superior to Linux, if it even is (yes, in some regards, but ymmv). But rather about my project for the summer.

Why write assembly

Most libc functions on other platforms already benefit from being handwritten in assembly, both scalar and SIMD variants. Using SIMD instructions for string functions are particularly unfit for a autovectorizing compiler as we make atypical use of SIMD instructions. For the scalar implementations we may use some Bit Twiddling Hacks such as those on Sean Eron Anderson’s site [5].

Compilers also struggle reasoning to decide which operations to do in GPRs and which to do in vector registers. Register allocation is also a problem for amd64 and the compiler may spill into the stack whereas for handwritten assembly you would be left with registers to spare. This is not really as extreme of a case on arm64 as we have way more registers than amd64 to play around with.

Another compelling reason to have performance critical libc functions written in assembly is that all programs that link against libc will benefit from these improvements. Although this will put some additional pressure on me as an implementer as the code cant break other peoples programs just because they abused libc in interesting ways. An example of that is how memcmp on FreeBSD differs from the ISO/IEC 9899:1999 requirement’s. In particular FreeBSD documents that memcmp returns the difference between the first two mismatching characters, as opposed to merely returning a negative/positive integer or zero.

But my project will solely deal with using Arm NEON instructions to simd-ify the string functions, although bit-twiddling GPRs does sound enticing.

Architectural levels

The AMD64 SysV ABI supplement defines the following architecture levels, where for FreeBSD we have implementations for most of the string functions in libc for scalar and baseline. Users are able to choose which level of enhancements to use using the ARCHLEVEL flag. A complete list of enhanced functions are available in the simd(7) manpage [6].

scalar    scalar enhancements only (no SIMD)

baseline      cmov, cx8, x87 FPU, fxsr, MMX, osfxsr, SSE, SSE2

x86-64-v2  cx16,    lahf/sahf, popcnt, SSE3, SSSE3, SSE4.1, SSE4.2

x86-64-v3  AVX, AVX2, BMI1, BMI2, F16C, FMA, lzcnt, movbe, osxsave

x86-64-v4  AVX-512F/BW/CD/DQ/VL

amd64 strlen implementation

The amd64 strlen implementation consists of two parts, a scalar one (using bit-twiddling) and a Vectorized one (baseline). I’ll focus on the SIMD one here, the interested reader can navigate to /usr/src/lib/libc/amd64/string or point their browser to [7] for the scalar implementation.

ARCHENTRY(strlen, baseline)
    mov %rdi, %rcx
    pxor    %xmm1, %xmm1
    and $~0xf, %rdi         # align string
    pcmpeqb (%rdi), %xmm1   # compare head (with junk before string)
    mov %rcx, %rsi          # string pointer copy for later
    and $0xf, %ecx          # amount of bytes rdi is past 16 byte alignment
    pmovmskb %xmm1, %eax
    add $32, %rdi           # advance to next iteration
    shr %cl, %eax           # clear out matches in junk bytes
    test    %eax, %eax      # any match? (can't use ZF from SHR as CL=0 is possible)
    jnz 2f

    ALIGN_TEXT
1:  pxor    %xmm1, %xmm1
    pcmpeqb -16(%rdi), %xmm1    # find NUL bytes
    pmovmskb %xmm1, %eax
    test    %eax, %eax          # were any NUL bytes present?
    jnz 3f

    /* the same unrolled once more */
    pxor    %xmm1, %xmm1
    pcmpeqb (%rdi), %xmm1
    pmovmskb %xmm1, %eax
    add $32, %rdi           # advance to next iteration
    test    %eax, %eax
    jz  1b

    /* match found in loop body */
    sub $16, %rdi           # undo half the advancement
3:  tzcnt   %eax, %eax      # find the first NUL byte
    sub %rsi, %rdi          # string length until beginning of (%rdi)
    lea -16(%rdi, %rax, 1), %rax    # that plus loc. of NUL byte: full string length
    ret

    /* match found in head */
2:  tzcnt   %eax, %eax      # compute string length
    ret
ARCHEND(strlen, baseline)

Most of these instructions aren’t anything odd, MOV,XOR,AND,SHR, but what stands out is PCMPEQB and PMOVMSKV.

PMOVMSKB [8] is one of the most useful instructions for finding where the the index of our NULL character. The string functions in libc obviously function on C style string so they are NULL terminated. So what we do there is a compare followed by figuring out where the match was.

But heres the kicker, there is no PMOVMSKB instruction for Aarch64 which has caused a whole lot of headache. Im basing this on the amount of posts online regarding substitutions for PMOVMSKB on Aarch64 whereas assembly instructions are otherwise often rarely complained about. [9][10][11]

Substituting missing instructions

The most promising substitution to PMOVMSKB appears to be SHRN. With it we can take a 128-bit vector, shift by #imm and truncate to 8 bits. So basically end up with a mask of either chunks of all 0’s or all 1’s, then by truncation we end up with single but halfbytes which correspond to whether we had a match or not. Hopefully thats comes out clearly, otherwise there is an excellent video courtesy of [10] which shows it in action.

strlen PoC

Here is an evolution of my attempt of porting strlen to Aarch64. They also come in the form of a git repository [12] where I keep my experiments before theyre ready to be integrated into libc at my fork of freebsd-src that I keep on github. [13]
It’s also good to know while reading this code that registers x9-x15 are “corruptible” meaning that a function can change them to whatever without having to restore them afterwards. Registers d0-d7 are “parameter and results registers”. So according to the ARM procedure call standard, x0 etc. is where the input to a function is stored and you can do whatever you want with x9-x15 without breaking anything.

Simple strlen

The most simple variant simply checks the first chunk without any loop. This simple example will only give valid results for short strings which are 16 byte aligned. That is that the data begins on a memory address that is a multiple of 16. You can create such a string to test it out like this:

#include <stdalign.h>
#include <stdio.h>
#include <string.h>

extern size_t _strlen(const char * ptr);

int
main() {
    alignas(16) char string[] = "str";
    printf("strlen: %zu\n", _strlen(string));
}

Now for the simple strlen implementation. Note that <machine/asm.h> has nice little macros ENTRY() and END() which are used throughout.

ENTRY(_strlen)
    BIC x10,x0,#0xf       // alignment
    LDR q0,[x10]          // load input to Vector register
    CMEQ v0.16b,v0.16b,#0 // look for 0's
    SHRN v0.8b,v0.8h,#4   // ^
    FMOV x0,d0            // move to GPR
    RBIT x0,x0            // reverse bits as NEON has no ctz
    CLZ x0,x0             // count leading zeros
    LSR x0,x0,#2          // get offset index
    RET
END(_strlen)

Naïve implementation

Now for a simple but naïve solution to creating a loop and handling strings which are not already 16 byte aligned. We simply calculate the offset to the nearest 16 byte boundary and traverse to the boundary with a Scalar implementation and then turn to the SIMD variant.

ENTRY(_strlen)
    BIC x10,x0,#0xf
    AND x9,x0,#0xf
    MOV x11,#0
    CBZ x9,.Laligned_loop

.Lunaligned_start:
    LDR x5,[x10]
    ADD x10,x10,#1
    CBZ x5,.Lfound_null
    SUB x9,x9,#1
    CBNZ x9,.Lunaligned_start
    B .Laligned_loop

.Lnext_it:
    ADD x11,x11,#16
    ADD x10,x10,#16

.Laligned_loop:
    LDR q0,[x10]
    CMEQ v0.16b,v0.16b,#0
    SHRN v0.8b,v0.8h,#4
    FMOV x0,d0
    CBZ x0,.Lnext_it
    RBIT x0,x0
    CLZ x0,x0
    LSR x0,x0,#2
    ADD x0,x0,x11
    RET

.Lfound_null:
    RET
END(_strlen)

This doesn’t make use of handy Aarch64 instruction paramaters such as LDR qreg,[xreg,#imm]! which increments by the immediate value before the load.

Improved implementation

Now we use improve our previous implementation by using SIMD Instructions for the first chunk. We use a GPR load after the first check to improve performance for short strings as a GPR move is required for the results and statistically short strings are the most common in real world scenarios. This statement is backed by a survey conducted by the LLVM project. I have not been able to find a direct link to those results, but I will update this post when I find them. See: https://code.ornl.gov/llvm-doe/llvm-project/-/tree/doe/libc/benchmarks/distributions

ENTRY(_strlen)
    BIC x10,x0,#0xf
    LDR q0,[x10]
    CMEQ    v0.16b,v0.16b,#0
    SHRN    v0.8b,v0.8h,#4
    FMOV    x1,d0       // move to GPR
    LSL x2,x0,#2        // get the byte offset of the last processed
    LSR x1,x1,x2        // align with offset
    CBZ x1,.Lloop       // jump if no hit
    RBIT    x1,x1
    CLZ x0,x1
    LSR x0,x0,#2
    RET

.Lloop:
    LDR q0,[x10,#16]!    // increment by 16 and load
    CMEQ v0.16b,v0.16b,#0
    SHRN v0.8b,v0.8h,#4
    fmov x1,d0           // get offset in case of hit
    cbz x1,.Lloop        // x1 is zero if no hit in segment
    .Ldone:
    SUB x0,x10,x0
    RBIT x1,x1
    CLZ x3,x1
    LSR x3,x3,#2
    ADD x0,x0,x3
    RET
END(_strlen)

FCMP to avoid GPR move

After improving on the naïve implementation I realized that we can avoid a move from a SIMD register to a GPR by using FCMP in the loop. I also realized that we can avoid a few instructions if the input is already 16 byte aligned (see .Laligned), although it does introduce a new branch to be resolved.
The later benchmarks will indicate whether or not this is a worthwhile improvement.

ENTRY(_strlen)
    BIC x10,x0,#0xf
    AND x9,x0,#0xf
    LDR q0,[x10]
    CMEQ    v0.16b,v0.16b,#0
    SHRN    v0.8b,v0.8h,#4
    CBZ x9,.Laligned
    FMOV    x1,d0
    LSL x2,x0,#2
    LSR x1,x1,x2
    CBZ x1,.Lloop
    RBIT    x1,x1
    CLZ x0,x1
    LSR x0,x0,#2
    RET

.Laligned:
    FMOV    x1,d0
    CBNZ    x1,.Ldone

.Lloop:
    LDR q0,[x10,#16]!
    CMEQ v0.16b,v0.16b,#0
    SHRN v0.8b,v0.8h,#4
    fcmp d0,#0.0
    B.EQ .Lloop
    FMOV x1,d0
.Ldone:
    SUB x0,x10,x0
    RBIT x1,x1
    CLZ x3,x1
    LSR x3,x3,#2
    ADD x0,x0,x3
    RET
END(_strlen)

Now we can look at what further improvements can be done. We could avoid loop carried dependencies, for each iteration a post increment is currently used to go to next iteration, we could unroll the loop twice and increment x10 every two iterations to make it easier for the CPU to run two iterations at once. Aarch64 also has instructions for loading several SIMD registers at once using the LD1,LD2… family of instructions. [16]

libc integration

Getting a string function written in assembly integrated into libc on FreeBSD isn’t as big of an ordeal as it may sound like. We simply use the ENTRY() and END() macros in the code and add the filenames to the associated Makefile.inc located at lib/libc/aarch64/string/Makefile.inc like the following:

@@ -15,11 +15,12 @@ AARCH64_STRING_FUNCS= \
        strchrnul \
        strcmp \
        strcpy \
-       memcmp \
        strncmp \
        strnlen \
        strrchr

+MDSRCS+= \
+       memcmp.S
 #
 # Add the above functions. Generate an asm file that includes the needed
 # Arm Optimized Routines file defining the function name to the libc name.

We can then build libc as a shared library and load it using LD_PRELOAD for running regression tests as running FreeBSD with a broken libc makes FreeBSD very sad and prone to severe errors. It’s always nice to avoid a broken install while debugging.

Building libc is as simple as

cd /usr/src/lib/libnetbsd && make
cd /usr/src/lib/libc && make

# OR

make -C /usr/src/lib/libc MAKEOBJDIRPREFIX=/tmp/objdir WITHOUT_TESTS=yes

As for debugging it’s as simple as loading up a test binary with lldb and setting a breakpoint for the string function being developed, strlen in our case.

Tests

FreeBSD comes bundled with an excellent Test Suite [14], they are written using a test framework Kyua with the ATF library. The FreeBSD wiki page has all the information necessary for this and there’s no need for my to repeat it here. :-)

Running the tests is as simple as going to /usr/src/lib/libc/tests/string and running make check. If you haven’t already run buildworld then you will need to build lib/libnetbsd first. This as FreeBSD also borrows some tests from upstream NetBSD located at /usr/src/contrib/netbsd-tests/lib/libc/string.

Benchmarks

Benchmarks are executed using fuz’ strperf program [17], it’s output is compatible with benchstat from /devel/go-perf. I benchmark all the implementations against the implementations in libc, I also tried running the code on a Raspberry Pi 5 running debian to see strlen holds up against glibc’s implementation.

I have benchmarked the previously described implementations. Seeing performance impact of substituting a GPR move followed by a bnz compared to a FCMP followed by a b.eq and the impact of branching immediately in the case of an already aligned string.

It’s also important to note that these implementations are hardware dependent as different cores may utilize more or fewer pipelines for specific instructions.

To test against glibc I borrowed a Raspbery Pi5 running debian and installed bsd-make to compile strperf, apt install bmake.

Then to generate benchmark results it’s as simple as:
for i in {1..20}; do ./strlen >> results/${TEST}; done

This produced the following results when evaluated with benchstat. You might need to scroll horizontally to view all the results.

os: FreeBSD
arch: arm64
cpu: ARM Cortex-A76 r4p1
        │ libc_Scalar  │              libc_ARM               │                 GPR                 │             GPR_aligned             │                FCMP                 │            FCMP_aligned             │
        │    sec/op    │   sec/op     vs base                │   sec/op     vs base                │   sec/op     vs base                │   sec/op     vs base                │   sec/op     vs base                │
Short      186.9µ ± 1%   134.6µ ± 0%  -28.01% (p=0.000 n=20)   121.0µ ± 0%  -35.26% (p=0.000 n=20)   118.5µ ± 0%  -36.62% (p=0.000 n=20)   121.8µ ± 0%  -34.85% (p=0.000 n=20)   119.8µ ± 0%  -35.91% (p=0.000 n=20)
Mid        45.05µ ± 1%   37.07µ ± 0%  -17.73% (p=0.000 n=20)   33.36µ ± 0%  -25.96% (p=0.000 n=20)   30.43µ ± 1%  -32.45% (p=0.000 n=20)   33.37µ ± 0%  -25.93% (p=0.000 n=20)   29.98µ ± 1%  -33.45% (p=0.000 n=20)
Long      13.894µ ± 0%   4.442µ ± 0%  -68.03% (p=0.000 n=20)   6.978µ ± 0%  -49.78% (p=0.000 n=20)   6.977µ ± 0%  -49.79% (p=0.000 n=20)   6.852µ ± 0%  -50.68% (p=0.000 n=20)   5.627µ ± 0%  -59.50% (p=0.000 n=20)
geomean    48.91µ        28.09µ       -42.58%                  30.43µ       -37.79%                  29.30µ       -40.09%                  30.31µ       -38.03%                  27.24µ       -44.31%

        │ libc_Scalar  │                libc_ARM                │                  GPR                  │              GPR_aligned              │                  FCMP                  │              FCMP_aligned              │
        │     B/s      │      B/s       vs base                 │      B/s       vs base                │      B/s       vs base                │      B/s       vs base                 │      B/s       vs base                 │
Short     637.7Mi ± 1%    885.9Mi ± 0%   +38.91% (p=0.000 n=20)    985.1Mi ± 0%  +54.47% (p=0.000 n=20)   1006.1Mi ± 0%  +57.77% (p=0.000 n=20)    978.9Mi ± 0%   +53.49% (p=0.000 n=20)    995.0Mi ± 0%   +56.02% (p=0.000 n=20)
Mid       2.584Gi ± 1%    3.141Gi ± 0%   +21.55% (p=0.000 n=20)    3.490Gi ± 0%  +35.07% (p=0.000 n=20)    3.825Gi ± 1%  +48.03% (p=0.000 n=20)    3.488Gi ± 0%   +35.01% (p=0.000 n=20)    3.883Gi ± 1%   +50.26% (p=0.000 n=20)
Long      8.379Gi ± 0%   26.210Gi ± 0%  +212.81% (p=0.000 n=20)   16.684Gi ± 0%  +99.12% (p=0.000 n=20)   16.686Gi ± 0%  +99.15% (p=0.000 n=20)   16.990Gi ± 0%  +102.77% (p=0.000 n=20)   20.690Gi ± 0%  +146.93% (p=0.000 n=20)
geomean   2.380Gi         4.145Gi        +74.15%                   3.826Gi       +60.76%                   3.973Gi       +66.92%                   3.841Gi        +61.37%                   4.274Gi        +79.56%

os: Linux
arch: aarch64
        │ strlen_glibc │
        │    sec/op    │
Short      132.1µ ± 0%
Mid        36.29µ ± 1%
Long       4.365µ ± 4%
geomean    27.55µ

        │ strlen_glibc │
        │     B/s      │
Short     902.7Mi ± 0%
Mid       3.208Gi ± 1%
Long      26.67Gi ± 4%
geomean   4.225Gi

What’s next

Despite resulting in worse perfomance for longer strings I will now continue the porting effort and translate memcmp to Aarch64 NEON. I will continue optimizing strlen by unrolling the main loop twice as previously mentioned. So instead of the current LDR q0,[x10,#16]! I’ll do a LDP q1, q2, [x10, #32]!

Hopefully I can get that strlen done by the end of the week and submit it for review to the FreeBSD Phabricator instance.

I also attended the LundLinuxCon [18] last week and saw a really interesting talk regarding safer flexible arrays in the Linux kernel [19]. It involved a new warning flag and some niceties that were recently merged into GCC15 [20] and LLVM18 [21], I will try building FreeBSD with those flags and see how many warning are present, but first I need to read up a littlebit whether or not FreeBSD even permits the use of flexible arrays in the kernel. :-)

References

[1] https://summerofcode.withgoogle.com/
[2] https://dflund.se/~getz/GSOC/FreeBSDproposal.txt
[3] https://wiki.freebsd.org/SummerOfCodeIdeas#Port_of_libc_SIMD_enhancements_to_other_architectures
[4] https://strajabot.com/
[5] https://graphics.stanford.edu/~seander/bithacks.html
[6] https://man.freebsd.org/cgi/man.cgi?query=simd&manpath=FreeBSD+15.0-CURRENT
[7] https://github.com/freebsd/freebsd-src/blob/main/lib/libc/amd64/string/strlen.S
[8] https://www.felixcloutier.com/x86/pmovmskb
[9] https://branchfree.org/2019/04/01/fitting-my-head-through-the-arm-holes-or-two-sequences-to-substitute-for-the-missing-pmovmskb-instruction-on-arm-neon/
[10] https://community.arm.com/arm-community-blogs/b/infrastructure-solutions-blog/posts/porting-x86-vector-bitmask-optimizations-to-arm-neon
[11] https://www.corsix.org/content/whirlwind-tour-aarch64-vector-instructions
[12] https://git.sr.ht/~getz/aarch64_string.h/
[13] https://github.com/soppelmann/freebsd-src/
[14] https://wiki.freebsd.org/TestSuite/
[15] https://developer.arm.com/documentation/PJDOC-466751330-593177/latest/
[16] https://www.scs.stanford.edu/~zyedidia/arm64/ld2_advsimd_mult.html
[17] https://github.com/clausecker/strperf
[18] https://lundlinuxcon.org
[19] https://embeddedor.com/slides/2024/llc/llc2024.pdf
[20] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108896
[21] https://github.com/llvm/llvm-project/pull/76348

Bonus: Hello World in Aarch64 Assembly

The FreeBSD Developers’ Handbook has a section on writing assembly, but it’s targeted towards x86 and is rather outdated. Using it’s suggestions Hello World would look like this.

    .text
    .global _start

kernel:
    int $0x80
    ret

_start:
    mov     $4, %rax
    mov     $1, %rdi
    mov     $message, %rsi
    mov     $13, %rdx
    call    kernel
    mov     $1, %rax
    mov     $69, %rdi
    syscall

message:
    .ascii "Hello, world\n"

But INT 80h is much slower than syscall or simply using svc #0 on arm64, this is because a lot of microcode is run when INT is executed, whereas the microcode for syscall is much simpler. Although INT 80h still works on amd64!

Whereas 0x80 is the i386 syscall interface it incidentally works for amd64 tasks because we don’t check if the process doing a syscall is a 32 bit or 64 bit one. But arguments are truncated to 32 bits, so you can’t e.g. pass pointers to the stack! And arm64 ofcourse doesn’t have the INT instruction.

In general, the method of doing syscalls is different on each architecture. FreeBSD is moving towards what win32 and solaris already pioneered: syscalls should be done by calling library functions so the kernel ABI and API can be adapted in the future. For this reason syscalls will be split into a new library libsys in FreeBSD 15.

If you’re wondering how to figure out what number each syscall corresponds to then you can check https://cgit.freebsd.org/src/tree/sys/kern/syscalls.master

/*
Compile with the following for non Aarch64 host:
aarch64-unknown-freebsd14.0-gcc13 --sysroot /usr/local/freebsd-sysroot/aarch64 hello.S -nostdlib

Run with:
qemu-aarch64-static ./a.out
*/

.text

/* Our application's entry point. */
.global _start

_start:
    /* syscall write(int fd, const void *buf, size_t count) */
    mov     x0, #1      /* fd := STDOUT_FILENO */
    ldr     x1, = msg    /* buf := msg */
    ldr     x2, = len    /* count := len */
    mov     w8, #4     /* write is syscall #4 */
    svc     #0          /* invoke syscall */

    /* syscall exit(int status) */
    mov     x0, #69      /* status := 69 */
    mov     w8, #1     /* exit is syscall #1 */
    svc     #0          /* invoke syscall */

/* Data segment: define our message string and calculate it's length. */
.data
msg:
    .ascii "Hello, world\n"
len = . - msg

Table of Contents

• GPU passthrough for bhyve on FreeBSD 14
  • Why even bother
  • Lets get started!
    • buildkernel
    • make bhyve
    • Bhyve passthrough
• Update for newer releases
• References

We recently purchased a GPU for our University computer club [1] for Machine Learning tasks. Being most comfortable on a FreeBSD system I decided that running a VM through bhyve would be most suitable. Especially as I expect that several students will connect to this machine concurrently using harmful software such as Visual Studio Codes remote development over SSH extension.

Why even bother

From experience I know that VSCode launches a gazillion processes and wont kindly terminate them. It also downloads binaries from microsoft servers and runs a little node session remote on the server for each user and doesnt reuse old ones. A bonus is that it doesnt run naively on FreeBSD (You can get this working using linuxulator [2] but dont bother).

But the problem is that meanwhile Bhyve does support PCI-e passthrough its GPU support is lacking according to people online. [3]

bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them.
Note: VGA / GPU pass-through devices are not currently supported.

That last part is not really true, the problem lies within some drivers (NVIDIA) only supporting hypervisors with the KVM signature, so we need to patch Bhyve.

Luckily patches are already available [4], and the instructions are as simple as following the FreeBSD developers handbook [5].

Lets get started!

First get the sources for FreeBSD, you probably only care about the branch with the patches applied. They reside on github from the Beckhoff github account under the tag phab/corvink/14.0/nvidia-wip.

cd /where/you/want/to/build
git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src

buildkernel

Now we can either use the script provided by corvink@ [6] or just follow the handbook.

cd freebsd-src
make -j64 buildkernel
make installkernel

Now the new kernel is installed, the previous kernel is located at /boot/kernel.old/kernel

make bhyve

Now its time to build bhyve with our patches, this will go quick. The provided script automates this for you. (really, use the script [6])

make -C include -j64
make -C include install  
make -C lib/libvmmapi -j64
make -C lib/libvmmapi install  
make -C sys/modules/vmm -j64
make -C sys/modules/vmm install  
make -C usr.sbin/bhyve -j64
make -C usr.sbin/bhyve install  
make -C usr.sbin/bhyvectl -j64
make -C usr.sbin/bhyvectl install  
make -C usr.sbin/bhyveload -j64
make -C usr.sbin/bhyveload install  

Really, just use the script. But now you should be ready to reboot into your freshly patches system!

For future FreeBSD releases you can simply go in and apply the patch manually if necessary.

Bhyve passthrough

Now to have our pci device to passthrough to bhyve we first figure out what id its assigned. This is basically following the wiki page at [3].

$ pciconf -v -l | grep -B 1 -A 3 NVIDIA  
ppt0@pci0:193:0:0:      class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2204 subvendor=0x1028 subdevice=0x3880
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 [GeForce RTX 3090]'
    class      = display
    subclass   = VGA
ppt1@pci0:193:0:1:      class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1aef subvendor=0x1028 subdevice=0x3880
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA

Now note down those numbers and insert them into /boot/loader.conf as following

# echo pptdevs="193/0/0 193/0/1" >> /boot/loader.conf

Now the host wont steal them when the system is started, now to map them into Bhyve. If you want to passthrough another device, such as an USB hub to the VM you can specify more devices by appending N to pptdevs such as pptdevs2="194/0/0".

We use the excellent bhyve-vm [7] utility to simplify vm management.

Simply add the following to the end of your $VM.conf file, they will now show up in pci slots 9:0 and 9:1 on the VM. You might want to make sure that you dont make them to a spot thats already occupied, if so just go higher.

passthru0="193/0/0=9:0"
passthru1="193/0/1=9:1"

And presto!

Now you can reboot your FreeBSD host and your VM will have access to the GPU.

On a linux VM you can verify that the patches were applied correctly by checking your hypervisor signator by running dmesg

# dmesg | grep KVM

Update for newer releases

The patch still works on 14.1 it just needed some tiny refactoring as read_config was renamed to pci_host_read_config. A patch file is provided nvidia.patch. Apply it by running git apply -v nvidia.patch after cloning 14.1 by running git clone --depth 1 --branch releng/14.1 https://github.com/freebsd/freebsd-src

References

[1] https://df.lth.se/
[2] https://wiki.freebsd.org/Linuxulator
[3] https://wiki.freebsd.org/bhyve/pci_passthru
[4] https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip
[5] https://docs.freebsd.org/en/books/developers-handbook/kernelbuild/
[6] http://okturing.com/src/19931/body
[7] https://github.com/churchers/vm-bhyve

First set your machine to developer mode and all that, there are about 100 guides for it so find one of those first.

I first used chrx to partition my disks, it’s a two step program, first you partition disks then you install ubuntu. I dont like Ubuntu so just do the first part and dont rerun the script, if you change your mind about your partition sizes you’ll need to edit the script because it assumes that partitioning is correct and continues with second part of the script if the linux partition (Unused Kernel C and Unused Rootfs) are greater than 0 or something like that. Or you can just factory reset your pixelbook. https://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format

Then install whatever distro you want really, I used void, arch also works well. Ctrl + L at the scary screen and install from your usb key, if you need to factory reset then you’ll need an usb c usb stick as dongles wont get detected that early. But installing linux with a usb stick and a dongle works fine sort of, give it some time to initialize before you hit Ctrl + L, atleast that was my experience.

If you cant get grub to show up you probably need to

fdisk /dev/nvme0n1 fdisk open /dev/nvme0n1

t changing type

6 choosing partition #6 (Unused Kernel C)

4 setting it to BIOS BOOT (4) get full list by pressing L

Now you should boot into whatever distro you installed

Time to install the latest chromiumos kernel.

We need the latest firmware, you can download it from the chromiumos git repo but it doesnt have what we need. So we need to get the recovery file and extract what we need from that then we can overwrite with updated firmware from the public chromiumos-linux-firmware repo if you’d like. Make sure EVERYTHING gets copied now from the location we mounted the firmware image /mnt/tmp/lib/firmware/ to the /lib/firmware on our machine.

wget https://dl.google.com/dl/edgedl/chromeos/recovery/chromeos_11021.81.0_eve_recovery_stable-channel_mp.bin.zip
unzip chromeos_11021.81.0_eve_recovery_stable-channel_mp.bin.zip
sudo kpartx -av chromeos_11021.81.0_eve_recovery_stable-channel_mp.bin
mkdir /mnt/tmp
mount -t ext2 /dev/mapper/loop0p3 -o ro /mnt/tmp
cp /mnt/tmp/lib/firmware/* /lib/firmware/*

You might also want to download the latest linux-firmware because some files are overlapping and then we will get the newest we can, until we get a new firmware image for the 5.4 kernel and can get the stuff that currently is not available on the repo.

https://chromium.googlesource.com/chromiumos/third_party/linux-firmware/

git clone https://chromium.googlesource.com/chromiumos/third_party/linux-firmware

and copy those files into /lib/firmware

or something like that, you might want to clone the factory-eve branch, do that like this

git clone https://chromium.googlesource.com/chromiumos/third_party/linux-firmware -b factory-eve-9667.B We also need to compile our new kernel, follow the arch linux traditional compilation guide or gentoo handbook or random github repos. https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation https://github.com/megabytefisher/eve-linux-hacks/ https://wiki.gentoo.org/wiki/Kernel/Configuration

Make sure we have a modified .config as the default wont work, its named config in this directory and you need to save it and name it .config and save it wherever you download the kernel I will attach one but you might want to look through it yourself. I added necessary flags for TPM to the default .config and necessary modules for unique chrome hardware drivers. Also added the default cpu governor to powersave instead of performance as another pixelbook config did, this lead to alot of coil whine on my machine, I believe it was coil whine it was unpleasant atleast. If it doesnt scare you feel free to change it or else just change it in userspace with the cpupower program or cpupower-gui.

Find what else might necessary here https://github.com/flantel/pixelbook-linux https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-5.4/chromeos/config/ pineview is the config we would be using on chromiumos, i believe. https://github.com/dnschneid/crouton/wiki/Build-chrome-os-kernel-and-kernel-modules https://chromium.googlesource.com/chromiumos/docs/+/master/kernel_faq.md

Also, important. Backlight wont work out of the box unless youre using the full rom firmware (I think), if youre dual booting with SEABIOS like me you will need a hack to force DPCD mode, otherwise i915 will try using PWM for backlight control due to pins not being connected blah blah, look it up. https://unrelenting.technology/articles/FreeBSD-and-custom-firmware-on-the-Google-Pixelbook Really, look it up it’s a good writeup about the pixelbook and linux/bsd.

So we will need to add the patch from said link to our the kernel we will download in the next step. After downloading open the file /drivers/gpu/drm/i915/display/inteldpaux_backlight.c and go to line 271 (might change in the future and this patch might not work depending on how much they change the i915 driver but for now its line 271) and merge this

So download the chromium-5.4 kernel like this

git clone https://chromium.googlesource.com/chromiumos/third_party/kernel -b chromeos-5.4

open the file and edit, here is the patch, apply it.

IMPORTANT: IF YOU ARE RUNNING THE UEFI FIRMWARE YOU PROBABLY DO NOT NEED TO APPLY THIS PATCH!!! TRY WITHOUT IT AND IF IT DOESNT WORK ADD THE PATCH AND JUST CONTINUTE FROM THE make -j15 modules COMMAND AGAIN.

--- i/drivers/gpu/drm/i915/intel_dp_aux_backlight.c
+++ w/drivers/gpu/drm/i915/intel_dp_aux_backlight.c
@@ -252,8 +252,12 @@ intel_dp_aux_display_control_capable(struct intel_connector *connector)
         * the panel can support backlight control over the aux channel
         */
        if (intel_dp->edp_dpcd[1] & DP_EDP_TCON_BACKLIGHT_ADJUSTMENT_CAP &&
-           (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP) &&
-           !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)) {
+           (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP)
+/* for Pixelbook (eve), simpler version of https://patchwork.kernel.org/patch/9618065/ */
+#if 0
+            && !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)
+#endif
+                       ) {
                DRM_DEBUG_KMS("AUX Backlight Control Supported!\n");
                return true;
        }

It should look like this after you have applied the patch

intel_dp_aux_display_control_capable(struct intel_connector *connector)
{
    struct intel_dp *intel_dp = enc_to_intel_dp(&connector->encoder->base);
    /* Check the eDP Display control capabilities registers to determine if
     * the panel can support backlight control over the aux channel
     */
    if (intel_dp->edp_dpcd[1] & DP_EDP_TCON_BACKLIGHT_ADJUSTMENT_CAP &&
                   (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP)
        /* for Pixelbook (eve), simpler version of https://patchwork.kernel.org/patch/9618065/ */
        #if 0
                    && !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)
        #endif
                               ) {
        DRM_DEBUG_KMS("AUX Backlight Control Supported!\n");
        return true;
    }
    return false;
} 

Now time to build the kernel, dont forget to add the .config file before you run these commands!

make -j15
make -j15 modules
sudo make modules_install

compilation took 45 minutes on my machine, I7 pixelbook.

Install modules with mkinitcpio or dracut, include some firmware files. i.e add FILES=(/lib/firmware/i915/kbl_dmc_ver1.bin /lib/firmware/i915/kbl_guc_ver9_14.bin) to /etc/mkinitcpio.conf I also added MODULES=(intel_agp i915) for good measure, but ymmv.

Make sure you also download xf86-video-intel and use some grub boot flags, I used these, you might want to use some other.

loglevel=4 slub_debug=P page_poison=1 pci=noaer i915.enable_dpcd_backlight=1 pcie_aspm=force i915.i915_enable_rc6=1 i915.i915_enable_fbc=1 i915.lvds_downclock=1 intel_idle.max_cstate=1 elevator=noop i915.semaphores=1 intel_iommu=on nmi_watchdog=0 i915.enable_guc=2 mitigations=off acpi_osi=linux acpi_backlight=vendor

You might not want to use the mitigations=off flag, also i915.enable.dbc=1 is enabled on chromeos which makes the colors washed out so, if you want that, enable it. I think I also missed some flags you can find them somewhere in the chromiumos repo. You can read more about the dbc flag in the following thread, you might get 20-40 minutes more battery life but you will also get washed out colors and screen flicker. https://www.reddit.com/r/PixelBook/comments/fqe2rj/how_to_fix_washed_out_colors_backlight_flickering/

And then add /etc/X11/xorg.conf.d files, I’ll add them to this repo. Also, if you want the touchpad to behave as on chromeos check out my other repo’s there are forks of the CMT driver with necessary patches. You might need to google to find all necessary dependencies but its not difficult to compile the driver yourself.

https://github.com/soppelmann/xf86-input-cmt

And necessary dependencies

https://github.com/soppelmann/libevdevc

https://github.com/soppelmann/libgestures/

Get natural scrolling working like this, you might want to add it to the xorg.conf.d file, 12 id the id of my touchpad, find yours by running xinput –list

xinput set-prop '12' 'Australian Scrolling' 1

Or add the following line to the /etc/X11/xorg.conf.d/50-touchpad-cmt-eve.conf file Option "Australian Scrolling" "1" with mkinitcpio it will be: sudo mkinitcpio -k 5.4.35-chromium -g /boot/initramfs-5.4.35-chromium.img or something like that. Check /lib/modules for what the actual directory name is called, you choose this in the config file

afterwards copy into /boot/ and run grub-mkconfig update-grub or use grub-customizer.

To install the kernel is literally to copy it into the /boot directory, grub reads that directory and creates boot entries for you. All that is needed is the vmlinuz file but we also want a initramfs (copied there by the mkinitcpio command) and a systemmap.

We copy it all like this

sudo cp arch/x86_64/boot/bzImage /boot/vmlinuz-somenumberandname sudo cp System.map /boot/System.map-somenumberandname Only problem now is that the backlight is messed up when entering sleep mode zzz, I changed the acpid event handler so it doesnt enter sleep mode. Problem was that after waking from sleep the max brightness is halved even though the sys file shows it to be at the highest.

Edit the file /etc/acpi/handler.sh and uncomment the line zzz after button/lid close

Also, brightness is set at 0 at startup, I fixed this by setting the brightness to 100 in /etc/rc.local Maybe you could try the i915.invert_brightness parameter but this also works, somewhat.

xbacklight -set 100

(this command will only work if youre using acpilight, otherwise you can echo something to /sys/class/backlight/intel_backlight/brightness but this is easier for me.

I’ll use this until I figure out how the get the backlight working properly, its good enough for now atleast. 50% backlight as maximum is ok and you can see it as a power saving measure. :-) Another option is to invoke rtcwake when closing/opening the lid, I’ll expirement with that later.

I also blacklisted some kernel modules I dont need, I attached the blacklist.conf I used, it goes in /etc/modprobe.d/

And now you can enjoy your pixelbook with the latest kernel with some minor caveats but it shouldnt be too hard to fix. Battery life is comparable to Chromeos too which is nice, but it all depends on your DE,WM and set of programs.

Tested at idle with chromium open and brightness at 40%

Powerstat on chromeos 4.4.211-17203-gbe96242e05a2

  Time    User  Nice   Sys  Idle    IO  Run Ctxt/s  IRQ/s Fork Exec Exit  Watts
----- ---- ---- ---- ------ 
 Average  16.9   0.0   8.8  74.0   0.3  2.8 4667.0 2536.9 37.3 14.1 37.3   5.80 
 GeoMean  16.5   0.0   8.7  73.8   0.2  2.1 4551.6 2030.1 34.5 13.4 35.4   5.77 
  StdDev   4.2   0.0   1.6   5.6   0.1  2.2 1149.8 2089.7 17.3  4.6 13.0   0.69 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
 Minimum  13.9   0.0   7.3  60.3   0.1  1.0 3934.0 1437.8 20.0  8.0 22.0   5.41 
 Maximum  27.0   0.0  12.6  78.2   0.5  8.0 7418.8 7746.6 83.0 21.0 65.0   7.74 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
Summary:
System:   5.80 Watts on average with standard deviation 0.69

Powerstat on Void with KDE5 as DE and i3-gaps as WM 5.4.35chromium+

  Time    User  Nice   Sys  Idle    IO  Run Ctxt/s  IRQ/s Fork Exec Exit  Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
 Average   4.4   0.0   1.7  93.7   0.2  1.4 1829.9 3209.3 22.4 12.1 11.2   5.57
 GeoMean   1.0   0.0   0.6  92.6   0.0  1.2 1207.8 1070.9  7.0 11.2  6.5   5.56
  StdDev   9.4   0.0   2.6  12.3   0.4  0.9 2143.5 4018.2 61.4  6.8 16.7   0.45
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
 Minimum   0.3   0.0   0.2  51.6   0.0  1.0  620.1  294.4  4.0 10.0  4.0   5.31
 Maximum  36.5   0.0   9.9  99.5   2.0  5.0 9585.4 11041.3 286.0 41.0 66.0   7.12
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Summary:
System:   5.57 Watts on average with standard deviation 0.45

Also when touching the touchpad there are alot of interrupts that you can find in /proc/interrupts, I couldnt figure out a way to solve that but it also happens on chromeos so, wontfix.

I also couldnt get the iwl7000 driver working, I believe its used together with the iwlfifi driver on chromeos but I think I need to point to some firmware files, it shouldnt be too hard as they are available in /lib/firmware/ if someone figures this out please write.

Also, bluetooth is working, I needed to add bluetoothd to my init system, you might need to do the same.
ln -s /etc/sv/bluetoothd /var/service on void. Also make sure you uncomment the blacklist of the bluetooth modules like so in /etc/modprobe.d/blacklist.conf if you added that file.

# blacklist bluetooth

# blacklist btintel

# blacklist btrtl

# blacklist btusb

There are some more things to add that I havent written, (tablet mode switch, audio, SuzyQ, TPM, EC, External displays) and I should rewrite/reformat all of this into a more readable format and fix grammar but this is good enough for now. Good luck