Files
linux/include/linux
Yury Norov e79864f316 lib/find_bit: optimize find_next_bit() functions
Over the past couple years, the function _find_next_bit() was extended
with parameters that modify its behavior to implement and- zero- and le-
flavors. The parameters are passed at compile time, but current design
prevents a compiler from optimizing out the conditionals.

As find_next_bit() API grows, I expect that more parameters will be added.
Current design would require more conditional code in _find_next_bit(),
which would bloat the helper even more and make it barely readable.

This patch replaces _find_next_bit() with a macro FIND_NEXT_BIT, and adds
a set of wrappers, so that the compile-time optimizations become possible.

The common logic is moved to the new macro, and all flavors may be
generated by providing a FETCH macro parameter, like in this example:

  #define FIND_NEXT_BIT(FETCH, MUNGE, size, start) ...

  find_next_xornot_and_bit(addr1, addr2, addr3, size, start)
  {
	return FIND_NEXT_BIT(addr1[idx] ^ ~addr2[idx] & addr3[idx],
				/* nop */, size, start);
  }

The FETCH may be of any complexity, as soon as it only refers the bitmap(s)
and an iterator idx.

MUNGE is here to support _le code generation for BE builds. May be
empty.

I ran find_bit_benchmark 16 times on top of 6.0-rc2 and 16 times on top
of 6.0-rc2 + this series. The results for kvm/x86_64 are:

                      v6.0-rc2  Optimized       Difference  Z-score
Random dense bitmap         ns         ns        ns      %
find_next_bit:          787735     670546    117189   14.9     3.97
find_next_zero_bit:     777492     664208    113284   14.6    10.51
find_last_bit:          830925     687573    143352   17.3     2.35
find_first_bit:        3874366    3306635    567731   14.7     1.84
find_first_and_bit:   40677125   37739887   2937238    7.2     1.36
find_next_and_bit:      347865     304456     43409   12.5     1.35

Random sparse bitmap
find_next_bit:           19816      14021      5795   29.2     6.10
find_next_zero_bit:    1318901    1223794     95107    7.2     1.41
find_last_bit:           14573      13514      1059    7.3     6.92
find_first_bit:        1313321    1249024     64297    4.9     1.53
find_first_and_bit:       8921       8098       823    9.2     4.56
find_next_and_bit:        9796       7176      2620   26.7     5.39

Where the statistics is significant (z-score > 3), the improvement
is ~15%.

According to the bloat-o-meter, the Image size is 10-11K less:

x86_64/defconfig:
add/remove: 32/14 grow/shrink: 61/782 up/down: 6344/-16521 (-10177)

arm64/defconfig:
add/remove: 3/2 grow/shrink: 50/714 up/down: 608/-11556 (-10948)

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
2022-09-21 12:21:32 -07:00
..
2022-04-20 12:59:50 +05:30
2022-05-22 20:44:29 +01:00
2022-03-23 19:58:38 +01:00
2022-06-30 13:40:35 +01:00
2022-02-01 14:25:50 +02:00
2022-07-05 20:25:39 +02:00
2022-01-22 08:33:34 +02:00
2022-07-14 12:14:30 -06:00
2022-06-29 13:21:51 -07:00
2022-03-15 10:32:44 +01:00
2021-12-10 17:10:55 -08:00
2022-06-28 10:37:25 -03:00
2021-12-10 12:51:28 +01:00
2022-07-01 14:53:01 +02:00
2022-08-31 14:57:28 +01:00
2022-01-20 08:52:54 +02:00
2022-06-09 21:53:09 -07:00
2022-06-09 21:53:12 -07:00
2022-06-09 21:53:09 -07:00
2022-02-28 23:26:27 -08:00
2022-06-27 06:29:12 -06:00
2022-06-29 17:42:28 -07:00
2022-07-17 17:31:38 -07:00
2022-05-02 14:06:20 -06:00
2022-01-27 13:53:26 +00:00
2022-05-03 16:09:03 -04:00
2022-06-19 10:38:26 +01:00
2022-07-27 14:04:52 +02:00
2022-08-28 14:02:44 -07:00
2022-04-28 23:16:14 -07:00
2022-07-29 18:07:19 -07:00
2022-08-02 12:34:04 -04:00
2021-12-16 22:22:20 +01:00
2022-04-28 16:31:10 +02:00
2022-05-17 13:32:46 -04:00
2022-08-09 14:11:34 -04:00
2022-07-29 20:16:58 -04:00
2022-02-09 09:24:40 -05:00
2022-01-12 10:14:09 -06:00
2022-07-01 16:38:35 -06:00
2022-02-02 07:49:59 -07:00
2022-02-09 08:04:44 +01:00
2022-02-09 08:04:44 +01:00
2022-01-22 08:33:37 +02:00
2022-01-08 12:43:57 -06:00
2022-01-24 14:45:02 +01:00
2022-03-08 14:33:36 -06:00
2022-03-17 20:16:29 -07:00
2022-03-23 19:58:41 +01:00
2022-05-22 21:03:01 +01:00
2022-04-07 12:53:54 +02:00
2022-06-27 14:41:31 +02:00
2022-02-24 15:04:51 +00:00
2022-05-08 01:33:08 -07:00
2022-02-25 09:36:06 +01:00
2022-04-11 19:18:27 -06:00
2022-03-22 15:57:11 -07:00
2021-11-25 18:35:23 +01:00
2022-01-27 13:53:27 +00:00
2022-08-08 22:37:24 -04:00
2022-08-11 04:31:14 -04:00
2022-06-13 09:54:52 -07:00
2022-07-10 21:17:30 -04:00
2021-10-19 23:44:30 +08:00